Neural network image compression using an adaptive VQ encoder and a basic encoder

By using the adaptive VQ encoder and basic encoder of the dual-branch NNIC framework, and utilizing the index graphs of the precursor codebook and basic codebook, the problem of balancing compression performance and perceptual quality in neural network image compression is solved, achieving efficient image compression and high-quality reconstruction.

CN123125885APending Publication Date: 2026-11-17HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202580026763.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-04-04
Filing Date
2025-04-03
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing deep learning-based neural network image compression methods struggle to balance compression performance and perceptual quality, and VQ-based compression methods are difficult to efficiently reduce bitrate and achieve high-quality reconstruction.

Method used

A dual-branch NNIC framework is adopted, combining a representation-adaptive VQ encoder and a basic encoder. By utilizing the precursor codebook and the basic codebook index map, a mask codebook index prediction is performed through a mask generation encoder, which reduces bit transmission and improves reconstruction quality.

Benefits of technology

It achieves high compression ratio and high reconstruction fidelity while maintaining high perceptual quality, solving the problem of balancing compression performance and perceptual quality in traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125885A_ABST
    Figure CN123125885A_ABST
Patent Text Reader

Abstract

An image compression method based on a vector quantization (VQ) neural network includes: a first encoder encoding a first image to obtain a first latent feature corresponding to the first image; generating a pre-codebook index map and a first codeword map corresponding to the pre-codebook index map based on the first latent feature, wherein the pre-codebook includes a codebook index, and the codebook index is assigned to the same vector in multiple codebooks; encoding the pre-codebook index map to generate an encoded pre-codebook index map; and sending the encoded pre-codebook index map to a decoder.
Need to check novelty before this filing date? Find Prior Art

Description

Cross-references to related applications

[0001] This application claims priority to Provisional Application No. 63 / 574,332, filed on April 4, 2024, and Provisional Application No. 63 / 574,334, also filed on April 4, 2024. The entire disclosure of the above applications is incorporated herein by reference. Technical Field

[0002] This invention relates to neural network-based image or video compression, and more specifically, to neural network image / video compression employing a representation of an adaptive vector quantization (VQ) encoder and a base encoder. Background Technology

[0003] With the rapid advancement of deep learning research, the development of neural network-based image compression (NNIC) using state-of-the-art deep learning techniques has shown promising prospects for achieving efficient image encoding and decoding. Unlike traditional image compression algorithms that rely on manually designed transforms (e.g., the discrete cosine transform in the Joint Photographic Experts Group (JPEG), such as JPEG, High Efficiency Video Coding (HEVC), or Versatile Video Coding (VVC), NNIC methods learn efficient image representations through training on large datasets. Summary of the Invention

[0004] The first aspect relates to an image compression method based on a VQ neural network implemented by an encoding device. The method includes: encoding a first image using a first encoder of the encoding device to obtain a first latent feature corresponding to the first image; generating a pre-codebook index map and a first codeword map corresponding to the pre-codebook index map based on the first latent feature, wherein the pre-codebook includes a codebook index, the codebook index being assigned to the same vector in multiple codebooks; encoding the pre-codebook index map to generate an encoded pre-codebook index map; and sending the encoded pre-codebook index map to a decoder.

[0005] Optionally, in a first implementation of the first aspect, the method further includes: encoding the second image using a second encoder of the encoding device to obtain a second latent feature and a hypertensor corresponding to the second image; generating a base codebook index map based on the hypertensor using a base codebook, wherein encoding the precursor codebook index map to generate the encoded precursor codebook index map includes: using the base codebook index map as a reference.

[0006] Optionally, in a second implementation of the first aspect or any of its implementations, the base codebook is trained to create a base codebook index graph similar to the precursor codebook index graph.

[0007] Optionally, in a third implementation of the first aspect or any of its implementations, the base codebook is trained to create a base codebook index graph similar to the precursor codebook index graph.

[0008] Optionally, in a fourth implementation according to the first aspect or any of its implementations, generating the basic codebook index graph based on the hypertensor using the basic codebook includes: quantizing and entropy encoding the hypertensor to obtain an encoded hypertensor; decoding the encoded hypertensor to obtain a reconstructed hypertensor; performing tensor prediction using the reconstructed hypertensor to obtain a predicted tensor of a first tensor; quantizing and entropy encoding the difference between the first tensor and the predicted tensor; performing entropy decoding and tensor prediction to generate a reconstructed first tensor; and calculating the basic codebook index graph based on the reconstructed first tensor using the basic codebook.

[0009] Optionally, in a fifth implementation of the first aspect or any of its implementations, the second image is a low-quality image of the first image.

[0010] Optionally, in a sixth implementation of the first aspect or any of its implementations, the second image is identical to the first image.

[0011] Optionally, in a seventh implementation of the first aspect or any of its implementations, the codewords in the precursor codebook or the base codebook are reordered in descending order of frequency, such that fewer bits are allocated to the more frequently used indices, thereby reducing the total code rate.

[0012] Optionally, in an eighth implementation of the first aspect or any of its implementations, the codewords in the precursor codebook or the basic codebook are generated by a channel-weighted codeword generation method, which considers the importance factor difference between channels.

[0013] Optionally, in a ninth implementation of the first aspect or any of its implementations, the codewords in the precursor codebook or the base codebook are generated by a codeword generation method that takes into account a code rate factor.

[0014] Optionally, according to the first aspect or any of its implementations, in the tenth implementation, the method further includes: performing a position matching method to minimize the difference between the precursor codebook index map and the base codebook index map to the greatest extent possible.

[0015] Optionally, in an eleventh implementation of the first aspect or any of its implementations, the method further includes: training the base codebook using a contrastive classification training method.

[0016] Optionally, in a twelfth implementation of the first aspect or any of its implementations, the method further includes: performing masking and reconstruction to further minimize information transmission of the encoded preamble index graph.

[0017] Optionally, in a thirteenth implementation of the first aspect or any of its implementations, the masking process applies a mask ratio based on image complexity.

[0018] The second aspect relates to an image compression method based on a VQ neural network implemented by a decoding device. The method includes: receiving an encoded pre-codebook index map, wherein the encoded pre-codebook index map includes indices that map to the same vectors in multiple codebooks; based on the encoded pre-codebook index map, retrieving codewords from N codebooks using the corresponding codebook indices in the encoded pre-codebook index map to generate N latent feature maps; combining the N latent feature maps with an N-channel weight map to generate reconstructed latent features; and decoding the reconstructed latent features to obtain a reconstructed image.

[0019] Optionally, in a first implementation of the second aspect, the method further includes: generating a control signal to guide a conditional generation reconstruction process when decoding the latent features of the reconstruction to obtain the reconstructed image.

[0020] The third aspect relates to an apparatus comprising: a memory for storing instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the apparatus to perform a method according to any of the foregoing aspects or any implementation thereof.

[0021] The fourth aspect relates to a computer program product comprising computer-executable instructions stored in a non-transitory computer-readable storage medium, which, when executed by one or more processors of a device, cause the device to perform the method according to any of the foregoing aspects or any implementation thereof.

[0022] For clarity, any of the above embodiments can be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0023] These and other features and their advantages will become clearer from the following detailed description in conjunction with the accompanying drawings and claims. Attached Figure Description

[0024] To gain a more complete understanding of the invention, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein similar reference numerals denote similar parts.

[0025] Figure 1 This is a schematic diagram of a VQ-based encoder framework using a single codebook or multiple codebooks according to an embodiment of the present invention.

[0026] Figures 2A to 2C This is a schematic diagram of a single codebook, multiple codebooks, and a lead codebook according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of a VQ-based encoder framework using a lead codebook according to an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the basic encoder framework according to an embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of the NNIC system framework according to an embodiment of the present invention.

[0030] Figure 6 This is a schematic diagram of masked generative transformer (MAGE) prediction according to an embodiment of the present invention.

[0031] Figure 7 This is a flowchart of an image compression method based on a VQ neural network implemented by an encoding device according to an embodiment of the present invention.

[0032] Figure 8 This is a flowchart of an image compression method based on a VQ neural network implemented by a decoding device according to an embodiment of the present invention.

[0033] Figure 9 This is a schematic diagram of the system framework according to an embodiment of the present invention. Detailed Implementation

[0034] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. The invention is by no means limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but can be modified within the full scope of the appended claims and their equivalents.

[0035] This invention describes various embodiments of systems and methods for improving image / video coding efficiency through a dual-branch NNIC framework, comprising a representation adaptive (RED)-vector quantization (VQ) encoder and a base encoder. The RED-VQ encoder uses a masked generative encoder (MAGE) for mask codebook index prediction. The disclosed embodiments are applicable to both neural network-based image compression and neural network-based video compression. In one embodiment, the RED-VQ encoder utilizes high-quality (HQ) visual representations to generate images with high perceptual quality, and the base encoder provides fidelity information to guide conditional image generation in the RED-VQ encoder. By leveraging the power of high-quality generative visual representation priors, the efficiency and robustness of integer transfer, and guided image generation with fidelity control, the disclosed embodiments simultaneously achieve high compression ratios and high reconstruction fidelity and perceptual quality.

[0036] In the context of image compression, some autoregressive scale super-prior NNIC frameworks use a variational autoencoder (VAE) as the master encoder to mine latent information from the input image, a super-prior model as the super encoder to mine additional super-latent information, an autoregressive model for context modeling to mine spatial relationships between adjacent latent coefficients, and a Gaussian mixture model (GMM) or Gaussian scale mixture (GSM) to generate the mean and scale associated with each latent coefficient. Generally, at the sending end, the input image... x Image embedding features are generated after passing through the VAE encoder. y The image embedding features yThe data is further compressed into a data string that is easy to store and transmit efficiently through quantization and entropy coding. Quantization (or quantizing) refers to the process of mapping a continuous range of values ​​to a smaller set of discrete values. Entropy coding is a lossless data compression technique that assigns variable-length codes to symbols based on their probability of occurrence. The goal of entropy coding is to represent data more efficiently by using shorter codes for frequently used symbols and longer codes for less frequently used symbols.

[0037] At the receiving end, the decoded image embedding features are recovered from the received string through entropy decoding and dequantization. Then, the VAE decoder uses the embedded features of the decoded image. Reconstructed output image The goal is to minimize the reconstructed output. The repair loss is compared with the original input x, and the reduction of the loss used to represent the image embedding features is minimized. y The bits are used for storage and transmission.

[0038] The current NNIC framework relies on learning a general, compact image representation (i.e., image embedding features). y It can capture the key points of input x to reconstruct The framework suffers from several serious limitations. First, compression performance is inherently limited by model capacity (e.g., the network structure and number of parameters of the input encoder and decoder) when learning a general prior P(x|y) in the image domain. During training and testing, it is difficult to further improve compression performance beyond a good baseline due to limited model capacity, limited training data, and limited computational resources. Second, the NNIC model is learned to balance mutually exclusive objectives in rate distortion (RD) loss, where reducing reconstruction distortion and reducing bit rate are contradictory. Therefore, it is difficult to simultaneously improve compression performance and perceptual quality due to the difficulty in balancing different loss terms during end-to-end training.

[0039] In contrast, VQ-based compression methods learn one or more highly compressed visual codebooks C through a vector-quantized autoencoder. i (1≤i≤K), the autoencoder is trained as follows: each codebook targets a semantic category of the image, such as outdoor, portrait, etc. VQ is a process used to map vectors from a large set (e.g., high-dimensional data such as images) to a finite set of representative vectors. Generally, input images or video frames can be encoded into a latent space as embedded features. y This embedding feature y Capture the content and features of the input image. Then, use the learned codebook C to... i (1≤i≤K) embedding featuresy The codeword integer indices are mapped to a sequence of codeword integer indices zi (1 ≤ i ≤ K). Then, these codeword integer indices are transmitted to the receiver for decoding. The decoder uses the same codebook, utilizing the codebook C corresponding to the codeword indices zi (1 ≤ i ≤ K). i Codewords in (1≤i≤K) are used to recover the feature set. i (1≤i≤K). Then, by analyzing the feature set... Apply a weighted sum to i (1≤i≤K) to generate reconstructed features. Finally, it is possible to base the reconstruction on features. To reconstruct the output .

[0040] By encoding codeword indices rather than latent features, VQ-based compression is not affected by the sensitivity to computational mismatches between the sender and receiver. Compared to transmitting latent features or residuals, transmitting indices can expand the dimensionality of latent features without increasing the bitrate (often associated with better representational power, leading to better reconstruction). Furthermore, generative VQ improves robustness to input degradation. Even with low-quality (LQ) inputs, high-quality (HQ) codebooks can generate realistic and rich textures. However, VQ-based compression methods suffer from the difficulty of efficiently reducing the bitrate, where multiple codebook index maps and their corresponding weights used in the weighted sum need to be encoded in the bitstream. Another issue with VQ-based compression is the difficulty in achieving HQ reconstruction; although the restored image may look good, significant fidelity and rich detail are often lost in the reconstruction result, especially when training the codebook using a generative adversarial network (GAN).

[0041] Therefore, this invention describes various embodiments of VQ-based compression systems and methods that improve image / video coding efficiency to solve one or more of the problems mentioned above.

[0042] Figure 1 This is a schematic diagram of a VQ-based encoder framework 100 using a single codebook or multiple codebooks according to an embodiment of the present invention. The VQ-based encoder framework 100 includes a VQ encoder 102, one or more representation (REP) adapters 110, a feature compressor 112, a feature fusion unit 114, and a decoder 116. Generally, the input image... x After passing through the VQ encoder 102, the VQ encoder 102 will process the input image. x Convert to embedded features y In one embodiment, the input imagex It is shaped as w × h × c The general three-dimensional (3D) tensor, where, w , h and c These represent the width, height, and number of channels of the image, respectively. A tensor is a multidimensional array that is larger than two dimensions. For example, a color image has c=3, a spectral image has c=1, or an RGB-D (color and depth) image has c=4.

[0043] In one embodiment, the VQ encoder 102 is a VAE encoder or model that has been trained using one or more highly compressed visual codebooks C. i (1≤i≤K) Transform the image x Encoding is done as discrete latent representations, where each codebook targets a semantic category of the image, such as outdoor, portrait, etc. Embedded features y Indicates the input image x The discrete structured representation, where, with the image x Each feature vector corresponding to a region or feature is replaced by the closest embedding learned from the visual codebook.

[0044] Embedded features y The latent features are passed to one or more REP adapters 110. Each REP adapter 110 may include (1 to N) before_quant_RED (BFR) 104, (1 to N) RED codebooks 106, and (1 to N) after_quant_RED (AFT) 108, where N is the number of codebooks included in the REP adapter 110. In one embodiment, before_quant_RED 104 is used to transfer latent features... y This is transformed into N new latent feature spaces or subspaces corresponding to N codebooks. RED codebook 106 is used to process latent features. y The codeword indexes are then retrieved from N REP adapters 110 using the corresponding codebook indexes, generating N codeword index maps. Each codeword index map stores an index pointing to the corresponding entry in the codebook. The codeword map includes the actual codewords, not their indices. After_quant_RED 108 further transforms the N codeword index maps into N new codebook-based latent features. vqs .

[0045] In one embodiment, when using a single codebook, RED codebook 106 is based on latent features. y and codebook C Computation codebook index diagram z q and codebook-based featuresy q Codebook-based features y q Instructions and Images x Embedding features y Corresponding codebook features. Codebook index diagram z q Stores pointers to the codebook C The index of the corresponding codebook feature term in the codebook. Specifically, the codebook C It includes N codewords, each codeword having d dimensions. The feature y based on the codebook... q Each VQ vector in (i=1, …, width ( w ) × height ( h )) corresponds to c q The code in (i=1, …, w × h ),in, The closest to the corresponding feature vector y i :

[0046] in, It is a distance metric, such as the L1 or L2 norm. That is, the entire codebook index graph. z q With w × h The index corresponding to each codeword w × h A number of integers.

[0047] In one embodiment, since there is only one codebook index graph in the system... z q A codebook-based latent feature y q No weighted graph needed W Therefore, feature compressor 112 is not required.

[0048] In one embodiment, when using multiple (N) codebooks, RED codebook 106 is used for feature-based embedding. y and N codebooks C n (n=1, …, N) Calculate the N codebook index graphs z q,n and shape w × h×d N latent features y based on codebooks ×N q,n ( vqs In these embodiments, feature compressor 112 is used to process latent features. y Output the weighted graphW In one embodiment, the weight graph W The shape is w × h ×N, where w and h With latent features y of w and h Similarly, N is the same as the number of before_quant_RED 104 / after_quant_RED 108 in REP adapter 110.

[0049] Feature fusion unit 114 receives weight map W and N codebook-based latent features vqs Weighted graph W Guide / instruct N codebook-based latent features vqs How to perform weighted combination to generate a shape w × h × d Combined latent features The potential characteristics of this combination It is passed to decoder 116. d It refers to depth, or the number of feature channels or filters. Combined latent features Through dense weighted graph W Leveraging the rich information from all codebooks in the system, decoder 116 uses a dense weight graph. W Generate the optimal reconstructed output image The feature fusion unit 114 can combine two input signals using a weighted sum operation, or it can use any other network architecture to combine two input signals. As a non-limiting example, in one embodiment, for (k=1, …, w × h For each vector in ), decoder 116 is used to... The calculation is as follows:

[0050] in, It is a weighted graph W The k-th vector of shape (N, 1) in the middle. There are N latent features based on the codebook. vqs The k-th tensor of shape (N, d).

[0051] In another embodiment, It can also be calculated as:

[0052] in,W and vqs The functions of are interchanged, where vqs Used as a weighted graph W Used as a highly compressed latent feature.

[0053] Figures 2A to 2C This is a schematic diagram of a single codebook, multiple codebooks, and a lead codebook according to an embodiment of the present invention. Figure 2A This illustrates the mapping from feature vectors to a single codebook. As mentioned above, a codebook is a predefined set of representative vectors (also called codewords) in a high-dimensional space, used to represent images. x Embedding features y Each vector in the codebook is associated with an index value. These index values ​​are used to reduce the number of bits stored or transmitted. For example, instead of storing or encoding the entire input vector, the system assigns the index value of the closest codebook vector to the input vector and stores / encodes only that index value. This reduces storage and computational complexity while achieving an approximate representation.

[0054] According to the disclosed embodiments, due to the way the codebooks are trained and used, the coverage of indices in a single codebook does not overlap. Therefore, for an input vector, the input vector will be assigned to one and only one corresponding codeword or representative vector in the codebook, along with the index associated with that representative vector. For example, when an input vector is received, it is compared with all codebook vectors, and the closest codebook vector is selected (e.g., based on some distance metric, typically Euclidean distance). Figure 2A As shown, for an input vector and the case of using only a single codebook X, the input vector can only be assigned to one and only one representative vector in codebook X (e.g., the representative vector associated with index A, the representative vector associated with index B, or the representative vector associated with index C). In the example shown, the input vector is assigned to the representative vector associated with index A, which is the representative vector closest to the input vector.

[0055] However, when using multiple codebooks (e.g., such as...) Figure 2B As shown in codebooks X, Y, and Z, the same input vector will be assigned to multiple indices, with each codebook corresponding to one index. For example, as... Figure 2B As shown, the indices covered by each of the multiple codebooks can overlap, such as... Figure 2B The bolded area is shown in the figure. Furthermore, it can be seen that all vectors within this overlapping space are assigned the same index in each corresponding codebook. Based on this observation, the present invention introduces a novel leading codebook, defined as a codebook including a codebook index that is assigned to a portion or all of the overlapping space of multiple codebooks. For example, Figure 2CThe representative vector in the preamble codebook is shown.

[0056] Figure 3 This is a schematic diagram of a VQ-based encoder framework 300 using a lead codebook according to an embodiment of the present invention. Similar to... Figure 1 The VQ-based encoder framework 100 and the VQ-based encoder framework 300 using a precursor codebook include, for example,... Figure 1 The diagram shows a VQ encoder 102, one or more REP adapters 110, a feature compressor 112, a feature fusion unit 114, and a decoder 116. Figure 1 Compared to the VQ-based encoder framework 100, the VQ-based encoder framework 300 using a preamble codebook may include a preamble codebook 320.

[0057] As described above, the input image x After passing through the VQ encoder 102, the VQ encoder 102 will process the input image. x Convert to embedded features y In one embodiment, the VQ encoder 102 is an HQ encoder. In one embodiment, embedded features... y The shape is w × h × d ,in, w and h Depending on the input width and height, and the network structure of the VQ encoder 102, d It represents the number of feature channels.

[0058] In this embodiment, embedded features y It is passed to the preamble 320. Preamble 320 includes before_quant_leading 322, preamble 324, and after_quant_leading 326. before_quant_leading 322 will pass the latent features. y This is transformed into a new latent feature space. Then, the precursor codebook 320 processes the latent features. y And by using the generated codebook index to retrieve codewords from the precursor codebook 324, a precursor codebook index diagram is generated. hq_idx And the corresponding codeword diagram. As mentioned above, the leading codebook 324 is a codebook that includes a codebook index, which is assigned to a portion or all of the overlapping space of multiple codebooks. after_quant_leading 326 further modifies the leading codebook index diagram. hq_idx Or, as detailed below, the reconstructed precursor codebook index diagram. Transform / encode into a discrete latent space to obtain new latent features vq0 This new potential feature vq0It is sent to the receiving end or the decoding end.

[0059] At the decoding or receiving end, latent features vq0 It is provided as input to one or more REP adapters 110. For example... Figure 1 As shown, the REP adapter 110 includes (1 to N) before_quant_RED 104, (1 to N) RED codebooks 106, and (1 to N) after_quant_RED 108, where N is the number of codebooks included in the REP adapter 110. In one embodiment, for a REP adapter 110 including N codebooks, N before_quant_RED 104 are used to encode latent features. vq0 The latent features are transformed into N discrete latent spaces. N RED codebooks (106) process these latent features and generate N codeword index maps using the corresponding codebook indices. Then, N after_quant_RED (108) processes further transform the N codeword index maps into N new latent features. vqs .

[0060] In some embodiments, a leading codebook index graph is generated for N codebooks using leading codebook 320. hq_ idx Therefore, for a REP adapter 110 containing N codebooks, N RED codebooks 106 can be removed, such as... Figure 3 The dashed lines in the diagram illustrate this. For example, in one embodiment, by directly assigning the corresponding input of each of the N RED codebooks 106 to the output of the corresponding RED codebook 106, the generation and lookup of the N codeword index maps can be skipped, effectively removing the N RED codebooks 106 from the REP adapter 110. In these embodiments, before_quant_RED 104 and after_quant_RED 108 can map a discrete latent precursor space to N discrete latent subspaces, where the characteristics of each discrete latent subspace are based on the distribution of each channel of the output of the feature compressor 112, allowing before_quant_RED 104 and after_quant_RED 108 to adapt to more diverse latent representation spaces. In these embodiments, the roles of the REP adapter 110 and the feature compressor 112 are interchanged. For example, since the output shape of the feature compressor 112 is... w × h The highly compressed latent features, representing ×N, retain the most fundamental N-channel information from the input of feature compressor 112, thus feature compressor 112 can be considered an efficient feature compressor. Since the highly compressed latent feature output shape generated by REP adapter 110 for feature compressor 112 is...w × h × d The weighted graph is ×N, so the REP adapter 110 can be regarded as a feature compressor / weight prediction.

[0061] In one embodiment, the precursor codebook 320 can be viewed as a step of transforming the latent features of the input into a discrete latent space, and the N REP adapters 110 further map the discrete latent precursor space into N discrete latent subspaces. In one embodiment, if the N RED codebooks 106 are not removed from the N REP adapters 110, the characteristics of each subspace are based on the categories and distributions of the dataset used to train the respective RED codebooks 106. However, if the N RED codebooks 106 are removed from the N REP adapters 110, then, as described above, the characteristics of each subspace are based on the distribution of each channel of the output of the feature compressor 112.

[0062] In some embodiments, Figure 3 The output image used to generate the reconstruction The rest of the process is as follows Figure 1 As shown. Alternatively, as Figure 3 As shown, in some embodiments, latent features vqs Potential features generated by the preamble codebook 320 vq0 Alternatively, the codeword index maps can be spliced ​​together at splicer 318, thus increasing the number of codeword index maps from N to N+1. In these embodiments, the feature fusion unit 114 outputs a shape of... w × h A weighted graph of ×(N+1). Additionally, in some embodiments, latent features... vq0 It can be used as a residual skip connection and added to each of the N latent features to form the final latent features. vqs .

[0063] By utilizing Figure 3 The illustrated preamble codebook demonstrates how embodiments of the present invention effectively transform a multi-codebook system into one that only requires sending a preamble codebook index map to the receiver. In a multi-codebook system, N codebook index maps need to be sent to the receiver. Since the index values ​​in the preamble codebook index map correspond to multiple codebooks, the VQ-based encoder framework 300 using the preamble codebook reduces the number of bits that need to be sent to the receiver, thereby improving efficiency without affecting the reconstructed output image. The quality.

[0064] In some embodiments, an optional base codebook is provided, which is expressed by a base codebook index graph. lq_idx Used as a reference to assist the preamble index diagram hq_idx The encoding. For example, as detailed below, in one embodiment, the basic codebook index diagram... lq_idx Used to generate a preamble index graph hq_idx The prediction p (e.g., let the basic codebook index graph) lq_idx The process involves a neural network (NN), which is then trained to make the output p as close as possible to the pre-codebook index graph. hq_idx In one embodiment, the precursor codebook index graph is located in the bitstream. hq_idx With the preamble index diagram hq_idx The residuals / differences between the predicted p values ​​(i.e., residual = hq_idx – p Encoding is performed. Compression is achieved when the residual approaches 0. During the decoding stage, The decoder decodes the residual and adds it. Basic codebook index diagram lq_idx To reconstruct the pre-codebook index graph hq_idx Reconstructed Precursor Codebook Index Diagram hq_idx Hereinafter referred to as In one embodiment, a base codebook is trained such that the base codebook index graph... lq_idx As close as possible to the preamble index graph hq_idx .

[0065] Figure 4 This is a schematic diagram of a basic encoder framework 400 according to an embodiment of the present invention. The basic encoder framework 400 includes a basic encoder 402, an AE / AD 406, a feature extractor 410, a before_quant_base 412, a basic codebook 414, and a code generator 416. In some embodiments, the before_quant_base 412, the basic codebook 414, or the code generator 416 is optional (e.g., ...). Figure 4 (As shown by the dashed line in the middle), which can be omitted from the basic encoder framework 400.

[0066] In one embodiment, the base encoder 402 is an LQ encoder. The base encoder 402 is used to process the input image. xb Transform into latent features yb and hypertensor zb In one embodiment, the input image xb and Figure 3 Input image in x The same. In other embodiments, the input image xb yes Figure 3 Input image in x A low-quality version. In one embodiment, latent features yb The shape is w × h × d ,in, w and hDepending on the input width and height, and the network structure of the base encoder 402, d This refers to the number of feature channels. A hypertensor is a four-dimensional or higher multidimensional array. In some embodiments, the hypertensor... zb Used to assist in the prediction and entropy coding process in AE / AD 406.

[0067] In one embodiment, the AE / AD 406 first quantizes the hypertensor. zb and the supertensor zb Entropy encoding is performed into the bitstream. Then, the AE / AD 406 decodes the bitstream to generate the reconstructed hypertensor. In one embodiment, the AE / AD406 is used to utilize the reconstructed hypertensor. Generating tensors yb The prediction version. Various prediction methods can be used. For example, in one embodiment, the autoregressive (AR) prediction method is used to predict tensors one by one. yb The elements in the table. In another embodiment, the chessboard prediction method is used to predict the tensor. yb Half of the elements are used, with the other half serving as anchors. In yet another embodiment, a mask-based prediction method is used to predict tensors based on fixed or dynamic mask scheduling. yb Elements in the AE / AD 406 quantized tensor. yb Compared with the prediction yb The difference between the two is calculated, and this difference (i.e., the quantization residual) is entropy encoded into the bitstream. Then, the AE / AD 406 performs entropy decoding and tensor prediction to generate the reconstructed tensor. yb In some embodiments, tensors can be generated using separate predictions. yb The predicted version.

[0068] In some embodiments, when input image xb The input image is encoded by traditional compression codecs such as JPEG / HEVC / VVC or by a neural network-based compression codec. x When reconstructing a low-quality version, it is because other codecs have already processed the input image. xb Compression has been performed, and AE / AD 406 is not required to compress / decompress again, so AE / AD 406 can be skipped.

[0069] In one embodiment, feature extractor 410 is used to enhance reconstruction. yb And generate tensors .

[0070] before_quant_base 412 will include potential features ybThis is transformed into a new latent feature space. Then, the basic codebook 414 is used to apply the basic codebook, based on the tensor. Calculate the base codebook index graph. As described above, in one embodiment, the base codebook is trained to generate the base codebook index graph. lq_idx The codebook, basic codebook index diagram lq_idx Similar to (i.e., as close as possible to) the preamble index graph hq_idx .

[0071] The code generator 416 receives the obtained basic codebook index map as input and generates an optimized basic index map. lq_idx The optimized base index graph lq_idx A graph that is closer to the pre-codebook index than previously trained. hq_idx .

[0072] Various neural networks can be used in the VQ-based encoder framework 300 and the basic encoder framework 400 using a precursor codebook. In one embodiment, a Visual Transformer (ViT) or Transformer blocks, such as Residual Swin Transformer Blocks (RSTB), are used to divide the input image or latent features into slices, and these slices are processed as a sequence. In another embodiment, a convolutional neural network (CNN) structure is used, and the entire image or latent features are encoded in parallel.

[0073] Figure 5 This is a schematic diagram of an NNIC system framework 500 according to an embodiment of the present invention. In one embodiment, the NNIC system framework 100 is a dual-branch NNIC framework, including a VQ-based encoder framework 300 using a precursor codebook, a basic encoder framework 400, and AE / AD 502 and ControlNet 506. Figure 5 In the first / preliminary processing branch, the input image x After passing through the VQ encoder 102, the VQ encoder 102 will process the input image. x Convert to embedded features y Embedded features y It is passed to the preamble codebook 320. For example... Figure 3 As shown, the 320-prefix codebook processes latent features. y And generate a preamble index graph. hq_idx .exist Figure 5 In the second / basic processing branch, the input image xbProcessed using the basic encoder framework 400 to generate, for example Figure 4 The optimized base index graph shown lq_idx .

[0074] like Figure 5 As shown, in the NNIC system framework 500, the AE / AD 502 is used to receive an optimized base index graph from the base processing branch. lq_idx And the preamble index diagram from preamble 320 hq_idx In one embodiment, the AE502 is used to perform a prediction operation to use an optimized base index graph. lq_idx As an optional reference, generate a precursor codebook index diagram. hq_idx The predicted version. As mentioned above, various prediction methods can be used. In one embodiment, once AE 502 generates the predicted version... hq_idx Then AE 502 is used for the preamble index graph. hq_idx Compared with the prediction hq_idx The quantization residuals between the two are quantized and entropy encoded. Then, the AE 502 performs entropy decoding and tensor prediction to generate a reconstructed precursor codebook index graph. Reconstructed Precursor Codebook Index Graph The preamble codebook 320 is returned to the preamble processing branch. Then, preamble codebook 320 is based on the reconstructed preamble codebook index graph. Generate the corresponding codeword index diagram vq0 after_quant_leading 326 will generate a codeword index diagram. vq0 Transform into a discrete latent space to obtain latent features vq0 .

[0075] like Figure 3 As shown, in some embodiments, then, latent features vq0 It is passed to one or more REP adapters 110. For example... Figure 3 As shown, the REP adapter 110 will have potential features vq0 The process is transformed into N discrete latent spaces, generating N codeword index maps, and then the N codeword index maps are transformed into N codebook-based latent features. vqs .

[0076] like Figure 4 As shown, in some embodiments, the feature extractor 410 in the basic encoder framework 400 is used to enhance the reconstruction. yb And generate tensors .like Figure 5 As shown, in some embodiments, tensors The input is provided to feature compressor 112 and ControlNet 506. Feature compressor 112 is used to work with tensors. Generate N-channel weight map W As described above, the feature fusion unit 114 receives the weight map. W and N codebook-based latent features vqs Weighted graph W Guide / instruct N codebook-based latent features vqs How to perform weighted combination to generate latent features of the combination? .

[0077] In one embodiment, tensors are used. As input, ControlNet 506 generates control signals. yc To be based on combined latent features The guiding condition generates the reconstruction process. Decoder 116 uses combined latent features. and control signals yc Both generate the reconstructed output image. Decoder 116 can use control signals in a variety of ways. yc To guide the reconstruction process. In some embodiments, the decoder 116 has a network structure including multiple CNN layers, for example, a VAE decoding network. Then, control signals... yc Adjusting combined latent features through affine transformation And generate new adjusted features. : , in, It is the preset guidance strength, affine parameters and It is given by the following formula: , in, It's a concatenation operation. This affine transformation is usually determined by a parameter set. The NN implementation is determined through training.

[0078] As described above, by using the pre-defined codebook, the NNIC system framework 500 can convert N codeword index maps into a single codeword index map, thereby reducing the number of bits that need to be sent to the receiver without affecting the reconstructed output image. The quality of the codewords. In some embodiments, to further reduce the overall code rate, the codewords in the codebook can be reordered in descending order based on the frequency of codeword usage in the training data. For example, for images depicting nature, the codewords corresponding to natural landscapes may be used more frequently than the codewords corresponding to human faces.

[0079] Furthermore, existing methods do not consider the importance factor differences between channels, resulting in suboptimal performance. For example, this invention confirms the following: (1) the correlation between latent channels is relatively constant for different images; (2) a few different channels are more sensitive to degradation operations such as quantization; and (3) these different channels have low correlation with all other channels, making them special and particularly important compared to other channels. Since the output tensor of a layer is generated by multiplying the input weight tensor and the input activation tensor of the layer, the correlation between latent channels is caused by the input weight tensor of the channel and is an inherent property of the pre-trained model. However, in current codebook codeword and index generation methods, codewords... (i=1, …, w×h) is closest to the corresponding latent feature y i , It's typing Index:

[0080] in, It is a distance metric, such as the L1 or L2 norm. Because Existing methods treat the distance between channels equally, but fail to consider the importance factor differences between channels, resulting in suboptimal performance. Therefore, some embodiments provide a channel-weighted codeword generation method to take into account the importance factor differences between channels. For example, in one embodiment, the vector is defined as a weighting factor. wf and to The output is multiplied element by element:

[0081] In one embodiment, vector wf It can be defined as a predefined vector, where constant values ​​are provided as channel importance factors for each channel, or a vector. wf It can be defined as a learnable vector so that the vector can be optimized during training. wf .

[0082] Furthermore, existing methods do not consider the rate factor, resulting in suboptimal rate distortion (RD) performance. Therefore, in some embodiments, the following RD-based codeword generation method is provided to take the rate factor into account:

[0083] in, It is an entropy measure used to estimate the entropy of codeword index k. It is a distance metric, such as the L1 or L2 norm, and λ is a hyperparameter used to control the weights between bit rate and distortion.

[0084] In some embodiments, since the precursor codebook and base codebook described herein are trained separately, and the input to the precursor codebook is the high-quality encoder output, while the input to the base codebook is the low-quality base encoder output, a mismatch occurs between the precursor and base codebooks after this training process. This means that two identical corresponding vectors in the precursor and base codebooks may be assigned different index values, and the difference between these two index values ​​may be large, which is detrimental to compression because the difference will be compressed. Therefore, it is desirable to minimize the difference.

[0085] In one embodiment, the following position matching method is provided to minimize the difference between high-quality and low-quality indices. First, a base codebook is trained independently. Then, the latent features of the training data are processed using a VQ encoder 102 and a precursor codebook. The position matching method iterates through the HQ indices, recording the vector position in the set R(n) of any HQ latent feature vectors assigned the same codeword index n for each codeword index n. Then, the same latent features of the training data are processed using the base encoder 402 and the base codebook. The position matching method records all LQ indices of all LQ latent feature vectors. Then, the position matching method iterates through the HQ indices, recording all LQ indices of LQ vectors in the position set R(n) for each HQ codeword index n, calculating a histogram of the LQ indices, and selecting the most frequently used LQ index as the matching index for HQ index n. Finally, the position matching method reorders the base codebook based on the generated LQ / HQ index matching lookup table.

[0086] By using independently trained precursor and base codebooks, the position matching method minimizes the difference between high-quality and low-quality indices by assigning the same index to matching positions in the HQ / LQ latent feature maps as much as possible.

[0087] In some embodiments, the position matching method can be viewed as a classification problem, where the HQ codeword index can be viewed as a classification label for LQ latent features.

[0088] In one embodiment, a contrastive classification training method is provided to train a base codebook, wherein, for an HQ codeword index n, if the HQ codeword index of the HQ feature vector corresponding to all LQ feature vectors is equal to n, then all LQ feature vectors are considered positive samples; if the HQ codeword index of the HQ feature vector corresponding to all LQ feature vectors is not equal to n, then all LQ feature vectors are considered negative samples.

[0089] Since the codeword index is determined by argmin(), and argmin() is not differentiable, a differentiable metric can be used instead during training.

[0090] For example, in one embodiment, softmax() can be used as a soft classification label:

[0091] In another embodiment, , Or their normalized versions can be used as ground truth for training.

[0092] Suppose z is an LQ latent feature, C is the basic codebook, and gt_dist is the precursor codebook. The output codebook loss can be defined using the following exemplary pseudocode: Distance calculation: q_dist=Dist( z.detach(), C ) e_dist=Dist( z, C.detach() ) Use softmax() for distance normalization: q_norm=torch.nn.functional.softmax( -q_dist / temp, dim=1 ) e_norm=torch.nn.functional.softmax( -e_dist / temp, dim=1 ) gt_norm=torch.nn.functional.softmax( -gt_dist.detach() / temp, dim=1 ) Codebook loss using L2 loss: q_dist_loss=self.mse( q_norm, gt_norm ) e_dist_loss=self.mse( e_norm, gt_norm ) Total codebook loss: codebook_loss=q_dist_loss+e_dist_loss beta In some embodiments, a masking and reconstruction method is provided to further minimize the amount of leading index graphs. hq_idx Information transmission. For example. Figure 6This is a schematic diagram of a masked generative encoder (MAGE) prediction method 600 according to an embodiment of the present invention.

[0093] Utilizing a masked schedule 602 equipped with a predefined, structured, and compression-friendly schedule, and with the aid of optional references lq_idx Only for hq_idx The unmasked portion ( hq_midx The unmasked portion is encoded and included in the bitstream. The selected schedule indicates the portion of the remaining information, which helps to achieve a more efficient compression strategy.

[0094] In one embodiment, the mask scheduling is a fixed schedule and therefore does not need to be included in the bitstream. In another embodiment, the mask scheduling is content-adaptive, so the mask / demask map needs to be included in the bitstream. In yet another embodiment, the mask scheduling is content-adaptive, but the content-based distribution only provides a limited number of fixed schedules, so only the schedule index needs to be included in the bitstream.

[0095] AE / AD 604 first performs a prediction operation to use lq_idx Generate as an optional reference hq_midx The predicted version. Then, the quantization residuals ( hq_midx Compared with the prediction hq_midx The difference between the two values ​​is encoded and the quantized residual is included in the bitstream. The AE / AD 604 decodes the bitstream to reconstruct the residual and uses the predicted value. hq_midx Generate refactored In one embodiment, an encoder-decoder Transformer architecture is employed, based on... Generate refactored The pre-trained MAGE is provided as the basic structure.

[0096] The indexed word segmenter 606 will Convert to its embedded version hq_mt Then, the MAGE Transformer encoder 608 will... hq_mt Convert to encoded version hq_emt and the encoded version hq_emt The data is transmitted to the improved MAGETransformer decoder 610. The improved MAGETransformer decoder 610 is used to process conditional boot control information. Predict masked words. A 612-layer linear classifier generates the predicted logit. hq_logits Finally, the Argmax() operation uses the predicted logit of 614. hq_logits To generate refactored .

[0097] In one embodiment, in order to combine the output from the base encoder The conditional control information is inspired by techniques commonly used in audio-to-text conversion tasks, where audio information is used to guide text lexical generation via cross-attention, inserting cross-attention blocks into each decoder block of the Transformer decoder. This allows lexical units to be generated from... Extracting relevant information facilitates guided lexical generation.

[0098] In one training embodiment, the cross-attention block inserted into the improved MAGE Transformer decoder 610 only affects the decoder block, while the remaining pre-trained parameters remain unchanged and frozen. Only the newly inserted cross-attention block, the multi-layer perceptron (MLP) block of each decoder block, and the outermost class linear layer are unfrozen. Following the training design of MAGE, the decoder is trained to accurately predict masked terms.

[0099] In one embodiment, there is no residual signal, or some or all of the residual signal (reconstructed) With the original hq_ idx The difference between them is encoded and included in the bitstream.

[0100] In some embodiments, due to the varying details and complexity of image regions, a statistical process is provided to identify complexity in low-bit compression, wherein a complexity evaluation function is constructed using image entropy, contrast, color diversity (histogram entropy), and spatial frequency (Fourier transform). The normalized complexity score integrates these aspects and reflects the image complexity. The complexity formula is as follows: F(X) = w1 entropy(x)+w2 contrast(X)+w3 color_entropy(X)+w4 s_freq(X), Among them, entropy(), contrast(), color_entropy(), and s_freq() represent image entropy, image contrast, color diversity (histogram entropy), and spatial frequency (Fourier transform).

[0101] Using various images, the image complexity distribution was obtained through a formula, and different thresholds for dynamic mask scheduling were set accordingly. A lower mask ratio was applied to complex image patches, while a very high mask ratio was applied to simple patches.

[0102] First, the image is divided into multiple blocks, and then different mask scheduling is applied to blocks with varying complexities. This ensures that, while maintaining image quality, more transmission bits are allocated to high-complexity regions, while lower bitrates are used for low-complexity regions. The complexity-aware masking process provides a direct and effective method to further dynamically reduce the transmission bitrate, enabling our proposed framework to better serve extreme low-bitrate image compression scenarios.

[0103] Figure 7 This is a flowchart of an image compression method 700 based on a VQ neural network implemented by an encoding device according to an embodiment of the present invention. In step 702, the encoding device encodes a first image using a first encoder to obtain a first latent feature corresponding to the first image. In step 704, based on the first latent feature, the encoding device generates a pre-codebook index map and a corresponding first codeword map using a pre-codebook. The pre-codebook includes a codebook index, which is assigned to the same vector in multiple codebooks. In step 706, the encoding device encodes the pre-codebook index map to generate an encoded pre-codebook index map. In step 708, the encoding device sends the encoded pre-codebook index map to the decoder.

[0104] Figure 8 This is a flowchart of an image compression method 800 based on a VQ neural network implemented by a decoding device according to an embodiment of the present invention. In step 802, the decoding device receives an encoded pre-codebook index map, wherein the encoded pre-codebook index map includes an index that maps to the same vector in multiple codebooks. In step 804, the decoding device, based on the encoded pre-codebook index map, retrieves codewords from N codebooks using the corresponding codebook index in the encoded pre-codebook index map, generating N latent feature maps. In step 806, the decoding device combines the N latent feature maps with an N-channel weight map to generate reconstructed latent features. In step 808, the decoding device decodes the reconstructed latent features to obtain a reconstructed image.

[0105] Figure 9 This is a schematic diagram of an apparatus 900 according to an embodiment of the present invention. The apparatus 900 can be used to implement embodiments of the present invention. For example, the apparatus 900 can be used to perform the functions of a transmitting / transmitting device or a receiving device according to any embodiment of the present invention. The apparatus 900 includes a receiving unit (RX) 920 or a receiving component for receiving data through an input port 910. The apparatus 900 also includes a transmitting unit (TX) 940 or a transmitting component for transmitting data through a data output port 950.

[0106] The apparatus 900 includes a memory 960 or data storage component for storing instructions and various types of data. The memory 960 can be any type or combination of storage components capable of storing data and / or instructions. For example, the memory 960 may include volatile and / or non-volatile memories such as read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The memory 960 may also include one or more disk drives, tape drives, and solid-state drives. In some embodiments, the memory 960 may be used as an overflow data storage device to store a program when a program is selected for execution, and to store instructions and data read during program execution. In some embodiments, the memory 960 may be a memory integrated with the processor 930.

[0107] The apparatus 900 includes one or more processors 930 or other processing components (such as a central processing unit, CPU) to process instructions. The one or more processors 930 may be implemented as one or more CPU chips, cores (such as multi-core processors), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The one or more processors 930 are communicatively coupled to ingress ports 910, RX 920, TX 940, egress ports 950, and memory 960 via a system bus. The one or more processors 930 may be used to execute instructions stored in memory 960. For example, in one embodiment, memory 960 stores a neural network image compression unit 970. The neural network image compression unit 970 includes data, executable instructions, and / or one or more subroutines for implementing the disclosed embodiments. Therefore, the one or more processors 930 provide a means for performing any calculation, comparison, determination, initiation, configuration, or any other operation corresponding to the claims when appropriate instructions are executed by the processor 930. Therefore, the addition of the neural network image compression unit 970 significantly improves the functionality of the device 900.

[0108] While this invention provides several embodiments, those skilled in the art will understand that the disclosed systems and methods can be embodied in many other specific forms without departing from the spirit or scope of the invention. The examples of this invention should be considered illustrative rather than restrictive, and the invention is not limited to the details given herein. For example, various elements or components may be combined or integrated into other systems, or certain features may be omitted or not implemented.

[0109] In summary, the disclosed embodiments include, but are not limited to, the following novel features: A dual-branch framework includes a RED-VQ encoder and a base encoder, wherein the RED-VQ encoder uses a masked generative encoder (MAGE) for mask codebook index prediction. In some embodiments, high mask ratio (50%, 75%, 89%, 94%) index predictions performed by the MAGE provide better index predictions and further reduce the bit rate. The complexity-aware masking technique described above further reduces the transmission bit rate while maintaining image reconstruction quality.

[0110] The introduction of a preamble codebook effectively transforms a multi-codebook system into a pipelined system combining a preamble codebook with one or more REP adapters. This further reduces the code rate.

[0111] Furthermore, some embodiments include codebook reordering, channel weight index generation, and RD-based index generation, achieving better codeword matching. Additionally, in some embodiments, position matching and contrastive classification-based training provide a better reference index graph for index prediction and compression.

[0112] Furthermore, without departing from the scope of the invention, the technologies, systems, subsystems, and methods described and illustrated as discrete or separate in the various embodiments can be combined or integrated with other systems, modules, technologies, or methods. Other items shown or described as coupled, directly coupled, or communicating with each other may be indirectly coupled or communicated electrically, mechanically, or otherwise through some interface, device, or intermediate component. Those skilled in the art can identify other examples of changes, substitutions, and modifications, and make changes, substitutions, and modifications without departing from the spirit and scope of the invention.

Claims

1. An image compression method based on vector quantization (VQ) neural networks implemented by an encoding device, characterized in that, The method includes: The first image is encoded using a first encoder of the encoding device to obtain a first latent feature corresponding to the first image; Based on the first latent feature, a pre-codebook index map and a first codeword map corresponding to the pre-codebook index map are generated using the pre-codebook, wherein the pre-codebook includes a codebook index, and the codebook index is assigned to the same vector in multiple codebooks; The preamble index map is encoded to generate an encoded preamble index map; The encoded preamble index map is sent to the decoder.

2. The image compression method based on VQ neural network according to claim 1, characterized in that, The second image is encoded using the second encoder of the encoding device to obtain a second latent feature and a hypertensor corresponding to the second image; Based on the hypertensor, a basic codebook index graph is generated using the basic codebook, wherein encoding the precursor codebook index graph to generate the encoded precursor codebook index graph includes using the basic codebook index graph as a reference.

3. The image compression method based on VQ neural network according to claim 2, characterized in that, The base codebook is trained to create a base codebook index graph similar to the precursor codebook index graph.

4. The image compression method based on VQ neural network according to claim 2 or 3, characterized in that, The base codebook is trained to create a base codebook index map that is as close as possible to the precursor codebook index map.

5. The image compression method based on a VQ neural network according to any one of claims 2 to 4, characterized in that, Based on the hypertensor, generating the basic codebook index graph using the basic codebook includes: The hypertensor is quantized and entropy encoded to obtain the encoded hypertensor; The encoded hypertensor is decoded to obtain the reconstructed hypertensor; Tensor prediction is performed using the reconstructed hypertensor to obtain the predicted tensor of the first tensor; The difference between the first tensor and the predicted tensor is quantized and entropy encoded. Perform entropy decoding and tensor prediction to generate the first reconstructed tensor; Based on the first tensor of the reconstruction, the basic codebook index graph is calculated using the basic codebook.

6. The image compression method based on a VQ neural network according to any one of claims 2 to 5, characterized in that, The second image is a low-quality version of the first image.

7. The image compression method based on a VQ neural network according to any one of claims 2 to 5, characterized in that, The second image is the same as the first image.

8. The image compression method based on a VQ neural network according to any one of claims 2 to 7, characterized in that, The codewords in the lead codebook or the base codebook are reordered in descending order of frequency, so that fewer bits are allocated to the indexes that are used more frequently, thereby reducing the total code rate.

9. The image compression method based on a VQ neural network according to any one of claims 2 to 8, characterized in that, The codewords in the lead codebook or the base codebook are generated by a channel-weighted codeword generation method, which takes into account the difference in importance factors between channels.

10. The image compression method based on a VQ neural network according to any one of claims 2 to 9, characterized in that, The codewords in the leading codebook or the basic codebook are generated by a codeword generation method that takes into account the code rate factor.

11. The image compression method based on a VQ neural network according to any one of claims 2 to 10, characterized in that, Also includes: A position matching method is performed to minimize the difference between the precursor codebook index map and the base codebook index map.

12. The image compression method based on a VQ neural network according to any one of claims 2 to 11, characterized in that, Also includes: The base codebook is trained using a contrastive classification training method.

13. The image compression method based on a VQ neural network according to any one of claims 1 to 12, characterized in that, Also includes: Perform masking and reconstruction to further minimize information transmission in the encoded preamble index graph.

14. The image compression method based on VQ neural network according to claim 13, characterized in that, The masking process applies a mask ratio based on image complexity.

15. An image compression method based on vector quantization (VQ) neural networks implemented by a decoding device, characterized in that, The method includes: Receive an encoded pre-codebook index map, wherein the encoded pre-codebook index map includes an index, the index being mapped to the same vector in multiple codebooks; Based on the encoded pre-codebook index map, codewords are retrieved from N codebooks by using the corresponding codebook index in the encoded pre-codebook index map to generate N potential feature maps; The N latent feature maps are combined with the N-channel weight map to generate reconstructed latent features; The latent features of the reconstruction are decoded to obtain the reconstructed image.

16. The image compression method based on VQ neural network according to claim 15, characterized in that, Decoding the latent features of the reconstruction to obtain the reconstructed image includes: A control signal is generated to guide the conditional generation reconstruction process when decoding the latent features of the reconstruction to obtain the reconstructed image.

17. An apparatus, characterized in that, include: Memory, used to store instructions; One or more processors are coupled to the memory and are used to execute the instructions to cause the device to perform the following operations: The first image is encoded using the first encoder of the device to obtain a first latent feature corresponding to the first image; Based on the first latent feature, a pre-codebook index map and a first codeword map corresponding to the pre-codebook index map are generated using the pre-codebook, wherein the pre-codebook includes a codebook index, and the codebook index is assigned to the same vector in multiple codebooks; The preamble index map is encoded to generate an encoded preamble index map; The encoded preamble index map is sent to the decoder.

18. The apparatus according to claim 17, characterized in that, The one or more processors execute the instructions to further cause the device to perform the following operations: The second image is encoded using the second encoder of the device to obtain a second latent feature and a hypertensor corresponding to the second image; Based on the hypertensor, a basic codebook index graph is generated using the basic codebook, wherein encoding the precursor codebook index graph to generate the encoded precursor codebook index graph includes using the basic codebook index graph as a reference.

19. The apparatus according to claim 18, characterized in that, The base codebook is trained to create a base codebook index graph similar to the precursor codebook index graph.

20. The apparatus according to claim 18 or 19, characterized in that, The base codebook is trained to create a base codebook index map that is as close as possible to the precursor codebook index map.

21. The apparatus according to any one of claims 18 to 20, characterized in that, The one or more processors execute the instructions to further cause the device to perform the following operations: The hypertensor is quantized and entropy encoded to obtain the encoded hypertensor; The encoded hypertensor is decoded to obtain the reconstructed hypertensor; Tensor prediction is performed using the reconstructed hypertensor to obtain the predicted tensor of the first tensor; The difference between the first tensor and the predicted tensor is quantized and entropy encoded. Perform entropy decoding and tensor prediction to generate the first reconstructed tensor; Based on the first tensor of the reconstruction, the basic codebook index graph is calculated using the basic codebook.

22. The apparatus according to any one of claims 18 to 21, characterized in that, The second image is a low-quality version of the first image.

23. The apparatus according to any one of claims 18 to 22, characterized in that, The codewords in the lead codebook or the base codebook are reordered in descending order of frequency, so that fewer bits are allocated to the indexes that are used more frequently, thereby reducing the total code rate.

24. The apparatus according to any one of claims 18 to 23, characterized in that, The codewords in the lead codebook or the base codebook are generated by a channel-weighted codeword generation method, which takes into account the difference in importance factors between channels.

25. The apparatus according to any one of claims 18 to 24, characterized in that, The codewords in the leading codebook or the basic codebook are generated by a codeword generation method that takes into account the code rate factor.

26. The apparatus according to any one of claims 18 to 25, characterized in that, The one or more processors execute the instructions to further cause the device to perform a position matching method to minimize the difference between the precursor codebook index map and the base codebook index map.

27. The apparatus according to any one of claims 18 to 26, characterized in that, The one or more processors execute the instructions to further enable the device to train the base codebook using a contrastive classification training method.

28. The apparatus according to any one of claims 18 to 21, characterized in that, The second image is the same as the first image.

29. The apparatus according to any one of claims 17 to 28, characterized in that, The one or more processors execute the instructions to further enable the apparatus to perform masking and reconstruction, thereby further minimizing the information transmission of the encoded preamble index graph.

30. The apparatus according to claim 29, characterized in that, The masking process applies a mask ratio based on image complexity.

31. An apparatus, characterized in that, include: Memory, used to store instructions; One or more processors are coupled to the memory and are used to execute the instructions to cause the device to perform the following operations: Receive an encoded pre-codebook index map, wherein the encoded pre-codebook index map includes an index, the index being mapped to the same vector in multiple codebooks; Based on the encoded pre-codebook index map, codewords are retrieved from N codebooks by using the corresponding codebook index in the encoded pre-codebook index map to generate N potential feature maps; The N latent feature maps are combined with the N-channel weight map to generate reconstructed latent features; The latent features of the reconstruction are decoded to obtain the reconstructed image.

32. The apparatus according to claim 31, characterized in that, The one or more processors execute the instructions to further cause the device to generate control signals to guide the conditional generation reconstruction process when decoding the potential features of the reconstruction to obtain the reconstructed image.

33. A computer program product, characterized in that, Includes computer-executable instructions stored in a non-transitory computer-readable storage medium, wherein, when executed by one or more processors of the device, the computer-executable instructions cause the device to perform the method according to any one of claims 1 to 17.

34. An apparatus, characterized in that, include: A first encoder or a first encoding component is used to encode a first image to obtain a first latent feature corresponding to the first image; A first generating component for performing the following operation or a component for performing the following operation: generating a pre-codebook index map and a first codeword map corresponding to the pre-codebook index map based on the first latent feature, wherein the pre-codebook includes a codebook index, and the codebook index is assigned to the same vector in multiple codebooks; The second encoding component is used to encode the preamble index map to generate an encoded preamble index map. A transmitting component or a component for performing the following operation: sending the encoded preamble index diagram to the decoder.

35. The apparatus according to claim 34, characterized in that, Also includes: A second encoder or encoding component is used to encode a second image to obtain a second latent feature and a hypertensor corresponding to the second image; A second generating component for performing the following operation or a component for performing the following operation: generating a base codebook index graph based on the hypertensor using the base codebook, wherein encoding the precursor codebook index graph to generate the encoded precursor codebook index graph includes: using the base codebook index graph as a reference.

36. The apparatus according to claim 35, characterized in that, The base codebook is trained to create a base codebook index graph similar to the precursor codebook index graph.

37. The apparatus according to claim 35 or 36, characterized in that, The base codebook is trained to create a base codebook index map that is as close as possible to the precursor codebook index map.

38. The apparatus according to any one of claims 35 to 37, characterized in that, Also includes: A first quantization and entropy encoding component is used to quantize and entropy encode the hypertensor to obtain the encoded hypertensor. A decoding component is used to decode the encoded hypertensor to obtain the reconstructed hypertensor; Tensor prediction component, used to perform tensor prediction using the reconstructed hypertensor to obtain the predicted tensor of the first tensor; The second quantization and entropy coding component is used to quantize and entropy code the difference between the first tensor and the prediction tensor. Entropy decoding and tensor prediction components are used to perform entropy decoding and tensor prediction to generate the first reconstructed tensor; A computational component for computing the base codebook index graph using the base codebook based on the first tensor of the reconstruction.

39. The apparatus according to any one of claims 35 to 38, characterized in that, The second image is a low-quality version of the first image.

40. The apparatus according to any one of claims 35 to 39, characterized in that, The codewords in the lead codebook or the base codebook are reordered in descending order of frequency, so that fewer bits are allocated to the indexes that are used more frequently, thereby reducing the total code rate.

41. The apparatus according to any one of claims 35 to 40, characterized in that, The codewords in the lead codebook or the base codebook are generated by a channel-weighted codeword generation method, which takes into account the difference in importance factors between channels.

42. The apparatus according to any one of claims 35 to 41, characterized in that, The codewords in the leading codebook or the basic codebook are generated by a codeword generation method that takes into account the code rate factor.

43. The apparatus according to any one of claims 35 to 42, characterized in that, It also includes a position matching component for minimizing the difference between the precursor codebook index map and the base codebook index map.

44. The apparatus according to any one of claims 35 to 43, characterized in that, It also includes a contrastive classification training component for training the base codebook.

45. The apparatus according to any one of claims 35 to 38, characterized in that, The second image is the same as the first image.

46. ​​The apparatus according to any one of claims 34 to 45, characterized in that, Also includes: Masking and reconstruction components are used to further minimize the information transmission of the encoded preamble index graph.

47. The apparatus according to claim 46, characterized in that, The masking and reconstruction components apply a mask ratio based on image complexity.

48. An apparatus, characterized in that, include: A receiving component is used to receive an encoded pre-codebook index map, wherein the encoded pre-codebook index map includes an index that maps to the same vector in multiple codebooks; A component is generated to retrieve codewords from N codebooks based on the encoded pre-codebook index map by using the corresponding codebook index in the encoded pre-codebook index map, and to generate N potential feature maps. A combination component is used to combine the N latent feature maps with the N-channel weight map to generate reconstructed latent features; A decoding component is used to decode the latent features of the reconstruction to obtain a reconstructed image.

49. The apparatus according to claim 48, characterized in that, It also includes a control signal generation component for generating control signals to guide the conditional generation reconstruction process when decoding the latent features of the reconstruction to obtain the reconstructed image.