Image coding, decoding and compression method and system for vector quantization entropy modeling

By using lattice vector quantizer and relaxed boundary entropy modeling in deep learning-based image compression, the training difficulties and consistency problems of vector quantization methods in image compression are solved, improving rate distortion performance and saving code stream overhead.

CN118984392AActive Publication Date: 2024-11-19SHANGHAI JIAOTONG UNIV

Patent Information

Application Number
CN202411119988.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-19
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

In deep learning-based image compression, the existing vector quantization method has problems such as convergence during training and inconsistent training and testing, resulting in poor rate distortion performance.

Method used

The image features are quantized using a lattice vector quantizer, and the quantized features are entropy-modeled through relaxed boundaries to reduce complexity and improve training stability and consistency.

Benefits of technology

The rate distortion performance of image compression is improved, the complexity is close to that of scalar quantizer, which avoids codebook collapse problem, and saves code stream overhead when the reconstructed image quality is the same.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984392A_ABST
    Figure CN118984392A_ABST
Patent Text Reader

Abstract

The invention discloses an image coding, decoding and compressing method and system for vector quantization entropy modeling. The method comprises the following steps: acquiring a first feature map of a to-be-coded image; obtaining a hyper-prior binary code stream and hyper-prior information of the first feature map; quantizing the first feature map by using a lattice vector quantizer; projecting the quantized first feature map to an integer coefficient domain of a lattice vector quantizer to obtain a second feature map; performing parametric modeling on the distribution of the second feature map to obtain mutually independent Gaussian distribution of each dimension, and predicting the mean value and variance of the Gaussian distribution by combining the hyper-prior information and a spatial context model; and dividing an integer coefficient domain of a lattice vector quantizer by adopting a relaxation boundary, performing probability estimation and arithmetic coding according to a predicted mean value and variance of Gaussian distribution to obtain a feature binary code stream, and combining the feature binary code stream with a super-prior feature binary code stream to obtain a compressed image binary code stream. According to the method, more efficient lattice vector quantization is adopted for the three-dimensional features, so that the rate distortion performance of image compression is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and specifically relates to an image encoding, decoding and compression method and system based on vector quantization entropy modeling in an image compression method based on deep learning. Background Art

[0002] In recent years, image compression methods based on deep learning have flourished. Image compression based on deep learning is mainly implemented in the form of transform coding, which is mainly divided into three parts: transform, quantization and entropy coding. At the same time, these three parts are optimized in an end-to-end manner. Specifically, the image to be encoded first obtains three-dimensional features through a nonlinear deep neural network, quantizes the three-dimensional features, and then uses an entropy model to estimate the distribution of quantized features and entropy encodes the features. Since the quantization operation is incompatible with the gradient back propagation in the image compression model training process, quantization has been less studied in image compression based on deep learning. But obviously, efficient quantization methods can improve the rate-distortion performance of image compression.

[0003] Scalar quantizers are widely used in deep learning-based image compression. The assumption behind the use of scalar quantizers is that powerful nonlinear transformations can be adapted to clumsy scalar quantization. After searching the literature on the prior art, it was found that scalar quantizers were first used in the paper "End-to-end Optimized Image Compression" published by Ballé et al. at the 2017 International Conference on Learning Representation. However, this assumption is flawed because scalar quantizers have been shown to be suboptimal even for independent sources in each dimension. In contrast, vector quantizers provide a more efficient way to partition space. However, very few prior arts consider the use of vector quantizers in deep learning-based image compression.

[0004] Agustsson et al. proposed a soft-to-hard vector quantization method in the paper "Soft-to-hard Vector Quantization for End-to-end Learning Compressible Representations" published at the 2017 Conference and Workshop on Neural Information Processing Systems. However, this method will cause inconsistencies between training and testing, and will also cause training instability, so the rate-distortion performance is not as good as the scalar quantizer. Zhu et al., in the paper "Unified Multivariate Gaussian Mixture for Efficient Neural Image compression" published at the 2022 IEEE Conference on Computer Vision and Pattern Recognition, adopted a cascaded vector quantization method based on soft-to-hard to improve the rate-distortion performance. However, this method is more complex, and because the codebook and model are trained together, problems such as codebook collapse will occur. Zhang et al. proposed using diamond lattice vector quantization in their paper “VQAC: Lattice Vector Quantization Coupled With Spatially Adaptive Companding for Efficient Learned Image Compression” published at the 2023 IEEE Conference on Computer Vision and Pattern Recognition. However, they still adopted a soft-to-hard approach and implemented it with two scalar quantizers.

[0005] In summary, due to the soft-to-hard training method, the existing vector quantization methods in deep learning-based image compression have problems with convergence during training and inconsistency between training and testing. Summary of the invention

[0006] In view of the above problems, the present invention proposes an image encoding, decoding, and compression method and system based on vector quantization entropy modeling, which uses a lattice vector quantizer to quantize features and adopts relaxed boundaries to perform entropy modeling on the quantized features, thereby reducing complexity and making training more stable. At the same time, it solves the problem of inconsistency between training and testing, thereby improving the rate-distortion performance of image compression.

[0007] A first aspect of the present invention provides an image encoding method using vector quantization entropy modeling, comprising:

[0008] Obtaining a first feature map of the image to be encoded through the encoding network, wherein the first feature map is a three-dimensional feature map, and the dimensions include a channel dimension, a height dimension, and a width dimension;

[0009] Obtaining a super-prior feature binary code stream and corresponding super-prior information of the first feature map;

[0010] quantizing the first feature map using a lattice vector quantizer to obtain a quantized first feature map;

[0011] Projecting the quantized first feature map to an integer coefficient domain of a lattice vector quantizer to obtain a second feature map;

[0012] Modeling the distribution parameter of the second feature map as a Gaussian distribution in which each dimension is independent from each other, and predicting the mean and variance of the Gaussian distribution by combining the super prior information and the spatial context model;

[0013] Using a relaxed boundary to partition the integer coefficient domain of the lattice vector quantizer, performing probability estimation on the second feature map according to the predicted mean and variance of the Gaussian distribution, performing arithmetic coding according to the estimated probability, and obtaining a feature binary code stream;

[0014] The feature binary code is combined with the super a priori feature binary code stream to obtain a compressed image binary code stream.

[0015] Optionally, obtaining a super a priori binary code stream of the first feature map and corresponding super a priori information includes:

[0016] Using a super priori coding network on the first feature map to obtain a super priori feature map;

[0017] Rounding off each feature in the super a priori feature map and quantizing it to an integer to obtain a quantized super a priori feature map;

[0018] Establishing an independent non-parametric distribution estimate for each channel of the quantized super-prior feature map;

[0019] Performing arithmetic coding on the super-a priori second feature map according to the non-parametric distribution estimation to obtain a super-a priori feature binary code stream;

[0020] arithmetically decoding the super-prior binary code stream using the non-parametric distribution estimation to obtain a reconstructed super-prior feature map;

[0021] A super-prior decoding network is used to obtain super-prior information for the reconstructed super-prior feature map.

[0022] Optionally, quantizing the first feature map by using a lattice vector quantizer to obtain a quantized first feature map includes:

[0023] Construct an n-dimensional lattice based on a pre-set arrangement

[0024] The feature vector of the i1th row and j1th column of the first feature graph with height H1, width W1 and number of channels M1 is Evenly divided into M1 / n vectors The k1th vector

[0025] For any vector Select the nearest lattice point in the lattice Λ according to the second norm As a quantized vector, the specific calculation process is:

[0026]

[0027] The quantized M1 / n vectors Splicing to obtain the first quantized feature map The eigenvector of the i1th row and j1th column is

[0028] Get the feature vectors of all H1 rows and W1 columns to get the first quantized feature map The quantized first feature map The height and width of are the same as the height and width of the first feature map.

[0029] Optionally, the construction of an n-dimensional lattice include:

[0030] Get a set of linearly independent n-dimensional basis {v1,v2,…,v n}, where the value of each basis is related to the specific arrangement of the lattice. Common lattice arrangements include cubic arrangement, rhombus arrangement, etc.

[0031] The set of linearly independent n-dimensional bases {v1,v2,…,v n} perform linear combination (where the combination coefficient is an integer) so that all the n-dimensional vectors after linear combination fill the entire space, and the lattice Λ is obtained, which is specifically expressed as a set:

[0032]

[0033] Among them, u=(u1,u2,…,u n ) T is an integer vector, V Λ =(v1,v2,…,vn ) T is the generator matrix corresponding to the lattice Λ arrangement, which is composed of a linearly independent n-dimensional basis {v1,v2,…,v n}composition.

[0034] Optionally, projecting the quantized first feature map to an integer coefficient domain of a lattice vector quantizer to obtain a second feature map includes:

[0035] The k1th feature vector of the i1th row and j1th column of the quantized first feature map The generator matrix V of the lattice Λ in the lattice vector quantizer is Λ The inverse matrix of n is the dimension of the lattice, and the specific calculation method is:

[0036]

[0037] in

[0038] The projected vector splicing to form the feature vector of the i2th row and j2th column of the second feature map Get the second feature map

[0039] Optionally, the parameterized modeling of the distribution of the second feature map as a Gaussian distribution in which each dimension is independent of each other, and predicting the mean and variance of the Gaussian distribution by combining the super prior information and the spatial context model, comprises:

[0040] Divide the second feature map into two spatial groups in a chessboard pattern, wherein the sum of the height dimension and the width dimension coordinates in the first spatial group is an odd number, and the sum of the height dimension and the width dimension coordinates in the second spatial group is an even number;

[0041] Generate an all-zero tensor according to the size of the first spatial grouping and concatenate it with the super-prior information, input it into a neural network composed of a cascade of multiple layers of convolution and ReLU activation functions, and obtain the mean and variance of the Gaussian distribution of the first spatial grouping, wherein the ReLU activation function returns a larger value between the input value and zero;

[0042] Input the first spatial grouping into a layer of convolution to obtain the context information of the second spatial grouping, concatenate the super prior information with the context information of the second spatial grouping, and input them into the above-mentioned neural network (a neural network composed of a cascade of multiple layers of convolution and ReLU activation functions) to obtain the mean and variance of the Gaussian distribution of the second spatial grouping;

[0043] The mean and variance of the Gaussian distribution of the first spatial grouping are combined with the mean and variance of the Gaussian distribution of the second spatial grouping to obtain the mean and variance of the Gaussian distribution of the second feature map.

[0044] Optionally, the adopting a relaxed boundary to partition the integer coefficient domain of the lattice vector quantizer, performing probability estimation on the second feature map according to the predicted mean and variance of the Gaussian distribution, performing arithmetic coding according to the estimated probability, and obtaining a feature binary code stream includes:

[0045] The integer coefficient domain of the lattice vector quantizer is divided by a hypercube of volume 1, which fills the entire space and has the integer vector as its center, so that the k2th eigenvector of the i2th row and j2th column of the second eigenmap is is the center of the hypercube to which it belongs; where 1≤i2≤H2, 1≤j2≤W2, 1≤k2≤M2 / n, H2, W2 and M2 are the height, width and number of channels of the second characteristic graph respectively, and n is the dimension of the lattice;

[0046] The boundary of the hypercube is used as the k2th feature vector of the i2th row and j2th column of the estimated second feature map According to the mean and variance of the predicted Gaussian distribution, the k2th feature vector of the i2th row and j2th column of the second feature map is estimated. Probability The specific calculation method is:

[0047]

[0048] in and They are The mean and variance of the Gaussian distribution of ;

[0049] According to the estimated probability, the k2th feature vector of the i2th row and j2th column of the second feature map is Perform arithmetic coding to obtain a code stream of the k2th feature vector of the i2th row and j2th column of the second feature graph;

[0050] The code streams of all feature vectors are combined to obtain the feature binary code stream.

[0051] A second aspect of the present invention provides an image decoding method using vector quantization entropy modeling, which is used to decode the code obtained by the above-mentioned image coding method, comprising:

[0052] Obtaining a compressed image binary code stream, and decomposing the compressed image binary code stream to obtain a feature binary code stream and a super-a priori feature binary code stream;

[0053] Performing arithmetic decoding on the super-prior binary code stream using non-parametric distribution estimation to obtain a super-prior feature map;

[0054] Using a super-prior decoding network to obtain super-prior information on the super-prior feature map;

[0055] Predicting a mean and a variance of a Gaussian distribution for each spatial grouping of a second feature map for decoding by combining the super prior information and the spatial context model;

[0056] Using a relaxed boundary to divide the integer coefficient domain of the lattice vector quantizer, performing arithmetical decoding on the second feature map according to the mean and variance of the Gaussian distribution, and obtaining the second feature map by spatial grouping in the order of spatial grouping;

[0057] Inversely projecting the second feature map to the feature domain to obtain a reconstructed first feature map;

[0058] The reconstructed first feature map is passed through a decoding network to obtain a reconstructed image.

[0059] Optionally, the k2-th projection feature vector of the i2-th row and j2-th column of the second feature map is The generator matrix V of the lattice Λ in the lattice vector quantizer Λ Back-projection into feature space The specific calculation method is:

[0060]

[0061] The inverse projected vector splicing to form the feature vector of the i2th row and j2th column of the reconstructed first feature map Obtain the reconstructed first feature map Among them, 1≤i2≤H2, 1≤j2≤W2, 1≤k2≤M2 / n, H2, W2 and M2 are the height, width and number of channels of the second characteristic map respectively, and n is the dimension of the lattice.

[0062] A third aspect of the present invention provides an image compression method using vector quantization entropy modeling, comprising:

[0063] Obtain the image to be compressed;

[0064] Encoding the image to be compressed based on the above-mentioned vector quantization entropy modeling image encoding method to obtain a compressed image binary code stream;

[0065] The compressed image binary code stream is decoded based on the above-mentioned image decoding method of vector quantization entropy modeling to obtain a reconstructed image.

[0066] Optionally, the parameters of the encoding network, decoding network, super a priori encoding network, super decoding neural network, vector quantization entropy modeling and spatial context model are obtained by rate distortion optimization:

[0067]

[0068] Among them, distortion is the image to be compressed x and the reconstructed image Mean square error, λ>0 is the preset Lagrangian coefficient for balancing distortion and bit rate, and the bit rate R is calculated based on the Gaussian distribution of the second feature map and the non-parametric distribution of the super prior feature map:

[0069]

[0070] where 1≤i2≤H2, 1≤j2≤W2, 1≤k2≤M2 / n, H2, W2 and M2 are the height, width and number of channels of the second feature map, n is the dimension of the lattice, and The k2th feature vector of the i2th row, j2th column and the second feature map is encoded respectively The mean and variance of the Gaussian distribution of is the super prior feature map The distribution estimate of the lth dimension, yes Uniform distribution on . Refers to the summation of each dimension of the super-prior feature map.

[0071] A fourth aspect of the present invention provides an image compression system, the image compression system comprising an encoder and a decoder, including:

[0072] Obtain the image to be compressed;

[0073] The encoder encodes the image to be compressed using the above-mentioned vector quantization entropy modeling image encoding method to obtain a compressed image binary code stream;

[0074] The decoder uses the above-mentioned vector quantization entropy modeling image decoding method to decompress the compressed image binary code stream to obtain a reconstructed image.

[0075] According to a fifth aspect of the present invention, an image compression device is provided, the image compression device comprising a memory and a processor, the memory storing a program executable by the processor, and the processor executing the program can perform any of the following methods:

[0076] - the above-mentioned vector quantization entropy modeling image coding method, or,

[0077] - the above-mentioned method for decoding an image by vector quantization entropy modeling, or,

[0078] -The above-mentioned vector quantization entropy modeling image compression method.

[0079] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0080] The image encoding, decoding and compression method based on vector quantization entropy modeling provided by the present invention utilizes a lattice vector quantizer to efficiently quantize features, has a complexity close to that of a scalar quantizer, has an infinite codeword codebook, and does not cause a codebook collapse problem.

[0081] The image encoding, decoding and compression method of vector quantization entropy modeling provided by the present invention does not adopt a training method from soft to hard, but performs entropy modeling based on a relaxed boundary. The convergence during training is stable, and there is no problem of inconsistency between training and testing.

[0082] The image encoding, decoding and compression methods based on vector quantization entropy modeling provided by the present invention can save 6.03% and 2.48% of the bit stream overhead respectively, compared with the scalar quantization method and the latest vector quantization method, while maintaining the same reconstructed image quality, thereby improving the rate-distortion performance.

[0083] The image compression method of vector quantization entropy modeling in the embodiment of the present invention can improve the image compression rate distortion performance on natural images of different resolutions collected in different scenes. The present invention has extremely strong practical application value, especially in actual image compression, it can perform efficient, stable, and low-complexity vector quantization on features with different statistical characteristics to improve rate distortion performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0085] Figure 1 The present invention is a flowchart of a method for encoding an image using vector quantization entropy modeling in accordance with an embodiment of the present invention.

[0086] Figure 2 The present invention is a flowchart of a method for image encoding based on vector quantization entropy modeling in a preferred embodiment of the present invention.

[0087] Figure 3 The present invention is a flowchart of an image decoding method based on vector quantization entropy modeling in a preferred embodiment of the present invention.

[0088] Figure 4 A module diagram of image compression based on vector quantization entropy modeling in one embodiment of the present invention.

[0089] Figure 5 The figure is a flow chart of an image compression method based on vector quantization entropy modeling in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0090] The present invention is described in detail below in conjunction with specific embodiments and accompanying drawings. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0091] Reference Figure 1 FIG. 1 is a flowchart of a method for encoding an image using vector quantization entropy modeling in an embodiment of the present invention. This embodiment provides a method for encoding an image using vector quantization entropy modeling, comprising steps S100-S700. Specifically:

[0092] S100, obtaining a first feature map of the image to be encoded through a coding network;

[0093] In this step, the image to be encoded is first determined, and then the first feature map of the image to be encoded is obtained through the encoding network. Specifically, the encoding network can adopt a deep neural network.

[0094] The first feature map is a three-dimensional feature map of the image to be encoded, and the dimensions of the three-dimensional features include a channel dimension, a height dimension, and a width dimension.

[0095] S200, obtaining a super-a priori feature binary code stream of a first feature map and corresponding super-a priori information;

[0096] In this step, the first feature map is sequentially passed through a super-prior coding network to obtain a super-prior first feature map, the super-prior feature map is quantized into integers to obtain a quantized super-prior feature map, further, an independent non-parametric distribution estimate is established for each channel of the quantized super-prior feature map, and the quantized super-prior feature map is arithmetically encoded using this non-parametric distribution estimate to obtain a super-prior feature binary code stream, a reconstructed super-prior feature map is obtained from the super-prior feature binary code stream, and a super-prior decoding network is used to obtain super-prior information.

[0097] S300, quantizing the first feature map using a lattice vector quantizer to obtain a quantized first feature map;

[0098] In this step, the quality of each spatial position of the first feature map is evenly divided into several vectors according to the lattice dimension size, each vector is quantized using a lattice vector quantizer, and the quantized first feature map is obtained after splicing.

[0099] S400, projecting the quantized first feature map to an integer coefficient domain of a lattice vector quantizer to obtain a second feature map;

[0100] In this step, the quantized first feature map is projected into the integer coefficient domain using the inverse matrix of the generator matrix of the lattice in the lattice vector quantizer to obtain the second feature map.

[0101] S500, parameterizing the distribution of the second feature map into a Gaussian distribution in which each dimension is independent of each other, and predicting the mean and variance of the Gaussian distribution by combining the hyper-prior information and the spatial context model;

[0102] In this step, the spatial context model is implemented by a deep neural network, which divides the second feature map into two spatial groups in a chessboard pattern. For each spatial group, the super-prior information and the spatial context model are combined to predict the mean and variance of the Gaussian distribution.

[0103] S600, using a relaxed boundary to partition the integer coefficient domain of the lattice vector quantizer, performing probability estimation on the second feature map according to the predicted mean and variance of the Gaussian distribution, performing arithmetic coding according to the estimated probability, and obtaining a feature binary code stream;

[0104] In this step, a hypercube with a volume of 1 is used to divide the integer coefficient field of the lattice vector quantizer. According to the boundary of the hypercube and the mean and variance of the predicted Gaussian distribution, the second feature map is probability estimated and arithmetic coded to obtain a feature binary code stream.

[0105] S700, merging the feature binary code with the super-a priori feature binary code stream to obtain a compressed image binary code stream.

[0106] In the above embodiments of the present invention, lattice vector quantization can be efficiently performed on three-dimensional features, and a relaxed boundary can be used to encode the three-dimensional features. In actual image compression, more efficient vector quantization can be achieved with a complexity close to that of scalar quantization, thereby improving rate-distortion performance. The features are efficiently quantized using a lattice vector quantizer, with a complexity close to that of a scalar quantizer, and the codebook has infinite codewords, without causing a codebook collapse problem.

[0107] In some embodiments, executing S200 to obtain the super a priori binary code stream of the first feature map and the corresponding super a priori information may include:

[0108] M101, using a super prior coding network to obtain a super prior feature map for the first feature map;

[0109] M102, rounding off each feature in the super-prior feature map to an integer to obtain a quantized super-prior feature map;

[0110] M103, establishes an independent non-parametric distribution estimate for each channel of the quantized hyper-prior feature map;

[0111] M104, performing arithmetic coding on the super-prior second feature map according to the non-parametric distribution estimation to obtain a super-prior feature binary code stream;

[0112] M105, using non-parametric distribution estimation to perform arithmetic decoding on the super-prior binary code stream to obtain a reconstructed super-prior feature map;

[0113] M106, a super-prior decoding network is used to obtain super-prior information for the reconstructed super-prior feature map.

[0114] The above-mentioned embodiment of the present invention adopts the above-mentioned steps to better obtain the required super-prior information and provide conditions for the final encoding and decoding.

[0115] Figure 2 Flow chart of the image encoding method of vector quantization entropy modeling in a preferred embodiment of the present invention. In the preferred embodiment, the vector quantization image encoding method may include steps S101-S109, specifically:

[0116] S101, feature extraction: the image to be encoded passes through the encoding network to obtain a first feature map, where the first feature map is a three-dimensional feature map, and the dimensions include channel dimension, height dimension and width dimension;

[0117] S102, super-prior feature extraction and quantization: the three-dimensional features of the first feature map are passed through a super-prior coding network to obtain a super-prior feature map; the super-prior features are rounded and quantized to integers;

[0118] S103, super-prior feature encoding and decoding: establishing an independent non-parametric distribution estimate for each channel of the quantized super-prior feature map, and using the non-parametric distribution estimate to perform arithmetic encoding on the super-prior feature to obtain a super-prior feature binary code stream; performing arithmetic decoding on the super-prior feature binary code stream to obtain a reconstructed super-prior feature map;

[0119] S104, super prior information acquisition: the reconstructed super prior feature map is passed through a super prior decoding network to obtain super prior information;

[0120] S105, lattice vector quantization: quantizing the vector of each spatial position of the three-dimensional feature using a lattice vector quantizer;

[0121] S106, feature projection: projecting the three-dimensional features to the integer coefficient domain using the inverse matrix of the generator matrix of the lattice in the lattice vector quantizer to obtain projected three-dimensional features;

[0122] S107, entropy modeling based on relaxed boundaries: parameterizing the distribution of the projected three-dimensional features into Gaussian distributions in which each dimension is independent of each other, and predicting the mean and variance of the Gaussian distribution by combining the hyper-prior information and the spatial context model; using a hypercube with a volume of 1 to partition the integer coefficient domain of the lattice vector quantizer;

[0123] S108, feature coding: performing probability estimation on the projected three-dimensional features according to the predicted mean and variance of the Gaussian distribution and the boundary division of the integer coefficient domain of the lattice vector quantizer, performing arithmetic coding according to the estimated probability, and obtaining a feature binary code stream;

[0124] S109, code stream synthesis: merging the feature binary code stream with the super-a priori feature binary code stream to obtain a compressed image binary code stream.

[0125] In the above embodiment of the present invention, through the lattice vector quantizer and the entropy modeling based on the relaxed boundary, the three-dimensional features can be more efficiently vectorized and quantized, thereby improving the rate-distortion performance of image compression. The entropy modeling based on the relaxed boundary does not adopt the training method from soft to hard, but performs entropy modeling based on the relaxed boundary. The convergence during training is stable, and there is no problem of inconsistency between training and testing.

[0126] In order to perform lattice vector quantization and entropy modeling more efficiently, in a possible implementation manner, S105 in the above embodiment may be implemented by the following method, specifically:

[0127] S1051, for a given n-dimensional lattice The feature vector of the i-th row and j-th column of the three-dimensional feature (1≤i≤H, 1≤j≤W, where H and W are the height and width of the three-dimensional feature respectively) is (M is the number of channels of the three-dimensional feature) is evenly divided into M / n vectors in

[0128] S1052, according to the second norm, M / n vectors Each of them is quantized to the nearest lattice point in the lattice Λ Get quantized M / n vectors The specific calculation process is:

[0129]

[0130] S1053, the quantized M / n vectors Splicing to form the feature vector of the quantized three-dimensional feature in the i-th row and j-th column The quantified three-dimensional features are obtained

[0131] In a possible implementation manner, S106 in the above embodiment may be implemented by the following method:

[0132] S1061, the quantified three-dimensional features The kth feature vector in the i-th row, j-th column The generator matrix V of the lattice Λ in the lattice vector quantizer Λ The inverse matrix of The specific calculation method is:

[0133]

[0134] in

[0135] S1062, the projected vector Splicing to form the feature vector of the i-th row and j-th column of the projected three-dimensional feature Get projected 3D features

[0136] In a possible implementation manner, S107 in the above embodiment may be implemented by the following method:

[0137] S1071, dividing the projected three-dimensional features into two spatial groups in a chessboard pattern, wherein the sum of the height dimension and the width dimension coordinates in the first spatial group is an odd number, and the sum of the height dimension and the width dimension coordinates in the second spatial group is an even number;

[0138] S1072, concatenate the hyper-prior information with the all-zero tensor of the same size as the first spatial grouping, and input it into a neural network consisting of a cascade of multiple layers of convolutions with a convolution kernel size of 1×1 and ReLU activation functions to obtain the mean and variance of the Gaussian distribution of the first spatial grouping; of course, other convolution kernel sizes are also applicable, such as 3×3, 5×5, etc., and are not limited to 1×1.

[0139] S1073, input the first spatial grouping into a layer of convolution to obtain the context information of the second spatial grouping, concatenate the super prior information and the context information of the second spatial grouping, input into a neural network composed of a cascade of multiple layers of convolution and ReLU activation functions, and obtain the mean and variance of the Gaussian distribution of the second spatial grouping; the neural network can be composed of a cascade of multiple layers of convolution and ReLU activation functions with a kernel size of 1×1. Of course, other convolution kernel sizes are also applicable, such as 3×3, 5×5, etc., and are not limited to 1×1. By adopting this step, the amount of calculation can be reduced, overfitting can be avoided, and the expression ability of the neurons of the neural network can be enhanced.

[0140] S1074, combining the mean and variance of the Gaussian distribution of the first spatial grouping with the mean and variance of the Gaussian distribution of the second spatial grouping to obtain the mean and variance of the Gaussian distribution of the projected three-dimensional feature.

[0141] S1075, partitioning the integer coefficient domain of the lattice vector quantizer with a hypercube of volume 1, all hypercubes fill the entire space and take the integer vector as their center, so that the projected three-dimensional feature has the kth feature vector in the i-th row and j-th column is the center of the hypercube to which it belongs;

[0142] S1076, using the boundary of the hypercube as the kth feature vector of the i-th row and j-th column of the estimated second feature map According to the mean and variance of the predicted Gaussian distribution, the kth feature vector of the i-th row, j-th column of the second feature map is estimated. The specific calculation method is:

[0143]

[0144] in and They are The mean and variance of the Gaussian distribution of .

[0145] The image encoding method of vector quantization entropy modeling in this embodiment utilizes a lattice vector quantizer to quantize features and combines it with a relaxed boundary for entropy modeling. It has the characteristics of low complexity, high stability, and an infinite codebook, and can therefore provide a more efficient vector quantization method.

[0146] Based on the same inventive concept, another embodiment of the present invention provides a method for decoding an image using vector quantization entropy modeling, which is used to decode the code obtained by the method for encoding an image using vector quantization entropy modeling in any of the above embodiments. Figure 3 As shown, the image decoding method of vector quantization entropy modeling includes:

[0147] S201, code stream decomposition: decomposing the compressed image binary code stream to obtain a feature binary code stream and a super-a priori feature binary code stream;

[0148] S202, super a priori feature decoding: performing arithmetic decoding on the super a priori feature binary code stream to obtain a reconstructed super a priori feature map;

[0149] S203, super prior information acquisition: inputting the reconstructed super prior feature map into the super prior decoding network to obtain super prior information;

[0150] S204, Entropy Modeling Based on Relaxed Bounds: The structure and parameter values ​​of this module are similar to Figure 2The relaxed bound-based entropy modeling in the illustrated embodiment is exactly the same;

[0151] Exemplarily, the projected three-dimensional features are first initialized with an all-zero tensor; the projected three-dimensional features are spatially divided into two spatial groups in a chessboard pattern, the sum of the height dimension and the width dimension coordinates in the first spatial grouping is an odd number, and the sum of the height dimension and the width dimension coordinates in the second spatial grouping is an even number. The distribution parameterization of each spatial grouping is modeled as a Gaussian distribution in which each dimension is independent of each other, and the mean and variance of the Gaussian distribution are predicted by combining the super-prior information and the spatial context model. The integer coefficient domain of the lattice vector quantizer is partitioned by a relaxed boundary, and a probability estimate is performed for each spatial grouping based on the mean and variance of the predicted Gaussian distribution.

[0152] S205, projected feature decoding: according to the predicted mean and variance of the Gaussian distribution and the integer coefficient domain boundary division of the lattice vector quantizer, the projected three-dimensional features are obtained by grouping them spatially in the order of spatial grouping;

[0153] S206, reverse projection of projection features: reversely project the projected three-dimensional features to the feature domain to obtain a three-dimensional feature map, that is, a reconstructed first feature map, wherein the feature domain is a real number domain where the first feature map is located;

[0154] S207, image reconstruction: input the reconstructed first feature map into the decoding network to obtain a reconstructed image.

[0155] In a possible implementation manner, S206 in the above embodiment may be implemented by the following method, specifically:

[0156] S2061, project the kth projection feature vector of the i-th row, j-th column of the three-dimensional feature The generator matrix V of the lattice Λ in the lattice vector quantizer Λ Back-projection into feature space The specific calculation method is:

[0157]

[0158] S2062, the vector after inverse projection Splicing to form the feature vector of the i-th row and j-th column of the three-dimensional feature Get 3D features

[0159] The image decoding method of vector quantization entropy modeling in this embodiment is used to correspond to Figure 2 The image encoding method of vector quantization entropy modeling shown in the figure uses a lattice vector quantizer to quantize features and combines the relaxation boundary for entropy modeling, thereby obtaining a more efficient quantization method. For the parts not described in detail above, please refer to Figure 2 The corresponding description in the illustrated embodiment will not be repeated here.

[0160] Based on the same inventive idea, in another embodiment, a method for image compression based on vector quantization entropy modeling is also provided, comprising:

[0161] M201, obtaining the image to be compressed;

[0162] M202, encoding the image to be compressed based on the image encoding method of any one of the above embodiments to obtain a compressed image binary code stream;

[0163] M203, decoding the compressed image binary code stream based on the image decoding method of any one of the above embodiments to obtain a reconstructed image.

[0164] The above-mentioned image encoding method and image decoding method are the image encoding method and decoding method based on vector quantization entropy modeling in the above-mentioned embodiments, which will not be described in detail here.

[0165] The above-mentioned image encoding method and image decoding method can be performed by using an encoder and a decoder. In a specific embodiment, 24 images with a resolution of 512×768 in the Kodak test set are compressed. The flowchart of the image compression method of vector quantization entropy modeling is as follows: Figure 4 As shown, including:

[0166] S301, image acquisition module: sequentially obtains images to be compressed;

[0167] S302, encoder: encode the image to be compressed according to steps S101-S109 to obtain a compressed image binary code stream.

[0168] For example, the parameters are as follows: the height of the three-dimensional feature is H=32, the width is W=48, and the number of channels is M=192; the specific values ​​of the preset parameters of the entropy modeling based on the relaxed boundary are: the dimension of the lattice Λ is n=2, and the generating matrix of the lattice Λ is

[0169] S303, decoder: decompress the compressed image binary code stream according to S201-S207 to obtain a reconstructed image.

[0170] Among them, the height, width, number of channels of the three-dimensional features and the preset parameters of entropy modeling based on relaxed boundaries are exactly the same as those of the encoder.

[0171] The compression method in the above embodiment of the present invention is based on Figure 1 , 2 Image coding method based on vector quantization entropy modeling and Figure 3The image decoding method of vector quantization entropy modeling shown in the figure is implemented. For the parts not described in detail above, refer to the corresponding description in the above embodiments and will not be repeated here.

[0172] The compression method in the above-mentioned embodiment of the present invention can improve the performance of vector quantization on natural images of different resolutions collected in different scenes, improve the image compression rate-distortion performance, and can perform efficient, stable, and low-complexity vector quantization on features with different statistical characteristics, thereby improving the rate-distortion performance.

[0173] For the above-mentioned image compression method, the compression effect can be improved by training the model. For example, the training set consists of 20,000 natural images of different resolutions collected in various natural scenes, and the trained model can be used for actual image encoding and decoding.

[0174] Specifically, Figure 5 As shown, in a preferred embodiment, the image compression method may include:

[0175] S401, feature extraction: randomly crop the images in the training set into images with a resolution of 256×256, and input 16 images into the encoding network in one step to obtain three-dimensional features, wherein: the encoding network is composed of a convolution layer, a GDN activation function, a convolution layer, a GDN activation function, a convolution layer, a GDN activation function, and a convolution layer cascaded in sequence, and the height and width of the three-dimensional features are both 16, and the number of channels is 192;

[0176] The above-mentioned image resolution, number of images trained in one step, structure of the encoding network and three-dimensional feature parameters in this embodiment can be adjusted according to actual conditions in other embodiments and are not limited to those described in this step. The same applies to the parameters in the following steps.

[0177] S402, feature noise addition: to avoid the lattice vector quantization operation causing the gradient to be unable to be back-propagated, noise uniformly distributed in the Voronoi region of the lattice vector quantizer is added to each vector of each spatial position of the three-dimensional feature to simulate the quantization operation;

[0178] S403, feature projection: Each vector of each spatial position of the three-dimensional feature after adding noise is generated by the generation matrix of the lattice Λ in the lattice vector quantizer The inverse matrix of is projected into the integer coefficient domain to obtain the projected three-dimensional features;

[0179] S404, super-prior feature extraction and noise addition: input the three-dimensional feature into the super-prior coding network to obtain the super-prior feature. In order to avoid the inability to back-propagate the gradient due to the quantization operation, a uniformly distributed noise of -0.5 to 0.5 is added to the super-prior feature to simulate the quantization operation. The super-prior coding network is composed of three convolutional layers in cascade.

[0180] S405, super a priori feature bit rate estimation: establishing an independent non-parametric distribution estimation for each channel of the super a priori feature, and using the non-parametric distribution estimation to obtain the super a priori feature bit rate estimation;

[0181] S406, obtaining super-prior information: inputting the noisy super-prior features into a super-prior decoding network to obtain super-prior information, wherein the super-prior decoding network is composed of three cascaded convolutional layers;

[0182] S407, Entropy Modeling Based on Relaxed Bounds: The structure of this module is similar to Figure 2 The entropy modeling based on the relaxed boundary in the illustrated embodiment is exactly the same, the distribution parameterization of the projected three-dimensional features is modeled as a Gaussian distribution in which each dimension is independent of each other, and the mean and variance of the Gaussian distribution are predicted by combining the super prior information and the spatial context model;

[0183] In this embodiment, a hypercube with a volume of 1 is used to divide the integer coefficient domain of the lattice vector quantizer; wherein the specific values ​​of the preset parameters are: the dimension of the lattice Λ n=2, the generator matrix of the lattice Λ

[0184] S408, feature rate estimation: performing rate estimation on the projected three-dimensional feature according to the predicted mean and variance of the Gaussian distribution and the integer coefficient domain boundary partition of the lattice vector quantizer to obtain a feature rate estimation;

[0185] S409, total bit rate estimation: adding the super a priori characteristic bit rate estimation and the characteristic bit rate estimation to obtain a total bit rate estimation;

[0186] S410, reverse projection of projected features: each vector of each spatial position of the projected three-dimensional feature is converted by the generator matrix V of the lattice Λ in the lattice vector quantizer Λ Back-project to the feature domain to obtain the reconstructed three-dimensional features after noise addition, where the generating matrix of the lattice Λ

[0187] S411, image reconstruction: input the reconstructed noisy three-dimensional features into the decoding network to obtain a reconstructed image, wherein: the decoding neural network is composed of a convolution layer, a GDN activation function, a convolution layer, a GDN activation function, a convolution layer, a GDN activation function and a convolution layer cascaded in sequence.

[0188] In some possible implementations, the above image compression method can be used as an image compression model, which can be further optimized using stochastic gradient descent, and its rate-distortion loss function is:

[0189]

[0190] Among them, distortion is the image to be compressed x and the reconstructed image Mean square error, λ>0 is the preset Lagrangian coefficient for balancing distortion and bit rate (one λ value corresponds to one image compression model), and the bit rate R is calculated based on the Gaussian distribution of the projected three-dimensional features and the non-parametric distribution of the hyper-prior features:

[0191]

[0192] in and They are the predicted projected three-dimensional features, row i, column j, and k-th feature vectors. The mean and variance of the Gaussian distribution of It is a super prior feature The distribution estimate of the nth dimension, yes Uniform distribution on .

[0193] Application implementation effect:

[0194] In the above embodiment, the Flicker dataset is used to train the image compression model of vector quantization entropy modeling, and λ is set to 0.0016, 0.0032, 0.0075, 0.015, 0.03 and 0.045, corresponding to 6 image compression models with different bit rates, respectively. The Adam optimizer is used for 1,000,000 steps of optimization, and the initial 950,000 steps of learning rate is set to 10 -4 , then dropped to 10 -5 .

[0195] In the above embodiment, the image encoder of the 6 trained vector quantization entropy modeling image compression models is used to compress 24 images with a resolution of 512×768 in the Kodak test set. The average bit rates of the 24 compressed images are 0.1443bpp, 0.2334bpp, 0.3366bpp, 0.4840bpp, 0.6842bpp and 0.9062bpp respectively;

[0196] In the above embodiment, the image encoder of the 6 trained vector quantization entropy modeling image compression models is used to compress 24 images with a resolution of 512×768 in the Kodak test set to obtain reconstructed images. The average peak signal-to-noise ratio (PSNR) of the reconstructed 24 images is 29.01dB, 30.79dB, 32.30dB, 34.02dB, 35.84dB and 37.42dB respectively;

[0197] Compared with the method using the benchmark neural network, the image compression method of vector quantization entropy modeling provided in the above embodiment of the present invention has better rate-distortion performance. Specifically, compared with the scalar quantization method and the current latest vector quantization method, under the premise of the same reconstructed image quality, 6.03% and 2.48% of the code stream overhead can be saved respectively. The image compression method of vector quantization entropy modeling in the embodiment of the present invention can improve the image compression rate-distortion performance on natural images of different resolutions collected in different scenes. The present invention has extremely strong practical application value, especially in actual image compression, it can perform efficient, stable, and low-complexity vector quantization on features with different statistical characteristics to improve rate-distortion performance.

[0198] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0199] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0200] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0202] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A method for image coding based on vector quantization entropy modeling, characterized in that: include: Obtaining a first feature map of the image to be encoded through the encoding network, wherein the first feature map is a three-dimensional feature map, and the dimensions include a channel dimension, a height dimension, and a width dimension; Obtaining a super-prior feature binary code stream and corresponding super-prior information of the first feature map; quantizing the first feature map using a lattice vector quantizer to obtain a quantized first feature map; Projecting the quantized first feature map to the integer coefficient domain of the lattice vector quantizer to obtain a second feature map; Modeling the distribution parameter of the second feature map as a Gaussian distribution in which each dimension is independent from each other, and predicting the mean and variance of the Gaussian distribution by combining the super prior information and the spatial context model; Using a relaxed boundary to divide the integer coefficient domain of the lattice vector quantizer, performing probability estimation on the second feature map according to the predicted mean and variance of the Gaussian distribution, performing arithmetic coding according to the estimated probability, and obtaining a feature binary code stream; The feature binary code is combined with the super a priori feature binary code stream to obtain a compressed image binary code stream.

2. The image encoding method based on vector quantization entropy modeling according to claim 1, characterized in that: The obtaining of the super-a priori binary code stream of the first feature map and the corresponding super-a priori information includes: Using a super priori coding network on the first feature map to obtain a super priori feature map; Rounding off each feature in the super a priori feature map and quantizing it to an integer to obtain a quantized super a priori feature map; Establishing an independent non-parametric distribution estimate for each channel of the quantized super-prior feature map; Performing arithmetic coding on the super-a priori second feature map according to the non-parametric distribution estimation to obtain a super-a priori feature binary code stream; arithmetically decoding the super-prior binary code stream using the non-parametric distribution estimation to obtain a reconstructed super-prior feature map; A super-prior decoding network is used to obtain super-prior information for the reconstructed super-prior feature map.

3. The image encoding method based on vector quantization entropy modeling according to claim 1, characterized in that: The step of quantizing the first feature map by using a lattice vector quantizer to obtain a quantized first feature map includes: Construct an n-dimensional lattice based on a pre-set arrangement The feature vector of the i1th row and j1th column of the first feature graph with height H1, width W1 and number of channels M1 is Evenly divided into M1 / n vectors; For any vector Select the nearest lattice point in the lattice Λ according to the second norm As a quantized vector; 1≤i1≤H1, 1≤j1≤W1, 1≤k1≤M1 / n; Concatenate the quantized M1 / n vectors to obtain the first quantized feature map The characteristic vector of the i1th row and j1th column; Get the feature vectors of all H1 rows and W1 columns to get the first quantized feature map The quantized first feature map The height and width of are the same as the height and width of the first feature map.

4. The image encoding method based on vector quantization entropy modeling according to claim 3, characterized in that: The method constructs an n-dimensional lattice based on a preset arrangement include: Get a set of linearly independent n-dimensional basis {v1,v2,…,v n }; The set of linearly independent n-dimensional bases {v1,v2,…,v n } are linearly combined so that all the n-dimensional vectors after linear combination cover the entire space, and the lattice Λ is obtained, which is specifically expressed as a set: Among them, u=(u1,u2,…,u n ) T is an integer vector, V Λ =(v1,v2,…,v n ) T is the generator matrix corresponding to the lattice Λ arrangement, which is composed of a linearly independent n-dimensional basis {v1,v2,…,v n }, n is the dimension of the lattice.

5. The image encoding method based on vector quantization entropy modeling according to claim 1, characterized in that: The step of projecting the quantized first feature map to an integer coefficient domain of a lattice vector quantizer to obtain a second feature map comprises: The k1th feature vector of the i1th row and j1th column of the quantized first feature map The generator matrix V of the lattice Λ in the lattice vector quantizer is Λ The inverse matrix of n is the dimension of the lattice; The projected vector splicing to form the feature vector of the i2th row and j2th column of the second feature map Get the second feature map 6. The image encoding method based on vector quantization entropy modeling according to claim 1, characterized in that: The parameterized modeling of the distribution of the second feature map is a Gaussian distribution in which each dimension is independent of each other, and the mean and variance of the Gaussian distribution are predicted by combining the super prior information and the spatial context model, including: Divide the second feature map into two spatial groups in a chessboard pattern, wherein the sum of the height dimension and the width dimension coordinates in the first spatial group is an odd number, and the sum of the height dimension and the width dimension coordinates in the second spatial group is an even number; Generate an all-zero tensor according to the size of the first spatial grouping and concatenate it with the super-prior information, input it into a neural network composed of a cascade of multiple layers of convolution and ReLU activation functions, and obtain the mean and variance of the Gaussian distribution of the first spatial grouping, wherein the ReLU activation function returns a larger value between the input value and zero; Input the first spatial grouping into a layer of convolution to obtain context information of the second spatial grouping, concatenate the super prior information with the context information of the second spatial grouping, input the above-mentioned neural network, and obtain the mean and variance of the Gaussian distribution of the second spatial grouping; The mean and variance of the Gaussian distribution of the first spatial grouping are combined with the mean and variance of the Gaussian distribution of the second spatial grouping to obtain the mean and variance of the Gaussian distribution of the second feature map.

7. The image encoding method based on vector quantization entropy modeling according to claim 1, characterized in that: The method of using a relaxed boundary to partition the integer coefficient domain of the lattice vector quantizer, performing probability estimation on the second feature map according to the predicted mean and variance of the Gaussian distribution, performing arithmetic coding according to the estimated probability, and obtaining a feature binary code stream includes: The integer coefficient domain of the lattice vector quantizer is divided by a hypercube of volume 1, which fills the entire space and has the integer vector as its center, so that the k2th eigenvector of the i2th row and j2th column of the second eigenmap is is the center of the hypercube to which it belongs; where 1≤i2≤H2, 1≤j2≤W2, 1≤k2≤M2 / n, H2, W2 and M2 are the height, width and number of channels of the second characteristic graph respectively, and n is the dimension of the lattice; The boundary of the hypercube is used as the k2th feature vector of the i2th row and j2th column of the estimated second feature map According to the mean and variance of the predicted Gaussian distribution, the k2th feature vector of the i2th row and j2th column of the second feature map is estimated. Probability According to the estimated probability, the k2th feature vector of the i2th row and j2th column of the second feature map is Perform arithmetic coding to obtain a code stream of the k2th feature vector of the i2th row and j2th column of the second feature graph; The code streams of all feature vectors of the second feature map are combined to obtain a feature binary code stream.

8. A method for decoding an image using vector quantization entropy modeling, used for decoding the code obtained according to any one of claims 1 to 7, characterized in that: include: Obtaining a compressed image binary code stream, and decomposing the compressed image binary code stream to obtain a feature binary code stream and a super-a priori feature binary code stream; Performing arithmetic decoding on the super-prior binary code stream using non-parametric distribution estimation to obtain a super-prior feature map; Using a super-prior decoding network to obtain super-prior information on the super-prior feature map; Predicting a mean and a variance of a Gaussian distribution for each spatial grouping of a second feature map for decoding by combining the super prior information and the spatial context model; Using a relaxed boundary to divide the integer coefficient domain of the lattice vector quantizer, performing arithmetical decoding on the second feature map according to the mean and variance of the Gaussian distribution, and obtaining the second feature map by spatial grouping in the order of spatial grouping; Inversely projecting the second feature map to a feature domain to obtain a reconstructed first feature map; the feature domain refers to a real number domain where the first feature map is located; The reconstructed first feature map is passed through a decoding network to obtain a reconstructed image.

9. The image decoding method based on vector quantization entropy modeling according to claim 8, characterized in that: The step of inversely projecting the second feature map to a feature domain to obtain a reconstructed first feature map includes: The k2th projection feature vector of the i2th row and j2th column of the second feature graph is The generator matrix V of the lattice Λ in the lattice vector quantizer Λ Back-projection into feature space The inverse projected vector splicing to form the feature vector of the i2th row and j2th column of the reconstructed first feature map Obtain the reconstructed first feature map Among them, 1≤i2≤H2, 1≤j2≤W2, 1≤k2≤M2 / n, H2, W2 and M2 are the height, width and number of channels of the second characteristic map respectively, and n is the dimension of the lattice.

10. A method for image compression based on vector quantization entropy modeling, characterized in that: include: Obtain the image to be compressed; Encoding the image to be compressed based on the image encoding method according to any one of claims 1 to 7 to obtain a compressed image binary code stream; The compressed image binary code stream is decoded based on the image decoding method described in any one of claims 8-9 to obtain a reconstructed image.

11. The image compression method based on vector quantization entropy modeling according to claim 10, characterized in that: The parameters of the encoding network, decoding network, super prior encoding network, super decoding neural network, vector quantization entropy modeling and spatial context model are obtained through rate distortion optimization: Among them, distortion is the image to be compressed x and the reconstructed image Mean square error, λ>0 is the preset Lagrangian coefficient for balancing distortion and bit rate, and the bit rate R is calculated based on the Gaussian distribution of the second feature map and the non-parametric distribution of the super prior feature map: where 1≤i2≤H2, 1≤j2≤W2, 1≤k2≤M2 / n, H2, W2 and M2 are the height, width and number of channels of the second feature map, n is the dimension of the lattice, and The k2th feature vector of the i2th row, j2th column and the second feature map is encoded respectively The mean and variance of the Gaussian distribution of is the super prior feature map The distribution estimate of the lth dimension, yes The uniform distribution on It refers to the summation of each dimension of the super-prior feature map.

12. An image compression system, comprising an encoder and a decoder, characterized in that: include: Encoder: the encoder adopts the image encoding method according to any one of claims 1 to 7 to encode the image to be compressed to obtain a compressed image binary code stream; Decoder: The decoder decodes the compressed image binary code stream based on the image decoding method described in any one of claims 8-9 to obtain a reconstructed image.

13. An image compression device, comprising a memory and a processor, wherein the memory stores a program executable by the processor, wherein: When executing the program, the processor may perform any of the following methods: - the image encoding method according to any one of claims 1 to 7, or, - the image decoding method according to any one of claims 8 to 9, or, -The image compression method according to any one of claims 10-11.

Citation Information

Patent Citations

  • Probability entropy modeling image coding, decoding and compression method based on conditional diffusion

    CN117119204A

Cited By

  • Image coding and decoding and transmission method and system for neural network

    CN119544979A

  • Image compression method and system based on variable code rate allocation, and storage medium

    CN119767013A