Image compression network, image coding method, image decoding method and image compression method

Abstract

The invention provides an image compression network, an image encoding method, an image decoding method and an image compression method, and the image compression network comprises an encoding network, a decoding network and an entropy model. The coding network comprises a main coding network and a bypass coding network, the coding network is used for coding an image to be compressed, and the bypass coding network adopts orthogonal linear constraint; the decoding network comprises a main decoding network and a bypass decoding network, the main decoding network and the bypass decoding network are connected in parallel, the decoding network is used for decoding a compressed image binary code stream, and the bypass decoding network adopts orthogonal linear constraint; the entropy model is used for predicting the mean value and variance of Gaussian distribution. According to the invention, the bypass coding network and the bypass decoding network are introduced into the image compression network, so that the training time of the end-to-end image compression network is efficiently reduced, and the quality of the reconstructed image is improved while the image compression code rate overhead is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to an image compression network and an image encoding method, a decoding method, and a compression method. Background Art

[0002] In recent years, with the rapid development of deep learning technology, end-to-end image compression models based on deep learning have received widespread attention and rapid progress. By constructing complex nonlinear transformation structures, such models can provide more adaptive quantization units for multidimensional quantizers, breaking through the structural limitations of traditional transform coding methods, and theoretically have potential advantages in image compression performance. However, existing image compression methods based on deep learning still face some significant challenges in practical applications, among which the long training time is one of the most prominent pain points. Existing end-to-end image compression models usually require a large amount of data and high-dimensional feature learning processes, which leads to an extremely long training cycle for the model. Especially when training on large-scale data sets, the complexity and computational requirements of deep neural networks are further increased, resulting in a significant increase in training time. This long training process not only brings huge pressure in terms of resource utilization and computing costs, but also seriously affects the development cycle and application promotion of the model. Therefore, how to effectively shorten the training time of the model while ensuring the image compression effect has become a key technical problem that needs to be solved in the field of image compression.

[0003] To solve this problem, recent studies have optimized the structure of the encoder and decoder of the image compression model, such as "ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding" published by Dailan Hen et al. at the 2022 Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. By using stacked residual blocks instead of normalization layers at the encoder and decoder end, efficient training and higher rate-distortion performance are achieved. However, this method still requires up to two million steps of iterative training to converge to a good performance result. Summary of the invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide an image compression network and an image encoding method, a decoding method, and a compression method.

[0005] To achieve the above-mentioned purpose, according to a first aspect of the present disclosure, there is provided an image compression network, comprising: an encoding network, a decoding network and an entropy model;

[0006] The coding network includes a main coding network and a bypass coding network, the coding network is used to encode the image to be compressed, and the bypass coding network adopts orthogonal linear constraints;

[0007] The decoding network includes a main decoding network and a bypass decoding network, the main decoding network and the bypass decoding network are connected in parallel, the decoding network is used to decode a compressed image binary code stream, and the bypass decoding network adopts orthogonal linear constraints;

[0008] The entropy model is used to predict the mean and variance of a Gaussian distribution.

[0009] Optionally, the main encoding network adopts an end-to-end image compression encoding network, including multiple convolution blocks or multiple Transformer blocks.

[0010] Optionally, the bypass coding network includes a plurality of orthogonal linear wavelet coding units, each of which includes a cascaded discrete wavelet forward transform and an orthogonal linear layer.

[0011] Optionally, the main decoding network adopts a decoding network of an end-to-end image compression network, and the main decoding network includes multiple convolution blocks or multiple Transformer blocks.

[0012] Optionally, the bypass decoding network comprises a plurality of orthogonal linear wavelet decoding units, each of which comprises a cascaded discrete wavelet inverse transform and an orthogonal linear layer.

[0013] Optionally, the training loss functions of the encoding network, the decoding network and the entropy model are:

[0014]

[0015] in, represents distortion, x represents the image to be compressed, represents the reconstructed image, λ1 represents the first balance parameter, λ2 represents the second balance parameter, R1 represents the bit rate, represents the loss term used to constrain the orthogonality of the orthogonal linear layer;

[0016]

[0017] in, represents the characteristics of the kth element, represents the predicted distribution of the kth element, θ k ={φk ,ψ k} represents a set of learnable parameters, Represents the predictive distribution of side information;

[0018]

[0019] Where I represents a matrix with diagonal values ​​of 1, and W enc,j represents the coefficients of the orthogonal linear layer of the orthogonal linear wavelet coding unit of the bypass coding network, W enc,j Represents coefficients of an orthogonal linear layer of an orthogonal linear wavelet decoding unit of the bypass decoding unit.

[0020] According to a second aspect of the present disclosure, there is provided an image encoding method, comprising:

[0021] Inputting the image to be compressed into the encoding network, determining a first feature map of the image to be compressed, wherein the first feature map represents a three-dimensional feature map;

[0022] quantizing each eigenvalue in the first feature map into an integer to determine a second feature map;

[0023] Inputting the second feature map into an entropy model, parameterizing the distribution of the second feature map into a Gaussian distribution, and determining the Gaussian distribution of the second feature map;

[0024] According to the Gaussian distribution of the second feature map, determining the predicted mean and variance of the Gaussian distribution,

[0025] Determining a probability estimate of the second feature map based on the predicted mean and variance of the Gaussian distribution;

[0026] The probability estimate of the second feature map is arithmetically encoded to determine a compressed image binary code stream.

[0027] Optionally, inputting the image to be compressed into the encoding network and determining the first feature map of the image to be compressed includes:

[0028] Inputting the image to be compressed into a main coding network to determine a main feature map;

[0029] Inputting the image to be compressed into a bypass coding network to determine a bypass feature map;

[0030] The main feature map and the bypass feature map are combined to determine a first feature map of the image to be compressed.

[0031] Optionally, inputting the image to be compressed into a bypass coding network to determine a first bypass feature map includes:

[0032] Input the image to be compressed into the first orthogonal linear wavelet coding unit, and in order, the output of each previous orthogonal linear wavelet coding unit is used as the input of the next orthogonal linear wavelet coding unit, and the last orthogonal linear wavelet coding unit outputs the first bypass feature;

[0033] Among them, the input amount of each of the orthogonal linear wavelet coding units is used as the input amount of the discrete wavelet forward transform of each of the orthogonal linear wavelet coding units, the discrete wavelet forward transform outputs a wavelet transform coding feature map, the output amount of the discrete wavelet forward transform is used as the input amount of the orthogonal linear layer of each of the orthogonal linear wavelet coding units, the orthogonal linear layer outputs a first transformation feature, and the output amount of the orthogonal linear layer is used as the output amount of each of the orthogonal linear wavelet coding units.

[0034] According to a third aspect of the present disclosure, there is provided an image decoding method for decoding the compressed image binary code stream obtained in the second aspect of the present disclosure, comprising:

[0035] Get the compressed image binary code stream;

[0036] Inputting the compressed image binary code stream into an entropy model to determine a mean and a variance of a Gaussian distribution of a predicted second feature map;

[0037] Performing arithmetic decoding according to the mean and variance of the Gaussian distribution of the predicted second feature map to determine the second feature map;

[0038] The second feature map is input into a decoding network to determine a reconstructed image.

[0039] Optionally, inputting the second feature map into a decoding network to determine the reconstructed image includes:

[0040] Inputting the second feature map into a main decoding network to determine a main decoded image;

[0041] Inputting the second feature map into a bypass decoding network to determine a bypass decoded image;

[0042] The main decoded image and the bypass decoded image are combined to determine the reconstructed image.

[0043] Optionally, inputting the second feature map into a bypass decoding network to determine a bypass decoded image includes:

[0044] Input the second feature map into the first orthogonal linear wavelet decoding unit, and in order, the output of each previous orthogonal linear wavelet decoding unit is used as the input of the next orthogonal linear wavelet decoding unit, and the last orthogonal linear wavelet decoding unit outputs the bypass decoded image;

[0045] Among them, the input amount of each of the orthogonal linear wavelet decoding units is used as the input amount of the discrete wavelet inverse transform of each of the orthogonal linear wavelet decoding units, the discrete wavelet inverse transform outputs discrete wavelet features, the output amount of the discrete wavelet inverse transform is used as the input amount of the orthogonal linear layer of each of the orthogonal linear wavelet decoding units, the orthogonal linear layer outputs the second transformation feature, and the output amount of the orthogonal linear layer is used as the output amount of each of the orthogonal linear wavelet decoding units.

[0046] According to a fourth aspect of the present disclosure, there is provided an image compression method, comprising:

[0047] Obtain the image to be compressed;

[0048] Encode the image to be compressed based on any one of the image encoding methods provided in the second aspect of the present disclosure to obtain a compressed image binary code stream;

[0049] Based on any one of the image decoding methods provided in the third aspect of the present disclosure, the compressed image binary code stream is decoded to obtain a reconstructed image.

[0050] According to a fifth aspect of the present disclosure, a terminal is provided, including a memory and a processor, wherein the memory stores a program executed by the processor, and the processor may perform any of the following methods when executing the program:

[0051] The image encoding method according to any one of the second aspects of the present disclosure, or,

[0052] The image decoding method according to any one of the third aspects of the present disclosure, or,

[0053] The fourth aspect of the present disclosure provides the image compression method.

[0054] According to the sixth aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, it can execute the image encoding method provided by the second aspect of the present disclosure, or the image decoding method provided by the third aspect of the present disclosure, or the image compression method provided by the fourth aspect of the present disclosure.

[0055] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0056] Through the above technical scheme, a bypass encoding network is introduced into the encoding network of the image compression network, and orthogonal linear constraints are adopted. A bypass decoding network is introduced into the decoding network of the image compression network, and orthogonal linear constraints are adopted. This can effectively reduce the training time of the end-to-end image compression network and save bit rate while obtaining the same reconstructed image quality.

[0057] The embodiments of the present disclosure introduce multiple orthogonal linear wavelet coding units in a bypass decoding network, which can effectively reduce the training time of an end-to-end image compression network. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0059] Figure 1 The figure is a schematic diagram showing the structure of an image compression network according to an exemplary embodiment.

[0060] Figure 2 The figure is a flowchart of an image encoding method according to an exemplary embodiment.

[0061] Figure 3 The figure is a flowchart of an image decoding method according to an exemplary embodiment.

[0062] Figure 4 The figure is a flowchart of an image compression method according to an exemplary embodiment.

[0063] Figure 5 It is a block diagram of an image compression system according to an exemplary embodiment.

[0064] Figure 6 The figure is a flow chart of an image compression network training method according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0066] Figure 1 The figure is a schematic diagram showing the structure of an image compression network according to an exemplary embodiment.

[0067] like Figure 1 As shown, the present disclosure provides an image compression network, including an encoding network, a decoding network and an entropy model.

[0068] The coding network includes a main coding network and a bypass coding network. The coding network is used for coding the compressed image, and the bypass coding network adopts orthogonal linear constraints.

[0069] The bypass coding network can accelerate the training and convergence of the coding network, and further accelerate the convergence of the coding network and entropy model based on orthogonal linear constraints.

[0070] The decoding network includes a main decoding network and a bypass decoding network, which are connected in parallel. The decoding network is used to decode the compressed image binary code stream, and the bypass decoding network adopts orthogonal linear constraints.

[0071] The bypass decoding network can accelerate the training and convergence of the decoding network, and further accelerate the convergence of the decoding network and entropy model based on orthogonal linear constraints.

[0072] The entropy model is used to predict the mean and variance of a Gaussian distribution.

[0073] In the present disclosure, the main encoding network and the main decoding network can adopt the encoding network and decoding network in the existing end-to-end image compression method, including but not limited to convolutional network, residual network, Transformer network; the upper model can adopt any entropy model, for example, super prior entropy model, spatial autoregressive entropy model and channel autoregressive entropy model.

[0074] Through the above technical scheme, a bypass encoding network is introduced into the encoding network of the image compression network, and orthogonal linear constraints are adopted. A bypass decoding network is introduced into the decoding network of the image compression network, and orthogonal linear constraints are adopted. This can effectively reduce the training time of the end-to-end image compression network and save bit rate while obtaining the same reconstructed image quality.

[0075] In a possible embodiment, the main encoding network adopts an end-to-end image compression encoding network, including multiple convolution blocks or multiple Transformer blocks.

[0076] Among them, the main encoding network can adopt the encoding network of any existing end-to-end image compression method, including but not limited to convolutional network, residual network, and Transformer network.

[0077] In the present disclosure, the main encoding network is composed of 4 convolutional layers with a downsampling multiple of 2 and 3 ReLU non-linear activation units alternately cascaded. Specifically, it is composed of a convolutional layer with upsampling, a ReLU non-linear activation unit, a convolutional layer with upsampling, a ReLU non-linear activation unit, a convolutional layer with upsampling, a ReLU non-linear activation unit, and a convolutional layer with upsampling, which are cascaded in sequence.

[0078] The bypass coding network includes a plurality of orthogonal linear wavelet coding units, each of which includes a cascaded discrete wavelet forward transform and an orthogonal linear layer.

[0079] Among them, the coefficient of the orthogonal linear layer of each orthogonal linear wavelet coding unit is W enc,j .

[0080] In the present disclosure, the bypass coding network is formed by cascading four orthogonal linear wavelet coding units.

[0081] In a possible embodiment, the main decoding network adopts a decoding network of an end-to-end image compression network, and the main decoding network includes multiple convolution blocks or multiple Transformer blocks.

[0082] In the present disclosure, the main decoding network is composed of 4 convolutional layers with an upsampling multiple of 2 and 3 ReLU non-linear activation units alternately cascaded. Specifically, it is composed of a convolutional layer with upsampling, a ReLU non-linear activation unit, a convolutional layer with upsampling, a ReLU non-linear activation unit, a convolutional layer with upsampling, a ReLU non-linear activation unit, and a convolutional layer with upsampling, which are cascaded in sequence.

[0083] The bypass decoding network includes a plurality of orthogonal linear wavelet decoding units, each of which includes a cascaded discrete wavelet inverse transform and an orthogonal linear layer.

[0084] Among them, the coefficient of the orthogonal linear layer of each orthogonal linear wavelet decoding unit is W dec,j .

[0085] In the present disclosure, the bypass decoding network is formed by cascading four orthogonal linear wavelet decoding units in sequence.

[0086] By introducing multiple orthogonal linear wavelet coding units in the bypass decoding network, the training time of the end-to-end image compression network can be effectively reduced.

[0087] An image compression network provided by the present disclosure can be used as an image compression model, and is further optimized in stages using a stochastic gradient descent method.

[0088] In a possible embodiment, the training loss functions of the encoding network, the decoding network, and the entropy model are:

[0089]

[0090] in, represents distortion, x represents the image to be compressed, represents the reconstructed image, λ1 represents the first balance parameter, λ2 represents the second balance parameter, R1 represents the bit rate, represents the loss term used to constrain the orthogonality of the orthogonal linear layer;

[0091]

[0092] in, represents the characteristics of the kth element, represents the predicted distribution of the kth element, θ k ={φ k ,ψ k} represents a set of learnable parameters, Represents the probability distribution of side information;

[0093]

[0094] Among them, I represents a matrix with diagonal values ​​of 1, W enc,j represents the coefficients of the orthogonal linear layer of the orthogonal linear wavelet coding unit of the bypass coding network, W enc,j Represents the coefficients of the orthogonal linear layer of the orthogonal linear wavelet decoding unit of the bypass decoding unit.

[0095] The image compression network provided by the present disclosure can be applied to any existing end-to-end image compression coding network and entropy model network to accelerate the training of the image compression network and improve the training efficiency of the image compression network.

[0096] Figure 2 The figure is a flowchart of an image encoding method according to an exemplary embodiment.

[0097] like Figure 2 As shown, the present disclosure provides an image encoding method, including S11 to S16.

[0098] S11, inputting the image to be compressed into the encoding network, and determining a first feature map of the image to be compressed.

[0099] The coding network may adopt a space-frequency modulation coding network, and the space-frequency modulation coding network is a coding network composed of a main coding network and a bypass coding network.

[0100] The first feature map represents a three-dimensional feature map, and the dimensions of the first feature map include a channel dimension, a height dimension, and a width dimension.

[0101] S12, quantize each eigenvalue in the first feature map into an integer to determine a second feature map.

[0102] S13, inputting the second feature map into the entropy model, parameterizing the distribution of the second feature map into a Gaussian distribution, and determining the Gaussian distribution of the second feature map.

[0103] S14, determining the mean and variance of the predicted Gaussian distribution according to the Gaussian distribution of the second feature map.

[0104] S15, determining a probability estimate of the second feature map according to the predicted mean and variance of the Gaussian distribution.

[0105] S16, performing arithmetic coding on the probability estimate of the second feature map to determine a compressed image binary code stream.

[0106] In a possible embodiment, S11 may include S101 to S103.

[0107] S101, input the image to be compressed into the main coding network to determine the main feature map.

[0108] Among them, the image to be compressed is represented as x, and the main feature map is represented as y main .

[0109] S102, inputting the image to be compressed into the bypass coding network to determine the bypass feature map.

[0110] Among them, the bypass feature map is represented by y aux .

[0111] S103, combining the main feature map and the bypass feature map to determine a first feature map of the image to be compressed.

[0112] Among them, the first feature map is represented by y.

[0113] Specifically, S102, inputting the image to be compressed into the bypass coding network and determining the bypass feature map may include:

[0114] The image to be compressed is input into the first orthogonal linear wavelet coding unit, and in order, the output of each previous orthogonal linear wavelet coding unit is used as the input of the next orthogonal linear wavelet coding unit, and the last orthogonal linear wavelet coding unit outputs the first bypass feature;

[0115] Among them, the input amount of each orthogonal linear wavelet coding unit is used as the input amount of the discrete wavelet forward transform of each orthogonal linear wavelet coding unit, the discrete wavelet forward transform outputs a wavelet transform coding feature map, the output amount of the discrete wavelet forward transform is used as the input amount of the orthogonal linear layer of each orthogonal linear wavelet coding unit, the orthogonal linear layer outputs the first transformation feature, and the output amount of the orthogonal linear layer is used as the output amount of each orthogonal linear wavelet coding unit.

[0116] As an example, the input of the first orthogonal linear wavelet coding unit is the image to be compressed x, the output of the first orthogonal linear wavelet coding unit is x1, and the output x1 of the first orthogonal linear wavelet coding unit is used as the input of the second orthogonal linear wavelet coding unit, and so on. The input of the jth orthogonal linear wavelet coding unit is the output x of the j-1th orthogonal linear wavelet coding unit. j-1, the output of the last orthogonal linear wavelet coding unit is the bypass feature map y aux .

[0117] As another example, the following operations are performed within the orthogonal linear wavelet coding unit:

[0118] For the jth orthogonal linear wavelet coding unit, the input x j-1 Input discrete wavelet forward transform, through wavelet forward transform, determine the wavelet transform coding feature map DWT (x j-1 ); The wavelet transform encoding feature map DWT(x j-1 ) Input coefficient is W enc,j The orthogonal linear layer determines the first conversion feature and serves as the output of the jth orthogonal linear wavelet coding unit. The specific formula is:

[0119] x j =DWT(x j-1 )W enc，j

[0120] By adopting a bypass coding network and multiple orthogonal linear wavelet coding units of the bypass coding network to encode the compressed image, effective decorrelation is performed, redundant information in the encoding is reduced, the bit rate overhead in the image compression process is reduced, the image reconstruction quality is improved, and the training process can be accelerated.

[0121] In a possible embodiment, S12 may include S201 to S202.

[0122] S201, inputting the first feature map into a preset super-a priori encoder, performing super-a priori feature extraction, and determining the super-a priori features of the first feature map.

[0123] S202, quantize each feature value of the super-prior feature of the first feature map into an integer by rounding, and determine a second feature map.

[0124] In a possible embodiment, the entropy model may adopt a super-prior entropy model and introduce a super-prior network, which includes a super-prior encoder and a super-prior decoder.

[0125] In a possible embodiment, an image encoding method may further include S17 to S20.

[0126] S17, establishing independent distribution estimates between features based on the super prior features in the second feature map.

[0127] S18, performing arithmetic coding on the super-prior features in the second feature map by using independent distribution estimation between features to determine the super-prior feature binary code stream.

[0128] Among them, arithmetic coding can be performed by a super-prior encoder of a super-prior network in an entropy model.

[0129] S19, performing arithmetical decoding on the super-a priori feature binary code stream to determine the restored super-a priori feature.

[0130] Among them, arithmetic decoding can be performed by a super-prior decoder in a super-prior network in an entropy model.

[0131] S20, inputting the restored super-prior features into a preset super-decoding neural network to determine the super-prior information.

[0132] Steps S17 to S20 may be performed in an entropy model, and S17 to S20 are performed before step S13.

[0133] In a possible embodiment, S14 may include:

[0134] According to the Gaussian distribution of the second feature map, the super prior information is used to predict the mean and variance of the Gaussian distribution of the second feature map, and the mean and variance of the predicted Gaussian distribution are determined.

[0135] In a possible embodiment, S15 may include:

[0136] According to the predicted mean and variance of the Gaussian distribution, a probability estimation is performed on each three-dimensional feature of the second feature map to determine a probability estimation of the second feature map.

[0137] In a possible embodiment, S16 may include S401 to S402.

[0138] S401, performing arithmetic coding on the probability estimate of the second feature map to obtain a feature binary code stream.

[0139] The feature binary code stream is a feature binary code stream of three-dimensional features.

[0140] S402, merging the feature binary code stream and the super a priori feature binary code stream to determine a compressed image binary code stream.

[0141] The super a priori feature binary code stream is the super a priori feature binary code stream of the super a priori feature obtained in step S18.

[0142] In a possible embodiment, an entropy model used in an image encoding method may adopt any one of a hyper-prior entropy model, a spatial autoregressive entropy model, and a channel autoregressive entropy model.

[0143] As an example, the entropy model adopts a hyper-prior entropy model, which is as follows:

[0144] A super-prior network is introduced, which includes a super-prior encoder and a super-prior decoder. The super-prior encoder is used to perform arithmetic coding on the super-prior in the second feature map, and the super-prior decoder is used to perform Gaussian modeling on each point in the second feature map with a mean of μ and a variance of σ to determine the Gaussian distribution of the second feature map, and entropy coding is performed on the probability estimate of the second feature map to determine the compressed image binary code stream.

[0145] As another example, the entropy model adopts a spatial autoregressive entropy model, which further introduces a spatial autoregressive model on the basis of the hyper-prior entropy model to model the local contextual correlation of the second feature map in space and predict the probability distribution of the eigenvalues ​​of the second feature map.

[0146] As another example, the entropy model adopts a channel autoregressive entropy model, which further introduces a channel conditional model based on the hyper-prior entropy model, models the characteristics of the channel according to the correlation between the channels of the second feature map, and determines the probability distribution of the current channel by regressing and predicting the characteristics of the previous channel.

[0147] In a possible embodiment, in an image encoding method, a hyper-prior entropy model and a basic encoding unit based on convolution and nonlinear units can be used, and a basic encoding unit of Transformer and an entropy model based on autoregression can also be used, both of which can achieve the purpose of image encoding disclosed in the present invention.

[0148] Figure 3 The figure is a flowchart of an image decoding method according to an exemplary embodiment.

[0149] like Figure 3 As shown, the present disclosure also provides an image decoding method, which can decode the compressed image binary code stream obtained by the above encoding method, including S21 to S24.

[0150] S21, obtaining a compressed image binary code stream.

[0151] S22, inputting the compressed image binary code stream into the entropy model to determine the mean and variance of the Gaussian distribution of the predicted second feature map.

[0152] S23, performing arithmetic decoding according to the mean and variance of the predicted Gaussian distribution of the second feature map to determine the second feature map.

[0153] S24, inputting the second feature map into a decoding network to determine a reconstructed image.

[0154] In a possible embodiment, S22 may include S221.

[0155] S221, inputting the compressed image binary code stream into the entropy model, decomposing the compressed image binary code stream, and obtaining a feature binary code stream and a super-a priori feature binary code stream.

[0156] In a possible embodiment, S22 may also include S222 to S223.

[0157] S222, performing arithmetic decoding on the super a priori feature binary code stream to obtain restored super a priori features;

[0158] S223, inputting the restored super-prior features into the super-decoding neural network to obtain super-prior information.

[0159] Steps S221 to S223 may be performed in an entropy model.

[0160] In a possible embodiment, S22 may further include S224.

[0161] S224, using the super prior information to predict the mean and variance of the Gaussian distribution of the predicted second feature map.

[0162] The Gaussian distribution of the predicted second feature map is a Gaussian model of each feature in the predicted second feature map.

[0163] The super a priori information used in step S224 is the super a priori information obtained in step S223.

[0164] In a possible embodiment, S23 may include:

[0165] The feature binary code stream is arithmetically decoded according to the mean and variance of the Gaussian of the predicted second feature map to determine the second feature map, where the second feature map contains quantized three-dimensional features.

[0166] The decoding network adopted in the present disclosure is a space-frequency modulation decoding network, which includes a main decoding network and a bypass decoding network.

[0167] In a possible embodiment, S24 may include S241 to S243.

[0168] S241, input the second feature map into the main decoding network to determine the main decoded image.

[0169] Among them, the main decoded image is expressed as

[0170] S242, input the second feature map into the bypass decoding network to determine the bypass decoded image.

[0171] Among them, the bypass decoded image is expressed as

[0172] S243, combining the main decoded image and the bypass decoded image to determine a reconstructed image.

[0173] The reconstructed image is expressed as

[0174] Specifically, S242, inputting the second feature map into the bypass decoding network to determine the bypass decoded image may include:

[0175] The second feature map is input into the first orthogonal linear wavelet decoding unit, and in order, the output of each previous orthogonal linear wavelet decoding unit is used as the input of the next orthogonal linear wavelet decoding unit, and the last orthogonal linear wavelet decoding unit outputs a bypass decoded image;

[0176] Among them, the input amount of each orthogonal linear wavelet decoding unit is used as the input amount of the discrete wavelet inverse transform of each orthogonal linear wavelet decoding unit, the discrete wavelet inverse transform outputs discrete wavelet features, the output amount of the discrete wavelet inverse transform is used as the input amount of the orthogonal linear layer of each orthogonal linear wavelet decoding unit, the orthogonal linear layer outputs the second transformation feature, and the output amount of the orthogonal linear layer is used as the output amount of each orthogonal linear wavelet decoding unit.

[0177] As an example, the input of the first orthogonal linear wavelet decoding unit is the second feature map, the input of the first orthogonal linear wavelet decoding unit is y1, and the output y1 of the first orthogonal linear wavelet decoding unit is used as the input of the second orthogonal linear wavelet decoding unit, and so on. The input of the jth orthogonal linear wavelet decoding unit is the output y of the j-1th orthogonal linear wavelet decoding unit. j-1 , the output of the last orthogonal linear wavelet coding unit is the bypass decoded image

[0178] As another example, the following operations are performed within the orthogonal linear wavelet decoding unit:

[0179] For the jth orthogonal linear wavelet decoding unit, the input y j-1 Input discrete wavelet inverse transform, determine the discrete wavelet features through wavelet inverse transform; input the discrete wavelet features into coefficients W dec,j The orthogonal linear layer determines the second conversion feature and serves as the output of the j-th orthogonal linear wavelet decoding unit.

[0180] The reconstructed image is obtained by adopting the main decoding network and the bypass decoding network, thereby effectively improving the reconstruction quality of the reconstructed image.

[0181] In a possible embodiment, the present disclosure further provides an image compression method, including:

[0182] Get the compressed image;

[0183] The image to be compressed is encoded by using the image encoding method provided by the present disclosure to obtain a compressed image binary code stream;

[0184] The image decoding method provided by the present disclosure is used to decode the compressed image binary code stream to obtain a reconstructed image.

[0185] Figure 4 The figure is a flowchart of an image compression method according to an exemplary embodiment.

[0186] like Figure 4 As shown, the present disclosure also provides an image compression method, including S31 to S45.

[0187] S31, obtaining an image to be compressed.

[0188] S32, inputting the image to be compressed into the encoding network, and determining a first feature map of the image to be compressed.

[0189] S33, quantize each eigenvalue in the first feature map into an integer to determine a second feature map.

[0190] S34, establishing independent distribution estimates between features based on the super prior features in the second feature map;

[0191] S35, performing arithmetic coding on the super-prior features in the second feature map by using independent distribution estimation between features to determine a super-prior feature binary code stream;

[0192] S36, performing arithmetic decoding on the super a priori feature binary code stream to determine the restored super a priori feature;

[0193] S37, inputting the restored super-prior features into a preset super-decoding neural network to determine the super-prior information.

[0194] S38, inputting the second feature map into the entropy model, parameterizing the distribution of the second feature map into a Gaussian distribution, and determining the Gaussian distribution of the second feature map.

[0195] S39, determining the mean and variance of the predicted Gaussian distribution according to the Gaussian distribution of the second feature map.

[0196] According to the Gaussian distribution of the second feature map, the super prior information is used to predict the mean and variance of the Gaussian distribution of the second feature map, so as to determine the mean and variance of the predicted Gaussian distribution.

[0197] S40, determining a probability estimate of the second feature map according to the predicted mean and variance of the Gaussian distribution.

[0198] Wherein, according to the predicted mean and variance of the Gaussian distribution, a probability estimation is performed on each three-dimensional feature of the second feature map, thereby determining the probability estimation of the second feature map.

[0199] S41, performing arithmetic coding on the probability estimate of the second feature map to determine a compressed image binary code stream.

[0200] Among them, firstly, the probability estimate of the second feature map is arithmetically encoded to obtain a feature binary code stream; secondly, the feature binary code stream and the super prior feature binary code stream are merged to determine the compressed image binary code stream.

[0201] S42, obtaining a compressed image binary code stream.

[0202] S43, inputting the compressed image binary code stream into the entropy model to determine the mean and variance of the Gaussian distribution of the predicted second feature map.

[0203] Among them, first, the compressed image binary code stream is input into the entropy model, and the compressed image binary code stream is decomposed to obtain the feature binary code stream and the super prior feature binary code stream; secondly, the super prior feature binary code stream is arithmetically decoded to obtain the restored super prior features; then, the restored super prior features are input into the super decoding neural network to obtain the super prior information; finally, the super prior information is used to predict the mean and variance of the Gaussian distribution of the predicted second feature map.

[0204] S44, performing arithmetic decoding according to the mean and variance of the predicted Gaussian distribution of the second feature map to determine the second feature map.

[0205] The characteristic binary code stream is arithmetically decoded according to the mean and variance of the Gaussian values ​​of the predicted second characteristic map to determine the second characteristic map, and the second characteristic map contains quantized three-dimensional features.

[0206] S45, inputting the second feature map into a decoding network to determine a reconstructed image.

[0207] Among them, first, the second feature map is input into the main decoding network to determine the main decoding image; secondly, the second feature map is input into the bypass decoding network to determine the bypass decoding image; finally, the main decoding image and the bypass decoding image are combined to determine the reconstructed image.

[0208] The present disclosure provides an image compression method which uses a trained image compression network to efficiently decompose image information and reduce redundant information, thereby reducing image compression bitrate overhead and improving image reconstruction quality.

[0209] Figure 5 It is a block diagram of an image compression system according to an exemplary embodiment.

[0210] like Figure 5 As shown, in a possible embodiment, the present disclosure provides an image compression system 1000, including:

[0211] The acquisition module 1100 is used to acquire the image to be compressed.

[0212] Encoder 1200, the encoder is used to encode the image to be compressed according to the image encoding method provided by the present disclosure, and obtain a compressed image binary code stream;

[0213] Decoder 1300, the decoder uses the image decoding method provided by the present disclosure to decode the compressed image binary code stream to determine a reconstructed image.

[0214] Regarding the embodiment of the above system, the specific manner in which each device performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0215] In a possible embodiment, an image compression system 1000 provided by the present disclosure is used to compress 24 images with a resolution of 512x768 in a Kodak test set, including:

[0216] The acquisition module sequentially acquires the images to be compressed in the Kodak test set.

[0217] Encoder: Encode the image to be compressed according to the above steps S31 to S35 to determine the compressed image binary code stream.

[0218] The three-dimensional feature of the first feature map of the image to be compressed has 230 channels, 32 heights, and 48 widths.

[0219] Decoder: Decode the compressed image binary code stream according to the above steps S36 to S45 to determine the reconstructed image.

[0220] The three-dimensional features of the decoded reconstructed image have 230 channels, 32 heights, and 48 widths.

[0221] Figure 6 The figure is a flow chart of an image compression network training method according to an exemplary embodiment.

[0222] like Figure 6 As shown, in a possible embodiment, one million natural images in the ImageNet-train data set are used to train the image compression network provided by the present disclosure, and the trained image compression network can be used to encode and decode images, including S51 to S59.

[0223] S51, 3D feature extraction: randomly crop the images in the training set into images with a resolution of 256×256, and input them into the encoding network consisting of the main encoding network and the bypass encoding network to obtain 3D features.

[0224] Among them, the number of three-dimensional feature channels is 192, and the height and width are both 16.

[0225] Of course, the above-mentioned image resolution and three-dimensional feature parameters in this embodiment can be adjusted according to actual conditions and are not limited to those described in this step. The same applies to the parameters in the following steps.

[0226] Correspondingly, the main encoding network may also adopt other network structures, such as cascade of Transformer blocks and residual blocks, etc., and is not limited to the cascade of convolution modules in this step.

[0227] S52, super-prior feature extraction and noise addition: The three-dimensional features are input into the super-prior encoding neural network to obtain the super-prior features. In order to avoid the gradient being 0 everywhere caused by the quantization operation, a uniformly distributed noise of -0.5 to 0.5 is added to the super-prior features to simulate the actual quantization.

[0228] Among them: the super prior coding neural network is composed of three convolutional layers in cascade; of course, the super prior coding neural network can also be other network structure forms, such as three residual modules in cascade, etc., and is not limited to the super prior coding neural network in this step.

[0229] S53, super a priori feature bit rate estimation: establish a distribution estimation for the super a priori features that is independent of each other, and use the distribution estimation to obtain the super a priori feature bit rate estimation.

[0230] S54, super-prior information acquisition: input the super-prior features after adding noise into the super-prior decoding neural network to obtain super-prior information, wherein: the super-prior decoding neural network is composed of three cascaded convolutional layers.

[0231] S55, feature noise addition: To avoid the gradient being zero everywhere due to the quantization operation, a uniformly distributed noise of -0.5 to 0.5 is added to the three-dimensional features to simulate actual quantization.

[0232] S56, Gaussian distribution probability modeling: The structure of this module is similar to Figure 2 The probability entropy modeling in the illustrated embodiment is exactly the same; the hyper-prior information is used to predict the mean and variance of the Gaussian distribution; of course, the entropy model network may also be in other forms, such as a spatial context-based autoregressive model and a reconstruction context-based autoregressive model.

[0233] S57, characteristic code rate estimation: obtain characteristic code rate estimation according to the mean and variance of the predicted Gaussian distribution.

[0234] S58, total bit rate estimation: add the super a priori characteristic bit rate estimation and the characteristic bit rate estimation to obtain a total bit rate estimation.

[0235] S59, image reconstruction: input the noisy three-dimensional features into a coding network composed of a main coding network and a bypass coding network to obtain a reconstructed image.

[0236] In a possible embodiment, the ImageNet dataset is used to train the four image compression networks provided by the present disclosure, and λ is set to 0.0018, 0.0035, 0.0067, and 0.0130, respectively, corresponding to image compression networks with four different bit rates, and the Adam optimizer is used for 1,000,000 steps of optimization. The initial 900,000 steps of learning rate is set to 10 -4 , and finally dropped to 10 at 900000 steps -5 .

[0237] The four image compression networks trained above are used to compress 24 images with a resolution of 512×768 in the Kodak test set. The average bit rates of the compressed images are 0.1284bpp, 0.1983bpp, 0.2963bpp, and 0.4332bpp respectively; the average distortion PSNR of the reconstructed images and the original images are 29.650, 31.142, 32.712, and 34.430 respectively.

[0238] Compared with the method using the benchmark neural network, the image compression network provided by the present disclosure has better rate-distortion performance and faster training convergence speed. Specifically, under the premise of the same reconstruction quality, it can reduce the encoding bit rate overhead by 1-2% and save 50% of the training time.

[0239] Based on the same concept as above, in another embodiment of the present disclosure, a terminal is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute an image encoding method, an image decoding method, or an image compression method when executing the program.

[0240] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0241] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0242] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0243] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0244] In an embodiment of the present disclosure, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of an image encoding method, an image decoding method, or an image compression method in any of the above embodiments are implemented.

[0245] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0246] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0247] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0248] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0249] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0250] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. An image compression network, characterized in that: include: Encoding network, decoding network and entropy model; The coding network includes a main coding network and a bypass coding network, the coding network is used to encode the image to be compressed, and the bypass coding network adopts orthogonal linear constraints; The decoding network includes a main decoding network and a bypass decoding network, the main decoding network and the bypass decoding network are connected in parallel, the decoding network is used to decode a compressed image binary code stream, and the bypass decoding network adopts orthogonal linear constraints; The entropy model is used to predict the mean and variance of a Gaussian distribution.

2. The image compression network according to claim 1, characterized in that include: The main encoding network adopts an end-to-end image compression encoding network, including multiple convolution blocks or multiple Transformer blocks; The bypass coding network includes a plurality of orthogonal linear wavelet coding units, each of which includes a cascaded discrete wavelet forward transform and an orthogonal linear layer.

3. The image compression network according to claim 1, characterized in that include: The main decoding network adopts a decoding network of an end-to-end image compression network, and the main decoding network includes multiple convolution blocks or multiple Transformer blocks; The bypass decoding network includes a plurality of orthogonal linear wavelet decoding units, each of which includes a cascaded discrete wavelet inverse transform and an orthogonal linear layer.

4. The image compression network according to claim 1, characterized in that The training loss functions of the encoding network, the decoding network and the entropy model are: in, represents distortion, x represents the image to be compressed, represents the reconstructed image, λ1 represents the first balance parameter, λ2 represents the second balance parameter, R1 represents the bit rate, represents the loss term used to constrain the orthogonality of the orthogonal linear layer; in, represents the characteristics of the kth element, represents the predicted distribution of the kth element, θ k ={φ k ,ψ k } represents a set of learnable parameters, Represents the predictive distribution of side information; Among them, I represents a matrix with diagonal values ​​of 1, W enc,j represents the coefficients of the orthogonal linear layer of the orthogonal linear wavelet coding unit of the bypass coding network, W enc,j Represents coefficients of an orthogonal linear layer of an orthogonal linear wavelet decoding unit of the bypass decoding unit.

5. An image encoding method, characterized in that: include: Inputting the image to be compressed into the encoding network, determining a first feature map of the image to be compressed, wherein the first feature map represents a three-dimensional feature map; quantizing each eigenvalue in the first feature map into an integer to determine a second feature map; Inputting the second feature map into an entropy model, parameterizing the distribution of the second feature map into a Gaussian distribution, and determining the Gaussian distribution of the second feature map; According to the Gaussian distribution of the second feature map, determining the predicted mean and variance of the Gaussian distribution, Determining a probability estimate of the second feature map based on the predicted mean and variance of the Gaussian distribution; The probability estimate of the second feature map is arithmetically encoded to determine a compressed image binary code stream.

6. The method according to claim 5, characterized in that The step of inputting the image to be compressed into the encoding network and determining the first feature map of the image to be compressed comprises: Inputting the image to be compressed into a main coding network to determine a main feature map; Inputting the image to be compressed into a bypass coding network to determine a bypass feature map; The main feature map and the bypass feature map are combined to determine a first feature map of the image to be compressed.

7. The method according to claim 6, characterized in that The step of inputting the image to be compressed into a bypass coding network and determining a first bypass feature map comprises: Input the image to be compressed into the first orthogonal linear wavelet coding unit, and in order, the output of each previous orthogonal linear wavelet coding unit is used as the input of the next orthogonal linear wavelet coding unit, and the last orthogonal linear wavelet coding unit outputs the first bypass feature; Among them, the input amount of each of the orthogonal linear wavelet coding units is used as the input amount of the discrete wavelet forward transform of each of the orthogonal linear wavelet coding units, the discrete wavelet forward transform outputs a wavelet transform coding feature map, the output amount of the discrete wavelet forward transform is used as the input amount of the orthogonal linear layer of each of the orthogonal linear wavelet coding units, the orthogonal linear layer outputs a first transformation feature, and the output amount of the orthogonal linear layer is used as the output amount of each of the orthogonal linear wavelet coding units.

8. An image decoding method, characterized in that: Used to decode the compressed image binary code stream obtained in claim 5, comprising: Get the compressed image binary code stream; Inputting the compressed image binary code stream into an entropy model to determine a mean and a variance of a Gaussian distribution of a predicted second feature map; Performing arithmetic decoding according to the mean and variance of the Gaussian distribution of the predicted second feature map to determine the second feature map; The second feature map is input into a decoding network to determine a reconstructed image.

9. The method according to claim 8, characterized in that The step of inputting the second feature map into a decoding network to determine a reconstructed image includes: Inputting the second feature map into a main decoding network to determine a main decoded image; Inputting the second feature map into a bypass decoding network to determine a bypass decoded image; The main decoded image and the bypass decoded image are combined to determine the reconstructed image.

10. The method according to claim 8, characterized in that The step of inputting the second feature map into a bypass decoding network to determine a bypass decoded image includes: Input the second feature map into the first orthogonal linear wavelet decoding unit, and in order, the output of each previous orthogonal linear wavelet decoding unit is used as the input of the next orthogonal linear wavelet decoding unit, and the last orthogonal linear wavelet decoding unit outputs the bypass decoded image; Among them, the input amount of each of the orthogonal linear wavelet decoding units is used as the input amount of the discrete wavelet inverse transform of each of the orthogonal linear wavelet decoding units, the discrete wavelet inverse transform outputs discrete wavelet features, the output amount of the discrete wavelet inverse transform is used as the input amount of the orthogonal linear layer of each of the orthogonal linear wavelet decoding units, the orthogonal linear layer outputs the second transformation feature, and the output amount of the orthogonal linear layer is used as the output amount of each of the orthogonal linear wavelet decoding units.

11. An image compression method, characterized in that: include: Obtain the image to be compressed; Encoding the image to be compressed based on the image encoding method described in any one of claims 5 to 7 to obtain a compressed image binary code stream; The compressed image binary code stream is decoded based on the image decoding method described in any one of claims 8 to 10 to obtain a reconstructed image.

12. A terminal, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a program executed by the processor, and the processor can perform any of the following methods when executing the program: The image encoding method according to any one of claims 5 to 7, or, The image decoding method according to any one of claims 8 to 10, or, The image compression method according to claim 11.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can execute the image encoding method described in any one of claims 5 to 7, or the image decoding method described in any one of claims 8 to 10, or the image compression method described in claim 11.

Images