A variable bit rate image compression method and system

Abstract

The present invention is aimed at the field of end-to-end optimized image compression, and discloses a variable bit rate image compression method and system. First, the bit rate control parameter λ is converted into a binary vector by one-hot encoding, and then multiple groups of expert voting weights are adaptively generated according to the encoding results through a fully connected layer. At the same time, the model will generate multiple groups of convolution kernel parameters corresponding to the same number of expert weights, and realize the efficient generation of the final dynamic convolution kernel by weighted summation according to the expert weights. Finally, a variational autoencoder is constructed with the dynamic parameter convolution kernel generated above. The variable bit rate model proposed by this method can effectively change the direction of data distribution change in the compression process according to the bit rate adjustment parameter, and the design of dynamic parameter adjustment can effectively reduce the computational complexity of the overall variable bit rate compression framework, so as to realize the control of multiple compression bit rates with a single compression model in the end-to-end optimized compression framework.

Description

Technical Field

[0001] The present invention relates to the field of end-to-end optimized image compression, and in particular to a variable bit rate image compression method and system. Background Art

[0002] Image compression is a fundamental technique in signal processing and computer vision. The continuous advancement of image and video compression methods has enabled the continuous innovation of new applications, including ultra-high-definition image data streaming and augmented reality. The goal of image compression, particularly lossy image compression, is to preserve the key visual information of the image signal while reducing the bit rate used to encode the image for efficient transmission and storage.

[0003] The traditional transform-based image compression pipeline consists of several basic modules and steps: transform, quantization, and entropy coding. First, a compressive transform operation is used to convert the image signal into compact and uncorrelated coefficients. Then, quantization discards the least important information by truncating the coefficient vector to smaller information dimensions. Finally, entropy coding is used to compress the quantized related coefficients. However, each module of a traditional hybrid image codec has complex dependencies on other modules, making it difficult to jointly optimize the entire codec. With the rapid development of deep learning, the potential of artificial neural networks has been further explored, and the concept of a learning-based image compression framework has been established. End-to-end optimized compression methods can jointly train the parameters of the entire framework, and improving the performance of each module will naturally promote the ultimate goal. However, most current image compression schemes only support a single bitrate per model, and designs for variable bitrate compression using a single model are rarely explored.

[0004] Building an effective variable-rate compression scheme that controls the model's rate-distortion trade-off through a rate adjustment parameter is a challenge. This method uses the Lagrange multiplier λ as the rate adjustment parameter. A neural network adaptively determines expert weights based on the value of λ. These weights are then assigned to a corresponding number of convolution kernels for weighted summation, ultimately participating in the overall data transformation and compression in the form of dynamic parameterized convolution. This approach, combined with the loss function of the overall end-to-end optimization framework, achieves variable bitrate functionality by adjusting the compression capability of the overall model through the rate adjustment parameter. Summary of the Invention

[0005] Purpose of the Invention: Most current learning-based image compression methods optimize networks by minimizing the weighted sum of rate-distortion pairs using Lagrangian multipliers. The lambda multiplier introduced in the Lagrangian matrix is used as a hyperparameter to train the network to achieve the desired balance between rate and quality. This optimization method requires the training and deployment of separate networks to adapt to the compression rate. Specifically, when performing rate-distortion optimization, different compression rates can only be achieved by training a corresponding number of models. Therefore, the deployment cost of this approach is quite high over a wide range of rate-distortion curves. This invention addresses this deployment difficulty of current end-to-end optimized image compression methods by providing a method for constructing a variable-rate image compression framework.

[0006] A variable bit rate image compression method, characterized in that:

[0007] Input images or features into the constructed and trained compression model containing the variational autoencoder;

[0008] The compression model compresses and optimizes the input image or feature and outputs the compressed image or feature;

[0009] In the above method, the construction of the variational autoencoder includes

[0010] The rate control parameter λ is converted into a binary vector through one-hot encoding, and then multiple groups of expert voting weights are adaptively generated according to the encoding results through a fully connected layer.

[0011] Generate multiple sets of convolution kernel parameters with the same number as the expert weights, and generate dynamic convolution kernels by weighted summation according to the expert weights.

[0012] The generated dynamic parameter convolution kernel replaces the ordinary convolution kernel to construct a variational autoencoder.

[0013] The gradient descent method is used to train the variational autoencoder, and the variable bit rate training strategy is used to optimize the variational autoencoder.

[0014] In the above method, the compression model construction steps are as follows:

[0015] The rate control parameter is encoded, and the value interval Λ of the encoding rate control parameter is defined, and Λ={256, 512, 1024, 2048, 4096, 6048} is taken, and the rate control parameter λ∈Λ.

[0016] A fully connected network is used to predict expert weights. The fully connected layer in the neural network is combined with the linear rectification activation function as a weight generation tool. The Lagrange multiplier λ is used as the bit rate control parameter to generate a set of adaptively adjusted expert weights for the next step of data variational direction guidance.

[0017] Generate multiple sets of learnable convolution kernel parameters, use convolutional neural networks to perform cascaded continuous variation on the input features, update the parameters of multiple sets of convolution kernels synchronously, and dynamically allocate the convolution kernels using the weights generated in step 2.2.

[0018] The convolution kernels are weighted and summed according to the simulated expert weights. The number of convolutions of the dynamic parameterized convolution is the linear combination of multiple convolution operations.

[0019] Reconstruct the compression model using dynamic convolution. Dynamic parameter convolution replaces all normal convolutions in the image compression framework. The convolution operations are all dynamic convolutions conditioned on the bitrate adjustment parameters. Under the guidance of training, the corresponding conditional variational encoding and decoding can be performed.

[0020] Construct a rate-distortion loss function for variable bitrate.

[0021] In the above method, during training, the gradient descent method is applied to the loss function, and a variable bit rate training strategy is used to optimize the overall model. Specifically, the main encoder obtains the potential expression y from the input image x, and then the super-potential expression z is obtained from the potential expression y through the super-prior encoder;.

[0022] The Lagrange multiplier λ is randomly selected to generate dynamic parameter convolution kernels and construct loss functions, making the model always sensitive to the bitrate adjustment parameters, thereby constructing an end-to-end optimized image compression model with variable bitrate inference.

[0023] In the above method, the specific process of encoding the rate control parameter is as follows:

[0024] The rate control parameter is encoded to eliminate the numerical distance prior. First, the value interval Λ of the encoding rate control parameter is defined. The rate control parameter λ∈Λ, the encoding result can be expressed as

[0025] λ′=encode Λ (λ)

[0026] Where λ′ represents the encoding result, encode Λ (λ) represents the encoding operation within the Λ set. In the variable bitrate model construction phase, encoding can ensure that the rate control parameters are reorganized in an encoded form under the premise of different bitrate types, avoiding the value of the rate control parameters from causing erroneous training results in the final model training.

[0027] In the above method, the specific process of simulating expert weight prediction is to use a neural network layer combined with a linear activation function as a weight generation tool. In the expert weight generation process, the length lambda_num of the rate control parameter list and the number of expert weights expert_num must be determined first. The weight generation process can be expressed as:

[0028]

[0029] Among them, w expert is a dynamically generated expert weight combination, This method uses a neural network to transform features with an input size of lambda_num into features with an output size of middle_channel. Act(x) is the activation function, and Softmax{X} is the normalized exponential function, which normalizes the module output so that the weighted sum is 1. At this point, the model has generated a set of adaptively adjusted expert weights using the Lagrange multiplier λ as a rate control parameter for the next step of data variational guidance.

[0030] In the above method, the specific process of generating the learnable convolution kernel parameters and the weighted sum of the corresponding convolution kernel is:

[0031] Use convolutional neural network to perform cascade continuous variation on input features;

[0032] The convolution kernels are weighted and summed according to the simulated expert weights. Considering the high computational complexity of linear combination of multiple convolution operations, the dynamic parameterized convolution with the same effect is optimized, that is,

[0033] (α1·W1+…+α n W n )*x=α1·(W1*x)+…+α n ·(W n *x),

[0034] where α i is the expert weight, W i is the convolution kernel parameter, and * represents the convolution operation. As can be seen from the formula, the two operations are exactly the same in terms of results, but in terms of overall computational efficiency, the number of convolutions of dynamic parameterized convolution is the linear combination of multiple convolution operations. The formula for dynamic parameterized convolution is defined as:

[0035]

[0036]

[0037] Since the Softmax function is used to normalize the weights, a constraint is introduced: the sum of the total weight terms is 1.

[0038] In the above method, the rate-distortion loss function for variable bitrate is constructed based on the following formula

[0039]

[0040] Where λ is the Lagrange multiplier, p x is the unknown distribution of natural images, represents rounding to the nearest integer (quantization), y=f(x) is the encoder, To quantify the potential expression, is a discrete entropy model, For the decoder, represents the reconstructed image. The first fraction in the loss function, the rate term, corresponds to the cross-entropy between the marginal distribution of the latent representation and the estimated result of the learned entropy model. Minimizing the cross-entropy makes the two distributions as similar as possible. The second fraction in the loss function, the distortion term, corresponds to the likelihood of an approximate form of the original image and the reconstructed result.

[0041] In the above method, the specific process of end-to-end training of the variable bit rate model is as follows:

[0042] Applying the gradient descent method to the loss function, the overall loss function of variable bitrate training can be expressed as the following function:

[0043]

[0044]

[0045] Loss total =R φ,θ (λ)+λ·D φ,θ (λ),

[0046] During training, the Lagrange multiplier λ is randomly selected to generate the dynamic parameter convolution kernel and construct the loss function, so that the model is always sensitive to the bitrate adjustment parameters, thereby constructing an end-to-end optimized image compression model with variable bitrate inference.

[0047] A variable bit rate image compression system, comprising:

[0048] Module 1: Input images or features into the constructed and trained compression model containing variational autoencoders;

[0049] The second module: The compression model compresses and optimizes the input image or feature and outputs the compressed image or feature.

[0050] Therefore, the present invention has the following advantages:

[0051] 1) The present invention constructs an end-to-end optimized image compression framework, which can adaptively adjust the bit rate of the compressed code stream according to the input conditions, achieving a trade-off between bit rate and image reconstruction quality.

[0052] 2) This paper proposes an efficient variable bitrate adjustment module for rate control of image compression models based on input Lagrange multipliers. By adaptively adjusting the weights of multiple convolution kernel parameters, the feature distribution can be modified according to conditions. Reparameterization is used to achieve efficient weighted convolution, resulting in a system with both high performance and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the overall framework diagram of the technology described in this invention.

[0054] Figure 2 Detailed diagram of the model of the technology described in this invention.

[0055] Figure 3 The overall flow chart of the method is shown in Figure 2. DETAILED DESCRIPTION

[0056] The overall flow chart of the present invention is as follows: Figure 3 The specific implementation steps are as follows:

[0057] Step 1: Overall end-to-end optimized compression model construction. As an end-to-end optimized lossy image compression method, the general operation process of the present invention is to map pixel data to a quantized latent representation and then perform lossless compression on the latent representation. The present invention adopts the form of a convolutional neural network, which approximates a nonlinear function and has the potential to map pixels to a latent space that is more compressible than the linear transformation used by traditional image codecs. The present invention uses multiple cascaded convolution operation layers and generalized split normalization to construct a variational autoencoder network. This nonlinear transform coding method consists of an encoder conversion and a decoder between data and the latent dimensional space. The goal of compression is to reduce the information entropy of the latent expression under the premise of constructing an entropy model of shared prior probabilities. Finally, the entropy model is used in combination with the entropy coding algorithm range coding to generate a compressed bit stream.

[0058] The present invention builds a basic framework based on the structure of a compressed variational autoencoder. The model input is the original image to be compressed, and the encoder output is a potential expression variable that conforms to a Gaussian distribution. This output result needs to be input into the super-prior module for further feature extraction and data distribution prediction, and serves as context information to assist the arithmetic encoder in entropy coding. The potential expression variable undergoes quantization and compression by the arithmetic encoder, and the compression result is written into the bitstream for storage. During decoding, the arithmetic decoder decodes the potential expression variable from the bitstream and restores the original image information through the decoding part of the compressed variational autoencoder.

[0059] The present invention provides an efficient variable bit rate module construction method. It generates dynamic weights according to the bit rate control parameters in a simulated expert manner, and performs weighted summation of multiple learnable convolution kernel parameters according to the weights to generate a dynamic parameter convolution kernel. The distribution of input features is changed by means of dynamic convolution operation. The steps of constructing the variable bit rate module are as follows:

[0060] Step 2.1: One-hot encode the rate control parameter. First, define the value range of the encoding rate control parameter Λ, and take Λ = {256, 512, 1024, 2048, 4096, 6048}. The rate control parameter λ∈Λ, the encoding result can be expressed as

[0061] λ′=onehot Λ (λ)

[0062] where λ ′ Indicates binary encoding result, onehot Λ (λ) represents the one-hot encoding operation within the Λ set. When building a variable-rate model, one-hot encoding ensures that the rate control parameters are reorganized in binary form for different bitrate types, preventing the numerical values of the rate control parameters from causing erroneous training results.

[0063] Step 2.2: Use a fully connected network to predict expert weights. The present invention uses expert weight generation to guide the direction of data distribution changes, and performs an efficient and deep data variation process by weighting the convolution kernel. The present invention uses the fully connected layer (FC) in the neural network and the rectified linear unit (ReLU) activation function as a weight generation tool. In the expert weight generation process, the length lambda_num of the rate control parameter list and the number of expert weights expert_num must be determined first. The weight generation process can be expressed as:

[0064]

[0065] Among them, w expert is a dynamically generated expert weight combination, This refers to using a fully connected layer to transform features with an input size of lambda_num into features with an output size of middle_channel. ReLU(x) is the linear rectification activation function, and Softmax{X} is the normalized exponential function, which normalizes the module output so that the weighted sum is 1. In this implementation, lambda_num is set to 6, middle_channel is set to 64, and expert_num is set to 4. At this point, the model has generated a set of adaptively adjusted expert weights using the Lagrange multiplier λ as a rate control parameter for the next step of data variational direction guidance.

[0066] Step 2.3: Generate multiple sets of learnable convolution kernel parameters. To perform an efficient variational process, the present invention uses a convolutional neural network to perform cascaded continuous variation on the input features and better explore the variational control capabilities of the bit rate control parameters. The present invention uses multiple sets of convolution kernels for synchronous parameter updates. Different convolution kernels will focus on different compression levels. The weights generated in step 2.2 will be used to dynamically allocate the effects of the convolution kernels.

[0067] Step 2.4: Perform weighted summation of the convolution kernels according to the simulated expert weights. Considering the high computational complexity of linear combination of multiple convolution operations, the dynamic parameterized convolution with the same effect is optimized, that is,

[0068] (α1·W1+…+α n W n )*x=α1·(W1*x)+…+α n ·(W n *x), where α i is the expert weight, W i is the convolution kernel parameter, and * represents the convolution operation. In this implementation, n is set to 4. As can be seen from the above formula, the two operations are completely consistent in terms of results, but in terms of overall computational efficiency, the number of convolutions of the dynamic parameterized convolution is the linear combination of multiple convolution operations. The present invention defines the formula of dynamic parameterized convolution as:

[0069]

[0070]

[0071] Since the Softmax function is used to normalize the weights in step 2.2, a constraint term is introduced: the sum of the total weight terms is 1.

[0072] Step 3: Reconstruct the compression framework using dynamic convolution. The dynamic parameter convolution, the core component invented in Step 2, replaces all standard convolutions in the image compression framework. This allows all convolution operations in the entire framework to be dynamic convolutions conditioned on the bitrate adjustment parameter. With proper training guidance, conditional variational encoding and decoding can be performed accordingly.

[0073] Step 4: Construct a rate-distortion loss function for variable bitrate. The training goal of the lossy compression model is to minimize the expected length of the bitstream and the distortion of the reconstructed image relative to the original image, which can be summarized as a rate-distortion optimization problem:

[0074]

[0075] Among them, λ is the Lagrange multiplier, which determines the rate-distortion trade-off expected by the model, p x is the unknown distribution of natural images, represents rounding to the nearest integer (quantization), y=f(x) is the encoder, To quantify the potential expression, is a discrete entropy model, For the decoder, Represents the reconstructed image. The first fraction in the loss function, namely the rate term, corresponds to the cross entropy between the marginal distribution of the potential expression and the estimated result of the learned entropy model. Minimizing the cross entropy makes the two distributions as similar as possible. The second fraction in the loss function, the distortion term, corresponds to the likelihood of the original image and the reconstructed result in an approximate form. For example, when When denoting the mean squared error (MSE), the model can be trained as a variational autoencoder.

[0076] Step 5: End-to-end training of variable bitrate model. The present invention applies the gradient descent method to the loss function in step 4 and uses a variable bitrate training strategy to optimize the overall model. Specifically, the present invention obtains the latent expression y from the input image x through the main encoder, and then obtains the super latent expression z from the latent expression y through the super prior encoder. Taking into account the bitrate consumption of the latent expression and the super latent expression and the overall reconstruction loss, the overall loss function of the variable bitrate training can be expressed as the following function:

[0077]

[0078]

[0079] Loss total =R φ,θ (λ)+λ·D φ,θ (λ),

[0080] This embodiment uses an autoregressive model for context prediction and construction of a hyper-prior module, but the autoregressive model is not within the scope of application of this patent and will not be described in detail.

[0081] During training, the Lagrange multiplier λ is randomly selected to generate the dynamic parameter convolution kernel and construct the loss function, so that the model is always sensitive to the bitrate adjustment parameters, thereby constructing an end-to-end optimized image compression model with variable bitrate inference.

[0082] The specific implementations described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the described specific implementations or replace them with similar methods without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.

Claims

1. A variable bit rate image compression method, characterized in that: Input an image or feature into a constructed and trained compression model that includes a variational autoencoder; wherein the variational autoencoder construction process includes: converting a rate control parameter λ into a binary vector through one-hot encoding, and then adaptively generating multiple sets of expert voting weights based on the encoding results through a fully connected layer; generating multiple sets of convolution kernel parameters with the same number of expert weights, and generating dynamic convolution kernels by weighted summation based on the expert weights; replacing ordinary convolution kernels with the generated dynamic convolution kernels to construct a variational autoencoder; training the variational autoencoder using a gradient descent method, and optimizing the variational autoencoder using a variable bit rate training strategy; During training, the gradient descent method is applied to the loss function, and a variable bitrate training strategy is used to optimize the overall model. Specifically, the main encoder obtains the latent expression y from the input image x, and then the super-latent expression z is obtained from the latent expression y through the super-prior encoder. The Lagrange multiplier λ is randomly selected to generate the dynamic parameter convolution kernel and construct the loss function, making the compression model always sensitive to the bitrate adjustment parameter, thereby constructing an end-to-end optimized image compression model with variable bitrate inference. The compression model compresses and optimizes the input image or feature and outputs the compressed image or feature; wherein, the construction process of the compression model includes: encoding the rate control parameter, defining the value interval Λ of the encoding rate control parameter, taking Λ={256,512,1024,2048,4096,6048}, and the rate control parameter λ∈Λ; using a fully connected network to predict expert weights, combining the fully connected layer in the neural network with the linear rectifier activation function as a weight generation tool, and using the Lagrange multiplier λ as the rate control parameter to generate a set of adaptively adjusted expert weights for the next step of data variation direction guidance; generating multiple sets of learnable convolution kernel parameters, using a convolutional neural network to perform cascaded continuous variation on the input features, synchronously updating the parameters of multiple sets of convolution kernels, and dynamically allocating the generated adaptively adjusted expert weights to the convolution kernel; performing weighted summation on the convolution kernel according to the simulated expert weights, and the number of convolutions of the dynamic parameterized convolution is a linear combination of multiple convolution operations. Use dynamic convolution to reconstruct the compression model; replace all ordinary convolutions in the image compression framework with dynamic parameter convolution. All convolution operations are dynamic convolutions conditioned on the bit rate adjustment parameters. Under the guidance of training, corresponding conditional variational encoding and decoding can be performed; construct a rate-distortion loss function for variable bit rate.

2. The variable bit rate image compression method according to claim 1, characterized in that: The specific process of encoding the rate control parameters is as follows: The rate control parameter is encoded to eliminate the numerical distance prior. First, the value interval Λ of the encoding rate control parameter is defined. The rate control parameter λ∈Λ, the encoding result can be expressed as λ′=encode Λ (l) Where λ′ represents the encoding result, encode Λ (λ) represents the encoding operation within the Λ set. In the construction of the variable bit rate model, encoding can ensure that the bit rate control parameters are reorganized in an encoded form under the premise of different bit rate types, so as to avoid the numerical size of the bit rate control parameters from causing erroneous training results in the final model training.

3. The variable bit rate image compression method according to claim 1, wherein: The specific process of simulating expert weight prediction is as follows: a neural network layer combined with a line activation function is used as a weight generation tool. During the expert weight generation process, the length lambda_num of the rate control parameter list and the number of expert weights expert_num must be determined first. The weight generation process can be expressed as: Among them, w expert is a dynamically generated expert weight combination, Refers to using a neural network to convert features with an input size of lambda_num into features with an output size of middle_channel. Act(x) is the activation function, and Softmax{X} is the normalized exponential function used to standardize the module output so that the sum of the weights is 1. At this point, the model has generated a set of adaptively adjusted expert weights using the Lagrange multiplier λ as the bitrate control parameter for the next step of data variational direction guidance.

4. The variable bit rate image compression method according to claim 1, wherein: The specific process of generating the learnable convolution kernel parameters and the weighted sum of the corresponding convolution kernels is as follows: Use convolutional neural network to perform cascade continuous variation on input features; The convolution kernels are weighted and summed according to the simulated expert weights; considering the high computational complexity of linear combination of multiple convolution operations, the dynamic parameterized convolution with the same effect is optimized, that is, (α1·W1+…+α n ·W n )*x=α1·(W1*x)+…+α n ·(W n *x), where α i is the expert weight, W i is the convolution kernel parameter, and * represents the convolution operation. As can be seen from the formula, the two operations are exactly the same in terms of results. However, in terms of overall computational efficiency, the number of convolutions of the dynamic parameterized convolution is the linear combination of multiple convolution operations. The formula for dynamic parameterized convolution is defined as: Since the Softmax function is used to normalize the weights, a constraint is introduced: the sum of the total weight terms is 1.

5. The variable bit rate image compression method according to claim 1, wherein: The rate-distortion loss function for variable bitrate is constructed based on the following formula Where λ is the Lagrange multiplier, p x is the unknown distribution of natural images, Indicates rounding to the nearest integer, y = f(x) is the encoder, To quantify the potential expression, is a discrete entropy model, For the decoder, Represents the reconstructed image; the first fraction in the loss function, that is, the rate term, corresponds to the cross entropy between the marginal distribution of the potential expression and the estimated result of the learned entropy model. Minimizing the cross entropy makes the two distributions as similar as possible; the second fraction in the loss function, the distortion term, corresponds to the likelihood of the approximate form of the original image and the reconstructed result.

6. The variable bit rate image compression method according to claim 1, characterized in that: The specific process of end-to-end training of the variable bit rate model is as follows: Applying the gradient descent method to the loss function, the overall loss function of variable bitrate training can be expressed as the following function: Loss total =R φ,θ (λ)+λ·D φ,θ (l), During training, the Lagrange multiplier λ is randomly selected to generate the dynamic parameter convolution kernel and construct the loss function, so that the model is always sensitive to the bitrate adjustment parameters, thereby constructing an end-to-end optimized image compression model with variable bitrate inference.

7. A variable bit rate image compression system, wherein the variable bit rate image compression system is used to implement the variable bit rate image compression method according to claim 1, characterized in that: include: Module 1: Input images or features into the constructed and trained compression model containing variational autoencoders; The second module: The compression model compresses and optimizes the input image or feature and outputs the compressed image or feature.

Images