Image compression post-processing method capable of regulating and controlling fidelity

Abstract

The invention discloses an image compression post-processing method capable of regulating and controlling fidelity, and the method comprises the steps: 1) constructing an image compression post-processing frame which comprises a consistency guide denoising diffusion model and a grammar vector generator; the consistency-guided denoising diffusion model comprises a diffusion feature encoder, a consistency feature encoder, a decoder and a syntax-driven feature fusion module SFF; the method comprises the steps of (1) training an image compression post-processing frame by using an image compression training data set, and (3) compressing a natural image to be compressed and inputting the compressed natural image into the trained image compression post-processing frame to obtain an enhanced image. According to the invention, an additional consistency guiding feature is introduced to constrain the deviation in the diffusion process; and an ultra-low bit rate stream syntax vector which is regulated and controlled by a tradeoff control factor and extracted from a coding end is utilized to guide adaptive fusion of a consistency guide feature and a diffusion feature, so that accurate fidelity-reality tradeoff control and high-quality image reconstruction are realized.

Description

Technical Field

[0001] The invention belongs to the field of digital image compression and relates to an image compression post-processing method using adjustable fidelity. Background Art

[0002] Image compression aims to reduce the storage space and transmission bandwidth required for images while maintaining the content and quality of the image. Image compression has always attracted attention from both industry and academia.

[0003] Traditional image compression frameworks are mostly composed of multiple manually designed modules coupled to reduce information redundancy and retain only key information. However, due to its reliance on artificial priors, the framework is not differentiable as a whole and difficult to optimize globally. Deep network-based image compression methods learn the probability distribution of image data through a rate-distortion loss function and optimize for fidelity to reduce the difference between the true distribution and the target distribution of entropy coding, thereby reducing the bitrate. However, traditional distortion metrics are often inconsistent with human subjective evaluation of image quality, resulting in the problem of better evaluation indicators but lower subjective quality.

[0004] Image compression methods for human vision give priority to increasing "realism" at the visual level, so as to give priority to preserving the image quality perceived by human vision as much as possible, rather than strictly using common fidelity measurement indicators as quality judgment criteria. Although these methods can produce satisfactory results, they often add some detailed textures when reconstructing images in order to increase realism, which introduces ambiguity in the content, making it difficult for users to distinguish which details are inherent in the original image and which are added to increase realism.

[0005] Based on the above, we can know that at the image level, under the same bit rate conditions, fidelity and realism cannot be achieved at the same time, and a trade-off must be made between the two. Therefore, an image compression method with adjustable fidelity is proposed, which hopes to take both fidelity and realism into account at the same time. By adjusting the bias of the decoder, the same main bitstream can be used to reconstruct images with different fidelity-realism trade-offs. However, these methods are still limited by the capabilities of existing generative models and cannot reconstruct higher quality images. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide an image compression post-processing method based on a denoising diffusion model, which introduces an additional consistency guide feature to constrain the deviation in the diffusion process. And use an ultra-low bit rate stream syntax vector that is regulated by a trade-off control factor and extracted from the encoding end to guide the adaptive fusion of the consistency guide feature and the diffusion feature to alleviate the uncertainty of the post-processing target. Accurate fidelity-realism trade-off control and high-quality image reconstruction are achieved. The overall framework is shown in the attached figure. Figure 1shown.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A method for post-processing image compression with adjustable fidelity, comprising the following steps:

[0009] 1) constructing an image compression post-processing framework, including a consistency-guided denoising diffusion model and a grammar vector generator; the consistency-guided denoising diffusion model includes a diffusion feature encoder, a consistency feature encoder, a decoder, and a grammar-driven feature fusion module SFF;

[0010] 2) Using the image compression training data set to train the image compression post-processing framework, the training method is:

[0011] 21) Compress the input image x to obtain a degraded image The realism control factor α, image x and degraded image Input the semantic vector generator to obtain a grammar vector s for weighing and regulating fidelity and realism and input it into the grammar-driven feature fusion module SFF; wherein by injecting different realism control factors α, the grammar vector generator outputs different grammar vectors s;

[0012] 22) The consistency-guided denoising diffusion model is used for degraded images Perform T-step prediction to obtain the reconstructed image x0;

[0013] t=1~T;

[0014] When predicting in step t, the noise map y input in step t is used t For degraded images Add noise to obtain a noisy image and input it into the diffusion feature encoder to obtain N diffusion feature maps with different resolutions. And input it into the syntax driven feature fusion module SFF, the different resolution diffusion feature maps d i Corresponding to different scales;

[0015] Where i∈{0,...,N};

[0016] During the prediction of step t, the consistency feature encoder is obtained from the degraded image Extract N-level feature maps

[0017] And input it into the grammar-driven feature fusion module SFF; the i-th level feature map e i The resolution diffusion feature map d of the i-th scale i The scales correspond to each other;

[0018] The syntax-driven feature fusion module SFF includes N+1 fusion units Nth fusion unit SFF N Used to get the grammar vector s from d N 、e N Fusion to obtain the initial feature u N ′,u N ′ and d N After fusion, we get the fusion feature u N And input to the N-1th fusion unit SFF N-1 , the jth fusion unit SFF j According to the grammatical vector s, the fusion feature u j+1 、e j Fusion to obtain the initial feature u j ′,u j ′ and d j After fusion, we get the fusion feature u j And input to the j-1th fusion unit SFF j-1 , j = N-1~0;

[0019] The initial feature u0′ output by the 0th fusion unit SFF0 is fused with d0 to obtain the fused feature u0 as the t-1th

[0020] The noise map y of the step input t-1 ; Use the fused features output from the last step as the reconstructed image x0;

[0021] 23) According to the set loss function L = λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R Calculate the loss value, and optimize the image compression post-processing framework according to the loss value; wherein, λ d , p and λ f is the weight term, α is the trade-off control factor, L d is the mean square error loss function, L p is the quality loss function perceived by human eyes, L R is the bit rate loss function used to measure the bit stream size required for encoding feature s;

[0022] 3) For the natural image to be compressed, the compressed image is input into the trained image compression post-processing framework to obtain an enhanced image.

[0023] Furthermore, a two-stage training strategy is used to train the image compression post-processing framework; wherein, in the first stage of training, the trade-off control factor α is set to a fixed value of 0.5, and the first stage is 2.0×10 5 In the second stage of training, the trade-off control factor α is set to take a random value in the range of [0.0, 1.0], and 5.0×10 5 Iterations.

[0024] Furthermore, the syntax vector generator generates the syntax vector code stream s by first upsampling the realism control factor α to obtain the image x and Features with the same scale α x ; Then α x , x, and Cascade to get features Then from the features Multiple layers of features with different scales are extracted, and the features of each scale are globally average pooled to obtain a one-dimensional vector. The one-dimensional vectors are cascaded and arithmetic encoded to obtain a syntax vector code stream s.

[0025] Furthermore, L p =L LPIPS-A (x,x0)+L LPIPS-V (x,x0); where L LPIPS To learn the loss function for perceptual image patch similarity, L LPIPS-A Indicates the use of pre-trained AlexNet network to extract image features, L LPIPS-V Indicates the use of pre-trained VGG network to extract image features, φ l (x) represents the l-th layer feature of image I extracted by the network, H l and W l Respectively represent φ l (x) The width and height of the

[0026] Furthermore, the jth fusion unit SFF j Get the fusion feature u j The method is as follows: first, the decoding result obtained after decoding the grammar vector s is concatenated with t and then input into a fully connected network to obtain two dynamic convolution kernels. and Then pass Calculate the fusion feature u j ; where * represents the convolution operation.

[0027] Furthermore, a gradient descent algorithm is used to optimize the image compression post-processing framework according to the loss value, and the gradient value is truncated in the interval [-0.5, 0.5] during gradient back propagation.

[0028] Furthermore, the image x is compressed using a lossy compression method optimized for fidelity to obtain a degraded image

[0029] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the above method.

[0030] A computer-readable storage medium stores a computer program, wherein the computer program implements the above method when executed by a processor.

[0031] Compared with the prior art, the present invention has the following positive effects:

[0032] The present invention significantly improves the performance of adjustable fidelity image compression. On the Kodak image compression benchmark dataset, compared with the traditional image compression method BPG, it can improve the peak signal-to-noise ratio BD-rate by 33.28 when biased towards fidelity, and improve the Vgg network-based perceptual image block similarity (LPIPS-V) BD-rate by 69.41 when biased towards realism. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Figure 2 is a diagram of the image compression post-processing framework with adjustable fidelity.

[0034] Figure 2 The framework diagram of the grammar vector generator submodule.

[0035] Figure 3 Framework diagram of the consistency-guided denoising diffusion model.

[0036] Figure 4 The framework diagram of the grammar-driven feature fusion submodule. DETAILED DESCRIPTION

[0037] In order to make the above features and advantages of the present invention more obvious and easy to understand, the following embodiments are specifically cited and described in detail with reference to the accompanying drawings. It should be noted that the specific number of layers, number of modules, number of functions, and settings of certain layers given in the following embodiments are only a preferred implementation method and are not intended to be limiting. It should be understood that those skilled in the art can select the number and set certain layers according to actual needs.

[0038] This embodiment discloses an image compression post-processing method based on a denoising diffusion model, which is specifically described as follows:

[0039] Step 1: Collect high-quality image training datasets for network training and testing.

[0040] Step 2: Build an image compression post-processing training framework.

[0041] The image compression post-processing framework with adjustable fidelity is as follows Figure 1 As shown, it includes two units, namely a consistency-guided denoising diffusion model (including four sub-units: a diffusion feature encoder, a consistency feature encoder, a decoder, and a grammar-driven feature fusion module SFF) and a grammar vector generator module. Figure 3 In the above example, the diffusion feature encoder is the region corresponding to feature d, the consistency feature encoder is the region corresponding to feature e, and the decoder is the region corresponding to feature u.

[0042] In terms of the process, starting from the input image x, we first use a lossy compression method optimized for fidelity to obtain a degraded image The specific lossy compression method is not within the scope of protection of this patent. Then, the semantic vector generator is used with a realistic control factor α, the original input image x, and the degraded image As input, a grammatical vector s is extracted, which will be used to regulate the trade-off between fidelity and realism. t The complete denoising diffusion process of the input noise map finally obtains a higher quality reconstructed image x0. By injecting different realism control factors α at the encoding end, the grammar vector generator can obtain different grammar vectors s, which can store the corresponding grammar prior information and fidelity control information. With different grammar vectors s as the input of the guide vector, the consistency guided diffusion model can Different enhancements are performed to reconstruct high-quality images x0 with different fidelity-realism trade-offs. The flexibility of the application is increased through the processing of this part. For example, when the user expects to use this method for compression oriented to fidelity, the realism control factor α is set to a smaller value (the smaller α is, the higher the fidelity of the reconstructed image and the lower the realism). In this case, the vector generator can generate a grammatical vector s that completely corresponds to the input α. This vector is used as the input of the branched guided diffusion model, and the reconstructed image x0 can have better fidelity. When the user expects to use this method for compression oriented to realism, the realism control factor α is set to a larger value (the larger α is, the higher the realism of the reconstructed image and the lower the fidelity). In this case, the vector generator can generate a grammatical vector s that completely corresponds to the input α. This vector is used as the input of the branched guided diffusion model, and the reconstructed image x0 can have better realism.

[0043] The specific framework of the grammar vector generator is as follows Figure 2 As shown, the image x and the degraded image after lossy compression are Both are used as inputs to the grammar vector generator, and a control factor α∈[0.0,1.0] is injected to provide guidance for the optimization target. In the grammar vector generator, the control factor α is first upsampled to obtain the same value as the input image x and Features with the same scale α x After that, α x , x, and Three features with the same size are concatenated to obtain the feature Then, the feature Injected into the encoder of the grammar vector generator to extract multiple layers of features with different scales Global average pooling is used at each scale to compress multiple layers of features into a compact one-dimensional vector, and these vectors are cascaded to obtain the final output syntax vector. The syntax vector is then arithmetically encoded using the rANS entropy encoder to obtain the syntax vector code stream s.

[0044] In the T-step prediction process, the t-step prediction is based on y t In the complete denoising diffusion process of the input noise image, the commonly used denoising process is adopted, that is, starting from an initial state q(xT)~N(0,I) of pure Gaussian noise, the target image x0 is obtained through multi-step prediction, where the probability of each step prediction is:

[0045]

[0046] Among them, a neural network is used to predict the mean of the distribution The framework diagram of the consistency-guided denoising diffusion model used in the prediction is as follows Figure 3 As shown, the diffusion feature encoder and the consistency feature encoder are connected through the UNet network (i.e. and The jump connection between them is the UNet network connection mode), and the diffusion feature encoder above converts the noisy input image into a set of multi-resolution diffusion feature maps Each feature map corresponds to a different scale, where N represents the depth of the U-Net network backbone, where N=4, i∈{0,...,N}. At the same time, the consistency feature encoder below is from the degraded image Extract feature maps These feature maps are similar in scale to The feature maps are kept in one-to-one correspondence. Afterwards, a set of grammars drive the feature fusion module To fuse the two branch features, first, use SFF to fuse d N and e N Get u N , the process can be expressed as:

[0047] uN =SFF N (u N ,e N ,s))

[0048] Afterwards, the syntax-driven feature fusion module SFF iteratively fuses the corresponding features u in each layer of the decoder i+1 and e i , and the corresponding fusion feature and the diffusion feature d of the corresponding scale i Fusion gets u i , the specific process can be expressed as:

[0049] u i =Conv(d i ,SFF i (u i+1 ,e i ,s))

[0050] Conv represents the convolution operation. After layer-by-layer iterative fusion, the final fusion feature u0 is the predicted probability distribution mean. The fused feature u0 is also used as the noise map y input in the t-1th step t-1 Enter the next iteration.

[0051] The fused features of the last step prediction output are used as the reconstructed image x0.

[0052] The framework diagram of the grammar-driven feature fusion module SFF is as follows Figure 4 As shown, by concatenating the grammatical vector obtained by decoding s and t and injecting them into a fully connected network, two dynamic convolution kernels can be obtained. and In the first step, d N and e N Get u N In the process, the two convolution kernels are respectively N and d N Perform convolution, that is:

[0053]

[0054] Then in the remaining layers, these two convolution kernels are respectively combined with e i and u i-1 Convolution is performed separately to achieve adaptive fusion of features in each layer:

[0055]

[0056] Among them, * represents the convolution operation.

[0057] Step 3: Use the collected data set to train the model. The total loss function of the image compression post-processing model based on the denoising diffusion model is:

[0058] L=λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R ,

[0059] Among them, λ d , p and λ f is a weight term, usually λ d Set to 65.0, λ p Set to 34.56, λ f Set to 0.00001. α in the loss function is the trade-off control factor of the input model. During training, its value is directly used as the proportional factor of each item in the loss function to allow the model to fully learn the correlation between this value and the trade-off ratio between fidelity and realism. The training batch size of the model is 2, using the Adam optimizer, and the initial learning rate is 1×10 -4 , optimizer hyperparameters are set to β1 = 0.9, β2 = 0.999. In addition, to avoid gradient explosion, the gradient value in the gradient back propagation will be truncated in the interval [-0.5, 0.5]. During the training process, 256×256 pixel blocks are randomly cropped and a two-stage training strategy is used.

[0060] Specifically, the training strategy is carried out in two stages:

[0061] In the first stage, the trade-off control factor of the input network is set to a fixed value of 0.5, and the corresponding proportional factor α in the loss function is also 0.5, that is, the loss function term is:

[0062] L=λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R ,

[0063] Through this process, the consistency-guided denoising diffusion model can be optimized under the guidance of a single optimization goal, gradually learn the data distribution, achieve good performance under the single optimization goal, and realize the warm-up process of the model. 5 Iterations.

[0064] After the completion of the first stage, the consistency-guided denoising diffusion model has been able to learn to fit a specific trade-off and achieve better performance. After that, the second stage of model training is launched. In the second stage of training, the model preheated in the first stage is guided to learn the correlation between the trade-off control factor α and the desired fidelity-realism trade-off. In the training operation, the change of α is released, and α is allowed to randomly take values ​​in the range of [0.0,1.0]. Through the random value of α, the control factor input of the model will change accordingly, and at the same time, the loss function L that optimizes the fidelity in the loss function d And the loss function L that optimizes the sense of reality p The ratio between the two will also change, resulting in changes in the optimization target, gradually allowing the network to learn the correlation between the trade-off control factor α and the desired trade-off, allowing the trade-off control factor α to have the ability to control the trade-off. 5 Iterations.

[0065] 1) L d To optimize the mean square error loss function between the target x and the reconstructed image x0:

[0066] L d =‖x-x0‖ 2 ,

[0067] 2) L p To optimize the human eye perception quality loss function between the target x and the reconstructed image x0:

[0068] L p =L LPIPS-A (x,x0)+L LPIPS-V (x,x0),

[0069] Where L LPIPS Learn the loss function for perceptual patch similarity:

[0070]

[0071] Here L LPIPS-A and L LPIPS-V Respectively represent the use of pre-trained AlexNet network and VGG network to extract image features, where φ l (x) represents the l-th layer feature of image I extracted by the network, H l and W l Respectively represent φ l (x) The width and height of the

[0072] 3) L R is the bit rate loss function associated with feature s, which is used to measure the bit stream size required to encode feature s:

[0073] In actual calculation, the probability distribution of the original quantized feature s' of the vector s is obtained by compression and the negative log-likelihood function between the Gaussian distribution, that is: L R =-log s' (s').

[0074] Step 4: In the inference phase, the image to be compressed is input, and the compressed degraded image is obtained using the existing compression framework. At the encoding end, the degraded image, the original image, and the trade-off control parameter α are used to obtain the guidance vector s through the syntax vector generator. At the decoding end, the vector s and the compressed degraded image are injected into the post-processing model to enhance the degraded image to obtain a high-quality image that is consistent with the user's desired trade-off.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for post-processing image compression with adjustable fidelity, comprising: 1) Construct an image compression post-processing framework, including a consistency-guided denoising diffusion model and a grammar vector generator; The consistency-guided denoising diffusion model includes a diffusion feature encoder, a consistency feature encoder, a decoder, and a syntax-driven feature fusion module SFF; 2) Using the image compression training data set to train the image compression post-processing framework, the training method is: 21) Compress the input image x to obtain a degraded image The realism control factor α, image x and degraded image Input the semantic vector generator to obtain a grammar vector s for weighing and regulating fidelity and realism and input it into the grammar-driven feature fusion module SFF; wherein by injecting different realism control factors α, the grammar vector generator outputs different grammar vectors s; 22) The consistency-guided denoising diffusion model is used for degraded images Perform T-step prediction to obtain the reconstructed image x0; t=1~T; When predicting in step t, the noise map y input in step t is used t For degraded images Add noise to obtain a noisy image and input it into the diffusion feature encoder to obtain N diffusion feature maps with different resolutions. And input it into the syntax driven feature fusion module SFF, the different resolution diffusion feature maps d i Corresponding to different scales; where i∈{0,...,N}; During the prediction of step t, the consistency feature encoder is obtained from the degraded image Extract N-level feature maps And input it into the grammar-driven feature fusion module SFF; the i-th level feature map e i The resolution diffusion feature map d of the i-th scale i The scales correspond to each other; The syntax-driven feature fusion module SFF includes N+1 fusion units Nth fusion unit SFF N Used to get the grammar vector s from d N 、e N Fusion to obtain the initial feature u N ′,u N ′ and d N After fusion, the fusion feature u is obtained N And input to the N-1th fusion unit SFF N-1 , the jth fusion unit SFF j According to the grammatical vector s, the fusion feature u j+1 、e j Fusion to obtain the initial feature u j ′,u j ′ and d j After fusion, the fusion feature u is obtained j And input to the j-1th fusion unit SFF j-1 , j = N-1~0; The initial feature u0′ output by the 0th fusion unit SFF0 is fused with d0 to obtain the fused feature u0 as the noise image y input in the t-1th step t-1 ; Use the fused features output from the last step as the reconstructed image x0; 23) According to the set loss function L = λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R Calculate the loss value, and optimize the image compression post-processing framework according to the loss value; wherein, λ d , p and λ f is the weight term, α is the trade-off control factor, L d is the mean square error loss function, L p is the quality loss function perceived by human eyes, L R is the bit rate loss function used to measure the bit stream size required for encoding feature s; 3) For the natural image to be compressed, the compressed image is input into the trained image compression post-processing framework to obtain an enhanced image.

2. The method according to claim 1, characterized in that The image compression post-processing framework is trained using a two-stage training strategy; in the first stage of training, the trade-off control factor α is set to a fixed value of 0.5, and the first stage is 2.0×10 5 In the second stage of training, the trade-off control factor α is set to take a random value in the range of [0.0, 1.0], and 5.0×10 5 Iterations.

3. The method according to claim 1, characterized in that: The method for generating the syntax vector code stream s by the syntax vector generator is: firstly, up-sampling the realism control factor α is obtained to obtain the same value as the image x and Features with the same scale α x ; Then α x , x, and Cascade to get features Then from the features Multiple layers of features with different scales are extracted, and the features of each scale are globally average pooled to obtain a one-dimensional vector. The one-dimensional vectors are cascaded and arithmetic encoded to obtain a syntax vector code stream s.

4. The method according to claim 1, 2 or 3, characterized in that: L p =L LPIPS-A (x,x0)+L LPIPS-V (x,x0); where L LPIPS To learn the loss function for perceptual image patch similarity, L LPIPS-A Indicates the use of pre-trained AlexNet network to extract image features, L LPIPS-V Indicates the use of pre-trained VGG network to extract image features, φ l (x) represents the l-th layer feature of image I extracted by the network, H l and W l Respectively represent φ l (x) The width and height of the 5. The method according to claim 1, 2 or 3, characterized in that: The jth fusion unit SFF j Get the fusion feature u j The method is as follows: first, the decoding result obtained after decoding the grammar vector s is concatenated with t and then input into a fully connected network to obtain two dynamic convolution kernels. and Then pass Calculate the fusion feature u j ; where · represents the convolution operation.

6. The method according to claim 1, 2 or 3, characterized in that: The image compression post-processing framework is optimized according to the loss value using a gradient descent algorithm, and the gradient value is truncated in the interval [-0.5, 0.5] during gradient back propagation.

7. The method according to claim 1, characterized in that Compress image x using a lossy compression method optimized for fidelity to obtain a degraded image 8. A server, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images