A post-processing method for image compression with adjustable fidelity

Abstract

The present invention discloses a method for image compression post-processing with adjustable fidelity, comprising the following steps: 1) constructing an image compression post-processing framework, including a consistency-guided denoising diffusion model and a grammar vector generator; the consistency-guided denoising diffusion model includes a diffusion feature encoder, a consistency feature encoder, a decoder, and a grammar-driven feature fusion module (SFF); 2) training the image compression post-processing framework using an image compression training dataset; and 3) compressing a natural image to be compressed and inputting it into the trained image compression post-processing framework to obtain an enhanced image. The present invention introduces an additional consistency-guided feature to constrain deviations in the diffusion process; and utilizes an ultra-low bitrate stream grammar vector, which is controlled by a trade-off control factor and extracted from the encoding end, to guide the adaptive fusion of the consistency-guided feature and the diffusion feature, thereby achieving precise fidelity-realism trade-off control and high-quality image reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of digital image compression and relates to an image compression post-processing method using adjustable fidelity. Background Art

[0002] Image compression aims to reduce the storage space and transmission bandwidth required for images while preserving the image content and quality. Image compression has always attracted attention from both industry and academia.

[0003] Traditional image compression frameworks are often constructed by coupling multiple manually designed modules to reduce information redundancy and retain only critical information. However, due to their reliance on artificial priors, these frameworks are not differentiable, making them difficult to optimize globally. Deep network-based image compression methods use a rate-distortion loss function to learn the probability distribution of image data and optimize for fidelity, minimizing the discrepancy between the true and target distributions of entropy coding, thereby reducing bitrate. However, traditional distortion metrics often align with human subjective assessments of image quality, often resulting in better evaluation metrics but lower subjective quality.

[0004] Image compression methods targeting human vision prioritize increasing "realism" at the visual level, prioritizing the preservation of image quality as perceived by the human eye, rather than strictly judging quality based on common fidelity metrics. While these methods can produce satisfactory results, they often introduce ambiguity in the content by adding detailed textures to the reconstructed image to enhance realism, making it difficult for users to distinguish which details are inherent in the original image and which are added to enhance realism.

[0005] Based on the above, we can see that at the image level, at the same bitrate, fidelity and realism cannot be achieved simultaneously; a trade-off is inevitable. Therefore, image compression methods with adjustable fidelity have been proposed, hoping to balance both fidelity and realism. By adjusting the decoder's bias, the same main bitstream can be used to reconstruct images with different fidelity-realism trade-offs. However, these methods are still limited by the capabilities of existing generative models and cannot reconstruct higher-quality images. Summary of the Invention

[0006] To address the above issues, the present invention aims to provide an image compression post-processing method based on a denoising diffusion model. This method introduces an additional consistency-guided feature to constrain the deviation in the diffusion process. Furthermore, an ultra-low bitrate stream syntax vector, which is regulated by a trade-off control factor and extracted from the encoding end, is used to guide the adaptive fusion of the consistency-guided feature and the diffusion feature, thereby alleviating the uncertainty of the post-processing target. This method achieves precise fidelity-realism trade-off control and high-quality image reconstruction. The overall framework is shown in the attached figure. Figure 1shown.

[0007] The technical solution adopted in the present invention is as follows:

[0008] A method for post-processing image compression with adjustable fidelity, comprising the following steps:

[0009] 1) Constructing an image compression post-processing framework, including a consistency-guided denoising diffusion model and a grammar vector generator; the consistency-guided denoising diffusion model includes a diffusion feature encoder, a consistency feature encoder, a decoder, and a grammar-driven feature fusion module (SFF);

[0010] 2) Using the image compression training dataset to train the image compression post-processing framework, the training method is as follows:

[0011] 21) Compress the input image x to obtain a degraded image The realism control factor α, image x and degraded image Input the semantic vector generator to obtain a grammatical vector s for balancing fidelity and realism, and input the grammatical vector s into the grammar-driven feature fusion module SFF; wherein by injecting different realism control factors α, the grammatical vector generator outputs different grammatical vectors s;

[0012] 22) The consistency-guided denoising diffusion model is used to degrade the image Perform T-step prediction to obtain the reconstructed image x0;

[0013] t=1~T;

[0014] When predicting the tth step, the noise map y input in the tth step is used t For degraded images Add noise to obtain a noisy image and input it into the diffusion feature encoder to obtain N diffusion feature maps with different resolutions. And input it into the syntax driven feature fusion module SFF, the different resolution diffusion feature maps d i Corresponding to different scales;

[0015] Where i∈{0,...,N};

[0016] During the t-th step prediction, the consistency feature encoder is trained from the degraded image Extract N-level feature maps

[0017] And input it into the grammar-driven feature fusion module SFF; the i-th level feature map e i and the resolution diffusion feature map d of the i-th scale i The scales are consistent;

[0018] The syntax-driven feature fusion module SFF includes N+1 fusion units Nth fusion unit SFF N Used to get the grammar vector s from the d N 、e N Fusion is performed to obtain the initial feature u N ′,u N ′ and d N After fusion, the fusion feature u is obtained N And input to the N-1th fusion unit SFF N-1 , the j-th fusion unit SFF j According to the grammatical vector s, the fusion feature u j+1 、e j Fusion is performed to obtain the initial feature u j ′,u j ′ and d j After fusion, the fusion feature u is obtained j And input to the j-1 fusion unit SFF j-1 , j = N-1~0;

[0019] The initial feature u0′ output by the 0th fusion unit SFF0 is fused with d0 to obtain the fusion feature u0 as the t-1th

[0020] Noise map y of the step input t-1 ; Use the fusion features output from the last step as the reconstructed image x0;

[0021] 23) According to the set loss function L = λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R Calculate the loss value and optimize the image compression post-processing framework according to the loss value; wherein, λ d ,λ p and λ f is the weight term, α is the trade-off control factor, L d is the mean square error loss function, L p is the quality loss function perceived by human eyes, L R is the rate loss function used to measure the bitstream size required for encoding feature s;

[0022] 3) For the natural image to be compressed, the compressed image is input into the trained image compression post-processing framework to obtain an enhanced image.

[0023] Furthermore, a two-stage training strategy is used to train the image compression post-processing framework; wherein, in the first stage of training, the trade-off control factor α is set to a fixed value of 0.5, and the first stage is 2.0×10 5 In the second stage of training, the weight control factor α is set to take a random value in the range of [0.0, 1.0]. The second stage is 5.0×10 5 Iterations.

[0024] Furthermore, the syntax vector generator generates the syntax vector code stream s by first upsampling the realism control factor α to obtain the image x and Features α with the same scale x ; Then α x , x and Cascade to obtain features Then from the features Multiple layers of features with different scales are extracted, and the features of each scale are globally average pooled to obtain a one-dimensional vector. The one-dimensional vectors are concatenated and arithmetic coded to obtain a syntax vector code stream s.

[0025] Furthermore, L p =L LPIPS-A (x,x0)+L LPIPS-V (x,x0); where L LPIPS To learn the loss function for perceptual image patch similarity, L LPIPS-A Indicates that the features of the image are extracted using the pre-trained AlexNet network, L LPIPS-V Indicates that the features of the image are extracted using the pre-trained VGG network, φ l (x) represents the l-th layer feature of image I extracted by the network, H l and W l Respectively represent φ l (x) The width and height of the

[0026] Furthermore, the j-th fusion unit SFF j Get the fusion feature u j The method is as follows: first, the decoding result of the grammatical vector s is concatenated with t and then input into a fully connected network to obtain two dynamic convolution kernels. and Then pass Calculate the fusion feature u j ; where * represents the convolution operation.

[0027] Furthermore, a gradient descent algorithm is used to optimize the image compression post-processing framework according to the loss value, and the gradient value is truncated in the interval [-0.5, 0.5] during gradient back propagation.

[0028] Furthermore, the image x is compressed using a lossy compression method optimized for fidelity to obtain a degraded image

[0029] A server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the above method.

[0030] A computer-readable storage medium stores a computer program thereon, wherein the computer program implements the above method when executed by a processor.

[0031] Compared with the prior art, the present invention has the following positive effects:

[0032] The present invention significantly improves the performance of adjustable fidelity image compression. On the Kodak image compression benchmark dataset, compared with the traditional image compression method BPG, it can improve the peak signal-to-noise ratio BD-rate by 33.28 when biased towards fidelity, and improve the Vgg network-based perceptual image block similarity (LPIPS-V) BD-rate by 69.41 when biased towards realism. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a diagram of the image compression post-processing framework with adjustable fidelity.

[0034] Figure 2 The framework diagram of the grammar vector generator submodule.

[0035] Figure 3 Framework diagram of the consistency-guided denoising diffusion model.

[0036] Figure 4 The framework diagram of the grammar-driven feature fusion submodule. DETAILED DESCRIPTION

[0037] To further enhance the aforementioned features and advantages of the present invention, the following embodiments are provided with accompanying drawings for detailed description. It should be noted that the specific number of layers, modules, functions, and layer configurations given in the following embodiments are merely preferred implementations and are not intended to be limiting. Persons skilled in the art will appreciate that they can select the number and configuration of layers based on actual needs.

[0038] This embodiment discloses an image compression post-processing method based on a denoising diffusion model, which is described in detail as follows:

[0039] Step 1: Collect high-quality image training datasets for network training and testing.

[0040] Step 2: Build an image compression post-processing training framework.

[0041] The image compression post-processing framework with adjustable fidelity is as follows Figure 1 As shown, it includes two units: a consistency-guided denoising diffusion model (including four sub-units: a diffusion feature encoder, a consistency feature encoder, a decoder, and a grammar-driven feature fusion module SFF) and a grammar vector generator module. Figure 3 In the middle, the diffusion feature encoder is the region corresponding to feature d, the consistency feature encoder is the region corresponding to feature e, and the decoder is the region corresponding to feature u.

[0042] In terms of the process, starting from the input image x, we first use a lossy compression method for fidelity optimization to obtain a degraded image The specific lossy compression method is not within the scope of protection of this patent. Then, the semantic vector generator is used to generate the realism control factor α, the original input image x, and the degraded image As input, a grammatical vector s is extracted, which will be used to regulate the trade-off between fidelity and realism. t The complete denoising diffusion process of the input noise map finally obtains a higher quality reconstructed image x0. By injecting different realism control factors α at the encoding end, the syntax vector generator can obtain different syntax vectors s, which can store the corresponding syntax prior information and fidelity control information. With different syntax vectors s as the input of the guidance vector, the consistency guided diffusion model can Different enhancements are performed to reconstruct high-quality reconstructed images x0 with different fidelity-realism trade-offs. The flexibility of the application is increased through the processing of this part. For example, when the user expects to use this method for compression oriented towards fidelity, the realism control factor α is set to a smaller value (the smaller α is, the higher the fidelity of the reconstructed image and the lower the realism). In this case, the vector generator can generate a grammatical vector s that completely corresponds to the input α. This vector is used as the input of the branched guided diffusion model, and the reconstructed image x0 can have better fidelity. When the user expects to use this method for compression oriented towards realism, the realism control factor α is set to a larger value (the larger α is, the higher the realism of the reconstructed image and the lower the fidelity). In this case, the vector generator can generate a grammatical vector s that completely corresponds to the input α. This vector is used as the input of the branched guided diffusion model, and the reconstructed image x0 can have better realism.

[0043] The specific framework of the grammar vector generator is as follows Figure 2 As shown, the image x and the degraded image after lossy compression are As the input of the grammar vector generator, a control factor α∈[0.0,1.0] is injected to provide guidance for the optimization target. In the grammar vector generator, the control factor α is first upsampled to obtain the same value as the input image x and Features α with the same scale x After that, α x , x and Three features with the same size are concatenated to obtain the feature Then, the features Injected into the encoder of the grammar vector generator to extract multiple layers of features with different scales Global average pooling is used at each scale to compress multiple layers of features into a compact one-dimensional vector, and these vectors are cascaded to obtain the final output syntax vector. The syntax vector is then arithmetically encoded using the rANS entropy encoder to obtain the syntax vector code stream s.

[0044] In the T-step prediction process, the t-step prediction is based on y t In the complete denoising diffusion process of the input noise image, the commonly used denoising process is adopted, that is, starting from an initial state q(xT)~N(0,I) of pure Gaussian noise, the target image x0 is obtained through multi-step prediction, where the probability of each step prediction is:

[0045]

[0046] Among them, a neural network is used to predict the mean of the distribution The framework diagram of the consistency-guided denoising diffusion model used in prediction is as follows Figure 3 As shown, the diffusion feature encoder and the consistency feature encoder are connected through the UNet network (i.e. and The jump connection between them is the UNet network connection mode), and the diffusion feature encoder above converts the noisy input image into a set of multi-resolution diffusion feature maps Each feature map corresponds to a different scale, where N represents the depth of the U-Net network backbone, and here N=4, i∈{0,...,N}. At the same time, the consistency feature encoder below is obtained from the degraded image Extract feature maps These feature maps are similar in scale to The feature maps are kept in one-to-one correspondence. Afterwards, a set of grammar-driven feature fusion modules are used To fuse the two branch features, first, use SFF to fuse d N and e N Get u N , the process can be expressed as:

[0047] uN =SFF N (u N ,e N ,s))

[0048] Afterwards, the syntax-driven feature fusion module SFF iteratively fuses the corresponding features u in each layer of the decoder i+1 and e i , and the corresponding fusion feature and the diffusion feature d of the corresponding scale i Fusion gets u i , the specific process can be expressed as:

[0049] u i =Conv(d i ,SFF i (u i+1 ,e i ,s))

[0050] Where Conv represents the convolution operation. After layer-by-layer iterative fusion, the final fusion feature u0 is the predicted probability distribution mean The fusion feature u0 is also used as the noise image y input in step t-1 t-1 Enter the next round of iteration.

[0051] The fused features of the last step prediction output are used as the reconstructed image x0.

[0052] The framework diagram of the grammar-driven feature fusion module SFF is as follows Figure 4 As shown, by concatenating the grammatical vector obtained after decoding s and t and injecting them into a fully connected network, two dynamic convolution kernels can be obtained. and In the first step, d N and e N Get u N In the process, the two convolution kernels are respectively N and d N Perform convolution, that is:

[0053]

[0054] Then in the remaining layers, these two convolution kernels are respectively combined with e i and u i-1 Perform convolution separately to achieve adaptive fusion of features in each layer:

[0055]

[0056] Among them, * represents the convolution operation.

[0057] Step 3: Use the collected data set to train the model. The total loss function of the image compression post-processing model based on the denoising diffusion model is:

[0058] L=λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R ,

[0059] Among them, λ d ,λ p and λ f is the weight term, usually λ d Set to 65.0, λ p Set to 34.56, λ f Set to 0.00001. α in the loss function is the trade-off control factor of the input model. During training, its value is directly used as the scaling factor of each term in the loss function to allow the model to fully learn the correlation between this value and the trade-off ratio between fidelity and realism. The model is trained with a batch size of 2, using the Adam optimizer, and an initial learning rate of 1×10 -4 The optimizer hyperparameters are set to β1 = 0.9 and β2 = 0.999. Furthermore, to prevent gradient explosion, gradient backpropagation is clipped to the interval [-0.5, 0.5]. During training, 256×256 pixel blocks are randomly cropped and a two-stage training strategy is used.

[0060] Specifically, the training strategy is carried out in two stages:

[0061] In the first stage, the weight control factor of the input network is set to a fixed value of 0.5, and the corresponding proportional factor α in the loss function is also 0.5, that is, the loss function term is:

[0062] L=λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R ,

[0063] Through this process, the consistency-guided denoising diffusion model can be optimized under the guidance of a single optimization objective, gradually learning the data distribution, achieving better performance under this single optimization objective, and completing the model warm-up process. 5 Iterations.

[0064] After the completion of the first phase, the consistency-guided denoising diffusion model has been able to learn to fit a specific trade-off and achieve better performance. After that, the second phase of model training is launched. In the second phase of training, the model preheated in the first phase is guided to learn the correlation between the trade-off control factor α and the desired fidelity-realism trade-off. In the training operation, the change of α is released and α is randomly selected in the range of [0.0, 1.0]. Through the random value of α, the control factor input of the model will change accordingly, and at the same time, the loss function L that optimizes the fidelity in the loss function d and optimize the loss function L for realism p The ratio between the two will also change, which will lead to changes in the optimization target, gradually allowing the network to learn the correlation between the trade-off control factor α and the desired trade-off, so that the trade-off control factor α can generate the ability to control the trade-off. 5 Iterations.

[0065] 1)L d To optimize the mean square error loss function between the target x and the reconstructed image x0:

[0066] L d =‖x-x0‖ 2 ,

[0067] 2)L p To optimize the human eye perception quality loss function between the target x and the reconstructed image x0:

[0068] L p =L LPIPS-A (x,x0)+L LPIPS-V (x,x0),

[0069] Among them L LPIPS Learn a loss function for perceptual patch similarity:

[0070]

[0071] The L here LPIPS-A and L LPIPS-V Respectively represent the use of pre-trained AlexNet network and VGG network to extract image features, where φ l (x) represents the l-th layer feature of image I extracted by the network, H l and W l Respectively represent φ l (x) The width and height.

[0072] 3)L R is the bitrate loss function associated with feature s, which is used to measure the bitrate size required to encode feature s:

[0073] In actual calculation, the negative log-likelihood function between the probability distribution of the original quantized feature s' of the vector s and the Gaussian distribution is expressed by compression, that is: L R =-log s' (s').

[0074] Step 4: In the inference phase, the image to be compressed is input and the existing compression framework is used to generate a compressed, degraded image. On the encoder side, the degraded image, the original image, and the trade-off parameter α are used to generate a guidance vector s through a syntax vector generator. On the decoder side, vector s and the compressed, degraded image are fed into a post-processing model to enhance the degraded image and produce a high-quality image consistent with the user's desired trade-off.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the claims.

Claims

1. A method for post-processing image compression with adjustable fidelity, comprising: 1) Build an image compression post-processing framework, including a consistency-guided denoising diffusion model and a grammar vector generator; The consistency-guided denoising diffusion model includes a diffusion feature encoder, a consistency feature encoder, a decoder, and a syntax-driven feature fusion module SFF; 2) Using the image compression training dataset to train the image compression post-processing framework, the training method is as follows: 21) Compress the input image x to obtain a degraded image The realism control factor α, image x and degraded image Input the grammar vector generator to obtain a grammar vector s for balancing fidelity and realism, and input the grammar vector s into the grammar-driven feature fusion module SFF; wherein the grammar vector generator outputs different grammar vectors s by injecting different realism control factors α; 22) The consistency-guided denoising diffusion model is used to degrade the image Perform T-step prediction to obtain the reconstructed image x0; t=1~T; When predicting the tth step, the noise map y input in the tth step is used t For degraded images Add noise to obtain a noisy image and input it into the diffusion feature encoder to obtain N diffusion feature maps with different resolutions. And input it into the syntax driven feature fusion module SFF, the different resolution diffusion feature maps d i Corresponding to different scales; where i∈{0,...,N}; During the t-th step prediction, the consistency feature encoder is trained from the degraded image Extract N-level feature maps And input it into the grammar-driven feature fusion module SFF; the i-th level feature map e i and the resolution diffusion feature map d at the i-th scale i The scales are consistent; The syntax-driven feature fusion module SFF includes N+1 fusion units Nth fusion unit SFF N Used to get the grammar vector s from the d N 、e N Fusion is performed to obtain the initial feature u N ′,u N ′ and d N After fusion, the fusion feature u is obtained N And input to the N-1th fusion unit SFF N-1 , the j-th fusion unit SFF j According to the grammatical vector s, the fusion feature u j+1 、e j Fusion is performed to obtain the initial feature u j ′,u j ′ and d j After fusion, the fusion feature u is obtained j And input to the j-1 fusion unit SFF j-1 , j = N-1~0; The initial feature u0′ output by the 0th fusion unit SFF0 is fused with d0 to obtain the fused feature u0 as the noise image y input in the t-1th step t-1 ; Use the fusion features output from the last step as the reconstructed image x0; 23) According to the set loss function L = λ d L d (x0,x)+α·λ p L p (x0,x)+λ f L R Calculate the loss value and optimize the image compression post-processing framework according to the loss value; wherein, λ d ,λ p and λ f is the weight term, α is the trade-off control factor, L d is the mean square error loss function, L p is the quality loss function perceived by human eyes, L R is the rate loss function used to measure the bitstream size required for encoding feature s; 3) For the natural image to be compressed, the compressed image is input into the trained image compression post-processing framework to obtain an enhanced image.

2. The method according to claim 1, characterized in that The image compression post-processing framework is trained using a two-stage training strategy; in the first stage of training, the trade-off control factor α is set to a fixed value of 0.5, and the first stage is 2.0×10 5 In the second stage of training, the weight control factor α is set to take a random value in the range of [0.0, 1.0]. The second stage is 5.0×10 5 Iterations.

3. The method according to claim 1, characterized in that The method for generating the syntax vector code stream s by the syntax vector generator is as follows: first, up-sampling the realism control factor α is performed to obtain the value of the syntax vector code stream s corresponding to the image x and the image x. Features α with the same scale x ; Then α x , x and Cascade to obtain features Then from the features Multiple layers of features with different scales are extracted, and the features of each scale are globally average pooled to obtain a one-dimensional vector. The one-dimensional vectors are concatenated and arithmetic coded to obtain a syntax vector code stream s.

4. The method according to claim 1, 2 or 3, characterized in that: L p =L LPIPS-A (x,x0)+L LPIPS-V (x,x0); where L LPIPS To learn the loss function for perceptual image patch similarity, L LPIPS-A Indicates that the features of the image are extracted using the pre-trained AlexNet network, L LPIPS-V Indicates that the features of the image are extracted using the pre-trained VGG network, φ l (x) represents the l-th layer feature of image I extracted by the network, H l and W l Respectively represent φ l (x) The width and height of the 5. The method according to claim 1, 2 or 3, characterized in that: The j-th fusion unit SFF j Get the fusion feature u j The method is as follows: first, the decoding result of the grammatical vector s is concatenated with t and then input into a fully connected network to obtain two dynamic convolution kernels. and Then pass Calculate the fusion feature u j ; where · represents the convolution operation.

6. The method according to claim 1, 2 or 3, characterized in that: The image compression post-processing framework is optimized according to the loss value using a gradient descent algorithm, and the gradient value is truncated in the interval [-0.5, 0.5] during gradient back propagation.

7. The method according to claim 1, characterized in that Compress image x using a lossy compression method optimized for fidelity to obtain a degraded image 8. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images