Distortion-perception tradeoff controllable image compression method, system, medium and equipment

Abstract

The invention provides a distortion-perception balance controllable image compression method, system, medium and device, and the method comprises the steps: inputting a to-be-processed image into a pre-trained baseline encoder, and determining a first hidden variable; performing entropy coding on the first hidden variable to determine a second hidden variable; inputting the second hidden variable into a preset controllable feature fusion module, and determining a third hidden variable; and inputting the third hidden variable into a pre-trained baseline decoder, and determining a reconstructed image. According to the invention, the parameter and the structure of the pre-trained baseline deep learning image codec are not changed, the distortion-perception tradeoff in the single hidden variable can be adjusted through the preset controllable feature fusion module, the flexible adjustment of the distortion-perception tradeoff is realized, and a user can obtain different decoded images according to requirements.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image compression method, system, medium and device with controllable distortion-perception trade-off. Background Art

[0002] As digital visual data continues to dominate Internet traffic, the development of efficient image and video codecs has become increasingly important. In recent years, deep learning-based codecs have made significant progress in both the image and video domains, demonstrating superior compression performance compared to traditional codecs such as JPEG, JPEG2000, and BPG.

[0003] Current learning-based image codecs mainly rely on the transform coding paradigm and the variational autoencoder (VAE) architecture. Most of these models adopt rate-distortion loss as the optimization objective and directly optimize for low-distortion performance. However, distortion-oriented codecs usually exhibit mode averaging behavior at low bitrates, resulting in blurring that significantly degrades the visual quality for human observers.

[0004] Recent studies have shown that optimizing perceptual quality can achieve greater compression gains with less perceptible distortion, thereby reducing bitrate. Some work has proposed using generative adversarial networks (GANs) and diffusion-based decoders to optimize neural image codecs. However, these schemes have difficulty achieving high pixel-level fidelity because they introduce high-frequency noise and unrealistic textures. Studies have shown that both distortion and perception goals cannot be fully achieved simultaneously in a given architecture. But both pixel-level fidelity and image-level realism are critical to image compression frameworks. The inability to achieve both metrics simultaneously is a significant limitation of current image codecs. Given this limitation, an ideal codec should be able to flexibly traverse between different distortion-perceptual trade-offs at a given bitrate. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present disclosure is to provide an image compression method, system, medium and device with controllable distortion-perception trade-off.

[0006] To achieve the above objective, according to a first aspect of the present disclosure, a method for image compression with controllable distortion-perception tradeoff is provided, comprising:

[0007] Input the image to be processed into the pre-trained baseline encoder to determine the first latent variable;

[0008] Performing entropy coding on the first latent variable to determine a second latent variable;

[0009] Inputting the second latent variable into a preset controllable feature fusion module to determine a third latent variable;

[0010] The third latent variable is input into a pre-trained baseline decoder to determine a reconstructed image.

[0011] Optionally, performing entropy coding on the first latent variable to determine the second latent variable includes:

[0012] quantifying the first latent variable to determine the quantified first latent variable;

[0013] The quantized first latent variable is entropy encoded to determine the second latent variable.

[0014] Optionally, inputting the second latent variable into a preset controllable feature fusion module to determine a third latent variable includes:

[0015] Performing diffusion model sampling processing on the second latent variable to determine the third latent variable;

[0016] During the diffusion model sampling process, a distortion-perception trade-off parameter is used to weigh the distortion-perception of the second latent variable.

[0017] Optionally, determining the preset controllable feature fusion module includes:

[0018] Determine the pre-trained auxiliary encoder;

[0019] Inputting the training image into the pre-trained auxiliary encoder and outputting a fourth latent variable;

[0020] The fourth latent variable is used as the desired feature, and noises of different intensities are added to the second latent variable used for training as the input of the controllable feature fusion module, and the third latent variable is output, and the controllable feature fusion model is trained to determine the trained controllable feature fusion module;

[0021] The trained controllable feature fusion module is optimized by using total optimization loss back propagation to determine the preset controllable feature fusion module.

[0022] Optionally, the total optimization loss includes diffusion loss and reconstruction loss, the diffusion loss represents the loss between the third latent variable and the fourth latent variable, and the reconstruction loss represents the loss between the training image and the reconstructed training image.

[0023] Optionally, determining a pre-trained auxiliary encoder includes:

[0024] Inputting the training image into a preset auxiliary encoder and outputting a fourth latent variable;

[0025] Inputting the fourth latent variable into the pre-trained baseline decoder to determine a reconstructed training image;

[0026] The preset auxiliary encoder is optimized according to the loss between the training image and the reconstructed training image to determine the pre-trained auxiliary encoder.

[0027] Optionally, the loss between the training image and the reconstructed training image includes a perceptual loss and a distortion loss;

[0028] If the pre-trained baseline deep learning image codec is distortion optimized, the loss between the training image and the reconstructed training image adopts the perceptual loss;

[0029] If the pre-trained baseline deep learning image codec is perceptually optimized, the loss between the training image and the reconstructed training image adopts a distortion loss.

[0030] According to a second aspect of the present disclosure, there is provided an image compression system with controllable distortion-perception trade-off, comprising:

[0031] A latent variable conversion module, used for inputting the image to be processed into a pre-trained baseline encoder to determine a first latent variable;

[0032] An encoding module, used for performing entropy encoding on the first latent variable to determine a second latent variable;

[0033] A weighing module, used for inputting the second latent variable into a preset controllable feature fusion module to determine a third latent variable;

[0034] The decoding module is used to input the third latent variable into a pre-trained baseline decoder to determine a reconstructed image.

[0035] According to a third aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method provided in the first aspect of the present disclosure are implemented.

[0036] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0037] a memory having a computer program stored thereon;

[0038] A processor is used to execute the computer program in the memory to implement the steps of the method provided in the first aspect of the present disclosure.

[0039] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0040] Through the above technical solution, a preset controllable feature fusion module is added without changing the parameters and structure of the pre-trained baseline deep learning image codec. The distortion-perception trade-off in a single latent variable is adjusted through the preset controllable feature fusion module to achieve flexible distortion-perception trade-off adjustment. For the same compressed bitstream, users can obtain different decoded images according to the required distortion-perception trade-off, achieve free switching between low distortion and high perceptual quality, and improve the flexibility of image compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0042] Figure 1 The invention is a flowchart of an image compression method with controllable distortion-perception trade-off according to an exemplary embodiment.

[0043] Figure 2 It is a schematic diagram of a reasoning process for determining a preset controllable feature fusion module according to an exemplary embodiment.

[0044] Figure 3 It is a schematic diagram showing a comparison between a preset controllable feature fusion module and a baseline deep learning image codec bit rate-index curve according to an exemplary embodiment.

[0045] Figure 4 It is a schematic diagram showing a comparison of bit rate-distortion-perception conversion curves of a preset controllable feature fusion module according to an exemplary embodiment.

[0046] Figure 5 The invention is a block diagram of an image compression system with controllable distortion-perception trade-off according to an exemplary embodiment. DETAILED DESCRIPTION

[0047] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0048] Figure 1 The invention is a flowchart of an image compression method with controllable distortion-perception trade-off according to an exemplary embodiment.

[0049] like Figure 1 As shown, the present disclosure provides an image compression method with controllable distortion-perception trade-off, including S11 to S14.

[0050] S11, input the image to be processed into a pre-trained baseline encoder to determine the first latent variable.

[0051] Among them, the image to be processed is represented as x, and the first latent variable is represented as y.

[0052] The pre-trained baseline encoder uses an autoencoder to perform a nonlinear transformation on the input image x to be processed and convert it into the first latent variable y.

[0053] S12, performing entropy coding on the first latent variable to determine the second latent variable.

[0054] Among them, the second latent variable is expressed as

[0055] S13, inputting the second latent variable into a preset controllable feature fusion module to determine the third latent variable.

[0056] Among them, the third latent variable is expressed as

[0057] The preset controllable feature fusion module disclosed in the present invention is an additional module added to the decoding end. The preset controllable feature fusion module is a detachable, plug-and-play module. The preset controllable feature fusion module is directly connected to the pre-trained baseline decoder of the decoding end. There is no need to change the architecture of the pre-trained baseline encoder and the pre-trained baseline decoder, and there is no need to retrain the pre-trained baseline codec.

[0058] S14, inputting the third latent variable into the pre-trained baseline decoder to determine the reconstructed image.

[0059] Among them, the reconstructed image of the image to be processed is expressed as

[0060] In this disclosure, the pre-trained baseline encoder and the pre-trained baseline decoder are collectively referred to as a pre-trained baseline deep learning image codec.

[0061] The pre-trained baseline encoder and pre-trained baseline decoder disclosed in the present invention are basic neural network codecs that are trained to optimize distortion / perception by using rate constraints and distortion / perception loss between the reconstructed image and the original image as loss functions, i.e., pre-trained baseline deep learning image codecs. After training, their parameters remain unchanged during subsequent use. Moreover, there are no restrictions on the pre-trained baseline encoder and pre-trained baseline decoder used in the present invention, and any end-to-end deep learning image codec can be used.

[0062] Through the above technical solution, a preset controllable feature fusion module is added without changing the parameters and structure of the pre-trained baseline deep learning image codec. The distortion-perception trade-off in a single latent variable is adjusted through the preset controllable feature fusion module to achieve flexible distortion-perception trade-off adjustment. For the same compressed bitstream, users can obtain different decoded images according to the required distortion-perception trade-off, achieve free switching between low distortion and high perceptual quality, and improve the flexibility of image compression.

[0063] In a possible embodiment, S12 may include S21 to S22.

[0064] S21, quantizing the first latent variable to determine the quantized first latent variable.

[0065] S22, performing entropy coding on the quantized first latent variable to determine a second latent variable.

[0066] Among them, the second latent variable Also represented as hidden variables to be unpacked

[0067] In a possible embodiment, S13 may include S31 to S32.

[0068] S31, performing diffusion model sampling processing on the second latent variable to determine the third latent variable.

[0069] Among them, the preset controllable feature fusion module performs a multi-step diffusion model sampling process on the input second latent variable.

[0070] The third hidden variable of the present disclosure Also expressed as a controllable hidden variable

[0071] S32, during the diffusion model sampling process, using a distortion-perception trade-off parameter to weigh the distortion-perception of the second latent variable.

[0072] The distortion-perception tradeoff parameter is represented as τ, and the distortion-perception tradeoff parameter controls the distortion-perception tradeoff during the multi-step diffusion model sampling process in step S31.

[0073] To train a plug-and-play controllable feature fusion module, an auxiliary encoder is first trained for the pre-trained baseline deep learning image codec. The auxiliary encoder is used to generate corresponding guidance information for the controllable feature fusion module during the training phase, and the auxiliary encoder is only used in the training phase of the controllable feature fusion module.

[0074] In a possible embodiment, S31 may also include S301 to S304.

[0075] S301, determining a pre-trained auxiliary encoder.

[0076] A pre-trained baseline deep learning image codec is obtained, and a preset auxiliary encoder is trained based on the pre-trained baseline deep learning image codec.

[0077] The pre-trained auxiliary encoder does not include an entropy model, an arithmetic coding process, and an arithmetic decoding process, and the pre-trained auxiliary encoder is directly connected to the pre-trained baseline decoder of the present invention.

[0078] S302, input the training image into the pre-trained auxiliary encoder, and output the fourth latent variable.

[0079] The fourth latent variable output by the pre-trained auxiliary encoder during training Also referred to as the expected characteristic

[0080] S303, taking the fourth latent variable as the desired feature, and adding noise of different intensities to the second latent variable used for training as the input of the controllable feature fusion module, outputting the third latent variable, training the controllable feature fusion model, and determining the trained controllable feature fusion module.

[0081] The fourth latent variable generated by the pre-trained auxiliary encoder It serves as prior knowledge to guide the training of the controllable feature fusion module.

[0082] The noises of different intensities include noises of multiple intensities t.

[0083] During the training process, the input of the controllable feature fusion module includes a second latent variable with noise of different intensities t added. The second latent variable without adding noise And also includes a pseudo-continuous variable t / T, the pseudo-continuous variable t / T is used to make the controllable feature fusion module perceive the current noise intensity, and add a second hidden variable of noise with different intensities t As the main input, the second latent variable without adding noise The forward calculation process of the controllable feature fusion module is controlled by the controller.

[0084] S304, optimizing the trained controllable feature fusion module by using total optimization loss back propagation to determine a preset controllable feature fusion module.

[0085] Among them, the total optimization loss includes diffusion loss and reconstruction loss. Diffusion loss represents the third latent variable With the fourth hidden variable The reconstruction loss represents the loss between the training image and the reconstructed training image.

[0086] Specifically, the diffusion loss represents the third latent variable With the fourth hidden variable The Euclidean distance between .

[0087] If the pre-trained baseline deep learning image codec is distortion optimized, then the reconstruction loss in the total optimization loss is the perceptual loss between the training image and the reconstructed training image, such as LPIPS; if the pre-trained baseline deep learning image codec is perceptually optimized, then the reconstruction loss in the total optimization loss is the distortion loss between the training image and the reconstructed training image, such as MSE.

[0088] The diffusion loss and reconstruction loss are weighted and summed by the trade-off parameters to obtain the total optimization loss.

[0089] In a possible embodiment, the controllable feature fusion module is trained, and the specific process is as follows:

[0090] The fourth latent variable generated by the pre-trained auxiliary encoder That is, the expected feature is used as the training target of the controllable feature fusion module. During the training process of the controllable feature fusion module, noise of different intensities is added to the second latent variable In the above equation, the diffusion input of the controllable feature fusion module is expressed as:

[0091]

[0092] in,

[0093] β t ∈(0,1) represents a monotonically increasing noise scheduling sequence. During the training of the controllable feature fusion module, t is randomly sampled from [0,T], and a pseudo-continuous variable t / T is used to represent the current noise intensity of the model.

[0094] The total optimization loss function for training the controllable feature fusion module is expressed as:

[0095]

[0096] Among them, g s represents the pre-trained baseline decoder, represents the controllable feature fusion module, λ represents the trade-off parameter;

[0097]

[0098] represents diffusion loss;

[0099]

[0100] represents the reconstruction loss.

[0101] Figure 2 It is a schematic diagram of a reasoning process for determining a preset controllable feature fusion module according to an exemplary embodiment.

[0102] like Figure 2 As shown, in a possible embodiment, in the inference process of image compression, that is, the image compression process, the controllable feature fusion module adopts a diffusion model sampling method, that is, a DDIM sampling method, and replaces the original Markov process with a deterministic generation process to improve the sampling speed. The controllable feature fusion module directly generates decodable features and calculates the predicted noise.

[0103] Among them, in the reasoning process of image compression, It is called the noise feature. The result of the diffusion model sampling process in time step t is:

[0104]

[0105] The predicted noise is:

[0106]

[0107] In summary, the diffusion model sampling process of the preset controllable feature fusion module is expressed as:

[0108]

[0109] The recursive equation of the above diffusion model sampling process is obtained from inference t gradually decreases from T to 0,

[0110] In the training and reasoning process of the preset controllable feature fusion module, using different values ​​for T can reduce the number of diffusion model sampling processing steps required for the reasoning process.

[0111] In practice, different scenarios have different requirements for distortion and perception. In the reasoning process of the preset controllable feature fusion module, the distortion-perception trade-off parameter τ is used to achieve the controllable distortion-perception trade-off, and the following equation is used to guide the diffusion model sampling process:

[0112]

[0113] Among them, when τ=0, the third latent variable output by the preset controllable feature fusion module is completely composed of perception-oriented latent variables.

[0114] When τ = 1, the third latent variable output by the preset controllable feature fusion module Corresponding to the pre-trained baseline deep learning image codec, the last third latent variable is input into the pre-trained baseline decoder to obtain the decoded reconstructed image.

[0115] In a possible embodiment, S301 may include S310 to S313.

[0116] S310, input the training image into a preset auxiliary encoder, and output a fourth latent variable.

[0117] A pre-trained baseline deep learning image codec is obtained, and an auxiliary encoder is trained for the pre-trained baseline deep learning image codec, and the preset auxiliary encoder is trained based on the pre-trained baseline deep learning image codec.

[0118] The pre-trained baseline deep learning image codec obtained in the present disclosure can be optimized for distortion or for perception.

[0119] If the pre-trained baseline deep learning image codec is distortion optimized, the pre-trained auxiliary encoder needs to be optimized for perception.

[0120] If the pre-trained baseline deep learning image codec is perceptually optimized, the pre-trained auxiliary encoder needs to be optimized for distortion.

[0121] Among them, the fourth latent variable is expressed as The fourth hidden variable It is used as the desired feature to perform target-guided optimization on the training process of the controllable feature fusion module.

[0122] S311, input the fourth latent variable into the pre-trained baseline decoder to determine the reconstructed training image.

[0123] S312, optimizing the preset auxiliary encoder according to the loss between the training image and the reconstructed training image, and determining the pre-trained auxiliary encoder.

[0124] Therein, the loss between the training image and the reconstructed training image is used as the optimization target of the preset auxiliary encoder.

[0125] In one possible embodiment, the loss between the training image and the reconstructed training image includes a perceptual loss and a distortion loss.

[0126] As an example, if the pre-trained baseline deep learning image codec is distortion optimized, the loss between the training image and the reconstructed training image uses a perceptual loss such as LPIPS.

[0127] As another example, if the pre-trained baseline deep learning image codec is perceptually optimized, the loss between the training image and the reconstructed training image adopts a distortion loss such as MSE.

[0128] In a possible embodiment, the preset controllable feature fusion module of the present disclosure uses the distortion-perception trade-off parameter τ to control the fusion of the compression feature and the conversion feature, that is, to control the second latent variable and the third latent variable The fusion of , thus obtaining a controllable third latent variable The third latent variable is input into a pre-trained baseline decoder to output a reconstructed image, wherein the reconstructed image has a desired distortion-perception trade-off.

[0129] In a possible embodiment, the output result of the controllable feature fusion module provided by the present disclosure is evaluated, the Kodak dataset is used as the test image dataset, and the three models of HiFiC, Cheng20 and MSHyper are used as baseline models to respectively use the controllable feature fusion module provided by the present disclosure for quantitative evaluation. Among them, HiFiC is a deep learning image codec with advanced perception performance, Cheng20 is a deep learning image codec with advanced compression performance, and MSHyper is a deep learning image codec with advanced compression performance.

[0130] The present disclosure uses a bit rate-index curve to evaluate the baseline model using the preset controllable feature fusion module of the present disclosure, and calculates the BD-rate savings corresponding to each baseline model.

[0131] Figure 3 It is a schematic diagram showing a comparison between a preset controllable feature fusion module and a baseline deep learning image codec bit rate-index curve according to an exemplary embodiment.

[0132] like Figure 3 As shown in the figure, the horizontal axis bpp (bits-per-pixel) represents the number of bits required for each pixel, that is, the bit rate. The vertical axis can be used to represent a variety of indicators, such as: the peak signal-to-noise ratio PSNR between the reconstructed image and the original image, the structural similarity index SSIM, the multi-scale similarity index MS-SSIM, the deep image structure and texture similarity DISTS, the perceptual image block similarity LPIPS, the generated image quality FID, the CLIP-IQA based on CLIP image quality assessment, and the natural image quality NIQE.

[0133] In the bit rate-index curve, if the index represented by the vertical axis is as large as possible, the curve located above the bit rate-index curve can achieve better reconstruction quality at the same bit rate and have better compression performance; if the index represented by the vertical axis is as small as possible, the curve located below the bit rate-index curve can achieve better reconstruction quality at the same bit rate and have better compression performance.

[0134] The BD-rate saving is used to quantitatively indicate the percentage of bit rate that can be saved under the same reconstruction quality. The larger the BD-rate saving is, the better the performance is.

[0135] like Figure 3 As shown, for the baseline codec optimized for distortion, Ours (Cheng2020, τ=0) disclosed in this paper significantly improves LPIPS-BDRate by 158.75% compared with the original Cheng20 method, and the average PSNR attenuation is 1.08dB. In terms of high perception (τ=0), compared with the HiFiC method, the DISTS indicator achieves comparable or better results, and the PSNR is also significantly better than it, with an average increase of 1.48dB.

[0136] For the perceptual-oriented baseline codec, our method Ours (HiFiC, τ = 0) achieves 22.59% bitrate savings on the Kodak dataset compared to the original HiFiC method. For low distortion (τ = 0), our performance matches or outperforms the distortion method Cheng20 in terms of SSIM and MS-SSIM, while also significantly outperforming it in terms of perceptual quality.

[0137] Figure 4 It is a schematic diagram showing a comparison of bit rate-distortion-perception conversion curves of a preset controllable feature fusion module according to an exemplary embodiment.

[0138] As shown in the figure, the impact of different τ values ​​(τ∈{0,0.3,0.5,0.8,1}) at different bit rates is evaluated. Our results Ours(MSHyper) show that the proposed image compression method with controllable distortion-perceptual trade-off performs a controllable and smooth traversal between low distortion (high PSNR) and high perceptual quality (low LPIPS).

[0139] Figure 5 The invention is a block diagram of an image compression system with controllable distortion-perception trade-off according to an exemplary embodiment.

[0140] Based on the same concept, the present disclosure also provides an image compression system 100 with controllable distortion-perception trade-off. Figure 5As shown, it includes: a latent variable conversion module 110, an encoding module 120, a weighing module 130, and a decoding module 140.

[0141] A latent variable conversion module 110 is used to input the image to be processed into a pre-trained baseline encoder to determine a first latent variable;

[0142] An encoding module 120, configured to perform entropy encoding on the first latent variable to determine a second latent variable;

[0143] A weighing module 130, configured to input the second latent variable into a preset controllable feature fusion module to determine a third latent variable;

[0144] The decoding module 140 is used to input the third latent variable into a pre-trained baseline decoder to determine a reconstructed image.

[0145] Through the above technical solution, a preset controllable feature fusion module is added without changing the baseline deep learning image codec parameters and structure. The distortion-perception trade-off in a single latent variable is adjusted through the preset controllable feature fusion module to achieve flexible distortion-perception trade-off adjustment. For the same compressed bitstream, users can obtain different decoded images according to the required rate, achieve free switching between low distortion and high perceptual quality, and improve the flexibility of image compression.

[0146] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0147] Based on the same concept as above, in another embodiment of the present disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute an image compression method with controllable distortion-perception trade-off when executing the program.

[0148] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0149] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0150] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0151] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0152] In an embodiment of the present disclosure, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of an image compression method with controllable distortion-perception trade-off in any of the above embodiments are implemented.

[0153] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0158] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for image compression with controllable distortion-perception tradeoff, characterized in that: include: Input the image to be processed into the pre-trained baseline encoder to determine the first latent variable; Performing entropy coding on the first latent variable to determine a second latent variable; Inputting the second latent variable into a preset controllable feature fusion module to determine a third latent variable; The third latent variable is input into a pre-trained baseline decoder to determine a reconstructed image.

2. The method according to claim 1, characterized in that The step of performing entropy coding on the first latent variable to determine the second latent variable includes: quantifying the first latent variable to determine the quantified first latent variable; The quantized first latent variable is entropy encoded to determine the second latent variable.

3. The method according to claim 1, characterized in that The step of inputting the second latent variable into a preset controllable feature fusion module to determine a third latent variable includes: Performing diffusion model sampling processing on the second latent variable to determine the third latent variable; During the diffusion model sampling process, a distortion-perception trade-off parameter is used to control the distortion-perception trade-off of the third latent variable.

4. The method according to claim 1, characterized in that: Determining the preset controllable feature fusion module includes: Determine the pre-trained auxiliary encoder; Inputting the training image into the pre-trained auxiliary encoder and outputting a fourth latent variable; The fourth latent variable is used as the desired feature, and noises of different intensities are added to the second latent variable used for training as the input of the controllable feature fusion module, and the third latent variable is output, and the controllable feature fusion model is trained to determine the trained controllable feature fusion module; The trained controllable feature fusion module is optimized by using total optimization loss back propagation to determine the preset controllable feature fusion module.

5. The method according to claim 4, characterized in that The total optimization loss includes a diffusion loss and a reconstruction loss, wherein the diffusion loss represents a loss between the third latent variable and the fourth latent variable, and the reconstruction loss represents a loss between a training image and a reconstructed training image.

6. The method according to claim 4, characterized in that The determining of the pre-trained auxiliary encoder comprises: Inputting the training image into a preset auxiliary encoder and outputting a fourth latent variable; Inputting the fourth latent variable into the pre-trained baseline decoder to determine a reconstructed training image; The preset auxiliary encoder is optimized according to the loss between the training image and the reconstructed training image to determine the pre-trained auxiliary encoder.

7. The method according to any one of claims 5 and 6, characterized in that The loss between the training image and the reconstructed training image comprises a perceptual loss and a distortion loss; If the pre-trained baseline deep learning image codec is distortion optimized, the loss between the training image and the reconstructed training image adopts the perceptual loss; If the pre-trained baseline deep learning image codec is perceptually optimized, the loss between the training image and the reconstructed training image adopts a distortion loss.

8. An image compression system with controllable distortion-perception tradeoff, characterized in that: include: A latent variable conversion module, used for inputting the image to be processed into a pre-trained baseline encoder to determine a first latent variable; An encoding module, used for performing entropy encoding on the first latent variable to determine a second latent variable; A weighing module, used for inputting the second latent variable into a preset controllable feature fusion module to determine a third latent variable; The decoding module is used to input the third latent variable into a pre-trained baseline decoder to determine a reconstructed image.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images