Image compression method, system, medium, and apparatus with controllable distortion-perception tradeoff
By introducing a controllable feature fusion module into the image codec, the problem of difficult adjustment of the distortion-perception trade-off in the existing technology is solved, flexible switching between low distortion and high perceptual quality is achieved, and the flexibility of image compression is improved.
Patent Information
- Application Number
- CN202411323995.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2044-09-23
AI Technical Summary
Existing image codecs struggle to achieve both high pixel-level fidelity and image-level realism at low bitrates, resulting in blur and visual quality degradation, and are unable to flexibly adjust the distortion-perception trade-off.
By introducing a controllable feature fusion module into the pre-trained baseline deep learning image codec, adjusting the distortion-perception trade-off of a single latent variable, and adding a preset controllable feature fusion module to flexibly adjust the distortion-perception trade-off, free switching between different distortion-perceptions can be achieved.
Under the same compression bitstream, users can choose different distortion-perception trade-offs as needed, freely switch between low distortion and high perceptual quality, and improve the flexibility of image compression.
Smart Images
Figure CN119211548B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of image processing, in particular to a distortion-perception trade-off controllable image compression method, system, medium and device. BACKGROUND
[0002] With digital visual data continuing to dominate internet traffic, the development of efficient image and video codecs becomes increasingly important. In recent years, deep learning-based codecs have made significant progress in both image and video domains. Compared with traditional codecs such as JPEG, JPEG2000 and BPG, they show superior compression performance.
[0003] Current learning-based image codecs mainly rely on the transform coding paradigm and variational autoencoder (VAE) architecture. Most of these models use rate-distortion loss as the optimization target, which directly optimizes for low distortion performance. However, distortion-oriented codecs usually exhibit mode-average behavior at low bitrates, resulting in blurring, which can significantly reduce the visual quality of human observers.
[0004] Recent studies have shown that optimizing perceptual quality can achieve greater compression gains through imperceptible distortion, thereby reducing bitrates. Some works propose to use generative adversarial networks (GANs) and diffusion-based decoders to optimize neural image codecs. However, these schemes are difficult to achieve high-pixel-level fidelity because they introduce high-frequency noise and unrealistic textures. Studies have shown that in a given architecture, the two objectives of distortion and perception cannot be fully achieved simultaneously. But pixel-level fidelity and image-level realism are both crucial for image compression frameworks. The inability to achieve both indicators simultaneously is a major limitation of current image codecs. Given this limitation, an ideal codec should be able to flexibly traverse between different distortion-perception trade-offs at a given bitrate. SUMMARY
[0005] In view of the defects in the prior art, the purpose of the present disclosure is to provide a distortion-perception trade-off controllable image compression method, system, medium and device.
[0006] To achieve the above-mentioned purpose, according to a first aspect of the present disclosure, a distortion-perception trade-off controllable image compression method is provided, comprising:
[0007] inputting a to-be-processed image into a pre-trained baseline encoder to determine a first latent variable;
[0008] entropy encoding the first latent variable to determine a second latent variable;
[0009] inputting the second latent variable into a pre-set controllable feature fusion module to determine a third latent variable;
[0010] inputting the third latent variable into a pre-trained baseline decoder to determine a reconstructed image.
[0011] Optionally, the entropy encoding of the first latent variable to determine a second latent variable comprises:
[0012] quantizing the first latent variable to determine a quantized first latent variable;
[0013] entropy encoding the quantized first latent variable to determine the second latent variable.
[0014] Optionally, the inputting of the second latent variable into a pre-set controllable feature fusion module to determine a third latent variable comprises:
[0015] diffusion model sampling processing the second latent variable to determine the third latent variable;
[0016] In the diffusion model sampling processing, a distortion-perception trade-off parameter is used to trade off the distortion-perception of the second latent variable.
[0017] Optionally, the pre-set controllable feature fusion module is determined by:
[0018] determining a pre-trained auxiliary encoder;
[0019] inputting a training image into the pre-trained auxiliary encoder to output a fourth latent variable;
[0020] using the fourth latent variable as an expected feature and adding noise of different intensities on the training second latent variable as an input quantity of the controllable feature fusion module to output a third latent variable, training the controllable feature fusion model, and determining a trained controllable feature fusion module;
[0021] optimizing the trained controllable feature fusion module by using total optimization loss back propagation to determine the pre-set controllable feature fusion module.
[0022] Optionally, the total optimization loss comprises a diffusion loss and a reconstruction loss, the diffusion loss represents a loss between the third latent variable and the fourth latent variable, and the reconstruction loss represents a loss between the training image and the reconstructed training image.
[0023] Optionally, the pre-trained auxiliary encoder is determined by:
[0024] inputting the training image into a pre-set auxiliary encoder to output a fourth latent variable;
[0025] inputting the fourth hidden variable into the pre-trained baseline decoder to determine a reconstructed training image;
[0026] optimizing the preset auxiliary encoder according to a loss between the training image and the reconstructed training image to determine the pre-trained auxiliary encoder.
[0027] Optionally, the loss between the training image and the reconstructed training image includes a perceptual loss and a distortion loss.
[0028] If the pre-trained baseline deep learning image codec is distortion-optimized, the loss between the training image and the reconstructed training image adopts the perceptual loss.
[0029] If the pre-trained baseline deep learning image codec is perceptual-optimized, the loss between the training image and the reconstructed training image adopts the distortion loss.
[0030] According to a second aspect of the present disclosure, an image compression system with controllable distortion-perceptual tradeoff is provided, comprising:
[0031] a hidden variable conversion module configured to input a to-be-processed image into a pre-trained baseline encoder to determine a first hidden variable;
[0032] an encoding module configured to entropy-encode the first hidden variable to determine a second hidden variable;
[0033] a tradeoff module configured to input the second hidden variable into a preset controllable feature fusion module to determine a third hidden variable;
[0034] a decoding module configured to input the third hidden variable into a pre-trained baseline decoder to determine a reconstructed image.
[0035] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the steps of the method provided in the first aspect of the present disclosure.
[0036] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:
[0037] a memory storing a computer program;
[0038] a processor configured to execute the computer program in the memory to implement the steps of the method provided in the first aspect of the present disclosure.
[0039] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:
[0040] By the technical solution, the preset controllable feature fusion module is added, without changing the parameters and structure of the pre-trained baseline deep learning image encoder-decoder, the distortion-perception trade-off in the single latent variable is adjusted through the preset controllable feature fusion module, flexible distortion-perception trade-off adjustment is realized, for the same compression bitstream, different decoded images can be obtained according to the required distortion-perception trade-off, free switching between low distortion and high perceptual quality is realized, and the flexibility of image compression is improved. BRIEF DESCRIPTION OF DRAWINGS
[0041] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:
[0042] Figure 1 is a flowchart of a distortion-perception trade-off controllable image compression method according to an example embodiment.
[0043] Figure 2 is a schematic diagram of an inference process of determining a preset controllable feature fusion module according to an example embodiment.
[0044] Figure 3 is a schematic diagram of a preset controllable feature fusion module and baseline deep learning image encoder-decoder rate-index curve comparison according to an example embodiment.
[0045] Figure 4 is a schematic diagram of a preset controllable feature fusion module rate-distortion-perception conversion curve comparison according to an example embodiment.
[0046] Figure 5 is a block diagram of a distortion-perception trade-off controllable image compression system according to an example embodiment. DETAILED DESCRIPTION
[0047] The present disclosure will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any form. It should be noted that, for those skilled in the art, without departing from the concept of the present disclosure, a number of modifications and improvements can be made. These all belong to the protection scope of the present disclosure.
[0048] Figure 1 is a flowchart of a distortion-perception trade-off controllable image compression method according to an example embodiment.
[0049] As Figure 1 shown, the present disclosure provides a distortion-perception trade-off controllable image compression method, comprising S11 to S14.
[0050] S11, input the to-be-processed image into a pre-trained baseline encoder to determine a first hidden variable.
[0051] The to-be-processed image is represented as x, and the first hidden variable is represented as y.
[0052] The pre-trained baseline encoder adopts a self-encoder to perform nonlinear transformation on the input to-be-processed image x to convert the to-be-processed image x into the first hidden variable y.
[0053] S12, entropy encoding the first hidden variable to determine a second hidden variable.
[0054] The second hidden variable is represented as
[0055] S13, inputting the second hidden variable into a pre-set controllable feature fusion module to determine a third hidden variable.
[0056] The third hidden variable is represented as
[0057] The pre-set controllable feature fusion module of the present disclosure is a module additionally added to the decoding end, and the pre-set controllable feature fusion module is a detachable, plug-and-play module. The pre-set controllable feature fusion module is directly connected with the pre-trained baseline decoder of the decoding end, without the need to change the architecture of the pre-trained baseline encoder and the pre-trained baseline decoder, and without the need to retrain the pre-trained baseline encoder and the pre-trained baseline decoder.
[0058] S14, inputting the third hidden variable into the pre-trained baseline decoder to determine a reconstructed image.
[0059] The reconstructed image of the to-be-processed image is represented as
[0060] In the present disclosure, the pre-trained baseline encoder and the pre-trained baseline decoder are collectively referred to as a pre-trained baseline deep learning image encoder-decoder.
[0061] The pre-trained baseline encoder and the pre-trained baseline decoder of the present disclosure are basic neural network encoders and decoders optimized for distortion / perception trained by using a code rate constraint and distortion / perception loss between a reconstructed image and an original image as a loss function, i.e., a pre-trained baseline deep learning image encoder-decoder. After the training is completed, the parameters of the pre-trained baseline deep learning image encoder-decoder remain unchanged in the subsequent use process. In addition, the pre-trained baseline encoder and the pre-trained baseline decoder adopted in the present disclosure are not limited, and any end-to-end deep learning image encoder-decoder can be adopted.
[0062] By the technical scheme, the preset controllable feature fusion module is added, without changing the pre-trained baseline deep learning image codec parameters and structure, the distortion-perception trade-off in the single latent variable is adjusted through the preset controllable feature fusion module, flexible distortion-perception trade-off adjustment is realized, for the same compression code stream, different decoding images can be obtained according to the required distortion-perception trade-off, free switching between low distortion and high perceptual quality is realized, and the flexibility of image compression is improved.
[0063] In a possible embodiment, S12 can include S21-S22.
[0064] S21, quantizing the first latent variable to determine a quantized first latent variable.
[0065] S22, entropy encoding the quantized first latent variable to determine a second latent variable.
[0066] The second latent variable is also denoted as a latent variable to be decompressed. The second latent variable is also denoted as a latent variable to be decompressed.
[0067] In a possible embodiment, S13 can include S31-S32.
[0068] S31, performing diffusion model sampling processing on the second latent variable to determine a third latent variable.
[0069] The preset controllable feature fusion module performs a multi-step diffusion model sampling processing process on the input second latent variable.
[0070] The third latent variable of the present disclosure The third latent variable of the present disclosure
[0071] S32, in the diffusion model sampling processing process, a distortion-perception trade-off parameter is used to trade off the distortion-perception of the second latent variable.
[0072] The distortion-perception trade-off parameter is denoted as τ, and the distortion-perception trade-off parameter controls the distortion-perception trade-off in the multi-step diffusion model sampling processing process in step S31.
[0073] To train the plug-and-play controllable feature fusion module, first, an auxiliary encoder is trained for the pre-trained baseline deep learning image codec, the auxiliary encoder is used to generate corresponding guide information for the controllable feature fusion module in the training stage, and the auxiliary encoder is only used in the training stage of the controllable feature fusion module.
[0074] In a possible embodiment, S31 can further include S301-S304.
[0075] S301, determine a pre-trained auxiliary encoder.
[0076] A pre-trained baseline deep learning image codec is obtained, and a pre-set auxiliary encoder is trained based on the pre-trained baseline deep learning image codec.
[0077] The pre-trained auxiliary encoder does not contain an entropy model, an arithmetic encoding process, and an arithmetic decoding process, and the pre-trained auxiliary encoder is directly connected with the pre-trained baseline decoder of the present disclosure.
[0078] S302, input the training image into the pre-trained auxiliary encoder, and output the fourth hidden variable.
[0079] The fourth hidden variable output by the pre-trained auxiliary encoder in the training process Also denoted as expected feature
[0080] S303, taking the fourth hidden variable as the expected feature, adding noise of different intensities to the second hidden variable as the input of the controllable feature fusion module, outputting the third hidden variable, training the controllable feature fusion model, and determining the trained controllable feature fusion module.
[0081] The fourth hidden variable generated by the pre-trained auxiliary encoder As prior knowledge, the controllable feature fusion module is provided with guidance for training.
[0082] Wherein, the noise of different intensities includes noise of multiple intensities t.
[0083] In the training process, the input of the controllable feature fusion module includes the second hidden variable to which noise of different intensities t is added The second hidden variable without adding noise And also includes a pseudo-continuous variable t / T, which is used to make the controllable feature fusion module perceive the intensity of the current noise, and the second hidden variable to which noise of different intensities t is added As the main input, the second hidden variable without adding noise The forward calculation process of the controllable feature fusion module is controlled by a controller.
[0084] S304, the trained controllable feature fusion module is optimized by using total optimization loss back propagation, and a pre-set controllable feature fusion module is determined.
[0085] Wherein, the total optimization loss includes a diffusion loss and a reconstruction loss, the diffusion loss represents the loss between the third hidden variable And the fourth hidden variable The reconstruction loss represents the loss between the training image and the reconstructed training image.
[0086] Specifically, the diffusion loss represents the Euclidean distance between the third hidden variable and the fourth hidden variable .
[0087] If the pre-trained baseline deep learning image codec is distortion-optimized, the reconstruction loss in the total optimization loss is the perceptual loss, such as LPIPS, between the training image and the reconstructed training image; if the pre-trained baseline deep learning image codec is perceptual-optimized, the reconstruction loss in the total optimization loss is the distortion loss, such as MSE, between the training image and the reconstructed training image.
[0088] The diffusion loss and the reconstruction loss are weighted and summed by a trade-off parameter to obtain the total optimization loss.
[0089] In one possible embodiment, the controllable feature fusion module is trained, and the specific process is as follows:
[0090] The fourth hidden variable generated by the pre-trained auxiliary encoder , that is, the expected feature, is taken as the training target of the controllable feature fusion module, and different intensities of noise are added to the second hidden variable during the training process of the controllable feature fusion module, and the diffusion input of the controllable feature fusion module is represented as:
[0091]
[0092] wherein,
[0093] β t represents a monotonically increasing noise scheduling sequence, t is randomly sampled from [0, T] during the training of the controllable feature fusion module, and a pseudo-continuous variable t / T is used to represent the current noise intensity of the model.
[0094] The total optimization loss function of the training of the controllable feature fusion module is represented as:
[0095]
[0096] wherein, g s represents the pre-trained baseline decoder, represents the controllable feature fusion module, and λ represents a trade-off parameter.
[0097]
[0098] represents the diffusion loss.
[0099]
[0100] represents the reconstruction loss.
[0101] Figure 2 The figure is a schematic diagram of a reasoning process for determining a preset controllable feature fusion module according to an exemplary embodiment.
[0102] like Figure 2 As shown, in a possible embodiment, in the inference process of image compression, that is, the image compression process, the controllable feature fusion module adopts a diffusion model sampling method, that is, a DDIM sampling method, and replaces the original Markov process with a deterministic generation process to improve the sampling speed. The controllable feature fusion module directly generates decodable features and calculates the predicted noise.
[0103] Among them, in the reasoning process of image compression, It is called the noise feature, and the result of the diffusion model sampling processing in time step t is:
[0104]
[0105] The predicted noise is:
[0106]
[0107] In summary, the diffusion model sampling process of the preset controllable feature fusion module is expressed as:
[0108]
[0109] The recursive equation of the above diffusion model sampling process is obtained from inference t gradually decreases from T to 0,
[0110] In the training and inference process of the preset controllable feature fusion module, using different values of T can reduce the number of diffusion model sampling processing steps required for the inference process.
[0111] In practice, different scenarios have different requirements for distortion and perception. In the inference process of the preset controllable feature fusion module, the distortion-perception trade-off parameter τ is used to achieve a controllable distortion-perception trade-off. The following equation is used to guide the diffusion model sampling process:
[0112]
[0113] Among them, when τ=0, the third latent variable output by the preset controllable feature fusion module is completely composed of perception-oriented latent variables.
[0114] When τ=1, the third latent variable output by the preset controllable feature fusion module is Corresponding to the pre-trained baseline deep learning image codec, the last third hidden variable is input into the pre-trained baseline decoder to obtain a decoded reconstructed image.
[0115] In a possible embodiment, S301 can include S310-S313.
[0116] S310, input the training image into the preset auxiliary encoder, and output a fourth hidden variable.
[0117] obtaining a pre-trained baseline deep learning image codec, training an auxiliary encoder for the pre-trained baseline deep learning image codec, and training the preset auxiliary encoder based on the pre-trained baseline deep learning image codec.
[0118] The pre-trained baseline deep learning image codec obtained by the present disclosure can be optimized for distortion or for perception.
[0119] If the pre-trained baseline deep learning image codec is optimized for distortion, the preset auxiliary encoder needs to be optimized for perception.
[0120] If the pre-trained baseline deep learning image codec is optimized for perception, the preset auxiliary encoder needs to be optimized for distortion.
[0121] The fourth hidden variable is represented as The fourth hidden variable for target guidance optimization of the training process of the controllable feature fusion module as the expected feature.
[0122] S311, input the fourth hidden variable into the pre-trained baseline decoder, and determine a reconstructed training image.
[0123] S312, according to the loss between the training image and the reconstructed training image, optimize the preset auxiliary encoder to determine the pre-trained auxiliary encoder.
[0124] The loss between the training image and the reconstructed training image is taken as the optimization target of the preset auxiliary encoder.
[0125] In a possible embodiment, the loss between the training image and the reconstructed training image includes a perception loss and a distortion loss.
[0126] As an example, if the pre-trained baseline deep learning image codec is optimized for distortion, the loss between the training image and the reconstructed training image adopts a perception loss, such as LPIPS.
[0127] As another example, if the pre-trained baseline deep learning image codec is perceptually optimized, the loss between the training image and the reconstructed training image employs a distortion loss, such as MSE.
[0128] In one possible embodiment, the preset controllable feature fusion module of the present disclosure employs a distortion-perception trade-off parameter τ to control the fusion of the compressed feature and the converted feature, i.e., to control the fusion of the second latent variable and the third latent variable , so as to obtain a controllable third latent variable and input the third latent variable into the pre-trained baseline decoder to output a reconstructed image, wherein the reconstructed image has a desired distortion-perception trade-off.
[0129] In one possible embodiment, the output result of the controllable feature fusion module provided by the present disclosure is evaluated, and the Kodak dataset is employed as the test picture dataset, and the HiFiC, Cheng20 and MSHyper three models are employed as the baseline models to perform quantitative evaluation by employing the controllable feature fusion module provided by the present disclosure. Among them, the HiFiC is a deep learning image codec with advanced perceptual performance, the Cheng20 is a deep learning image codec with advanced compression performance, and the MSHyper is a deep learning image codec with advanced compression performance.
[0130] The preset controllable feature fusion module of the present disclosure employs a rate-distortion curve to evaluate the baseline model employing the preset controllable feature fusion module, and calculates the BD-rate saving amount corresponding to each baseline model.
[0131] Figure 3 is a preset controllable feature fusion module and baseline deep learning image codec rate-distortion curve comparison diagram according to an exemplary embodiment.
[0132] As shown in Figure 3 , the horizontal coordinate bpp (bits-per-pixel) represents the number of bits required for each pixel, i.e., the code rate. The vertical coordinate can be used to represent a variety of indicators, such as: the peak signal-to-noise ratio PSNR of the reconstructed image and the original image, the structural similarity index SSIM, the multi-scale similarity index MS-SSIM, the depth image structure and texture similarity DISTS, the perceptual image block similarity LPIPS, the generated image quality FID, the CLIP-based image quality evaluation CLIP-IQA, and the natural image quality NIQE.
[0133] In the rate-distortion curve, if the index represented by the ordinate is the larger the better, then the curve located in the upper part of the rate-distortion curve can achieve better reconstruction quality at the same code rate, and has better compression performance; if the index represented by the ordinate is the smaller the better, then the curve located in the lower part of the rate-distortion curve can achieve better reconstruction quality at the same code rate, and has better compression performance.
[0134] The BD-rate saving is used to quantitatively indicate the percentage of code rate that can be saved under the same reconstruction quality, and the larger the BD-rate saving is, the better the performance is.
[0135] As shown in Figure 3 For the distortion-oriented baseline codec, compared with the original Cheng20 method, the LPIPS-BDRate of Ours (Cheng2020, τ = 0) of the present disclosure is increased by 158.75%, and the average PSNR attenuation is 1.08dB. In the high perception aspect (τ = 0), compared with the HiFiC method, the DISTS index achieves the same or better effect, and the PSNR is also significantly better than it, with an average of 1.48dB higher.
[0136] For the perception-oriented baseline codec, compared with the original HiFiC method, the bit rate of our method Ours (HiFiC, τ = 0) is saved by 22.59% on the Kodak dataset. In the low distortion aspect (τ = 0), our performance in SSIM and MS-SSIM matches or is better than the distortion method Cheng20, and is also significantly better than it in the aspect of perceptual quality.
[0137] Figure 4 is a rate-distortion-perception conversion curve comparison diagram of a preset controllable feature fusion module according to an example embodiment.
[0138] As shown in the figure, the influence of different τ values (τ ∈ {0, 0.3, 0.5, 0.8, 1}) under different bit rates is evaluated. The result of our Ours (MSHyper) shows that the proposed distortion-perception trade-off controllable image compression method performs controllable and smooth traversal between low distortion (high PSNR) and high perceptual quality (low LPIPS).
[0139] Figure 5 is a block diagram of a distortion-perception trade-off controllable image compression system according to an example embodiment.
[0140] Based on the same concept, the present disclosure also provides a distortion-perception trade-off controllable image compression system 100, as shown in Figure 5As shown, the system comprises: a latent variable conversion module 110, an encoding module 120, a trade-off module 130, and a decoding module 140.
[0141] The latent variable conversion module 110 is configured to input an image to be processed into a pre-trained baseline encoder to determine a first latent variable.
[0142] The encoding module 120 is configured to perform entropy encoding on the first latent variable to determine a second latent variable.
[0143] The trade-off module 130 is configured to input the second latent variable into a pre-set controllable feature fusion module to determine a third latent variable.
[0144] The decoding module 140 is configured to input the third latent variable into a pre-trained baseline decoder to determine a reconstructed image.
[0145] By the above technical solution, the pre-set controllable feature fusion module is added, without changing the parameters and structure of the baseline deep learning image encoding and decoding module, the distortion-perception trade-off in the single latent variable is adjusted through the pre-set controllable feature fusion module, flexible distortion-perception trade-off adjustment is realized, for the same compressed bitstream, a user can obtain different decoded images according to the required rate, free switching between low distortion and high perceptual quality is realized, and the flexibility of image compression is improved.
[0146] As to the embodiments of the above system, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.
[0147] Based on the same concept, in another embodiment of the present disclosure, an electronic device is also provided, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is configured to execute the distortion-perception trade-off controllable image compression method when executing the program.
[0148] Optionally, the memory is configured to store a program; the memory can include volatile memory (e.g., random access memory (RAM), such as static random access memory (SRAM), Double Data Rate synchronous dynamic random access memory (DDR SDRAM), and the like) or non-volatile memory (e.g., flash memory). The memory is configured to store computer programs (e.g., application programs, functional modules, and the like for implementing the above-described methods), computer instructions, and the like. The computer programs, computer instructions, and the like described above can be stored in one or more memories in a partitioned manner. Moreover, the computer programs, computer instructions, and the like described above can be invoked by the processor.
[0149] The computer programs, computer instructions, and the like described above can be stored in one or more memories in a partitioned manner. Moreover, the computer programs, computer instructions, and the like described above can be invoked by the processor.
[0150] The processor is configured to execute the computer programs stored in the memory, so as to implement each step in the methods involved in the above-described embodiments. Details can be referred to the related descriptions in the method embodiments.
[0151] The processor and the memory can be independent structures, or can be integrated into an integrated structure. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.
[0152] In the embodiments of the present disclosure, a non-transitory computer-readable storage medium is also provided, which stores a computer program. The computer program is executed by a processor to implement the steps of the distortion-perception trade-off controllable image compression method in any of the above-described embodiments.
[0153] Those skilled in the art will appreciate that embodiments of the present disclosure can be provided as methods, systems, or computer program products. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, and the like) containing computer-usable program code.
[0154] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the one or more flowcharts and / or blocks
[0155] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the one or more flowcharts and / or blocks
[0156] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the one or more flowcharts and / or blocks
[0157] While the preferred embodiments of the present disclosure have been described, additional variations and modifications can be employed by those skilled in the art. Therefore, the claimed disclosure is intended to cover all modifications and variations of the preferred embodiments of the present disclosure. It is intended that the scope of the disclosure should not be limited by the particular disclosed embodiments.
[0158] Obviously, numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the disclosure, the disclosure can be practiced otherwise than as specifically described. Thus, the disclosure includes all modifications and variations from the example constructions and methods already described and such as can naturally occur to those skilled in the art.
Claims
1. An image compression method with controllable distortion-perception trade-off, characterized in that: include: Input the image to be processed into the pre-trained baseline encoder to determine the first latent variable; Performing entropy coding on the first latent variable to determine a second latent variable; Inputting the second latent variable into a preset controllable feature fusion module to determine a third latent variable; Inputting the third latent variable into a pre-trained baseline decoder to determine a reconstructed image; The step of inputting the second latent variable into a preset controllable feature fusion module to determine the third latent variable includes: Performing diffusion model sampling processing on the second latent variable to determine the third latent variable; During the diffusion model sampling process, a distortion-perception trade-off parameter is used to control the distortion-perception trade-off of the third latent variable; Determining the preset controllable feature fusion module includes: Determine the pre-trained auxiliary encoder; Inputting the training image into the pre-trained auxiliary encoder and outputting a fourth latent variable; The fourth latent variable is used as the desired feature, and noise of different intensities is added to the second latent variable used for training as the input of the controllable feature fusion module, the third latent variable is output, the controllable feature fusion model is trained, and a trained controllable feature fusion module is determined; The trained controllable feature fusion module is optimized by using total optimization loss back propagation to determine the preset controllable feature fusion module; The loss between the training image and the reconstructed training image includes a perceptual loss and a distortion loss; If the pre-trained baseline deep learning image codec is distortion optimized, the loss between the training image and the reconstructed training image adopts the perceptual loss; If the pre-trained baseline deep learning image codec is perceptually optimized, the loss between the training image and the reconstructed training image adopts a distortion loss.
2. The method according to claim 1, characterized in that The entropy encoding of the first latent variable to determine the second latent variable includes: quantizing the first latent variable to determine the quantified first latent variable; Perform entropy coding on the quantized first latent variable to determine the second latent variable.
3. The method according to claim 1, characterized in that The total optimization loss includes a diffusion loss and a reconstruction loss, wherein the diffusion loss represents a loss between the third latent variable and the fourth latent variable, and the reconstruction loss represents a loss between the training image and the reconstructed training image.
4. The method according to claim 1, wherein The determining of the pre-trained auxiliary encoder includes: Inputting the training image into a preset auxiliary encoder and outputting a fourth latent variable; Inputting the fourth latent variable into the pre-trained baseline decoder to determine a reconstructed training image; The preset auxiliary encoder is optimized according to the loss between the training image and the reconstructed training image to determine the pre-trained auxiliary encoder.
5. An image compression system with controllable distortion-perception tradeoff, characterized in that include: A latent variable conversion module is used to input the image to be processed into a pre-trained baseline encoder to determine a first latent variable; an encoding module, configured to perform entropy encoding on the first latent variable to determine a second latent variable; a weighing module, configured to input the second latent variable into a preset controllable feature fusion module to determine a third latent variable; A decoding module, configured to input the third latent variable into a pre-trained baseline decoder to determine a reconstructed image; Wherein, the weighing module includes: a diffusion model sampling processing submodule, configured to perform diffusion model sampling processing on the second latent variable to determine the third latent variable; a trade-off control submodule, configured to use a distortion-perception trade-off parameter to perform a distortion-perception trade-off control on the third latent variable during the diffusion model sampling process; The image compression system with controllable distortion-perception tradeoff further includes: The controllable feature fusion module training submodule is used to: determine a pre-trained auxiliary encoder; input a training image into the pre-trained auxiliary encoder and output a fourth latent variable; use the fourth latent variable as the desired feature, add noise of different intensities to the second latent variable used for training as the input of the controllable feature fusion module, output a third latent variable, train the controllable feature fusion model, and determine a trained controllable feature fusion module; optimize the trained controllable feature fusion module using total optimization loss back propagation to determine the preset controllable feature fusion module; The loss between the training image and the reconstructed training image includes a perceptual loss and a distortion loss; If the pre-trained baseline deep learning image codec is distortion optimized, the loss between the training image and the reconstructed training image adopts the perceptual loss; If the pre-trained baseline deep learning image codec is perceptually optimized, the loss between the training image and the reconstructed training image adopts a distortion loss.
6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
7. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 4.