A low bitrate image compression method, system, terminal, and storage medium based on a diffusion model for human perception.

Abstract

This invention discloses a low-bitrate image compression method, system, terminal, and storage medium based on a diffusion model, geared towards human perception. The method includes: inputting a source image into a low-bitrate image compression framework and compressing it into dense features using a compression model; compressing the dense features into a bitstream and transmitting the bitstream to a receiver; decompressing the bitstream into dense features and using a lightweight decompressor to decompress the dense features into low-frequency image features; injecting text and high-frequency image information into the low-frequency image features using a diffusion model to obtain target image information; performing diffusion and denoising processing on the target image information to obtain target image features; and converting the target image features from the feature domain to the image domain to obtain the target image. This invention applies a weighted loss to the encoder, achieving better bitrate allocation, reducing the amount of data required for image storage and transmission, and enhancing the visual quality of the reconstructed image by enhancing the high-frequency information of the decoded features.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a low bit-rate image compression method, system, terminal, and computer-readable storage medium based on a diffusion model for human perception. Background Technology

[0002] Low bitrate image compression is a data compression technique used to reduce the amount of data, or bits, required to represent an image while ensuring that the image can be recognized and used normally.

[0003] Because a significant amount of information is lost at low bitrates, generative models are typically used to fill in the missing information and restore and reconstruct images at a level perceptible to humans. A generative model is a model that learns the probability distribution of data and generates new data samples based on the learned distribution. Currently, there are two main types: Generative Adversarial Models (GANs) and Diffusion Models.

[0004] Currently, the HiFiC framework has been proposed, using GANs to compensate for lost information and optimizing the reconstructed image for human vision, achieving excellent results. The ILLM framework has also been proposed, further improving the performance of GANs in low bitrate reconstruction. However, all these frameworks use GANs as the base model for information compensation, which can lead to artifacts in the reconstructed image. Furthermore, the insufficient generative performance of these models limits the exploration of image compression to even lower bitrates.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a low bitrate image compression method, system, terminal, and computer-readable storage medium based on a diffusion model for human perception. This invention aims to solve the problems in existing low bitrate image compression frameworks, such as image artifacts during reconstruction and insufficient model generation performance, which limit the exploration of image compression to even lower bitrates.

[0007] To achieve the above objectives, this invention provides a low bitrate image compression method based on a diffusion model oriented towards human perception. The low bitrate image compression method based on a diffusion model oriented towards human perception includes the following steps:

[0008] The source image from the sender is input into the low bitrate image compression framework, and the source image is compressed into dense features by the compression model in the low bitrate image compression framework.

[0009] The dense features are compressed into a bit stream using a lossless encoding method, and the bit stream is transmitted to the receiver through a channel.

[0010] The bitstream is decompressed into dense features using lossless decoding, and the dense features are then decompressed into low-frequency image features using a lightweight decompressor.

[0011] The target image information is obtained by injecting text and high-frequency image information into the low-frequency image features through the diffusion model in the low bit-rate image compression framework.

[0012] The target image information is subjected to diffusion and denoising processing to obtain target image features. The target image features are then transformed from the feature domain to the image domain to obtain the target image.

[0013] Optionally, the aforementioned low bitrate image compression method based on a diffusion model for human perception further includes, before inputting the sender's source image into the low bitrate image compression framework:

[0014] A low bitrate image compression framework is constructed, which includes a compression model and a diffusion model, and the low bitrate image compression framework is trained using an end-to-end training method.

[0015] Optionally, the low bitrate image compression method based on a diffusion model for human perception includes a first-stage training and a second-stage training.

[0016] The first stage of training is used to train some parameters of the encoder of the compression model, the conditional controller of the diffusion model, the time-aware decoupling module, and the decoder of the diffusion model in the low bit-rate image compression framework.

[0017] The second stage of training is used to freeze the compression model in the low bitrate image compression framework and train only some parameters of the conditional controller, time-aware decoupling module and decoder of the diffusion model.

[0018] Optionally, the aforementioned low bitrate image compression method based on a diffusion model for human perception, wherein the first stage of training includes:

[0019] The training image x is obtained. After the training image x passes through the encoder ε of the diffusion model, intermediate features z0 and multi-scale features f1 and f2 are obtained. The intermediate features z0 and multi-scale features f1 and f2 are then passed through the multi-scale compressor of the compression model to obtain the latent features of the diffusion model. latent features Transmitted to the receiver;

[0020] The latent features are processed through a time-aware decoupling module and a control module ICCN. The sample is fed into the diffusion model, and z is obtained by Gaussian sampling through the diffusion model. tAfter passing through the diffusion model, the first output z is obtained. t-1 After t iterations, the final output of the diffusion model is obtained.

[0021] Final output Input to the decoder of the diffusion model In the process, the reconstructed image is obtained.

[0022] Optionally, in the aforementioned low bitrate image compression method based on a diffusion model for human perception, the variance-weighted mean square error based on variance is expressed as follows during the first stage of training:

[0023]

[0024] in, This represents the mean of z0. express The mean of L, where w represents a trainable hyperparameter. imp L represents the weighted variance loss function. mse This represents the mean squared error loss function;

[0025] Using a time-aware decoupling module, the control factor is extracted from the diffusion model, and the noise prediction is as follows:

[0026]

[0027] in, This represents the noise in the prediction. This represents the original diffusion model denoising network. The parameters of the diffusion model are represented by ∈, where ∈ represents Gaussian noise, t represents the time step, and TAD represents the time step. η This indicates a time-aware decoupling module;

[0028] In the first phase of training, the training loss function is:

[0029]

[0030] in, λ1 represents the total loss of the first stage of training, λ2 represents the control bit rate, and λ3 represents the hyperparameters of the distortion. This represents the compression ratio loss function. This represents the loss in the diffusion model. Represents the random variable z0, The mean of ,t,∈.

[0031] Optionally, in the aforementioned low bitrate image compression method based on a diffusion model for human perception, the second stage of training includes:

[0032] The training image x is obtained. After the training image x passes through the encoder ε of the diffusion model, intermediate features z0 and multi-scale features f1 and f2 are obtained. The intermediate features z0 and multi-scale features f1 and f2 are then passed through the multi-scale compressor of the compression model to obtain the latent features of the diffusion model. latent features Transmitted to the receiver;

[0033] The latent features are processed through a time-aware decoupling module and a control module ICCN. The sample is fed into the diffusion model, and z is obtained by Gaussian sampling through the diffusion model. t After passing through the diffusion model, the first output z is obtained. t-1 ,

[0034] latent features Input to the decoder of the diffusion model Obtain visual information Visual information After passing through the encoder, the text information is input into the query device to perform image annotation on the training image x, obtain text information, and input the text information into the query device.

[0035] Introducing textual information as semantic embedding, and using latent features Visual information The semantic information from the text is integrated through the queryer to obtain a hybrid semantic output s. x Mixed semantic outputs s x The semantics are modulated through a projection layer and guided by the diffusion model.

[0036] According to z t-1 and mixed semantic outputs x After t iterations, the final output of the diffusion model is obtained. Final output Input to the decoder of the diffusion model In the process, the reconstructed image is obtained.

[0037] Optionally, in the aforementioned low bitrate image compression method based on a diffusion model for human perception, the training loss function in the second stage of training is:

[0038]

[0039] in, This indicates the total loss during the second phase of training. Represents the random variable z0, s x The mean of ,t,∈, where∈ represents Gaussian noise andt represents the time step. This represents the original diffusion model denoising network. TAD represents the parameters of the diffusion model. η This indicates a time-aware decoupling module.

[0040] Furthermore, to achieve the above objectives, the present invention also provides a low bitrate image compression system based on a diffusion model for human perception, wherein the low bitrate image compression system based on a diffusion model for human perception includes:

[0041] An image compression module is used to input the sender's source image into a low bitrate image compression framework and compress the source image into dense features through the compression model in the low bitrate image compression framework.

[0042] The feature compression module is used to compress the dense features into a bit stream using lossless encoding, and then transmit the bit stream to the receiver via the channel.

[0043] The feature decompression module is used to decompress the bitstream into dense features through lossless decoding, and to decompress the dense features into low-frequency image features through a lightweight decompressor;

[0044] The feature injection module is used to inject text and high-frequency image information into the low-frequency image features through the diffusion model in the low bit-rate image compression framework to obtain target image information;

[0045] The feature conversion module is used to perform diffusion and denoising processing on the target image information to obtain target image features, and to convert the target image features from the feature domain to the image domain to obtain the target image.

[0046] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a human-perception-oriented diffusion model-based low bitrate image compression program stored in the memory and executable on the processor, wherein when the human-perception-oriented diffusion model-based low bitrate image compression program is executed by the processor, it implements the steps of the human-perception-oriented diffusion model-based low bitrate image compression method as described above.

[0047] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a low bitrate image compression program based on a diffusion model for human perception, and the low bitrate image compression program based on a diffusion model for human perception, when executed by a processor, implements the steps of the low bitrate image compression method based on a diffusion model for human perception as described above.

[0048] In this invention, a source image from the sender is input into a low-bitrate image compression framework. The source image is compressed into dense features using a compression model within the framework. These dense features are then compressed into a bitstream using lossless encoding and transmitted to the receiver via a channel. The bitstream is decompressed into dense features using lossless decoding, and then decompressed into low-frequency image features using a lightweight decompressor. Text and high-frequency image information are injected into the low-frequency image features using a diffusion model within the low-bitrate image compression framework to obtain target image information. This target image information is then subjected to diffusion and denoising processing to obtain target image features. Finally, these target image features are transformed from the feature domain to the image domain to obtain the target image. This invention achieves better bitrate allocation by applying weighted loss to the encoder, reducing the amount of data required for image storage and transmission. Furthermore, it enhances the high-frequency information of the decoded features, improving the visual quality of the reconstructed image and effectively avoiding artifacts in the reconstructed image. Attached Figure Description

[0049] Figure 1 This is a flowchart of a preferred embodiment of the low bitrate image compression method based on a diffusion model for human perception according to the present invention;

[0050] Figure 2 This is a schematic diagram illustrating the image compression and decompression process in a preferred embodiment of the low bit rate image compression method based on a diffusion model for human perception according to the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the principle of training the low bitrate image compression framework using an end-to-end training method in a preferred embodiment of the diffusion model-based low bitrate image compression method for human perception according to the present invention.

[0052] Figure 4 This is a comparison chart of the compression performance of the low bitrate image compression framework and the accuracy of existing image compression methods in the FID index, based on a diffusion model and a preferred embodiment of the low bitrate image compression method of the present invention, which is geared towards human perception.

[0053] Figure 5 This is a comparison chart of the compression performance of the low bitrate image compression framework and the accuracy of existing image compression methods in a preferred embodiment of the diffusion model-based low bitrate image compression method of the present invention, tested on the LPIPS index.

[0054] Figure 6 This is a structural diagram of a preferred embodiment of the low bit-rate image compression system based on a diffusion model for human perception according to the present invention.

[0055] Figure 7 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] The preferred embodiment of the present invention describes a low bitrate image compression method based on a diffusion model, oriented towards human perception. Figure 1 and Figure 2 As shown, the low bit-rate image compression method based on a diffusion model, geared towards human perception, includes the following steps:

[0058] Step S10: Input the source image from the sender into the low bitrate image compression framework, and compress the source image into dense features through the compression model in the low bitrate image compression framework.

[0059] Specifically, this invention establishes a low bitrate image compression framework (DiffPC) based on a diffusion model oriented towards human perception. The low bitrate image compression framework (DiffPC) includes a compression model and a diffusion model, and is trained using an end-to-end training method.

[0060] The DiffPC framework consists of two parts. In the first part, the DiffPC framework achieves better bitrate allocation and reconstructs more accurate low-frequency image control branches under the guidance of importance-weighted loss. In the second part, the DiffPC framework effectively integrates a high-level semantic control branch, achieving more precise control and reconstruction results that are more in line with human perception.

[0061] The sending direction submits the source image to the DiffPC framework. The compression model in the DiffPC framework achieves efficient bitrate allocation through the image distribution obtained during training, compressing the source image into dense features (dense features correspond to sparse features, indicating that there is little redundancy between features, thus making the information more dense).

[0062] Step S20: Compress the dense features into a bit stream using lossless encoding, and transmit the bit stream to the receiver via a channel.

[0063] Step S30: Decompress the bitstream into dense features using lossless decoding, and then decompress the dense features into low-frequency image features using a lightweight decompressor.

[0064] Specifically, after the receiver obtains the bitstream, it decompresses the bitstream into dense features using lossless decoding, and then decompresses the dense features into low-frequency image features using a lightweight decompressor, while simultaneously transmitting text describing the image to the receiver.

[0065] Step S40: Inject text and high-frequency image information into the low-frequency image features through the diffusion model in the low bit-rate image compression framework to obtain the target image information.

[0066] Specifically, by using the pre-trained diffusion model in the DiffPC framework, text and high-frequency image information generated by the diffusion model are continuously injected into low-frequency image features to meet the aesthetic requirements of human perception.

[0067] Step S50: Perform diffusion and denoising processing on the target image information to obtain target image features, and convert the target image features from the feature domain to the image domain to obtain the target image.

[0068] Specifically, the diffused and denoised features are restored to the image domain by the decoder, and the receiver obtains the image transmitted by the sender through the DiffPC framework.

[0069] This invention achieves better bitrate allocation by applying weighted loss to the encoder during training, thereby reducing the amount of data required for image storage and transmission. In addition, at the receiver, the diffusion model in the DiffPC framework is used to enhance the high-frequency information of the decoded features, improving the visual quality of the reconstructed image and effectively avoiding the problem of artifacts in the reconstructed image.

[0070] Existing deep learning image compression models geared towards human perception are not optimal in terms of compression performance, especially at low bitrates where both compression and perception performance need improvement. This invention proposes the DiffPC framework, which employs importance-weighted loss during training to enhance the allocation of bitrates for better human perception, and uses a diffusion model for enhanced image decoding, thereby improving the performance of the DiffPC framework at low bitrates.

[0071] The present invention employs an end-to-end training method to train the low bitrate image compression framework (DiffPC), the training of which includes a first-stage training and a second-stage training.

[0072] The first stage of training is used to train some parameters of the encoder of the compression model, the conditional controller of the diffusion model, the time-aware decoupling module, and the decoder of the diffusion model in the low bit-rate image compression framework.

[0073] like Figure 3 As shown in (a), a training image x is obtained. After the training image x passes through the encoder ε of the diffusion model, intermediate features z0 and multi-scale features f1 and f2 are obtained. The intermediate features z0 and multi-scale features f1 and f2 are then passed through the multi-scale compressor of the compression model to obtain the latent features of the diffusion model. latent features Transmitted to the receiver; and the latent features are processed through the time-aware decoupling module and the control module ICCN. The sample is fed into the diffusion model, and z is obtained by Gaussian sampling through the diffusion model. t After passing through the diffusion model, the first output z is obtained. t-1 After t iterations, the final output of the diffusion model is obtained. Then the final output Input to the decoder of the diffusion model In the process, the reconstructed image is obtained.

[0074] Experimental observations show that significant high-frequency regions tend to have smaller variances, while less important flat regions tend to have larger variances. Therefore, an importance-weighted mean square error (MSE) based on variance is proposed as a measure.

[0075]

[0076] in, This represents the mean of z0. express The mean of z0, where w represents a trainable hyperparameter initialized to the variance of z0. L imp L represents the weighted variance loss function. mse This represents the mean squared error loss function.

[0077] At low bitrates, aggressive quantization strategies can lead to the reconstruction of latent features. Pathological degradation increases the challenge of diffusion models predicting z0, and may even cause diffusion models to abandon the distribution of the true data and instead generate distributions similar to the true data. The sample size is low. This phenomenon is called conditional leakage. To address this problem, this invention uses a Time-Aware Decoupling Module (TAD) to extract the control factor from the diffusion model. The noise prediction scheme proposed in this invention is as follows:

[0078]

[0079] in, This represents the noise in the prediction. This represents the original diffusion model denoising network. The parameters of the diffusion model are represented by ∈, where ∈ represents Gaussian noise, t represents the time step, and TAD represents the time step. η This indicates a time-aware decoupling module.

[0080] In the first phase of training, the training loss function is:

[0081]

[0082] in, λ1 represents the total loss of the first stage of training, λ2 represents the control bit rate, and λ3 represents the hyperparameters of the distortion. This represents the compression ratio loss function. This represents the loss in the diffusion model. Represents the random variable z0, The mean of ,t,∈.

[0083] The second stage of training is used to freeze the compression model in the low bitrate image compression framework and train only some parameters of the conditional controller, time-aware decoupling module and decoder of the diffusion model.

[0084] like Figure 3 As shown in (b), a training image x is obtained. After the training image x passes through the encoder ε of the diffusion model, intermediate features z0 and multi-scale features f1 and f2 are obtained. The intermediate features z0 and multi-scale features f1 and f2 are then passed through the multi-scale compressor of the compression model to obtain the latent features of the diffusion model. latent features Transmitted to the receiver; latent features are processed through the time-aware decoupling module and the control module ICCN. The sample is fed into the diffusion model, and z is obtained by Gaussian sampling through the diffusion model. t After passing through the diffusion model, the first output z is obtained. t-1 , latent features Input to the decoder of the diffusion model Obtain visual information Visual information After passing through the encoder, the text is input into the query unit. Image annotation is performed on the training image x to obtain text information, which is then input into the query unit. This text information is introduced as semantic embedding, integrating the latent features... Visual information The semantic information from the text is integrated through the queryer to obtain a hybrid semantic output s. x Mixed semantic outputs s x The semantics are modulated through a projection layer and guided by the diffusion model; based on z t-1 and mixed semantic outputs x After t iterations, the final output of the diffusion model is obtained. Final output Input to the decoder of the diffusion model In the process, the reconstructed image is obtained.

[0085] In the second phase of training, the training loss function is:

[0086]

[0087] in, This indicates the total loss during the second phase of training. Represents the random variable z0, s x The mean of ,t,∈, where∈ represents Gaussian noise andt represents the time step. This represents the original diffusion model denoising network. TAD represents the parameters of the diffusion model. η This indicates a time-aware decoupling module.

[0088] Furthermore, to address the color shift issue in the diffusion model, this invention performs color correction on the decoded image and normalizes the colors of the decoded image to make their mean and variance consistent with... Alignment.

[0089] To address the need for low bitrates, this invention designs a time-aware decoupling module, a conditional control module, and a multi-scale compressor. Through the design of these modules and the importance-weighted loss function, bit allocation at low bitrates can be optimized for human perception, and the visual effect of the reconstructed image can be improved at the decoding end.

[0090] Figure 4 and Figure 5 This chart compares the compression performance of the DiffPC framework provided in this invention with the accuracy of existing image compression methods. The selected existing image compression frameworks are the GAN-based HiFiC framework, the ILLM framework, the Diffusion-based CDC framework, and the non-generative ELIC framework based on convolutional neural networks. Testing was conducted on the CLIC test set. The CLIC test set is the dataset used in the 2020 Supervised Image Lossy Compression Challenge, consisting of 428 high-resolution images, used to train and evaluate the rate-distortion performance of image compression models. Figure 4 The test is based on the Frechet Inception Distance (FID) metric. FID measures the similarity between generated and real images by calculating the statistical distance between them in the feature space of the pre-trained network. The smaller the distance value, the more similar the generated image is to the real image. Figure 5 This test is based on the Learned Perceptual Image Patch Similarity (LPIPS) metric. LPIPS is a deep learning-based image similarity metric used to measure the perceptual similarity between two images; a smaller value indicates a higher degree of perceptual similarity. Figure 4 and Figure 5As can be clearly seen, this invention significantly improves the compression performance of perception metrics compared to previous solutions, while also demonstrating excellent performance at lower bit rates.

[0091] Furthermore, such as Figure 6 As shown, based on the above-mentioned low bitrate image compression method based on a diffusion model oriented towards human perception, this invention also provides a low bitrate image compression system based on a diffusion model oriented towards human perception, wherein the low bitrate image compression system based on a diffusion model oriented towards human perception includes:

[0092] Image compression module 51 is used to input the sender's source image into the low bitrate image compression framework and compress the source image into dense features through the compression model in the low bitrate image compression framework.

[0093] The feature compression module 52 is used to compress the dense features into a bit stream using lossless encoding, and transmit the bit stream to the receiver through the channel;

[0094] The feature decompression module 53 is used to decompress the bitstream into dense features through lossless decoding, and to decompress the dense features into low-frequency image features through a lightweight decompressor;

[0095] Feature injection module 54 is used to inject text and high-frequency image information into the low-frequency image features through the diffusion model in the low bit rate image compression framework to obtain target image information;

[0096] The feature conversion module 55 is used to perform diffusion and denoising processing on the target image information to obtain target image features, and to convert the target image features from the feature domain to the image domain to obtain the target image.

[0097] Furthermore, such as Figure 7 As shown, based on the above-mentioned low bit rate image compression method and system based on diffusion model for human perception, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0098] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a low-bitrate image compression program 40 based on a diffusion model for human perception. This low-bitrate image compression program 40 based on a diffusion model for human perception can be executed by the processor 10, thereby implementing the low-bitrate image compression method based on a diffusion model for human perception in this application.

[0099] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the low bit rate image compression method based on a diffusion model for human perception.

[0100] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminal's processor 10, memory 20, and display 30 communicate with each other via a system bus.

[0101] In one embodiment, when the processor 10 executes the human-perception-oriented diffusion-based low bit-rate image compression program 40 in the memory 20, it implements the steps of the human-perception-oriented diffusion-based low bit-rate image compression method as described above.

[0102] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a low bitrate image compression program based on a diffusion model for human perception, and the low bitrate image compression program based on a diffusion model for human perception, when executed by a processor, implements the steps of the low bitrate image compression method based on a diffusion model for human perception as described above.

[0103] In summary, this invention provides a low-bitrate image compression method, system, terminal, and computer-readable storage medium based on a diffusion model for human perception. The method includes: inputting a source image from a sender into a low-bitrate image compression framework; compressing the source image into dense features using a compression model within the low-bitrate image compression framework; compressing the dense features into a bitstream using lossless encoding; transmitting the bitstream to a receiver via a channel; decompressing the bitstream into dense features using lossless decoding; decompressing the dense features into low-frequency image features using a lightweight decompressor; injecting text and high-frequency image information into the low-frequency image features using a diffusion model within the low-bitrate image compression framework to obtain target image information; performing diffusion and denoising processing on the target image information to obtain target image features; and converting the target image features from the feature domain to the image domain to obtain the target image. This invention achieves better bitrate allocation by applying weighted loss to the encoder, reducing the amount of data required for image storage and transmission, and enhances the visual quality of the reconstructed image by enhancing the high-frequency information of the decoding features, effectively avoiding the problem of artifacts in the reconstructed image.

[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0105] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0106] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A low bitrate image compression method based on a diffusion model for human perception, characterized in that, The low bit-rate image compression method based on a diffusion model, which is geared towards human perception, includes: The source image from the sender is input into the low bitrate image compression framework, and the source image is compressed into dense features by the compression model in the low bitrate image compression framework. The dense features are compressed into a bit stream using a lossless encoding method, and the bit stream is transmitted to the receiver through a channel. The bitstream is decompressed into dense features using lossless decoding, and the dense features are then decompressed into low-frequency image features using a lightweight decompressor. The target image information is obtained by injecting text and high-frequency image information into the low-frequency image features through the diffusion model in the low bit-rate image compression framework. The training of the low bitrate image compression framework includes a first-stage training and a second-stage training. The first stage of training is used to train some parameters of the encoder of the compression model, the conditional controller of the diffusion model, the time-aware decoupling module, and the decoder of the diffusion model in the low bit-rate image compression framework. The second stage of training is used to freeze the compression model in the low bitrate image compression framework and train only some parameters of the conditional controller, time-aware decoupling module and decoder of the diffusion model. The first phase of training includes: Acquire training images When training images Encoder after diffusion model Then, intermediate features are obtained. and multi-scale features and intermediate features and multi-scale features and The latent features of the diffusion model are obtained through a multi-scale compressor of the compression model. , latent features Transmitted to the receiver; The latent features are decoupled through a time-aware decoupling module and a control module. The data is fed into the diffusion model, and Gaussian sampling is performed using the diffusion model to obtain... The first output is obtained after passing through the diffusion model. ,go through The final output of the diffusion model is obtained after the second iteration. ; Final output Input to the decoder of the diffusion model In the process, color correction and color normalization are performed to obtain the reconstructed image. ; The target image information is subjected to diffusion and denoising processing to obtain target image features. The target image features are then transformed from the feature domain to the image domain to obtain the target image.

2. The low bitrate image compression method based on a diffusion model for human perception as described in claim 1, characterized in that, The process of inputting the sender's source image into the low bitrate image compression framework also includes: A low bitrate image compression framework is constructed, which includes a compression model and a diffusion model, and the low bitrate image compression framework is trained using an end-to-end training method.

3. The low bitrate image compression method based on a diffusion model for human perception as described in claim 1, characterized in that, In the first phase of training, the variance-weighted mean square error based on the importance of variance is expressed as: ； in, express The mean, express The mean, This represents a trainable hyperparameter. This represents the weighted variance loss function. This represents the mean squared error loss function; Using a time-aware decoupling module, the control factor is extracted from the diffusion model, and the noise prediction is as follows: ； in, This represents the noise in the prediction. This represents the original diffusion model denoising network. The parameters represent the diffusion model. Indicates Gaussian noise. Indicates a time step. This indicates a time-aware decoupling module; In the first phase of training, the training loss function is: ； ； in, This represents the total loss during the first phase of training. This indicates control over the bitrate. Hyperparameters indicating distortion This represents the compression ratio loss function. This represents the loss in the diffusion model. Represents a random variable The mean.

4. The low bitrate image compression method based on a diffusion model for human perception as described in claim 1, characterized in that, The second phase of training includes: Acquire training images When training images Encoder after diffusion model Then, intermediate features are obtained. and multi-scale features and intermediate features and multi-scale features and The latent features of the diffusion model are obtained through a multi-scale compressor of the compression model. , latent features Transmitted to the receiver; The latent features are decoupled through a time-aware decoupling module and a control module. The data is fed into the diffusion model, and Gaussian sampling is performed using the diffusion model to obtain... The first output is obtained after passing through the diffusion model. , latent features Input to the decoder of the diffusion model Obtain visual information Visual information After passing through the encoder, the data is input to the query unit for training images. Perform image annotation to obtain text information, and input the text information into the query device; Introducing textual information as semantic embedding, and using latent features Visual information The semantic information from the text is integrated through the queryer to obtain a hybrid semantic output. Hybrid semantic output The semantics are modulated through a projection layer and guided by the diffusion model. according to and mixed semantic output ,go through The final output of the diffusion model is obtained after the second iteration. The final output will be Input to the decoder of the diffusion model In the process, the reconstructed image is obtained. .

5. The low bitrate image compression method based on a diffusion model for human perception according to claim 4, characterized in that, In the second phase of training, the training loss function is: ； in, This indicates the total loss during the second phase of training. Represents a random variable The mean, Indicates Gaussian noise. Indicates a time step. This represents the original diffusion model denoising network. The parameters represent the diffusion model. This indicates a time-aware decoupling module.

6. A low bitrate image compression system based on a diffusion model, oriented towards human perception, characterized in that, The diffusion-based low-bit-rate image compression system for human perception is used to implement the diffusion-based low-bit-rate image compression method for human perception as described in any one of claims 1-5, wherein the diffusion-based low-bit-rate image compression system for human perception comprises: An image compression module is used to input the sender's source image into a low bitrate image compression framework and compress the source image into dense features through the compression model in the low bitrate image compression framework. The feature compression module is used to compress the dense features into a bit stream using lossless encoding, and then transmit the bit stream to the receiver via the channel. The feature decompression module is used to decompress the bitstream into dense features through lossless decoding, and to decompress the dense features into low-frequency image features through a lightweight decompressor; The feature injection module is used to inject text and high-frequency image information into the low-frequency image features through the diffusion model in the low bit-rate image compression framework to obtain target image information; The feature conversion module is used to perform diffusion and denoising processing on the target image information to obtain target image features, and to convert the target image features from the feature domain to the image domain to obtain the target image.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a low bitrate image compression program based on a diffusion model for human perception stored in the memory and executable on the processor. When the low bitrate image compression program based on a diffusion model for human perception is executed by the processor, it implements the steps of the low bitrate image compression method based on a diffusion model for human perception as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a low bitrate image compression program based on a diffusion model for human perception, which, when executed by a processor, implements the steps of the low bitrate image compression method based on a diffusion model for human perception as described in any one of claims 1-5.

Images