Decoding method, training method, decoder, training device and chip

Abstract

This application provides a decoding method, a training method, a decoder, a training device, and a chip, which can improve the fidelity of images restored based on a diffusion model. The decoding method includes: receiving a compression result corresponding to a first image from a transmitting end; using the compression result as input to a diffusion model, and using the diffusion model to denoise the quantization noise in the compression result to obtain a reconstructed image of the first image.

Description

Technical Field

[0001] This application relates to the field of encoding and decoding technology, and in particular to a decoding method, training method, decoder, training device and chip. Background Technology

[0002] With the development of deep learning technology, many researchers are now using deep learning-based methods for image compression. One such deep learning-based compression method is the diffusion model-based image compression method.

[0003] The diffusion model progressively adds noise to the original image, gradually diffusing it into Gaussian noise. During inference, it starts with the Gaussian noise and gradually removes the noise to recover the original image. However, the image recovered in this way has low fidelity. Summary of the Invention

[0004] This application provides a decoding method, a training method, a decoder, a training device, and a chip, which can improve the fidelity of the recovered image.

[0005] In a first aspect, a decoding method is provided, comprising: receiving a compression result corresponding to a first image from a transmitting end; using the compression result as input to a diffusion model, and using the diffusion model to denoise the quantization noise in the compression result to obtain a reconstructed image of the first image.

[0006] Secondly, a method for training a diffusion model is provided, comprising: acquiring a training set, the training set including an original image and a compression result corresponding to the original image; adding a first quantization noise to the original image during forward diffusion to obtain a compression result corresponding to the original image; predicting the quantization noise in the compression result during reverse generation; performing denoising processing on the compression result based on the predicted quantization noise to obtain a predicted image of the original image; and training the diffusion model based on a first difference, the first difference including the difference between the first quantization noise and the predicted quantization noise.

[0007] Thirdly, a decoder is provided, comprising: a receiving unit configured to receive a compression result corresponding to a first image from a sending end; and a processing unit configured to use the compression result as input to a diffusion model, and to use the diffusion model to denoise the quantization noise in the compression result to obtain a reconstructed image of the first image.

[0008] Fourthly, a decoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method of the first aspect when running the computer program.

[0009] Fifthly, a training apparatus for a diffusion model is provided, comprising: an acquisition unit configured to acquire a training set, the training set including an original image and a compression result corresponding to the original image; an addition unit configured to add first quantization noise to the original image during a forward diffusion process to obtain a compression result corresponding to the original image; a prediction unit configured to predict the quantization noise in the compression result during a reverse generation process; a denoising unit configured to perform denoising processing on the compression result based on the predicted quantization noise to obtain a predicted image of the original image; and a training unit configured to train the diffusion model based on a first difference, the first difference including the difference between the first quantization noise and the predicted quantization noise.

[0010] In a sixth aspect, a training apparatus for a diffusion model is provided, comprising: a memory for storing a computer program; and a processor for executing the method as described in the second aspect when running the computer program.

[0011] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program that, when executed, implements the method as described in the first or second aspect.

[0012] Eighthly, a computer program product is provided, including a computer program that, when executed, implements the method as described in the first or second aspect.

[0013] In a ninth aspect, a non-volatile computer-readable storage medium is provided for storing a bitstream, the bitstream being decoded by utilizing a decoding method of a decoder, wherein the decoding method is the method described in the first aspect.

[0014] In a tenth aspect, a decoding chip is provided, the chip including a memory and a processor, the processor being configured to retrieve and execute a program or instructions from the memory to implement some or all of the steps described in the first aspect above.

[0015] In an eleventh aspect, a chip is provided, the chip including a memory and a processor, the processor being used to call and run a program or instructions from the memory to implement some or all of the steps described in the second aspect above.

[0016] In some embodiments of this application, the quantization noise introduced in the quantization stage can be used as noise added to the forward process of the diffusion model. Therefore, the diffusion model in some embodiments of this application can construct a diffusion process from the original image to the compressed result. By adding quantization noise of different intensities, the original image is gradually noised to the compressed result provided by the sending end.

[0017] During the inference process, it is not necessary to start inference from Gaussian noise. Instead, inference can start directly from the compressed result provided by the sender. Since the compressed result contains rich image information, starting inference from the compressed result can preserve the image information in the compressed result, thereby achieving fine image reconstruction and improving the fidelity of the recovered image. Attached Figure Description

[0018] Figure 1 Here are some structural example diagrams of video encoders that can be applied to some embodiments of this application;

[0019] Figure 2 Here are some structural example diagrams of video decoders that can be applied to some embodiments of this application;

[0020] Figure 3 This is a schematic diagram of the diffusion strategy in a traditional diffusion model.

[0021] Figure 4 This is a schematic diagram of the U-Net network architecture;

[0022] Figure 5 A schematic diagram illustrating the diffusion strategies provided in some embodiments of this application;

[0023] Figure 6 Network architecture diagrams for encoding and decoding provided in some embodiments of this application;

[0024] Figure 7 Flowcharts of methods for eliminating quantization noise provided in some embodiments of this application;

[0025] Figure 8 A schematic flowchart illustrating a decoding method provided in some embodiments of this application;

[0026] Figure 9 A schematic flowchart of a training apparatus for a diffusion model provided for some embodiments of this application;

[0027] Figure 10 This is a schematic diagram showing the results of evaluating solutions from some embodiments of this application and other related technologies using PSNR on the Kodak dataset;

[0028] Figure 11 This is a schematic diagram illustrating the results of evaluating solutions from some embodiments of this application and other related technologies using MS-SSIM on the Kodak dataset;

[0029] Figure 12 This is a schematic diagram illustrating the results of evaluating some embodiments of this application and other related technologies using PSNR on the CLIC2020 test set;

[0030] Figure 13 This is a schematic diagram illustrating the results of evaluating some embodiments of this application and other related technologies using FID on the CLIC2020 test set;

[0031] Figure 14 This application provides schematic diagrams of the decoder structure for some embodiments.

[0032] Figure 15 Schematic diagrams of the decoder provided in other embodiments of this application;

[0033] Figure 16 A schematic diagram of the structure of a training device for a diffusion model provided in some embodiments of this application;

[0034] Figure 17 This is a schematic diagram of the structure of a training device for a diffusion model provided in other embodiments of this application. Detailed Implementation

[0035] With the rapid development of modern internet technology, people have increasingly higher requirements for the quality of images and videos, as well as for transmission speed and storage space. Image compression technology has become particularly important. Image compression can reduce the size of data in an image, saving storage space and transmission bandwidth without sacrificing too much visual quality.

[0036] The following section describes the image encoding and decoding processes.

[0037] Figure 1 This is a schematic block diagram of a video encoder involved in some embodiments of this application.

[0038] It should be understood that the video encoder 100 can be used for lossy compression of images, or for lossless compression of images. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0039] This video encoder 100 can be applied to image data in luminance / chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents luminance (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) used to describe color and saturation. For example, in color format, 4:2:0 means that there are 4 luminance components and 2 chrominance components (YYYYCbCr) per 4 pixels; 4:2:2 means that there are 4 luminance components and 4 chrominance components (YYYYCbCrCbCr) per 4 pixels; and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0040] For example, the video encoder 100 reads video data and, for each image in the video data, divides the image into several coding tree units (CTUs). In some examples, CTUs may be called "tree blocks," "largest coding units" (LCUs), or "coding tree blocks" (CTBs). Each CTU can be associated with a pixel block of equal size within the image. Each pixel can correspond to one luminance (luma) sample and two chrominance (chroma) samples. Therefore, each CTU can be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU can be further divided into several coding units (CUs) for encoding. CUs can be rectangular or square blocks. CUs can correspond to prediction units (PUs) and transform units (TUs).

[0041] In some embodiments, the current block may be referred to as the current coding unit (CU). A prediction block may also be referred to as a prediction image block or an image prediction block, and a reconstructed image block may also be referred to as a reconstruction block or an image reconstruction block. Due to the need for parallel processing, an image can be divided into slices. Slices within the same image can be processed in parallel, meaning there is no data dependency between them. A "frame" is a commonly used term, generally understood to mean that one frame is an image. In this document, "frame" can also be replaced with "image" or "slice," etc.

[0042] The following is combined with Figure 1 The video encoder 100 in some embodiments of this application will be described.

[0043] In some embodiments, the video encoder 100 may include a prediction module 110. The prediction module 110 includes an inter-frame prediction module 111 and an intra-frame prediction module 112. Because there is a strong correlation between adjacent pixels in an image of a video, intra-frame prediction is used in video encoding and decoding techniques to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent images in a video, inter-frame prediction is used in video encoding and decoding techniques to eliminate temporal redundancy between adjacent images, thereby improving encoding efficiency.

[0044] The inter-frame prediction module 111 can be used for inter-frame prediction, which can include motion estimation and motion compensation. It can reference image information from different images. Inter-frame prediction uses motion information to find reference blocks in the reference images and generates prediction blocks based on these reference blocks to eliminate temporal redundancy. The motion information includes a list of reference images, the reference image index, and motion vectors. Motion vectors can be integer-pixel or fractional-pixel. If the motion vector is fractional-pixel, interpolation filtering needs to be used in the reference image to create the required fractional-pixel blocks. Here, the integer-pixel or fractional-pixel blocks in the reference image found based on the motion vectors are called reference blocks. Some techniques use reference blocks as prediction blocks, while others process the reference blocks further to generate prediction blocks. Processing the reference blocks further to generate prediction blocks can also be understood as using the reference blocks as prediction blocks and then processing them to generate new prediction blocks.

[0045] The intra-frame prediction module 112 refers only to information from the same image to predict pixel information within the current image block, thereby eliminating spatial redundancy.

[0046] Intra-frame prediction has multiple prediction modes. Taking the international digital video coding standards H-series as an example, the H.264 / AVC standard has 8 angular prediction modes and 1 non-angular prediction mode, while H.265 / HEVC extends this to 33 angular prediction modes and 2 non-angular prediction modes. High Efficiency Video Coding (HEVC) uses Planar, DC, and 33 angular modes, for a total of 35 prediction modes. Versatile Video Coding (VVC) uses Planar, DC, and 65 angular modes, for a total of 67 prediction modes.

[0047] It should be noted that with the increase in angle modes, intra-frame prediction will be more accurate and better meet the needs of the development of high-definition and ultra-high-definition digital video.

[0048] In some embodiments, the video encoder 100 may include a residual module 120. The residual module 120 may generate a residual block of the CU based on pixel blocks of the CU and prediction blocks of the CU. For example, the residual module 120 may generate a residual block of the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel block of the CU and the corresponding sample in the prediction block of the CU. The residual block can also be understood as residual information.

[0049] In some embodiments, the video encoder 100 may include a transformation module 125. The transformation module 125 can transform the residual information generated by the residual module 120 to obtain transformation coefficients.

[0050] In some embodiments, the video encoder 100 may include a quantization module 130. The quantization module 130 quantizes transform coefficients. The quantization module 130 quantizes the transform coefficients associated with the CU based on a quantization parameter (QP) value associated with the CU. The video encoder 100 can adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0051] In some embodiments, the video encoder 100 may include an inverse quantization module 135. The inverse quantization module 135 can apply inverse quantization to the quantized transform coefficients to obtain the transform coefficients.

[0052] In some embodiments, the video encoder 100 may include an inverse transform module 140. The inverse transform module 140 applies an inverse transform to the transform coefficients, which can be used to reconstruct the residual block.

[0053] In some embodiments, the video encoder 100 may include a reconstruction module 150. The reconstruction module 150 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by the prediction module 110 to generate reconstructed image blocks associated with the CU. By reconstructing each sample block of the CU in this manner, the video encoder 100 can reconstruct the pixel blocks of the CU.

[0054] In some embodiments, the video encoder 100 may include a loop filter module 160. The loop filter module 160 is used to process the pixels after inverse transform and inverse quantization to compensate for distortion information and provide a better reference for subsequent encoded pixels. For example, it may perform deblocking filtering operations to reduce the block effect of pixel blocks associated with the CU.

[0055] In some embodiments, the loop filtering module 160 includes a deblocking filtering module. The deblocking filtering module is used to remove block artifacts. Block artifacts refer to the discontinuities at the boundaries of coded blocks in an image. The deblocking filtering module can smooth the block boundaries to effectively reduce or remove block artifacts.

[0056] In some embodiments, the loop filtering module 160 includes a sample adaptive offset (SAO) module. The SAO module can be used to remove ringing effects. For strong edges in an image, due to quantization distortion of the high-frequency AC coefficients, a ripple effect will be generated around the edges after decoding; this distortion can be called ringing. The SAO module can suppress ringing effects from the pixel domain perspective.

[0057] In some embodiments, the loop filtering module 160 includes an adaptive loop filter (ALF) module, which is used to remove ringing effects. ALF technology can include luma ALF, chroma ALF, and inter-component ALF. ALF can be based on the Wiener filtering principle, using the original image information and reconstructed image information to establish the Wiener-Hough equation and solve for a series of filter coefficients with the minimum mean square error to remove ringing effects.

[0058] In some embodiments, the video encoder 100 may include a decoded image buffer 170. The decoded image buffer 170 may store reconstructed pixel blocks. The inter-frame prediction module 111 may use a reference image containing the reconstructed pixel blocks to perform inter-frame prediction on PUs of other images. In addition, the intra-frame prediction module 112 may use the reconstructed pixel blocks in the decoded image buffer 170 to perform intra-frame prediction on other PUs in the same image as the CU.

[0059] In some embodiments, the video encoder 100 may include an entropy coding module 180. The entropy coding module 180 may receive quantized transform coefficients from the quantization module 130. The entropy coding module 180 may perform one or more entropy coding operations on the quantized transform coefficients to produce entropy-coded data.

[0060] Figure 2 This is a schematic block diagram of a video decoder involved in some embodiments of this application.

[0061] Video decoder 200 can receive a bitstream. In some embodiments, video decoder 200 may include an entropy decoding module 210. Entropy decoding module 210 can parse the bitstream to extract syntax elements from it. As part of parsing the bitstream, entropy decoding module 210 can parse the entropy-encoded syntax elements in the bitstream. Prediction module 220, inverse quantization module 230, inverse transform module 235, reconstruction module 240, and loop filtering module 250 can decode video data based on the syntax elements extracted from the bitstream, i.e., generate decoded video data.

[0062] In some embodiments, the video decoder 200 may include a prediction module 220. The prediction module 220 includes an intra-frame prediction module 222 and an inter-frame prediction module 221.

[0063] Intra-prediction module 222 can perform intra-prediction to generate prediction blocks for PUs. Intra-prediction module 222 can use intra-prediction modes to generate prediction blocks for PUs based on pixel blocks of spatially adjacent PUs. Intra-prediction module 222 can also determine the intra-prediction mode of PUs based on one or more syntax elements parsed from the bitstream.

[0064] Inter-frame prediction module 221 can construct a first reference image list (list 0) and a second reference image list (list 1) based on the syntax elements parsed from the bitstream. Furthermore, if the PU uses inter-frame prediction coding, entropy decoding module 210 can parse the motion information of the PU. Inter-frame prediction module 221 can determine one or more reference blocks of the PU based on the motion information of the PU. Inter-frame prediction module 221 can generate prediction blocks for the PU based on one or more reference blocks of the PU.

[0065] In some embodiments, the video decoder 200 may include an inverse quantization module 230. The inverse quantization module 230 applies inverse quantization to the quantized transform coefficients to obtain the transform coefficients. The inverse quantization module 230 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0066] In some embodiments, the video decoder 200 may include an inverse transform module 235. The inverse transform module 235 can inversely transform the transform coefficients associated with the TU. After performing the inverse transform on the transform coefficients, the inverse transform module 235 can generate a residual block associated with the TU.

[0067] In some embodiments, the video decoder 200 may include a reconstruction module 240. The reconstruction module 240 uses a residual block associated with the TU of the CU and a prediction block of the PU of the CU to reconstruct pixel blocks of the CU. For example, the reconstruction module 240 may add samples of the residual block to the corresponding samples of the prediction block to reconstruct pixel blocks of the CU, thereby obtaining reconstructed image blocks.

[0068] In some embodiments, the video decoder 200 may include a loop filter module 250. The loop filter module 250 may perform deblocking filtering operations to reduce the block artifacts of pixel blocks associated with the CU.

[0069] In some embodiments, the video decoder 200 may include a decoded image buffer 260. The video decoder 200 may store the reconstructed image of the CU in the decoded image buffer 260. The video decoder 200 may use the reconstructed image in the decoded image buffer 260 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0070] The basic process of video encoding and decoding is as follows: At the encoding end, an image is divided into blocks. For the current block, the prediction module 110 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual module 120 can calculate a residual block based on the prediction block and the original block of the current block, that is, the difference between the prediction block and the original block of the current block. This residual block can also be called residual information. Through the transformation process of the transformation module 125 and the quantization process of the quantization module 130, the residual block can remove information that is not sensitive to the human eye, thereby eliminating visual redundancy. In some embodiments, the residual block before transformation by the transformation module 125 can be called a temporal residual block, and the temporal residual block after quantization by the quantization module 130 can be called a frequency residual block or a frequency domain residual block. The entropy coding module 180 receives the quantized transformation coefficients output by the quantization module 130 and can perform entropy coding on the quantized transformation coefficients to output a bitstream. For example, the entropy coding module 180 can eliminate character redundancy based on the target context model and the probability information of the binary bitstream.

[0071] At the decoding end, the entropy decoding module 210 parses the bitstream to obtain the prediction information and quantization coefficient matrix of the current block. The prediction module 220 uses the prediction information to generate a prediction block for the current block using intra-frame prediction or inter-frame prediction. The inverse quantization module 230 uses the quantization coefficient matrix obtained from the bitstream to inverse quantize the quantization coefficient matrix to obtain transform coefficients; the inverse transform module 235 performs an inverse transform on the transform coefficients to obtain a residual block. The reconstruction module 240 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks form a reconstructed image. The loop filtering module 250 performs loop filtering on the reconstructed image based on the image or based on the blocks to obtain a decoded image. The encoding end also requires similar operations to the decoding end to obtain a decoded image. This decoded image can also be called a reconstructed image, which can be used as a reference image for inter-frame prediction of subsequent images.

[0072] It should be noted that the block partitioning information determined at the encoding end, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, are carried in the bitstream when necessary. The decoding end determines the same block partitioning information, prediction, transform, quantization, entropy coding, and loop filtering mode information or parameter information as the encoding end by parsing the bitstream and analyzing existing information, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end.

[0073] It is understandable that the "inverse transformation" of the transform coefficients at the decoding end can also be referred to as "transformation" in the standard text. In some embodiments of this application, "transformation" and "inverse transformation" correspond to two opposite processes. For example, "transformation" converts the spatial domain values ​​to frequency domain coefficients, while "inverse transformation" converts the frequency domain coefficients back to spatial domain values. If the standard only specifies decoding, then "transformation" in the standard text refers to the decoding part, specifically the "inverse transformation" in this document. The "inverse transformation" of the transform coefficients at the decoding end can also be referred to as "transformation" in the standard text.

[0074] For ease of description, the encoder mentioned above will be referred to as the transmitter, and the decoder mentioned above will be referred to as the receiver.

[0075] Traditional image compression relies heavily on manual design, which easily leads to noticeable compression artifacts at high compression ratios, resulting in compression products with low perceived quality. Compression artifacts refer to the distortions or imperfections that occur after image compression.

[0076] With the development of deep learning technology, many researchers are now using deep learning-based methods for image compression.

[0077] For example, deep learning-based compression methods include neural image compressors based on variational autoencoder (VAE) architectures. VAE-based neural image compressors aim to minimize the trade-off between rate and distortion. However, models optimized only for distortion bitrate will produce blurry compressed results at low bitrates.

[0078] For example, deep learning-based compression methods can include diffusion compression, which compresses images based on a diffusion model. Compared to traditional neural compression methods that optimize for distortion metrics, diffusion compression fully utilizes the generative capabilities of the generative model itself to enhance the realism of image compression. It sacrifices distortion to improve perceptual quality, focusing more on enhancing human perceptual quality rather than faithful reconstruction.

[0079] The diffusion model will be introduced below.

[0080] Diffusion models are a class of deep learning methods based on probabilistic generative models, which have achieved significant results in recent years in fields such as image generation, speech synthesis, and text generation. Diffusion models simulate the physical diffusion process, gradually transforming data into noise, and then learn the reverse process to gradually recover the original data from the noise, achieving high-quality generation results.

[0081] The diffusion model includes two processes: a forward diffusion process and a reverse generation process, such as... Figure 3As shown. The forward diffusion process can be understood as a noise-adding process, while the reverse generation process can be understood as a noise-reducing process or a sampling process.

[0082] See Figure 3 The forward diffusion process transforms the original data into Gaussian noise (pure noise) by gradually adding noise to it. This process is typically a Markov chain, with a small amount of noise added at each step.

[0083] The reverse generation process can learn to gradually remove noise from the noise and recover the original data. This process is also a Markov chain, but in the opposite direction to the forward diffusion process.

[0084] The forward diffusion process and the reverse generation process will be described in detail below.

[0085] forward diffusion process

[0086] The forward diffusion process is the process of adding noise to the data. Suppose we have a batch of training data, and the data distribution is x0~q(x0), where 0 represents the initial state, that is, the state before noise diffusion has started.

[0087] The forward diffusion process is divided into multiple discrete diffusion steps T. For each diffusion step t, the data x from the previous diffusion step t-1 is given. t-1 Add Gaussian noise to generate noisy data x. t Meanwhile, data x t It will also be fed into the next diffusion step t+1 to continue adding noise. The variance of the noise is determined by a fixed value β located in the interval (0, 1). t The "noisy" data distribution is determined at the current diffusion step size. After iterating and adding noise T times, as long as T is large enough, a purely random noise distribution can be obtained according to the properties of Markov chains. Here, the diffusion step size is also called the time step.

[0088] The data expression for a single-step diffusion plus noise process from diffusion step size t-1 to diffusion step size t is as follows:

[0089]

[0090] Where q(x) t |x t-1 () represents the data x at the t-th diffusion step. t The probability distribution of x t This represents the data corresponding to the diffusion step size t, β t The coefficient represents the noise intensity at the t-th diffusion step, N represents a Gaussian distribution, I represents the original data, 1≤t≤T, and T is an integer greater than 1.

[0091] By recursively applying the above probability transition, we can obtain the result from the original data x0 to the final noisy data x. T Joint distribution:

[0092]

[0093] Where q(x) 1:T |x0) represents the transition from the initial data x0 to the final noisy data x. T The joint probability distribution, q(x) represents the product of all steps from t=1 to t=T, i.e., the combined product of all steps in the entire forward diffusion process. t |x t-1 ) indicates that given the previous step data x t-1 Under the condition that the current step x t The conditional probability distribution.

[0094] Reverse generation process

[0095] The reverse generation process is a denoising process, that is, the process of iteratively recovering clear data from random noise. This involves retrieving data from sampled Gaussian noise x. t To recover the original data x0 from a random noise of ~N(0,1), it is necessary to know the image distribution state transition at each step of the reverse generation process.

[0096] The formula for the single-step reverse denoising process from diffusion step size t to diffusion step size t-1 is as follows:

[0097]

[0098] Where q(x) t-1 |x t (x, 0) means that given x t Given x0, x t-1 The conditional probability distribution; N represents the Gaussian distribution; It is a mean function; The coefficient represents the noise intensity predicted at the t-th diffusion step.

[0099] Traditional diffusion models primarily work by spreading noise from the original image to a standard normal distribution through a forward diffusion process. The reverse generation process then starts with noise sampled from a standard Gaussian distribution and continuously uses the network to predict the noise to achieve denoising. This diffusion strategy, which requires sampling from noise containing no image information during inference, is more suitable for generation tasks.

[0100] It should be noted that the original images in some embodiments of this application may also be referred to as ground truth images.

[0101] The network architecture used in the diffusion model can be varied. For example, the network architecture used in the diffusion model can be the U-Net network architecture, the Transform network architecture, or other network architectures. Some embodiments of this application do not specifically limit this. The U-Net network architecture will be used as an example below.

[0102] U-Net is a convolutional neural network architecture for image segmentation. The overall structure of U-Net is a symmetrical U-shape; therefore, it can also be called a U-shaped network. The U-Net network consists of an encoder (downsampling path) and a decoder (upsampling path), as follows... Figure 4 As shown. The encoder is used to downsample the data, and the decoder is used to upsample the data.

[0103] The encoder includes multiple layers of convolution and pooling operations to extract high-level features of the image by progressive downsampling, while reducing spatial resolution.

[0104] The decoder includes multiple layers of deconvolution (transposed convolution) and upsampling operations, which restore the spatial resolution of the image by progressively upsampling, while generating the final segmentation result.

[0105] A notable feature of the U-Net network is the skip connections between the encoder and decoder. These connections directly pass feature maps from each layer of the encoder to the corresponding layer of the decoder, helping to recover more detailed information. The feature maps passed through skip connections can also be called skip connection features.

[0106] Skip connections concatenate high-resolution features from the encoder with upsampled features from the decoder. This structure helps preserve edge and detail information in the image, thereby improving segmentation accuracy.

[0107] By using skip connections, U-Net can fuse feature information at different scales. This multi-scale feature fusion enables the network to better capture and segment target objects at different scales, thereby improving segmentation performance.

[0108] The encoder and decoder in U-Net are symmetrical, meaning that the number of layers in the decoder is the same as the number of layers in the encoder. This symmetry makes the network structure more regular and easier to design and adjust.

[0109] See Figure 4 The left side 410 represents the downsampling process, and the right side 420 represents the upsampling process. Figure 4The U-Net architecture shown includes four downsampling layers (also called convolutional layers) and four corresponding upsampling layers. Each downsampling layer in the U-Net network is connected to its corresponding downsampling layer via skip connections. This means that the feature map obtained from each downsampling layer is connected to its corresponding upsampling layer, ensuring that each layer's feature map can be effectively used in subsequent computations. The feature maps passed through skip connections are also called skip connection features. Figure 4 The four downsampling layers in the network pass four feature maps to their corresponding upsampling layers via four skip connections. In this way, compared to other network structures (such as fully convolutional networks (FCNs)), U-Net avoids direct supervision and loss calculation on high-level feature maps. Instead, it combines features from low-level feature maps, resulting in a final feature map that contains both high-level and many low-level features. This achieves feature fusion at different scales and improves the accuracy of the model results.

[0110] The diffusion model described above is primarily used for generation tasks, but it is also employed in image compression techniques in some other applications. For example, the diffusion model can be deployed at the receiving end, and then used to reconstruct the original image.

[0111] For example, a diffusion model is used at the receiving end to reconstruct the image from the compression result at the sending end. This technique employs a two-stage image compression method. The first stage involves the sending end compressing the original image using an initial neural compressor to obtain a compressed result (also known as the initial compression result). The second stage involves the receiving end using a diffusion model to reconstruct the image and recover the original image. During the image reconstruction process, the fidelity of the reconstruction result from the initial neural compressor is enhanced based on the diffusion model.

[0112] However, in the above scheme, the generation process of the diffusion model still requires sampling from a Gaussian distribution. That is, the noise sampled from the Gaussian distribution is used as the input of the diffusion model, and the compression result is used as a condition to recover the original image. This diffusion model is also called a conditional diffusion model.

[0113] The above method cannot fully utilize the information from the initial neural compressor's compression results, limiting the improvement in fidelity and increasing sampling time costs. Therefore, for refined image reconstruction at the receiving end, the above diffusion strategy is not the optimal choice and may even lead to low-fidelity results.

[0114] To address the aforementioned issues, some embodiments of this application provide a decoding method, comprising: receiving a compression result corresponding to a first image from a transmitting end; using the compression result as input to a diffusion model, and using the diffusion model to denoise the quantization noise in the compression result to obtain a reconstructed image of the first image.

[0115] Some embodiments of this application also provide a method for training a diffusion model, comprising: acquiring a training set, the training set including an original image and a compression result corresponding to the original image; adding a first quantization noise to the original image during forward diffusion to obtain a compression result corresponding to the original image; predicting the quantization noise in the compression result during reverse generation; performing denoising processing on the compression result based on the predicted quantization noise to obtain a predicted image of the original image; and training the diffusion model based on a first difference, the first difference including the difference between the first quantization noise and the predicted quantization noise.

[0116] exist Figure 1 In the encoding process shown, during the quantization stage, continuous analog signal amplitudes are mapped to a finite discrete set. Therefore, quantization error exists during the quantization stage. The presence of quantization error also introduces noise, which is called quantization noise. Quantization noise can be understood as the residual between the compressed result and the original image.

[0117] The applicant believes that the quantization noise introduced in the quantization stage can be similar to the noise added in the forward process of the diffusion model. Therefore, some embodiments of this application can construct a diffusion process from the original image to the compression result. By adding quantization noise of different intensities, the original image is gradually annoyed to the compression result of the initial neural compressor. In the backward inference process, it is not necessary to randomly sample noise from the standard Gaussian distribution to generate the original image. Instead, inference can be started directly from the compression result containing rich image information. That is, the compression result is used as the input of the diffusion model to perform backward inference, making full use of the image information in the compression result to achieve refined reconstruction of image compression and improve the quality of the reconstructed image.

[0118] The diffusion strategies of the diffusion models provided in some embodiments of this application can be as follows: Figure 5 As shown. During the forward diffusion process, noise can be added to the original image to transform it into a compressed result; during the reverse generation process, noise can be gradually removed from the compressed result through learning to recover the original image.

[0119] and Figure 3 compared to, Figure 5The diffusion strategy shown can use the compressed result as the basis for reconstructing the image and directly denoise the quantization noise in the compressed result, thereby retaining more image information in the compressed result, which is beneficial to enhancing the fidelity of the reconstructed image.

[0120] This application proposes a method to eliminate quantization noise by improving the diffusion strategy by diffusing the ground truth image to the initial reconstruction map domain of the initial neural compressor. Based on this, noise removal is achieved directly on the quantization-noise image during inference. It is demonstrated that the diffusion model can significantly improve the perceptual quality of image compression at any bit rate while maintaining high quality, without incurring significant costs. Furthermore, this application performs adaptive high-frequency filtering on the skip connection features of the U-Net architecture diffusion model in the frequency domain, combined with frequency domain loss from wavelet transform to predict high-frequency details. In addition, a zero-sample sampling-guided enhancement method is designed to further improve image fidelity, ultimately achieving high-fidelity and low-distortion efficient image compression.

[0121] The network architecture diagrams of some embodiments of this application can be as follows: Figure 6 As shown. At the transmitting end, in step S610, the original image I0 is compressed using the initial neural compressor to obtain the compression result I. T The initial neural compressor's compression of the original image may include a quantization operation, which introduces quantization noise. q The introduced quantization noise ∈ q After undergoing a nonlinear transformation by the decoder in the initial neural compressor, the quantization noise can be understood as the compression result I. T The residual between the original image I0 and the original image I0.

[0122] At the receiving end, in step S620, the compression result is used as a condition for the diffusion model to guide sampling. Inference is then performed directly from the compression result without sampling Gaussian noise. The predicted image of the original image can be obtained through asynchronous inference.

[0123] For example, a compression model can be used to compress the original image, generating an initial compressed result containing both low-frequency information and the main high-frequency information of the original image. A conditional diffusion model is then used to model the residual between the original image and the initial compressed result, constructing a diffusion process for quantization noise to achieve a conditional diffusion model for removing quantization noise. Furthermore, while the backbone network of the diffusion model focuses on denoising, and the skip connections provide the decoder with prior high-frequency information, adaptive high-frequency filtering is applied to the skip connection features of the diffusion network, and wavelet domain high-frequency loss is combined to improve the network's ability to learn high-frequency details. Finally, a zero-shot sampling-guided enhancement method is used to effectively enhance the image fidelity.

[0124] Figure 7 The overall design flow provided by some embodiments of this application is illustrated.

[0125] See Figure 7 In step S710, an image compression model is trained. This image compression model is used to compress the original image to obtain a compressed result. The image compression model can be a neural compressor based on a VAE architecture or an ELIC image compression model.

[0126] In step S720, a diffusion model for eliminating quantization noise is provided.

[0127] An image compression model is used to initially compress the original image. This model is optimized using a distortion bitrate loss and can generate initial image compression results at various bitrates. Based on the initial compression results, a diffusion process between the initial compression results and the ground truth image is constructed. By designing a set of hyperparameters, following a Markov chain diffusion process, quantization noise of different intensities is gradually added to the original image, allowing it to diffuse into the initial compression result domain. The diffusion model is trained to predict the quantization noise added at each step of the forward diffusion process. The posterior distribution of the forward diffusion process is predicted through the diffusion network. Denoising is performed step by step starting from the initial compression results, making full use of the recovered low-frequency information and the main high-frequency information in the initial compression results to sample and generate high-fidelity and low-distortion compression results.

[0128] In step S730, the high-frequency information is enhanced.

[0129] High-frequency enhancement methods may include high-frequency filtering of skip connection features (step S734) and constructing a high-frequency loss in the wavelet domain (step S732).

[0130] Some embodiments of this application provide a plug-and-play frequency-domain adaptive filtering module that can be applied to any diffusion model using a U-Net architecture network to perform high-frequency filtering on skip connection features, mitigating the loss of texture details caused by enhanced denoising. For example, the multi-level skip features of a U-Net architecture diffusion model can be transformed to the Fourier frequency domain using Fourier transform and then subjected to spectral modulation to achieve high-frequency filtering.

[0131] The quantization noise predicted by the diffusion model can be transformed into a predicted ground truth image through a formula utilizing the special properties of the forward diffusion process. The ground truth image and the network-predicted ground truth image are then transformed into one low-frequency sub-band and three high-frequency sub-bands in the frequency domain using discrete wavelet transform. Based on the differences between the high-frequency sub-bands, the frequency domain detail preservation loss is determined to improve the distortion performance.

[0132] In step S740, a zero-sample sampling guidance enhancement method is provided.

[0133] The CLIP image encoder transforms the ground truth image predicted by the diffusion model and the compressed result to the feature space, where the CLIP perceptual loss between the two is calculated. At each step of the reverse generation process, the gradient of this CLIP perceptual loss is applied to the mean, forcing the network's predictions to retain the content of the compressed result during iteration, thus achieving impressive fidelity in the final generated result.

[0134] The following section will first provide detailed examples of the decoding methods of some embodiments of this application.

[0135] Figure 8 This is a flowchart illustrating a decoding method provided in some embodiments of this application. Figure 8 The method shown can be applied to the receiving end.

[0136] See Figure 8 In step S810, the compression result corresponding to the first image is received from the sending end.

[0137] The first image can be a single image or a frame from a video; some embodiments of this application do not specifically limit this. In other words, the solutions of some embodiments of this application can be used to decode compressed images or compressed videos.

[0138] The first image can refer to the original image or the ground truth image mentioned above.

[0139] The sending end can be used to compress the first image to obtain the compressed result corresponding to the first image. The sending end can compress the first image in a traditional way. For example, the sending end can compress the first image according to... Figure 1 The first image is compressed in the manner shown, such as by predicting, calculating residuals, transforming, quantizing, and entropy encoding the pixel information in the first image, to obtain the compression result corresponding to the first image.

[0140] The sending end can compress the first image using any bitrate. For example, the sending end can compress the first image at a low bitrate or at a high bitrate.

[0141] In some implementations, the sending end can also be called the encoding end or encoder, etc.

[0142] In some implementations, the compression result can also be replaced with a compressed image, or a compressed reconstructed image.

[0143] The compressed result corresponding to the first image is carried in the bitstream. In other words, the sending end can send a bitstream to the receiving end, which includes the compressed result corresponding to the first image. The receiving end can obtain the compressed result corresponding to the first image by receiving the bitstream.

[0144] In step S820, the compression result is used as the input of the diffusion model, and the quantization noise in the compression result is denoised using the diffusion model to obtain the reconstructed image of the first image.

[0145] The diffusion model can be deployed at the receiving end to denoise the received compressed result in order to obtain the reconstructed image.

[0146] A reconstructed image refers to the image obtained by the receiving end after decoding the compressed result. A reconstructed image can be understood as the first image recovered by the receiving end; it can also be called a predicted image.

[0147] As can be seen from the preceding text, during the compression of the first image by the sending end, quantization noise is introduced during the quantization stage. This process is similar to the forward noise addition process in the diffusion model. Therefore, some embodiments of this application can use the diffusion model to remove quantization noise in the compression result.

[0148] The forward diffusion process can be understood as diffusing the first image to the corresponding compressed result. In some embodiments of this application, the diffusion model can perform a reverse denoising process, meaning it can directly start from the compressed result for reverse reasoning, fully utilizing the image information in the compressed result of the first image to achieve refined reconstruction of the first image. Furthermore, compared to reasoning from noise, reasoning from the compressed result also helps reduce reasoning time costs.

[0149] In some implementations, the compression result can be used as the input to the diffusion model and as a condition for the diffusion model to guide the denoising process of the diffusion model, thereby controlling the image generated by the diffusion model.

[0150] The forward diffusion process and the reverse generation process of the diffusion model are introduced below.

[0151] For image compression tasks, starting inference directly from noise that contains no information reduces distortion quality and increases inference time cost. This application addresses the introduction of uncontrollable quantization noise in image compression tasks by designing a diffusion method for quantization noise elimination specific to the compression task.

[0152] The compressed image is obtained at the receiving end. The image compression network at the sending end quantizes the features of the original image and performs entropy coding. The sending end then sends the entropy-coded bitstream to the receiving end. The image encoding / decoding and quantization processes can be represented by the following formula:

[0153]

[0154] Where E represents the encoding process of the image compression network, D represents the decoding process of the image compression network, Q represents the quantization process before entropy encoding, and I represents the original image. This represents the compressed result of the original image.

[0155] Assuming there is no quantization noise and the encoder and decoder networks are ideally trained, the final reconstruction error will be zero. However, in actual compression, quantization inevitably introduces quantization noise, leading to errors in the compression result. There exists an approximate quantization noise residual between the original image I and the original image I. The variation trend of the quantization noise is nonlinear.

[0156] Compression results The residual between the original image I and the original image I can be expressed by the following formula:

[0157]

[0158] Where, ∈ n Let f(·) represent the uncontrollable quantization noise introduced when quantizing image features E(I), and let f(·) represent the nonlinear transformation function of the decoder in the image compression network. Therefore, the noise introduced by quantization can be represented by the residual between the original image and the compressed result.

[0159] In order to effectively remove quantization noise introduced in image compression, some embodiments of this application construct a diffusion method for removing quantization noise in an image, namely, constructing a diffusion strategy that diffuses the true image domain to the image domain containing quantization noise.

[0160] For example, suppose in, Also recorded as I T (Right now (This is the result obtained by adding noise to I through T steps), and I is also denoted as I0, ∈ q This represents an approximation of quantized noise. To construct a quantized noise-filled image from the original image I. The diffusion process of quantization noise involves applying different intensities of ∈ to the original image I according to a pre-defined hyperparameter schedule. q The formula is as follows:

[0161] I t = I t-1 + α t ∈ q (6)

[0162] Where t represents the diffusion step size, 1≤t≤T, T is the total diffusion step size, and T is an integer greater than 1. t This represents the image corresponding to the diffusion step size t, α tα represents the coefficient indicating the applied noise intensity at a diffusion step size of t. t Let be a covariance coefficient. From the above formula, we can obtain the direct diffusion from I0 to I. t The marginal probability distribution is given by the following formula:

[0163]

[0164] in, 1≤t≤T, where T is an integer greater than 1, α i It increases gradually, when the diffusion step size t = T. It equals 1.

[0165] By constructing the above-described forward diffusion process, the true image I0 can be gradually diffused into the image I containing quantized noise. T I T The compressed result is obtained from an image compression network. This is achieved by training a network ∈ θ (I t ,t,I T To predict ∈ q This allows us to obtain the predicted image from the ground truth image. The formula is as follows:

[0166]

[0167] Due to the edge distribution of the forward diffusion process Therefore, the simplified diffusion loss L can be used as follows. diff Training the diffusion model:

[0168]

[0169] Among them, L diff Indicates diffusion loss, Let E represent the square of the L2 norm, and E represent the expected value.

[0170] During the inference process, the image I containing noise can be transformed. T Denoising is performed step by step to obtain the corresponding clean original image I0. In contrast to the forward diffusion process, during inference, the value of t gradually decreases from T to t=1.

[0171] The model input for the (t-1)th diffusion step during inference is as follows:

[0172]

[0173] Among them, I t-1 This represents the model input corresponding to the (t-1)th diffusion step. Represents the original image for prediction, ∈ θ (It ,t,I T ) represents the quantization noise in the prediction. 1≤t≤T, where T is an integer greater than 1.

[0174] Substituting formula (8) into formula (10), we get:

[0175]

[0176] As can be seen from the above formula, based on the known parameter I t , ∈ θ (I t ,t,I T The prediction result corresponding to the (t-1)th diffusion step can be obtained. Denoising is continuously performed iteratively, and the reconstructed image I0 can be obtained when t=1.

[0177] During the denoising process of the compressed result, the receiving end may lose image texture details due to denoising. In order to better preserve texture details, the receiving end in some embodiments of this application can also perform adaptive high-frequency filtering on the skip connection features in the diffusion model, so that the network focuses more on the recovery of high-frequency information, thereby preserving more texture details and increasing the realism of the image.

[0178] In some implementations, the receiver can convert the skip connection features in the diffusion model into frequency domain features and perform high-frequency filtering on these features. The filtered features are then used to generate the reconstructed image. As mentioned earlier, the receiver denoises the compression result iteratively. For each iteration, high-frequency filtering is performed on the skip connection features, ensuring that each iteration retains more high-frequency information and guarantees the realism of the final reconstructed image.

[0179] There are various ways to convert skip connection features into frequency domain features, and some embodiments of this application do not specifically limit this. As an example, skip connection features can be converted into Fourier frequency domain features through Fourier transform, and the Fourier frequency domain features can be spectral modulated to achieve high-frequency filtering. As another example, skip connection features can be converted into frequency domain features through wavelet transform, and the frequency domain features can be filtered for high frequency.

[0180] This application does not specifically limit the network architecture used in some embodiments of the diffusion model; the diffusion model can use any network architecture. For example, the network architecture used in the diffusion model can be a U-Net network architecture or a Transform network architecture. The following mainly uses the U-Net network architecture as an example for description.

[0181] The aforementioned skip connection features can be found in the U-Net network architecture, which includes an encoder and a decoder. Skip connection features are image features passed between the encoder and decoder at the same level. The backbone of the U-Net network primarily aids in denoising, while the skip connection features are mainly responsible for passing high-frequency features from the encoder to the decoder. In other words, skip connection features contain high-frequency image features. Therefore, high-frequency filtering of these skip connection features can mitigate the loss of texture details caused by denoising.

[0182] In some implementations, skip connection features include skip connection features generated by multiple residual blocks of the encoder in the U-Net network architecture. In other words, skip connection features can include skip connection features of multiple levels in the encoder, with one residual block corresponding to one level.

[0183] The following section uses Fourier frequency conversion as an example to introduce the high-frequency filtering process for skip connection features.

[0184] F(s k ) = FFT(s k (12)

[0185] F′(s k )= F(s k ) ⊙ α k (13)

[0186] s′ k =IFFT(F′(s) k (14)

[0187] Among them, s k s′ represents the skip connection feature generated by the k-th residual block of the U-Net architecture network encoder, where k is an integer greater than or equal to 1. k This represents the skip connection feature after high-frequency filtering. FFT(·) represents Fourier transform, IFFT(·) represents inverse Fourier transform, ⊙ represents element-wise multiplication, and α represents the skip connection feature after high-frequency filtering. k It is a size and s k A consistent, learnable mask is used to adaptively suppress low frequencies in an image to achieve high-frequency filtering.

[0188] In some implementations, α k The value of can be determined by the following formula:

[0189]

[0190] Where, γ k Let r represent the learnable tensor matrix, and r represent the frequency radius. thThis indicates the frequency threshold.

[0191] For skip connection features of different sizes, r th Different. Typically, for deep skip connection features, r th The value is smaller; however, for shallow skip connection features, because they represent a wider frequency range, the corresponding r is smaller. th Relatively large.

[0192] In some implementations, α k It can be a parameter in the diffusion model; by training the diffusion model, α can be obtained. k .

[0193] The above methods can produce high-quality reconstructed images, but the methods described above mainly focus on the distortion quality of the images. In order to further improve the quality of the reconstructed images, it is necessary to improve the perceptual quality of the reconstructed images.

[0194] Currently, there is a sampling-guided perceptual quality enhancement method that borrows from classifier-guided techniques. This method guides the diffusion generation process by using an image classifier trained on noisy images. Adding the classifier's gradient causes the sampling process to continuously generate results in the direction of a specified class, thus producing high-fidelity results. However, this method requires training an additional image classifier, increasing training costs.

[0195] Based on this, some embodiments of this application provide a zero-shot sampling-guided enhancement method, which guides the sampling process towards high fidelity by constructing a perceptual loss between the compression result and the predicted image. This ensures the perceptual quality of the reconstructed image without increasing training costs.

[0196] In some implementations, the receiver can use a diffusion model to predict quantization noise and obtain a prediction result for the first image based on the predicted quantization noise. Since there is a perceptual loss between the prediction result and the compression result, the receiver can determine the perceptual loss between the image features in the compressed result and the image features in the prediction result, and adjust the prediction result of the diffusion model based on the perceptual loss to obtain the reconstructed image. This prediction result can also be referred to as the predicted image.

[0197] By adjusting the prediction results using perceptual loss, the expanded model can retain the image feature information in the compressed result during the denoising process, thereby enabling the generated reconstructed image to have impressive fidelity performance.

[0198] In some implementations, the compression result can be transformed to a feature space to obtain the feature information of the compression result, and the predicted image can be transformed to a feature space to obtain the feature information of the predicted image. Based on the feature information of the compression result and the feature information of the predicted image, the perceptual loss can be determined.

[0199] In some implementations, the aforementioned feature information can be semantic information.

[0200] In some implementations, a CLIP image encoder can be used to construct the perceptual loss between the predicted image and the compressed result. The CLIP image encoder can extract features containing semantic information from the image; therefore, it can be used to construct the perceptual loss between the predicted image and the compressed result.

[0201] For example, this can be illustrated using a diffusion model network ∈ θ (I t ,t,I T Predicting quantization noise, the predicted quantization noise can be denoted as ∈ θ (I t ,t,I T The predicted image obtained based on the predicted quantization noise can be represented as:

[0202]

[0203] in, Represents the predicted image, I t Let α represent the model input corresponding to the t-th diffusion step. i I represents the coefficient representing the noise intensity corresponding to the i-th diffusion step size. T This indicates the compression result.

[0204] The formula for calculating the perceptual loss using the CLIP image encoder is as follows:

[0205]

[0206] Among them, L CLIP D represents perceived loss. clip (·) represents the feature containing semantic information of the image obtained by the CLIP image encoder, and E represents the expected value. This represents the square of the L2 norm.

[0207] In some implementations, adjusting the prediction results of the diffusion model based on perceptual loss may include: using the perceptual loss to calculate the gradient of the predicted image to obtain a first gradient value, and adjusting the prediction results corresponding to each diffusion step size based on the first gradient value to obtain the reconstructed image.

[0208] For example, the first gradient value can be subtracted from the prediction result corresponding to each diffusion step to obtain the adjusted prediction result.

[0209] In some implementations, the receiver can determine the prediction result corresponding to the t-th diffusion step based on the predicted quantization noise, determine the perceptual loss between the predicted image and the compressed result, and use this perceptual loss to adjust the prediction result corresponding to the t-th diffusion step to obtain the model input for the (t-1)-th diffusion step. For example, the gradient of the prediction result can be calculated using the perceptual loss to obtain a first gradient value, and the prediction result corresponding to the t-th diffusion step can be adjusted based on the first gradient value to obtain the reconstructed image.

[0210] By using perceptual loss, the prediction result corresponding to the t-th diffusion step can be enhanced and adjusted, thereby obtaining a reconstructed image with enhanced perceptuality.

[0211] For example, the formula for adjusting the prediction result using perceptual loss is as follows:

[0212]

[0213] Among them, I t-1 I represents the enhanced model input corresponding to the (t-1)th diffusion step. t Let s represent the model input corresponding to the t-th diffusion step, and s represent the weight coefficients. Indicates the use of L CLIP right Calculate the gradient value after the gradient is applied.

[0214] Compared to formula (11), formula (19) has I t-1 The solution formula has been increased This term, thus making the enhanced prediction result I corresponding to the t-th diffusion step size... t-1 More perceptual information can be retained, improving the fidelity of the reconstructed image.

[0215] In some implementations, each diffusion step can correspond to a prediction result, and a perceptual loss can be obtained based on the prediction result. This perceptual loss can be adjusted to the prediction result (or the mean in the sampling process) corresponding to the diffusion step. By introducing the perceptual loss in each iteration, that is, applying the gradient of the CLIP perceptual loss to the mean, the prediction result of the network can retain the image features in the compressed result, thereby making the final generated result have impressive fidelity performance.

[0216] The compression result provided by the transmitter to the receiver includes low-frequency information of the first image and key high-frequency detail information. The diffusion method for eliminating quantization noise described above aims to reconstruct the more complex high-frequency information lost during the compression process of the first image. In some implementations, to further enhance the ability of the diffusion network to recover high-frequency information, some embodiments of this application also introduce high-frequency loss to train the diffusion model.

[0217] In some implementations, the original image can be transformed to the high-frequency domain to obtain the first high-frequency feature, and the predicted original image can be transformed to the high-frequency domain to obtain the second high-frequency feature. Based on the difference between the first and second high-frequency features, a high-frequency loss is determined. This high-frequency loss, along with the diffusion loss mentioned above, is used to train the diffusion model.

[0218] In some implementations, mathematical operations can be performed on the difference between the first and second high-frequency features to obtain the high-frequency loss. For example, the mean squared error (MSE) of this difference can be calculated to determine the high-frequency loss; that is, the high-frequency loss can be determined based on the MSE between the first and second high-frequency features. The high-frequency loss can also be called the frequency domain detail preservation loss.

[0219] Some embodiments of this application do not specifically limit the method of transforming the image to the high-frequency domain. As an example, the image can be transformed to the high-frequency domain using wavelet transform. The wavelet transform can be a Haar-based wavelet transform, and the Haar-based wavelet transform can be, for example, a multi-level 2D discrete wavelet transform based on Haar wavelets.

[0220] For example, suppose the original image is I0∈R H×W×C The predicted image obtained by the diffusion model is The image is decomposed into four sub-bands using a multi-level 2D discrete wavelet transform based on Haar wavelets, as shown in the following formula:

[0221]

[0222] Among them, A i H represents the low-frequency subband of the original image. i V represents the high-frequency subband of the original image in the horizontal direction. i D represents the high-frequency subband of the original image in the vertical direction. i This represents the high-frequency subband of the original image along the diagonal direction. This represents the low-frequency subband of the predicted image. This represents the high-frequency subband of the predicted image in the horizontal direction. This represents the high-frequency subband of the predicted image in the vertical direction. Represents the high-frequency subbands of the predicted image along the diagonal direction, 2D DWT(·) Let i represent the 2D discrete wavelet transform, where i represents the level of the wavelet transform, i∈[1-K], and K is a positive integer.

[0223] In the above formula,

[0224] For the original image I0 and the predicted image The frequency domain detail preservation loss L is determined by calculating the MSE between each high-frequency subband. fre The formula is as follows:

[0225]

[0226] In some implementations, the loss in frequency domain detail preservation can also be determined based on a portion of the high-frequency subbands mentioned above.

[0227] As an example, the frequency domain detail preservation loss can be determined based on the high-frequency subbands in the horizontal direction of the original and predicted images, as shown in the following formula:

[0228]

[0229] As an example, the frequency domain detail preservation loss can be determined based on the high-frequency subbands in the vertical direction of the original and predicted images, as shown in the following formula:

[0230]

[0231] As an example, the frequency domain detail preservation loss can be determined based on the high-frequency subbands in the diagonal direction of the original and predicted images, as shown in the following formula:

[0232]

[0233] As an example, the frequency domain detail preservation loss can be determined based on the high-frequency subbands of the original image in the horizontal and vertical directions, and the high-frequency subbands of the predicted image in the horizontal and vertical directions, as shown in the following formula:

[0234]

[0235] As an example, the frequency domain detail preservation loss can be determined based on the high-frequency subbands of the original image in the horizontal and diagonal directions, and the high-frequency subbands of the predicted image in the horizontal and diagonal directions, as shown in the following formula:

[0236]

[0237] As an example, the frequency domain detail preservation loss can be determined based on the high-frequency subbands of the original image in the diagonal and vertical directions, and the high-frequency subbands of the predicted image in the diagonal and vertical directions, as shown in the following formula:

[0238]

[0239] When training a diffusion model, it can be trained based on frequency domain detail preservation loss and diffusion loss. By introducing frequency domain detail preservation loss, the network's ability to recover high-frequency details of an image can be further enhanced on top of the recovered low-frequency content.

[0240] In some implementations, different weights can be assigned to different losses to adjust their impact on the diffusion model. In some implementations, the weight of the diffusion loss can be greater than the weight of the frequency domain detail preservation loss. For example, the weight of the diffusion loss can be 1, while the weight of the frequency domain detail preservation loss can be less than 1.

[0241] For example, the total training loss of the diffusion model can be:

[0242] L total = L diff + β L fre (29)

[0243] Where β represents the weight, and β < 1.

[0244] The above text combined Figure 8 This document describes in detail some of the decoding methods provided in embodiments of this application. The following section, in conjunction with... Figure 9 This application describes in detail the training methods of diffusion models provided in some embodiments. It should be noted that... Figure 8 The methods described and Figure 9 The methods described can be used in combination, and any content not described in detail below can be found in the description above.

[0245] Figure 9 The method shown can be applied to any terminal device or server, as long as the device is capable of training the diffusion model. This diffusion model can be deployed at the receiving end to denoise the compressed result of the first image received at the receiving end, thereby obtaining a reconstructed image of the first image.

[0246] Terminal devices may include, but are not limited to, personal computers, laptops, smartphones, tablets, etc. Servers may be standalone servers or server clusters consisting of multiple servers.

[0247] See Figure 9In step S910, a training set is obtained. This training set includes the original images and the corresponding compressed results. This training set can also be called a pairwise training set. The original images can also refer to ground truth images.

[0248] In some implementations, the compression result can also be replaced with a compressed image, or a compressed reconstructed image.

[0249] This training set can be generated by compressing the original images using a compression model. For example, the compression model can compress the original image to obtain a compressed result. This compressed result can include the compression results of images at various bitrates.

[0250] The compression model can be located at the sending end described above, or it can be located in any kind of device; some embodiments of this application do not specifically limit this. The sending end can also be referred to as the encoding end or encoder, etc.

[0251] The compression model can be any type of compression model. For example, it can be the ELIC neural image compression model. The ELIC neural image compression model uses a spatial channel context adaptive model for encoding, which can achieve state-of-the-art compression speed and greatly reduce the cost of obtaining paired training sets.

[0252] In some embodiments of this application, the paired training set contains 20,745 pairs of samples. Generally, the larger the number of samples, the better the performance of the trained diffusion model. In some implementations, a pre-trained ELIC neural image compression model can be used to perform the actual compression inference process on the Flicker2W dataset, thereby generating a paired training set of 20,745 images.

[0253] In step S920, during the forward diffusion process, a first quantization noise is added to the original image to obtain the compression result corresponding to the original image.

[0254] The forward diffusion process is equivalent to the encoding and compression process of the original image at the transmitting end. During the quantization stage, due to the existence of quantization error, there is a first quantization noise between the compressed result and the original image. The first quantization noise can be understood as the residual between the compressed result and the original image.

[0255] The forward diffusion process can diffuse noise from the original image to the compressed result by gradually adding noise. For example, noise can be added to the original image in T diffusion steps. For each diffusion step t, noise can be added based on the noise added in the previous diffusion step t-1, thus noisily diffusing the original image to the compressed image. The intensity of the noise added in each diffusion step can be different. Where 1 ≤ t ≤ T, and T is an integer greater than 1.

[0256] The process of adding noise can be found in formula (6) above.

[0257] In step S930, during the reverse generation process, the quantization noise in the compression result is predicted.

[0258] In the reverse generation process, a network can be trained to predict quantization noise.

[0259] In step S940, the compression result is denoised based on the predicted quantization noise to obtain the predicted image of the original image.

[0260] During the reverse generation process, the original image can be recovered from the compressed result based on the predicted quantization noise.

[0261] In some embodiments of this application, the diffusion model can progressively add noise to the original image to achieve a compressed result during the forward diffusion process. During the reverse generation process, it can start denoising from the compressed result to generate a predicted image of the original image. The diffusion model obtained in this way can fully utilize the image information in the compressed result of the original image during inference, achieving refined image reconstruction. Furthermore, it can reduce inference time costs.

[0262] The reverse generation process can also employ a stepwise denoising approach to gradually remove quantization noise from the compressed result. Unlike the forward diffusion process, the reverse generation process can denoise the compressed result using T diffusion steps. For each diffusion step t-1, noise can be further removed based on the prediction result of the previous diffusion step t, restoring the compressed result to the original image. Here, 1 ≤ t ≤ T, where T is an integer greater than 1.

[0263] The noise reduction process can be found in formulas (10) and (11) above.

[0264] Since there are differences between the original image recovered by the reverse generation process and the original image, some embodiments of this application also refer to the recovered original image as the predicted image.

[0265] In step S950, the diffusion model is trained based on the first difference. The first difference includes the difference between the first quantization noise and the predicted quantization noise, or in other words, the first difference includes the difference between the quantization noise added during the forward diffusion process and the quantization noise predicted during the reverse generation process.

[0266] Some embodiments of this application can reduce the first difference by training a diffusion model, thereby enabling the diffusion model to achieve a better performance state. The smaller the first difference, the better the performance of the diffusion model, and the higher the fidelity of the image predicted using the diffusion model.

[0267] In some implementations, the diffusion loss can be determined based on the first difference, and the diffusion model can be trained based on this loss. For example, the diffusion loss can be determined based on the MSE between the quantization noise added during the forward diffusion process and the quantization noise predicted during the reverse generation process.

[0268] The diffusion loss can be calculated using formula (9) above.

[0269] The compression result provided by the compression network can include low-frequency information and key high-frequency detail information of the original image. The diffusion model, during reverse generation, aims to reconstruct the more complex high-frequency details lost by the compression network. Therefore, some embodiments of this application, in addition to using the aforementioned diffusion loss, can also introduce a high-frequency loss. This high-frequency loss is used to further enhance the network's ability to recover high-frequency details based on the recovered low-frequency content. The high-frequency loss can also be called the frequency domain detail preservation loss. Some embodiments of this application can train the diffusion model based on the diffusion loss and the high-frequency loss.

[0270] High-frequency loss can be determined based on the difference between the high-frequency information corresponding to the original image and the high-frequency information corresponding to the predicted image. In some implementations, the training method of the diffusion model provided in some embodiments of this application further includes: performing a frequency domain transformation on the original image to obtain first high-frequency information corresponding to the original image; performing a frequency domain transformation on the predicted image to obtain second high-frequency information corresponding to the predicted image; and determining a second difference between the first high-frequency information and the second high-frequency information. The second difference is used to determine the high-frequency loss.

[0271] In some implementations, the second difference can refer to the MSE between the first high-frequency information and the second high-frequency information. The high-frequency loss can be determined based on the MSE between the first high-frequency information and the second high-frequency information.

[0272] The determination of high-frequency loss can be found in formulas (22) to (28) above.

[0273] In some implementations, the diffusion model can be trained based on the first difference and the second difference. The first difference can be used to determine the diffusion loss, and the second difference can be used to determine the high-frequency loss. Therefore, the diffusion model can be trained based on the diffusion loss and the high-frequency loss.

[0274] In some implementations, the total training loss can be determined based on the diffusion loss and the high-frequency loss, and the diffusion model can be trained based on the total training loss. The method for determining the total training loss can be found in formula (29) above.

[0275] In some implementations, the aforementioned frequency domain transformation can be a wavelet transform. Extracting high-frequency information from an image using wavelet transform is relatively simple to implement and does not have high cost requirements.

[0276] If the frequency domain transform is a wavelet transform, the first high-frequency information may include one or more of the following: high-frequency subbands of the original image in the horizontal direction, high-frequency subbands of the original image in the vertical direction, and high-frequency subbands of the original image in the diagonal direction.

[0277] If the frequency domain transform is a wavelet transform, then the second high-frequency information includes one or more of the following: the high-frequency subband of the predicted image in the horizontal direction, the high-frequency subband of the predicted image in the vertical direction, and the high-frequency subband of the predicted image in the diagonal direction.

[0278] In some implementations, the first high-frequency information includes the high-frequency subband of the original image in the horizontal direction (denoted as high-frequency subband a), and the second high-frequency information includes the high-frequency subband of the predicted image in the horizontal direction (denoted as high-frequency subband a'). The high-frequency loss can be determined based on the difference between high-frequency subband a and high-frequency subband a'. For example, the high-frequency loss can be determined based on the MSE between high-frequency subband a and high-frequency subband a', as shown in formula (23) above.

[0279] In some implementations, the first high-frequency information includes the high-frequency subband of the original image in the vertical direction (denoted as high-frequency subband b), and the second high-frequency information includes the high-frequency subband of the predicted image in the vertical direction (denoted as high-frequency subband b'). The high-frequency loss can be determined based on the difference between high-frequency subband b and high-frequency subband b'. For example, the high-frequency loss can be determined based on the MSE between high-frequency subband b and high-frequency subband b', as shown in formula (24) above.

[0280] In some implementations, the first high-frequency information includes the high-frequency subband of the original image in the diagonal direction (denoted as high-frequency subband c), and the second high-frequency information includes the high-frequency subband of the predicted image in the diagonal direction (denoted as high-frequency subband c'). The high-frequency loss can be determined based on the difference between high-frequency subband c and high-frequency subband c'. For example, the high-frequency loss can be determined based on the MSE between high-frequency subband c and high-frequency subband c', as in formula (25) above.

[0281] In some implementations, the first high-frequency information includes the high-frequency subbands of the original image in the horizontal direction (denoted as high-frequency subband a) and the high-frequency subbands of the original image in the vertical direction (denoted as high-frequency subband b), and the second high-frequency information includes the high-frequency subbands of the predicted image in the horizontal direction (denoted as high-frequency subband a') and the high-frequency subbands of the predicted image in the vertical direction (denoted as high-frequency subband b'). The high-frequency loss can be determined based on the differences between high-frequency subbands a and a', and between high-frequency subbands b and b'. For example, the high-frequency loss can be determined based on the MSE between high-frequency subbands a and a', and between high-frequency subbands b and b', as shown in formula (26) above.

[0282] In some implementations, the first high-frequency information includes the high-frequency subbands of the original image in the horizontal direction (denoted as high-frequency subband a) and the high-frequency subbands of the original image in the diagonal direction (denoted as high-frequency subband c), and the second high-frequency information includes the high-frequency subbands of the predicted image in the horizontal direction (denoted as high-frequency subband a') and the high-frequency subbands of the predicted image in the diagonal direction (denoted as high-frequency subband c'). The high-frequency loss can be determined based on the differences between high-frequency subbands a and a', and between high-frequency subbands c and c'. For example, the high-frequency loss can be determined based on the MSE between high-frequency subbands a and a', and between high-frequency subbands c and c', as shown in formula (27) above.

[0283] In some implementations, the first high-frequency information includes the high-frequency subbands of the original image in the vertical direction (denoted as high-frequency subband b) and the high-frequency subbands of the original image in the diagonal direction (denoted as high-frequency subband c), and the second high-frequency information includes the high-frequency subbands of the predicted image in the vertical direction (denoted as high-frequency subband b') and the high-frequency subbands of the predicted image in the diagonal direction (denoted as high-frequency subband c'). The high-frequency loss can be determined based on the differences between high-frequency subbands b and b', and between high-frequency subbands c and c'. For example, the high-frequency loss can be determined based on the MSE between high-frequency subbands b and b', and between high-frequency subbands c and c', as shown in formula (28) above.

[0284] In some implementations, the first high-frequency information includes the high-frequency subbands of the original image in the horizontal direction (denoted as high-frequency subband a), the high-frequency subbands of the original image in the vertical direction (denoted as high-frequency subband b), and the high-frequency subbands of the original image in the diagonal direction (denoted as high-frequency subband c). The second high-frequency information includes the high-frequency subbands of the predicted image in the horizontal direction (denoted as high-frequency subband a'), the high-frequency subbands of the predicted image in the vertical direction (denoted as high-frequency subband b'), and the high-frequency subbands of the predicted image in the diagonal direction (denoted as high-frequency subband c'). The high-frequency loss can be determined based on the differences between high-frequency subbands a and a', between high-frequency subbands b and b', and between high-frequency subbands c and c'. For example, the high-frequency loss can be determined based on the MSE between high-frequency subbands a and a', between high-frequency subbands b and b', and between high-frequency subbands c and c', as shown in formula (22) above.

[0285] In some implementations, the wavelet transform described above can be a multi-level 2D discrete wavelet transform.

[0286] Of course, in addition to wavelet transform, the frequency domain transformation mentioned above can also be other transformation methods, such as Fourier transform. By using Fourier transform, the image is transformed to the frequency domain, and then the frequency domain features are filtered at high frequencies to obtain high-frequency information.

[0287] In some implementations, high-frequency information in the image can also be extracted using a high-pass filter or a neural network.

[0288] The applicant trained and tested the diffusion model provided in some embodiments of this application. The diffusion model was deployed on the PyTorch platform, and training and testing of the model were completed on a single NVIDIA A100-SXM4-40GB GPU.

[0289] The training set collected above was used as the overall input to the diffusion model. The backbone network of the diffusion model adopted a U-Net architecture combining residual networks and attention mechanisms, and the noise timetable adopted a linear growth strategy. The wavelet transform scale was set to 4. Training was performed on images with a size of 256*256, with a batch size of 1, a learning rate of 8e-5, and optimization using the Adam optimizer. The number of training iterations was 60k. The 20718 paired dataset images were randomly cropped into 256×256 image patches and used as the training input to the model.

[0290] The proposed method was comprehensively evaluated by testing on the Kodak dataset and the CLIC2020 test set. These datasets contain images with different resolutions and content types. The Kodak dataset consists of 24 images with a resolution of 768×512; the CLIC2020 test dataset consists of 428 high-resolution images with a resolution of 2016×1512. Distortion metrics, perceptual metrics, and bit rate were used to evaluate the schemes and related techniques provided in some embodiments of this application. Peak signal-to-noise ratio (PSNR) was used for distortion measurement, and multi-scale structural similarity (MS-SSIM) and FID metrics were used for perceptual quality evaluation.

[0291] Due to the limited number of images in the Kodak dataset, MS-SSIM was used to measure perceptual quality on this dataset. Similar to other model methods, for inference results on the CLIC2020 test set, 256×256 resolution image patches were randomly cropped, and the FID metric was evaluated on these cropped patches. Bitrate was measured by performing actual entropy coding to obtain the compressed file size. Overlapping block segmentation was used to divide the test dataset into 256×256 image patches for inference, with an overlap of 64, to overcome the memory overflow issue when inferring high-resolution images. The DDIM method was used for skip sampling acceleration, with an actual inference step count of 1 step. Test results are as follows: Figures 10-13 As shown in the results, some embodiments of this application outperform the methods of related technologies in various metrics on the Kodak dataset and CLIC2020 test set.

[0292] from Figure 10 As can be seen, on the Kodak dataset, the PSNR values ​​obtained by testing some embodiments of this application are greater than the PSNR values ​​obtained by testing other related technologies. Therefore, the performance of the solutions provided by some embodiments of this application is superior to the performance of solutions in related technologies.

[0293] from Figure 11 As can be seen, on the Kodak dataset, the MS-SSIM values ​​obtained by testing some embodiments of this application are greater than the MS-SSIM values ​​obtained by testing other related technologies. Therefore, the performance of the solutions provided by some embodiments of this application is superior to that of related technologies.

[0294] from Figure 12It can be seen that, on the CLIC2020 test set, the PSNR values ​​obtained by testing some embodiments of this application are greater than the PSNR values ​​obtained by testing other related technologies. Therefore, the performance of the solutions provided by some embodiments of this application is superior to the performance of solutions in related technologies.

[0295] from Figure 13 It can be seen that, on the CLIC2020 test set, the FID values ​​obtained by testing some embodiments of this application are lower than the FID values ​​obtained by testing other related technologies. Therefore, the performance of some embodiments of this application is superior to that of related technologies.

[0296] In summary, this application addresses the quantization noise in neural compression results by constructing a diffusion model designed to eliminate it. The inference process begins with the compression result of an initial neural compressor containing both low-frequency and major high-frequency information, rather than sampling from standard Gaussian noise devoid of any information, thus improving image compression quality. Furthermore, by combining wavelet domain high-frequency loss and an adaptive high-frequency filtering module, the network's ability to learn high-frequency details is further enhanced. A zero-shot sampling enhancement method guides the diffusion model's inference to generate high-fidelity compression results. Experiments demonstrate that this method represents a significant improvement over previous methods.

[0297] The above text combined Figures 1 to 13 The method embodiments of this application are described in detail below, in conjunction with... Figures 14 to 17 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0298] Figure 14 This is a schematic diagram of the decoder structure provided in one embodiment of this application. Figure 14 As shown, the decoder 1400 includes a receiving unit 1410 and a processing unit 1420. The receiving unit 1410 is configured to receive the compression result corresponding to the first image from the transmitting end; the processing unit 1420 is configured to use the compression result as input to a diffusion model, and use the diffusion model to denoise the quantization noise in the compression result to obtain a reconstructed image of the first image.

[0299] In some implementations, the decoder 1400 further includes a conversion unit and a filtering unit. The conversion unit is configured to convert the skip connection features in the diffusion model into frequency domain features; the filtering unit is configured to perform high-frequency filtering on the frequency domain features, and the high-frequency filtered features are used to generate the reconstructed image.

[0300] In some implementations, the network architecture of the diffusion model is a U-Net network architecture, which includes an encoder and a decoder, and the skip connection feature is an image feature passed between the encoder and the decoder at the same level.

[0301] In some implementations, the skip connection feature includes skip connection features generated by multiple residual blocks in the encoder.

[0302] In some implementations, the processing unit is configured to: predict the quantization noise using the diffusion model; determine the prediction result of the first image based on the predicted quantization noise; determine the perceptual loss between the image features in the compression result and the image features in the prediction result; and adjust the denoising result of the diffusion model based on the perceptual loss to obtain the reconstructed image.

[0303] In some implementations, the processing unit is configured to: calculate the gradient of the prediction result using the perceptual loss to obtain a first gradient value; and adjust the prediction result corresponding to each diffusion step size based on the first gradient value to obtain the reconstructed image.

[0304] In some implementations, the processing unit is configured to: determine the predicted image corresponding to the t-th diffusion step size based on the predicted quantization noise; and enhance and adjust the prediction result corresponding to the t-th diffusion step size based on the first gradient value to obtain a reconstructed image with enhanced perceptuality; wherein 1≤t≤T, T is the total diffusion step size, and T is an integer greater than 1.

[0305] Understandably, in some embodiments of this application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated unit described above can be implemented in hardware or as a software functional module.

[0306] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0307] Therefore, some embodiments of this application provide a computer-readable storage medium applied to a decoder 1400, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned decoding method.

[0308] Based on the composition of the decoder 1400 and the computer-readable storage medium described above, see [link to documentation]. Figure 15 This illustrates a schematic diagram of the specific hardware structure of the decoder provided in some embodiments of this application. For example... Figure 15 As shown, the decoder 1500 may include a communication interface 1510, a memory 1520, and a processor 1530; the various components are coupled together via a bus system 1540. It is understood that the bus system 1540 is used to implement communication between these components. In addition to a data bus, the bus system 1540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 15 The general designated all buses as Bus System 1540. Among them,

[0309] The communication interface 1510 is used for receiving and sending signals during the process of sending and receiving information with other external network elements.

[0310] Memory 1520 is used to store computer programs.

[0311] Processor 1530, when running the computer program, performs the following:

[0312] Receive the compression result corresponding to the first image from the sending end;

[0313] The compression result is used as input to the diffusion model, and the diffusion model is used to denoise the quantization noise in the compression result to obtain the reconstructed image of the first image.

[0314] It is understood that the memory 1520 in some embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1520 of the systems and methods described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0315] The processor 1530 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1530 or by instructions in software form. The processor 1530 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in some embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in some embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1520. Processor 1530 reads the information in memory 1520 and completes the steps of the above method in conjunction with its hardware.

[0316] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), DSP devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0317] In some embodiments, as another example, the processor 1530 is also configured to perform the decoding method described in the foregoing embodiments when running the computer program.

[0318] In some embodiments of this application, a decoding chip is also provided, which includes a processor and a memory. The processor is configured to call and run a computer program from the memory, enabling a device equipped with the decoding chip to perform any of the decoding methods described above.

[0319] Figure 16 This is a schematic diagram of the structure of a training device for a diffusion model provided in one embodiment of this application. Figure 16 As shown, the device 1600 includes: an acquisition unit 1610, an addition unit 1620, a prediction unit 1630, a denoising unit 1640, and a training unit 1650. The acquisition unit 1610 is configured to acquire a training set, which includes an original image and a compression result corresponding to the original image; the addition unit 1620 is configured to add first quantization noise to the original image during forward diffusion to obtain a compression result corresponding to the original image; the prediction unit 1630 is configured to predict the quantization noise in the compression result during reverse generation; the denoising unit 1640 is configured to denoise the compression result based on the predicted quantization noise to obtain a predicted image of the original image; the training unit 1650 is configured to train the diffusion model based on a first difference, where the first difference includes the difference between the first quantization noise and the predicted quantization noise.

[0320] In some implementations, the apparatus 1600 further includes a transformation unit and a determination unit. The transformation unit is configured to perform a frequency domain transformation on the original image to obtain first high-frequency information corresponding to the original image; and to perform a frequency domain transformation on the predicted image to obtain second high-frequency information corresponding to the predicted image. The determination unit is configured to determine a second difference, wherein the second difference is the difference between the first high-frequency information and the second high-frequency information. The training unit 1650 is configured to train the diffusion model based on the first difference and the second difference.

[0321] In some implementations, the frequency domain transform is a wavelet transform, and the first high-frequency information includes one or more of the following: the high-frequency sub-band of the original image in the horizontal direction, the high-frequency sub-band of the original image in the vertical direction, and the high-frequency sub-band of the original image in the diagonal direction; the second high-frequency information includes one or more of the following: the high-frequency sub-band of the predicted image in the horizontal direction, the high-frequency sub-band of the predicted image in the vertical direction, and the high-frequency sub-band of the predicted image in the diagonal direction.

[0322] Understandably, in some embodiments of this application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated unit described above can be implemented in hardware or as a software functional module.

[0323] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0324] Therefore, some embodiments of this application provide a computer-readable storage medium applied to a training device 1600, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method in the foregoing embodiments.

[0325] Based on the composition of the training device 1600 and the computer-readable storage medium described above, see [link to documentation]. Figure 17 This illustrates a schematic diagram of the specific hardware structure of the training device provided in some embodiments of this application. For example... Figure 17 As shown, the training device 1700 may include a communication interface 1710, a memory 1720, and a processor 1730; the various components are coupled together via a bus system 1740. It is understood that the bus system 1740 is used to implement communication between these components. In addition to a data bus, the bus system 1740 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 17 The general labeled all buses as Bus System 1740. Among them,

[0326] The communication interface 1710 is used for receiving and sending signals during the process of sending and receiving information with other external network elements.

[0327] Memory 1720 is used to store computer programs.

[0328] Processor 1730, when running the computer program, performs the following:

[0329] Obtain a training set, which includes the original image and the compression result corresponding to the original image;

[0330] During the forward diffusion process, a first quantization noise is added to the original image to obtain the compression result corresponding to the original image;

[0331] During the reverse generation process, the quantization noise in the compression result is predicted;

[0332] Based on the predicted quantization noise, the compression result is denoised to obtain the predicted image of the original image;

[0333] The diffusion model is trained based on a first difference, which includes the difference between the first quantization noise and the predicted quantization noise.

[0334] It is understood that the memory 1720 in some embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1720 of the systems and methods described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0335] The processor 1730 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1730 or by instructions in software form. The processor 1730 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in some embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in some embodiments of this application can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1720. Processor 1730 reads the information in memory 1720 and completes the steps of the above method in conjunction with its hardware.

[0336] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), DSP devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0337] In some embodiments, as another example, the processor 1730 is also configured to perform the training method described in the foregoing embodiments when running the computer program.

[0338] In some embodiments of this application, a chip is also provided, the chip including a processor and a memory, the processor being configured to call and run a computer program from the memory, such that a device equipped with the chip can perform any of the training methods described above.

[0339] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0340] The serial numbers in some embodiments of this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0341] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0342] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0343] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0344] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method, characterized in that, include: Receive the compression result corresponding to the first image from the sending end; The compression result is used as input to the diffusion model, and the diffusion model is used to denoise the quantization noise in the compression result to obtain the reconstructed image of the first image.

2. The method according to claim 1, characterized in that, The method further includes: The skip connection features in the diffusion model are converted into frequency domain features; The frequency domain features are subjected to high-frequency filtering, and the filtered features are used to generate the reconstructed image.

3. The method according to claim 2, characterized in that, The network architecture of the diffusion model is a U-Net network architecture, which includes an encoder and a decoder. The skip connection feature is the image feature passed between the encoder and the decoder at the same level.

4. The method according to claim 3, characterized in that, The skip connection feature includes skip connection features generated by multiple residual blocks in the encoder.

5. The method according to any one of claims 1-4, characterized in that, The step of using the diffusion model to denoise the quantization noise in the compression result to obtain the reconstructed image of the first image includes: The diffusion model is used to predict the quantization noise; Based on the predicted quantization noise, the prediction result of the first image is determined; Determine the perceptual loss between the image features in the compression result and the image features in the prediction result; The denoising result of the diffusion model is adjusted based on the perceptual loss to obtain the reconstructed image.

6. The method according to claim 5, characterized in that, The step of adjusting the denoising result of the diffusion model based on the perceptual loss to obtain the reconstructed image includes: The gradient of the prediction result is obtained by using the perceptual loss to obtain a first gradient value; The prediction results corresponding to each diffusion step size are adjusted based on the first gradient value to obtain the reconstructed image.

7. The method according to claim 6, characterized in that, Determining the prediction result of the first image based on the predicted quantization noise includes: Based on the predicted quantization noise, determine the prediction result corresponding to the t-th diffusion step size; The step of adjusting the prediction result corresponding to each diffusion step size based on the first gradient value to obtain the reconstructed image includes: Based on the first gradient value, the prediction result corresponding to the t-th diffusion step size is adjusted to obtain the reconstructed image; Where 1≤t≤T, T is the total diffusion step size, and T is an integer greater than 1.

8. A training method for a diffusion model, characterized in that, include: Obtain a training set, which includes the original image and the compression result corresponding to the original image; During the forward diffusion process, a first quantization noise is added to the original image to obtain the compression result corresponding to the original image; During the reverse generation process, the quantization noise in the compression result is predicted; Based on the predicted quantization noise, the compression result is denoised to obtain the predicted image of the original image; The diffusion model is trained based on a first difference, which includes the difference between the first quantization noise and the predicted quantization noise.

9. The method according to claim 8, characterized in that, The method further includes: The original image is subjected to frequency domain transformation to obtain the first high-frequency information corresponding to the original image; The predicted image is subjected to frequency domain transformation to obtain the second high-frequency information corresponding to the predicted image; A second difference is determined, which is the difference between the first high-frequency information and the second high-frequency information; The step of training the diffusion model based on the first difference includes: The diffusion model is trained based on the first difference and the second difference.

10. The method according to claim 9, characterized in that, The frequency domain transform is a wavelet transform, and the first high-frequency information includes one or more of the following: the high-frequency sub-band of the original image in the horizontal direction, the high-frequency sub-band of the original image in the vertical direction, and the high-frequency sub-band of the original image in the diagonal direction. The second high-frequency information includes one or more of the following: high-frequency subbands of the predicted image in the horizontal direction, high-frequency subbands of the predicted image in the vertical direction, and high-frequency subbands of the predicted image in the diagonal direction.

11. A decoder, characterized in that, include: Memory, used to store computer programs; A processor, configured to perform the method as described in any one of claims 1-7 when running the computer program.

12. A training device for a diffusion model, characterized in that, include: Memory, used to store computer programs; A processor, configured to perform the method as described in any one of claims 8-10 when running the computer program.

13. A decoding chip, characterized in that, include: It includes a processor and a memory, the processor being configured to invoke a program or instructions from the memory to cause the method as described in any one of claims 1-7 to be executed.

14. A chip, characterized in that, include: It includes a processor and a memory, the processor being configured to invoke a program or instructions from the memory to cause the method as described in any one of claims 8-10 to be executed.

Images