A method, apparatus and medium for decoding images

Abstract

Embodiments of this disclosure relate to a method, apparatus, and medium for decoding images. The method includes: receiving a bitstream; acquiring the bitstream; decoding the bitstream to acquire low-frequency information and a noise representation of the current frame, wherein the low-frequency information is obtained by performing low-pass filtering on the image data of the current frame using an encoding device, and the noise representation is obtained by adding noise to the difference between the image data and the low-frequency information using the encoding device; performing noise reduction processing on the noise representation based on the low-frequency information using a first diffusion model to generate high-frequency information; and reconstructing the current frame by fusing the low-frequency information and the high-frequency information. In this manner, embodiments of this disclosure can improve compression efficiency and effectively improve the reconstruction quality of the current frame.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to video processing technology, and more specifically, to methods, apparatus and media for decoding images. Background Technology

[0002] Today, digital video capabilities are being applied to all aspects of people's lives. Traditional video coding techniques (such as H.264 / AV1 / H.266) mainly rely on inter-frame prediction, transform quantization, and entropy coding to achieve compression. These video coding techniques are limited in their compression ratio and reconstruction quality for high-frequency details, complex textures, and unstructured scenes, resulting in lower video quality. Summary of the Invention

[0003] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] In a first aspect, a method for decoding an image is proposed. The method includes: acquiring a bitstream; acquiring low-frequency information and a noise representation of the current frame by decoding the bitstream, wherein the low-frequency information is obtained by performing low-pass filtering on the image data of the current frame using an encoding device, and the noise representation is obtained by adding noise to the difference between the image data and the low-frequency information using an encoding device; performing noise reduction processing on the noise representation based on the low-frequency information using a first diffusion model to generate high-frequency information; and reconstructing the current frame by fusing the low-frequency information and the high-frequency information.

[0005] In a second aspect, an apparatus for decoding an image is proposed. The apparatus includes a processor and a non-transitory memory having instructions thereon. When executed by the processor, the instructions cause the processor to perform the method according to the first aspect of this disclosure.

[0006] In a third aspect, a non-transitory computer-readable storage medium is proposed. This non-transitory computer-readable storage medium stores instructions that cause a processor to perform the method according to the first aspect of this disclosure.

[0007] The embodiments of this disclosure can carry more detailed image information, such as high-frequency information, through noise representation, thereby improving compression efficiency. Furthermore, noise reduction processing can be performed on the noise representation by referencing low-frequency information using a first diffusion model, thereby improving the quality of high-frequency information and further improving the quality of the current frame reconstructed by fusing low-frequency information and such high-frequency signals. Thus, the embodiments of this disclosure can improve compression efficiency and effectively enhance the reconstruction quality of the current frame.

[0008] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0009] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0010] Figure 1 A block diagram illustrating an example video codec system according to some embodiments of the present disclosure is shown;

[0011] Figure 2 A flowchart illustrating an example process for decoding an image according to some embodiments of this disclosure is shown;

[0012] Figure 3 A flowchart illustrating an example process for encoding an image according to some embodiments of this disclosure is shown;

[0013] Figure 4 A block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0014] Throughout all the accompanying figures, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0015] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0016] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0017] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0018] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0020] As mentioned above, digital video capabilities are now being applied to all aspects of people's lives. Traditional video coding techniques (such as H.264 / AV1 / H.266) mainly rely on inter-frame prediction, transform quantization, and entropy coding to achieve compression. These video coding techniques are limited in their compression ratio and reconstruction quality for high-frequency details, complex textures, and unstructured scenes, resulting in lower video quality.

[0021] Embodiments of this disclosure propose a scheme for decoding an image. According to this scheme, a bitstream can be obtained. Further, by decoding the bitstream, low-frequency information and a noise representation of the current frame can be obtained, wherein the low-frequency information is obtained by performing low-pass filtering on the image data of the current frame by an encoding device, and the noise representation is obtained by adding noise to the difference between the image data and the low-frequency information by the encoding device. Additionally, a first diffusion model can be used to denoise the noise representation based on the low-frequency information to generate high-frequency information. Furthermore, the current frame can be reconstructed by fusing the low-frequency and high-frequency information.

[0022] In this way, embodiments of the present disclosure can carry more detailed image information, such as high-frequency information, through noise representation, thereby improving compression efficiency. Noise reduction processing can be performed on the noise representation by referencing low-frequency information through a first diffusion model, thereby improving the quality of high-frequency information and further improving the quality of the current frame reconstructed by fusing low-frequency information and such high-frequency signals.

[0023] Therefore, the embodiments of this disclosure can improve compression efficiency and effectively improve the reconstruction quality of the current frame.

[0024] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0025] Example Environment

[0026] Figure 1 This is a block diagram illustrating an example video encoding / decoding system 100 from which the techniques of this disclosure may be utilized. As shown, the video encoding / decoding system 100 may include encoding devices (e.g., source device 110) and decoding devices (e.g., destination device 120). The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and a first I / O interface 116.

[0027] Video source 112 may include sources such as video capture devices. Examples of video capture devices include, but are not limited to, interfaces for receiving video data from video content providers, computer graphics systems for generating video data, and / or combinations thereof.

[0028] Video data may include one or more images. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming an encoded representation of the video data. The bitstream may include encoded images and associated data. An encoded image is an encoded representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax structures. First I / O interface 116 may include a modulator / demodulator and / or a transmitter. Encoded video data can be directly transmitted to destination device 120 via network 130A through first I / O interface 116. Encoded video data may also be stored on storage medium / server 130B for access by destination device 120.

[0029] The destination device 120 may include a second I / O interface 126, a video decoder 124, and a display device 122. The second I / O interface 126 may include a receiver and / or a modem. The second I / O interface 126 may acquire encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120, or it may be external to the destination device 120, which is configured to interface with an external display device.

[0030] The video encoder 114 and the video decoder 124 can operate according to video compression standards such as the High Efficiency Video Codec (HEVC) standard, the Multi-Functional Video Codec (VVC) standard, and other existing and / or future standards.

[0031] It should be understood that the structure and function of the various elements in the example video codec system 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0032] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0033] Example process

[0034] Figure 2 A flowchart of an example process 200 for decoding an image according to some embodiments of the present disclosure is shown. Process 200 can be implemented at the destination device 120. Reference is made below. Figure 1 The process 200 is described using the video encoding / decoding system 100 shown. The video encoding / decoding system 100 may include encoding devices (e.g., source device 110) and decoding devices (e.g., destination device 120).

[0035] like Figure 2 As shown, in box 210, the destination device 120 acquires the bitstream. As an example, the bitstream can also be a compressed data stream obtained after compressing video frames (e.g., the current frame), which can be, for example, a series of decodable binary data sequences. As an example, such a bitstream can be acquired by the destination device 120 in any suitable scenario. For example, such a bitstream can support video transmission in applications such as end-to-end streaming media, video conferencing, surveillance, virtual reality (VR), and augmented reality (AR). The following description will exemplify this by assuming that such a bitstream is generated by the source device 110.

[0036] In box 220, the destination device 120 obtains the low-frequency information and noise representation of the current frame by decoding the bitstream. As an example, the low-frequency information is obtained by the encoding device performing low-pass filtering on the image data of the current frame, and the noise representation is obtained by the encoding device adding noise to the difference between the image data and the low-frequency information.

[0037] In some examples, the target device 120 may utilize the video decoder 124 to decode such a bitstream to obtain low-frequency information and noise representation of the current frame. As an example, such low-frequency information may be image information with frequencies below a threshold in the image data of the current frame, which may, for example, characterize the image structure of the current frame.

[0038] Furthermore, the source device 110 can obtain such low-frequency information by performing low-pass filtering on the image data of the current frame. As an example, the source device 110 can utilize any suitable low-pass filtering tool to perform such a low-pass filtering process, and this disclosure is not intended to limit the specific implementation process of such low-pass filtering.

[0039] Alternatively, the source device 110 can also obtain the low-frequency information by performing other processing on the image data of the current frame. For example, the source device 110 can obtain the low-frequency information by downsampling such image data.

[0040] In some examples, the noise representation can be a feature representation that satisfies a preset distribution function. As an example, source device 110 can obtain such a noise representation by adding noise to the difference between the image data of the current frame and low-frequency information (e.g., referred to as a first difference). Compared to such a first difference, the information entropy distribution of such a noise representation can be more uniform, and the noise representation contains less structured information. Therefore, compression of such a noise representation can have higher compression efficiency.

[0041] In some embodiments, to improve the reconstruction quality of the current frame while maintaining compression efficiency, the bitstream can indicate the low-frequency information of the current frame and compressed data regarding the noise representation, whereby the low-frequency information of the current frame may not need to be compressed. Based on this, the destination device 120 can obtain the low-frequency information of the current frame and the compressed data regarding the noise representation from the decoded bitstream. Additionally, the destination device 120 can perform decompression processing on the compressed data to obtain the noise representation of the current frame.

[0042] As an example, the destination device 120 can decode the acquired bitstream and obtain low-frequency information of the current frame and compressed data regarding the noise representation from the decoded bitstream. As an example, such compressed data can be obtained by compressing such a noise representation. Compared to the noise representation, such compressed data can occupy less bandwidth, thereby reducing the bandwidth required by the bitstream and further saving bit rate.

[0043] Alternatively or additionally, such compressed data can be generated by the source device 110 through processing the noise representation. Specifically, the source device 110 can perform singular value decomposition on the noise representation to construct a reference matrix. Further, the source device 110 can generate a residual representation based on the difference between the noise representation and the reference matrix (e.g., referred to as a second difference). Additionally, the source device 110 can generate compressed data based on the reference matrix and the residual representation.

[0044] As an example, such a reference matrix can be a low-rank approximation matrix of the noise representation, the rank of which can be, for example, a preset number described above (an appropriate positive integer less than the rank of the noise representation, e.g., 100, etc.). As an example, a specific implementation of the source device 110 compressing the noise representation can be found in the exemplary description of the encoding process below, and will not be repeated here. Therefore, embodiments of this disclosure can reduce the storage and transmission costs of the noise representation while maintaining the quality of subsequent video reconstruction.

[0045] In some embodiments, after obtaining compressed data representing the noise from the decoded bitstream, the destination device 120 may perform decompression processing on such compressed data to obtain the noise representation of the current frame. In some examples, such decompression processing may be a reverse reconstruction process of the noise representation compression process by the source device 110 to reduce the degree of data loss.

[0046] As an example, the destination device 120 can determine the residual representation from the compressed data and reconstruct the reference matrix based on the compressed data. Additionally, the destination device 120 can construct a noise representation of the current frame based on the residual representation and the reference matrix.

[0047] As an example, the destination device 120 can perform singular value deconstruction on the compressed data to obtain a reference matrix. From this, the destination device 120 can obtain the low-rank approximation matrix of the noise representation. Further, the destination device 120 can use the residual representation to reconstruct the noise representation of the current frame from the reference matrix. For example, the destination device 120 can sum the residual representation and the reference matrix to obtain such a noise representation.

[0048] Based on this compression and decompression method, the embodiments of this disclosure can balance the degree of compression of the noise representation and the reconstruction quality of the reconstructed noise representation after compression, thereby not only improving compression efficiency and reducing bit rate, but also maintaining a high reconstruction quality of the noise representation.

[0049] In box 230, the target device 120 uses a first diffusion model to denoise the noise representation based on low-frequency information to generate high-frequency information. As an example, such a first diffusion model can be an appropriate diffusion model that, for example, denoises the input noise representation to obtain higher quality information. For example, such a first diffusion model can be a conditional diffusion model, etc.

[0050] In some examples, the target device 120 may provide low-frequency information as a first conditional signal to the conditional diffusion model. Further, the target device 120 utilizes the conditional diffusion model to perform a predetermined number of noise reduction cycles on the noise representation, at least based on the first conditional signal, to generate high-frequency information.

[0051] As an example, such a first conditional signal can be used to guide the denoising process of a conditional diffusion model so that the denoising of the noise representation is consistent with the low-frequency information, thereby making the structure of the generated high-frequency information consistent with the structure indicated by the low-frequency information.

[0052] As an example, such a predetermined number of rounds can be a pre-set appropriate number of rounds, which in some examples may also be referred to as the number of iterations. As an example, the number of rounds in which the destination device 120 performs the noise reduction process on the noise representation can be equivalent to the number of rounds in which the source device 110 performs noise processing on the image data of the current frame, so as to ensure the reconstruction quality for the current frame.

[0053] In some embodiments, the target device 120 may provide richer conditional signals to the conditional diffusion model to further improve the reconstruction quality for the current frame. Specifically, the target device 120 may determine the descriptive information of the current frame by decoding the bitstream. The descriptive information includes at least one of the following: edge information, depth information, and semantic segmentation information. Further, the target device 120 may provide the descriptive information as a second conditional signal to the conditional diffusion model. Additionally, the target device 120 may utilize the conditional diffusion model to perform a predetermined number of noise reduction cycles on the noise representation based on the first and second conditional signals to generate high-frequency information.

[0054] In some examples, such descriptive information can be information describing the structure of the current frame. As examples, such edge information can include edge contours, edge boundaries, etc., in the current frame. As examples, such depth information can indicate the depth of image content in the current frame; for example, such depth information can include a depth map, etc. As examples, such semantic segmentation information can indicate multiple semantic regions in the current frame, the distribution of these semantic regions, etc. Based on this, embodiments of this disclosure can further constrain the denoising process of the first diffusion model, thereby further improving the reconstruction quality for the current frame.

[0055] In some examples, the description information of the current frame can be compressed and embedded into such a bitstream. For example, source device 110 can compress such description information using an appropriate compression method (e.g., binarization, feature quantization, etc.) to embed it into the bitstream. As an example, the compressed description information can be included as an extension segment or a separate sub-bitstream of such a bitstream. For example, destination device 120 can determine the description information of the current frame by decoding such an extension segment or such a separate sub-bitstream.

[0056] Based on this, the target device 120 can obtain low-frequency information and higher-quality high-frequency information of the current frame without requiring a high bit rate.

[0057] In frame 240, the target device 120 reconstructs the current frame by fusing low-frequency and high-frequency information.

[0058] As an example, the reconstructed current frame can possess both the image structure indicated by low-frequency information and the texture, detail, and other features indicated by high-frequency information. For instance, the reconstructed current frame can have the image content distribution, overall image outline, etc., indicated by low-frequency information, and can also have higher-quality textures and details indicated by high-frequency information.

[0059] As an example, the target device 120 can fuse such first and second feature information through various feasible fusion methods. Based on this, the reconstructed current frame can have a clearer structure, more natural texture, richer details, etc. In addition, high-frequency and low-frequency information can have more consistent semantics, thereby enabling the reconstructed current frame to have better temporal continuity.

[0060] In some embodiments, the target device 120 may fuse such low-frequency and high-frequency information along the feature dimension to further improve the reconstruction quality for the current frame. Specifically, the target device 120 may determine first feature information corresponding to the low-frequency information and second feature information corresponding to the high-frequency information. Further, the target device 120 may determine fused feature information by fusing the first and second feature information. Additionally, the target device 120 may construct the current frame based on the fused feature information.

[0061] As an example, the target device 120 can utilize any suitable feature extraction tool to determine the first feature information corresponding to the low-frequency information and the second feature information corresponding to the high-frequency information, such as convolutional neural networks, image multi-scale decomposition tools, etc. As an example, the first and second feature information can be single-level feature information or feature information corresponding to multiple levels. To ensure fusion accuracy, the first and second feature information can correspond to the same number of levels.

[0062] Furthermore, the target device 120 can determine fused feature information by fusing the first feature information and the second feature information, and construct the current frame based on the fused feature information. In order to preserve richer detail quality and higher image sharpness, in some examples, the target device 120 can fuse such first and second feature information at the hierarchical dimension of feature information to further improve the reconstruction quality of the current frame.

[0063] Taking the example that the first and second feature information correspond to multiple levels, in some embodiments, the target device 120 can fuse the feature representations corresponding to the same level in the first and second feature information to determine multiple fused feature representations corresponding to multiple levels. Additionally, the target device 120 can determine fused feature information based on multiple fused feature representations.

[0064] As an example, the first and second feature information corresponding to multiple levels can be data structures such as a Laplacian pyramid structure, which can correspond to different resolutions at different levels. Furthermore, the target device 120 can fuse the feature representations corresponding to the same level from the first and second feature information to determine multiple fused feature representations corresponding to multiple levels.

[0065] Taking three layers as an example, the first feature information includes, for example, feature representation A1 corresponding to the first layer, feature representation A2 corresponding to the second layer, and feature representation A3 corresponding to the third layer, while the second feature information includes, for example, feature representation B1 corresponding to the first layer, feature representation B2 corresponding to the second layer, and feature representation B3 corresponding to the third layer.

[0066] Furthermore, the target device 120 can fuse feature representation A1 and feature representation B1 to obtain fused feature representation C1, fuse feature representation A2 and feature representation B2 to obtain fused feature representation C2, and fuse feature representation A3 and feature representation B3 to obtain fused feature representation C3. Thus, embodiments of this disclosure can fuse comprehensive structural information and detail information at different resolutions to improve the reconstruction quality of the current frame.

[0067] Additionally, the target device 120 can fuse such fused feature representations C1, C2, and C3 to determine fused feature information. As an example, the target device 120 can directly fuse such feature representations C1, C2, and C3 to obtain such fused feature information.

[0068] To improve fusion quality, in some embodiments, the target device 120 can combine weight information corresponding to multiple levels to fuse multiple fusion feature representations to determine fusion feature information. Specifically, the target device 120 can determine weight information corresponding to multiple levels. Further, the target device 120 can fuse multiple fusion feature representations based on the weight information to determine fusion feature information.

[0069] As an example, such weight information can indicate the degree of bias towards low-frequency and high-frequency information. As another example, weight information corresponding to multiple levels can include weight distributions, weight graphs, etc. (e.g., corresponding to the current frame).

[0070] For example, in a weighted graph at a certain level, the target device 120 can assign higher weights to high-frequency information in image regions with richer details. As another example, in a weighted graph at a certain level, the target device 120 can assign higher weights to low-frequency information in image regions where structural information is more important. Therefore, embodiments of this disclosure can adaptively adjust the weight information corresponding to multiple levels to balance the visual consistency of low-frequency and high-frequency information.

[0071] In some embodiments, the target device 120 can fuse such low-frequency and high-frequency information at the pixel level to ensure reconstruction quality. Specifically, the target device 120 can generate first pixel information based on low-frequency information; generate second pixel information based on high-frequency information; and reconstruct the current frame by fusing the first pixel information and the second pixel information.

[0072] As an example, such first pixel information can indicate clearer image structure, such as brightness, region distribution, etc. As an example, such second pixel information can have richer detail information, such as hair, grass, etc. Based on this, the target device 120 can fuse such first pixel information and second pixel information so that the reconstructed current frame has both detail information and structural information.

[0073] In this way, embodiments of the present disclosure can carry more detailed image information, such as high-frequency information, through noise representation, thereby improving compression efficiency. Noise reduction processing can be performed on the noise representation by referencing low-frequency information through a first diffusion model, thereby improving the quality of high-frequency information and further improving the quality of the current frame reconstructed by fusing low-frequency information and such high-frequency signals.

[0074] In some embodiments, the bitstream acquired by the destination device 120 may be generated by the source device 110 (also referred to as the encoding device). As an example, the following will be illustrated in conjunction with the appendix... Figure 3 This document describes an example process for encoding images according to some embodiments of the present disclosure. Figure 3 A flowchart of an example process 300 for encoding an image according to some embodiments of the present disclosure is shown. Process 300 can be implemented at source device 110. Reference is made below. Figure 1 The process 300 is described using the video encoding and decoding system 100 shown.

[0075] like Figure 3 As shown in block 310, source device 110 performs low-pass filtering on the image data of the current frame to obtain low-frequency information.

[0076] As an example, during video encoding, source device 110 can encode each frame of the received video to generate a corresponding bitstream. The embodiments of this disclosure use the encoding and decoding processes for the current frame as examples for illustrative purposes. It should be understood that for each frame of the video received by source device 110, the encoding and decoding implementations for the current frame in the embodiments of this disclosure can be referenced.

[0077] Furthermore, the source device 110 can obtain the corresponding low-frequency information by performing low-pass filtering on the image data of the current frame. As an example, the source device 110 can utilize any suitable low-pass filtering tool to perform such a low-pass filtering process, and this disclosure is not intended to limit the specific implementation process of such low-pass filtering.

[0078] Alternatively, the source device 110 can also obtain the low-frequency information by performing other processing on the image data of the current frame. For example, the source device 110 can obtain the low-frequency information by downsampling such image data.

[0079] As an example, such low-frequency information can include structural information about the current frame, such as the overall brightness of the current frame, coarse structure (e.g., region distribution, basic color distribution, etc.). Such low-frequency information can be compressed with high efficiency and occupy a small bandwidth or bit rate.

[0080] Based on this, the source device 110 can directly compress such low-frequency information for embedding into the bitstream.

[0081] In box 320, source device 110 determines residual information based on the difference between image data and low-frequency information.

[0082] As an example, source device 110 can determine residual information between the current frame and low-frequency information by comparing the image data of the current frame and the low-frequency information (e.g., referred to as a second difference). As an example, source device 110 can obtain such residual information by subtracting the low-frequency information from the image data of the current frame.

[0083] In some examples, such residual information can contain details such as texture, noise, and minute pixel variations. Compared to low-frequency information, this residual information requires more resources to compress, resulting in lower compression efficiency.

[0084] In box 330, source device 110 uses a second diffusion model to add noise to the residual information to generate a noise representation.

[0085] As an example, the second diffusion model and the first diffusion model can correspond to the same model parameters. For instance, the first diffusion model and the second diffusion model can correspond to the forward noise addition process and the reverse noise reduction process of a diffusion model sharing the same model parameters. As an example, the first diffusion model can correspond to such a reverse noise reduction process, and the second diffusion model can correspond to such a forward noise addition process. Thus, the embodiments of this disclosure can guarantee the reconstruction quality for high-frequency information.

[0086] As an example, the second diffusion model's process of adding noise to the residual information can be an unconditional diffusion process, designed to generate a noise representation that is easier to compress, such as a noise representation that follows a normal distribution. For example, such a noise representation has a more uniform information entropy distribution, less structured information, and higher compression efficiency.

[0087] Therefore, embodiments of this disclosure can save bitrate during transmission by adding noise to such residual information and compressing the generated noise representation for transmission, thereby enabling bitstream transmission at a lower bitrate. Furthermore, through subsequent noise reduction processing of the noise representation, embodiments of this disclosure can ensure the reconstruction quality of video frames while saving bitrate.

[0088] In box 340, source device 110 generates a bitstream based on low-frequency information and noise representation.

[0089] As an example, source device 110 can compress such low-frequency information and such noise representation to generate such a bitstream. Specifically, source device 110 can use an appropriate coding scheme to compress such low-frequency information and compress the noise representation by means of singular value decomposition to generate a bitstream for the current frame based on the compressed low-frequency information and the compressed noise representation.

[0090] In some examples, source device 110 may perform singular value decomposition on the noise representation to obtain a set (or more) of singular values ​​and a feature representation corresponding to such a set of singular values. Further, source device 110 may sort such a set of singular values ​​based on their magnitude (e.g., from largest to smallest). Thus, source device 110 may obtain a predetermined number of principal singular values ​​and their corresponding feature representations based on the sorting results, and construct a reference matrix based on such a predetermined number of principal singular values ​​and their corresponding feature representations.

[0091] Furthermore, the source device 110 can generate a residual representation based on the difference between the noise representation and the reference matrix (e.g., referred to as a second difference). As an example, such a second difference can be obtained by subtracting the noise representation from the reference matrix.

[0092] As an example, such a residual representation has less information content compared to the reference matrix. Based on this, during the encoding of such a residual representation, the source device 110 can perform vector quantization, encoding, and other processing on such a second difference to further reduce the bandwidth required for compressed data.

[0093] Additionally, source device 110 can encapsulate such a reference matrix and residual representation to form such compressed data. Thus, embodiments of this disclosure can reduce the storage and transmission costs of noise representation while maintaining the quality of subsequent video reconstruction.

[0094] Furthermore, the source device 110 can use such compressed data and compressed low-frequency information to construct such a bitstream, thereby reducing bandwidth usage and improving transmission efficiency.

[0095] Alternatively or additionally, the source device 110 may embed richer conditional information into such a bitstream to help improve the reconstruction quality in subsequent decoding processes. Specifically, the source device 110 may obtain descriptive information of the current frame from the current frame, such descriptive information may include at least one of the following: edge information, depth information, and semantic segmentation information. As an example, an example description of the descriptive information can be referred to in the above embodiments, and will not be repeated here.

[0096] Furthermore, the source device 110 can compress such descriptive information using appropriate compression methods (e.g., binarization, feature quantization, etc.) and embed this descriptive information into extended segments or independent sub-bitstreams of the bitstream for subsequent decoding reference. Based on this, at lower bitrates, the source device 110 can transmit more information about the current frame through the bitstream, thereby improving the reconstruction quality of subsequent video frames.

[0097] In this way, embodiments of this disclosure can improve compression efficiency by constructing a noise representation corresponding to detailed information such as high-frequency information of the current frame, and ensure the quality of subsequent reconstruction of the current frame through a diffusion model. Furthermore, embodiments of this disclosure can also save bitrate by transmitting compressed data corresponding to the noise representation, thereby achieving higher reconstruction quality at a lower bitrate.

[0098] Example devices and equipment

[0099] Figure 4 A block diagram of a computing device 400 in which various embodiments of the present disclosure may be implemented is shown. The computing device 400 may be implemented as a source device 110 (or video encoder 114) or a destination device 120 (or video decoder 124), or may be included in a source device 110 (or video encoder 114) or a destination device 120 (or video decoder 124).

[0100] It should be understood that, Figure 4 The computing device 400 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0101] like Figure 4 As shown, computing device 400 includes general-purpose computing device 400. Computing device 400 may include at least one or more processors or processing units 410, memory 420, storage unit 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460.

[0102] In some embodiments, the computing device 400 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, a large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 400 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0103] Processing unit 410 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 420. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 400. Processing unit 410 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0104] Computing device 400 typically includes various computer storage media. Such media can be any media accessible by computing device 400, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 420 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 430 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 400.

[0105] The computing device 400 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 4 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0106] Communication unit 440 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in computing device 400 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, computing device 400 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0107] Input device 450 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 460 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 440, computing device 400 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 400 can also communicate with one or more devices that enable a user to interact with computing device 400, or, if necessary, with any device that enables computing device 400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via an input / output (I / O) interface (not shown).

[0108] In some embodiments, some or all of the components of computing device 400 may be arranged in a cloud computing architecture, rather than integrated into a single device. In a cloud computing architecture, components may be remotely provided and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (WAN), such as the Internet, using suitable protocols. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across remote data center locations. Cloud computing infrastructure may provide services through shared data centers, although they appear as a single access point to users. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from service providers at remote locations. Alternatively, the components and functionality described herein may be provided by conventional servers or installed directly or otherwise on client devices.

[0109] In embodiments of this disclosure, computing device 400 may be used to implement video encoding / decoding. Memory 420 may include one or more video encoding / decoding modules 425 having one or more program instructions. These modules are accessible and executable by processing unit 410 to perform the functions of the various embodiments described herein.

[0110] In an example embodiment of performing video encoding, input device 450 may receive video data as input 470 to be encoded. The video data may be processed, for example, by video codec module 425 to generate an encoded bitstream. The encoded bitstream may be provided as output 480 via output device 460.

[0111] In an example embodiment of performing video decoding, input device 450 may receive an encoded bitstream as input 470. The encoded bitstream may be processed, for example, by video codec module 425 to generate decoded video data. The decoded video data may be provided as output 480 via output device 460.

[0112] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. A method for decoding an image, characterized in that, The method includes: Obtain the bitstream; By decoding the bitstream, low-frequency information and noise representation of the current frame are obtained, wherein the low-frequency information is obtained by the encoding device performing low-pass filtering on the image data of the current frame, and the noise representation is obtained by the encoding device adding noise to the difference between the image data and the low-frequency information; Using a first diffusion model, noise reduction processing is performed on the noise representation based on the low-frequency information to generate high-frequency information. The first diffusion model is a conditional diffusion model. The process of using the first diffusion model to reduce noise based on the low-frequency information to generate the high-frequency information includes: providing the low-frequency information as a first conditional signal to the conditional diffusion model; determining the description information of the current frame by decoding the bitstream, the description information including at least one of the following: edge information, depth information, and semantic segmentation information; providing the description information as a second conditional signal to the conditional diffusion model; and performing a predetermined number of noise reduction rounds on the noise representation based on the first conditional signal and the second conditional signal using the conditional diffusion model to generate the high-frequency information. The current frame is reconstructed by fusing the low-frequency information and the high-frequency information.

2. The method according to claim 1, characterized in that, Decoding the bitstream to obtain the low-frequency information and noise representation of the current frame includes: The low-frequency information of the current frame and compressed data regarding the noise representation are obtained from the decoded bitstream; and The compressed data is decompressed to obtain the noise representation of the current frame.

3. The method according to claim 2, characterized in that, Performing decompression processing on the compressed data to construct the noise representation of the current frame includes: The residual representation is determined from the compressed data, and the reference matrix is ​​reconstructed based on the compressed data; and Based on the residual representation and the reference matrix, the noise representation of the current frame is constructed.

4. The method according to claim 3, characterized in that, The compressed data is generated by the encoding device based on the following process: Singular value decomposition is performed on the noise representation to construct the reference matrix; The residual representation is generated based on the difference between the noise representation and the reference matrix; as well as The compressed data is generated based on the reference matrix and the residual representation.

5. The method according to claim 1, characterized in that, Reconstructing the current frame by fusing the low-frequency information and the high-frequency information includes: Determine the first feature information corresponding to the low-frequency information and the second feature information corresponding to the high-frequency information; By fusing the first feature information and the second feature information, the fused feature information is determined; and The current frame is constructed based on the fused feature information.

6. The method according to claim 5, characterized in that, The first feature information and the second feature information correspond to multiple levels, and by fusing the first feature information and the second feature information, the fused feature information is determined to include: By fusing feature representations corresponding to the same level from the first feature information and the second feature information, multiple fused feature representations corresponding to the multiple levels are determined; and The fusion feature information is determined based on the multiple fusion feature representations.

7. The method according to claim 6, characterized in that, Based on the multiple fusion feature representations, the fusion feature information is determined as follows: Determine the weight information corresponding to the multiple levels; and Based on the weight information, the multiple fusion feature representations are fused to determine the fusion feature information.

8. The method according to claim 1, characterized in that, Reconstructing the current frame by fusing the low-frequency information and the high-frequency information includes: Based on the low-frequency information, first pixel information is generated; Based on the high-frequency information, second pixel information is generated; and The current frame is reconstructed by fusing the first pixel information and the second pixel information.

9. The method according to claim 1, characterized in that, The bitstream is generated by the encoding device based on the following process: Low-pass filtering is performed on the image data of the current frame to obtain the low-frequency information; Based on the difference between the image data and the low-frequency information, residual information is determined; The residual information is noise-added using a second diffusion model to generate the noise representation; as well as The bitstream is generated based on the low-frequency information and the noise representation.

10. The method according to claim 9, characterized in that, The first diffusion model and the second diffusion model correspond to the same model parameters.

11. An apparatus for decoding images, characterized in that, The apparatus includes a processor and a non-transitory memory having instructions thereon, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium for storing instructions, characterized in that, The instructions cause the processor to execute the instructions of the method according to any one of claims 1 to 10.

Images