Image processing method, image processing device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device for image super-resolution, electronic equipment and a storage medium. The image processing method comprises the following steps: performing multi-stage wavelet transform processing on a low-resolution noise-containing first spatial domain image to obtain a first multi-stage wavelet spectrogram; performing block processing on the first multi-level wavelet spectrogram to obtain a plurality of noise-added data blocks including a noise-added low-frequency component and a noise-added high-frequency component; performing de-noising processing on the plurality of noise-added data blocks to obtain a plurality of de-noised data blocks including de-noising low-frequency components and de-noising high-frequency components; performing inverse blocking processing on the plurality of de-noised data blocks to obtain a second multi-level wavelet spectrogram; and performing inverse multi-stage wavelet transform processing on the second multi-stage wavelet spectrogram to obtain a high-resolution second spatial domain image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision (CV), and in particular to an image processing method, an image processing device, an electronic device and a storage medium for image super-resolution. Background Art

[0002] In the field of computer image processing, super-resolution (SR) for a single image is a hot and challenging task. In today's world of rapid information flow, images, as important information carriers, may need to sacrifice image quality. In this scenario, the process of degrading an image from a high-resolution (HR) image to a smaller low-resolution (LR) image may result in the loss of some high-frequency information. In order to obtain better image quality, image super-resolution technology came into being. Simply put, image super-resolution technology is a technology that upsamples LR images to restore the corresponding HR images as much as possible.

[0003] For the Single Image Super Resolution (SISR) task, there are related research works based on deep learning that aim to obtain clearer and more realistic HR images from LR images. However, in order to achieve the transformation from LR images to HR images, the networks of these studies need to learn the mapping relationship between LR images and HR images through a large number of sample pairs. Generally, existing research works model the image super-resolution task as an implicit mapping from LR images to HR images. However, this implicit mapping from LR images to HR images makes the model lack of interpretability, resulting in poor fidelity of super-resolution results.

[0004] Therefore, in order to achieve better super-resolution effects, an image processing technology with better fidelity and quality is needed. Summary of the invention

[0005] In order to at least solve the above problems and / or disadvantages, embodiments of the present disclosure provide an image processing method, an image processing device, an electronic device, and a storage medium for image super-resolution.

[0006] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, which includes: obtaining a first multi-level wavelet spectrum diagram by performing multi-level wavelet transform processing on a low-resolution, noisy first spatial domain image; obtaining a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum diagram; obtaining a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components by performing denoising processing on the plurality of noisy data blocks; obtaining a second multi-level wavelet spectrum diagram by performing inverse block processing on the plurality of denoised data blocks; and obtaining a high-resolution second spatial domain image by performing inverse multi-level wavelet transform processing on the second multi-level wavelet spectrum diagram.

[0007] Optionally, the maximum number of levels of the multi-level wavelet transform processing is determined based on the super-resolution magnification scale for the first spatial domain image.

[0008] Optionally, the step of obtaining a plurality of noisy data blocks including a noisy low-frequency component and a noisy high-frequency component by performing block processing on the first multi-level wavelet spectrum graph includes: dividing the first multi-level wavelet spectrum graph into a plurality of image blocks, wherein the block size of each of the plurality of image blocks is determined based on the level of the wavelet spectrum graph where the image block is located in the first multi-level wavelet spectrum graph and the minimum block size of the low-frequency component of the first multi-level wavelet spectrum graph; obtaining a plurality of noisy data blocks with spectral information by performing four-dimensional position encoding on each of the plurality of image blocks, wherein the plurality of noisy data blocks include a low-frequency noisy data block corresponding to the noisy low-frequency component and a high-frequency noisy data block corresponding to the noisy high-frequency component.

[0009] Optionally, the step of obtaining multiple denoised data blocks including denoised low-frequency components and denoised high-frequency components by denoising the multiple noisy data blocks includes: acquiring a noise-free low-resolution image corresponding to the first spatial domain image; based on the low-resolution image, obtaining multiple denoised data blocks by denoising the low-frequency noisy data blocks and the high-frequency noisy data blocks respectively, wherein the multiple denoised data blocks include low-frequency denoised data blocks corresponding to the denoised low-frequency components and high-frequency denoised data blocks corresponding to the denoised high-frequency components.

[0010] Optionally, based on the low-resolution image, the step of obtaining multiple denoised data blocks by denoising low-frequency noisy data blocks and high-frequency noisy data blocks respectively includes: under the guidance of the low-resolution image, denoising the low-frequency noisy data blocks by a low-frequency decoder to obtain low-frequency denoised data blocks corresponding to the denoised low-frequency components; under the guidance of the low-resolution image, denoising the high-frequency noisy data blocks by a high-frequency decoder to obtain high-frequency denoised data blocks corresponding to the denoised high-frequency components.

[0011] Optionally, the step of obtaining a second multi-level wavelet spectrum diagram by performing inverse block processing on the multiple denoised data blocks includes: obtaining the second multi-level wavelet spectrum diagram by performing full connection on the multiple denoised data blocks.

[0012] Optionally, the step of obtaining a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum graph also includes: obtaining a plurality of noisy data blocks with a fixed number for denoising processing by performing a cross-attention operation on a plurality of noisy data blocks with a first number, wherein the step of obtaining a second multi-level wavelet spectrum graph by performing inverse block processing on the plurality of denoised data blocks also includes: obtaining a plurality of denoised data blocks with a first number for full connection by performing a grid sampling operation on a plurality of denoised data blocks with a fixed number obtained by inverse block processing.

[0013] Optionally, the image processing method is iteratively executed by using a diffusion model until a maximum number of iterations is completed, wherein the noisy first spatial domain image in the current iteration step is a noisy image of the initial low-resolution image or a noisy image of the second spatial domain image in the previous iteration step.

[0014] According to a first aspect of an embodiment of the present disclosure, an image processing device is provided, which includes: a wavelet transform unit, configured to obtain a first multi-level wavelet spectrum diagram by performing multi-level wavelet transform processing on a low-resolution noisy first spatial domain image; a block processing unit, configured to obtain a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum diagram; a denoising processing unit, configured to obtain a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components by performing denoising processing on the plurality of noisy data blocks; an inverse blocking processing unit, configured to obtain a second multi-level wavelet spectrum diagram by performing inverse blocking processing on the plurality of denoised data blocks; and an inverse wavelet transform unit, configured to obtain a high-resolution second spatial domain image by performing inverse multi-level wavelet transform processing on the second multi-level wavelet spectrum diagram.

[0015] Optionally, the maximum number of levels of the multi-level wavelet transform processing is determined based on the super-resolution magnification scale for the first spatial domain image.

[0016] Optionally, the block processing unit is configured to obtain a plurality of noisy data blocks including a noisy low-frequency component and a noisy high-frequency component by the following operations: dividing the first multi-level wavelet spectrum map into a plurality of image blocks, wherein the block size of each of the plurality of image blocks is determined based on the level of the wavelet spectrum map where the image block is located in the first multi-level wavelet spectrum map and the minimum block size of the low-frequency component of the first multi-level wavelet spectrum map; obtaining a plurality of noisy data blocks with spectral information by performing four-dimensional position encoding on each of the plurality of image blocks, wherein the plurality of noisy data blocks include a low-frequency noisy data block corresponding to the noisy low-frequency component and a high-frequency noisy data block corresponding to the noisy high-frequency component.

[0017] Optionally, the denoising processing unit is configured to obtain a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components by the following operations: obtaining a noise-free low-resolution image corresponding to the first spatial domain image; based on the low-resolution image, obtaining a plurality of denoised data blocks by performing denoising processing on low-frequency noisy data blocks and high-frequency noisy data blocks respectively, wherein the plurality of denoised data blocks include low-frequency denoised data blocks corresponding to the denoised low-frequency components and high-frequency denoised data blocks corresponding to the denoised high-frequency components.

[0018] Optionally, the denoising processing unit is configured to obtain multiple denoised data blocks by the following operations: under the guidance of the low-resolution image, denoising the low-frequency noisy data blocks through a low-frequency decoder to obtain low-frequency denoised data blocks corresponding to the denoised low-frequency components; under the guidance of the low-resolution image, denoising the high-frequency noisy data blocks through a high-frequency decoder to obtain high-frequency denoised data blocks corresponding to the denoised high-frequency components.

[0019] Optionally, the inverse blocking processing unit is configured to obtain the second multi-level wavelet spectrum diagram by the following operation: obtaining the second multi-level wavelet spectrum diagram by fully connecting multiple denoising data blocks.

[0020] Optionally, the block processing unit is also configured to obtain a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by the following operations: by performing a cross-attention operation on a plurality of noisy data blocks having a first number, a plurality of noisy data blocks having a fixed number for denoising processing are obtained, wherein the inverse block processing unit is further configured to obtain a second multi-level wavelet spectrum graph by the following operations: by performing a grid sampling operation on a plurality of denoised data blocks having a fixed number obtained through the inverse block processing, a plurality of denoised data blocks having a first number for full connection are obtained.

[0021] Optionally, the image processing method is iteratively executed by using a diffusion model until a maximum number of iterations is completed, wherein the noisy first spatial domain image in the current iteration step is a noisy image of the initial low-resolution image or a noisy image of the second spatial domain image in the previous iteration step.

[0022] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the method described above.

[0023] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein, when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the method as described above.

[0024] According to the embodiments of the present disclosure, the image processing method, image processing device, electronic device and storage medium have at least the following technical effects: better fidelity: the mapping relationship from the LR image to the low-frequency component spectrum diagram is simpler and more direct, and the mapping will be explicitly supervised; better quality: the high-frequency component data can be directly and explicitly processed, generated and supervised, so that more details can be generated, thereby improving the quality of the processing results; better interpretability: the image information is split into high- and low-frequency parts, which is consistent with the image super-resolution process.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings.

[0027] Figure 1 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure.

[0028] Figure 2 is a diagram of a MWDT framework according to an embodiment of the present disclosure.

[0029] Figure 3 and Figure 4 is a diagram illustrating an example of MDWT processing and iMDWT processing according to an embodiment of the present disclosure.

[0030] Figure 5 is a diagram illustrating an example of pyramid blocking processing according to an embodiment of the present disclosure.

[0031] Figure 6 is a diagram illustrating an example of a process of 4D position encoding according to an embodiment of the present disclosure.

[0032] Figure 7 is a diagram illustrating a multi-scale data block sampling process according to an embodiment of the present disclosure.

[0033] Figure 8 and Fig. 9 A diagram illustrating a denoising process according to an embodiment of the present disclosure.

[0034] Fig.10 is a diagram illustrating a multi-scale data block generation process according to an embodiment of the present disclosure.

[0035] Fig.11 is a block diagram illustrating an image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] The terms "include" or "comprising" used in this specification indicate the existence of the stated features, wholes, steps, operations, elements, components and / or combinations thereof, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof. It should be understood that although the terms "first", "second", "third", etc. are used to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of the present disclosure, the first information may be referred to as the second information; and similarly, the second information may also be referred to as the first information. As used herein, the term "in response to..." may be understood to mean "when...", "at..." or "if," depending on the context.

[0038] In addition, the phrase "at least one of..." that appears in the present disclosure includes three types of parallel situations: "any one of...", "any combination of...", and "the whole of..." For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; (3) performing step 1 and step 2.

[0039] Unless otherwise defined, all terms used herein (including technical terms and scientific terms) have the same meaning as those generally understood by ordinary technicians in the field to which the present disclosure belongs based on the understanding of the disclosure of this application, and should not be interpreted in an idealized or overly formal way. Here, the use of the term "may" with respect to examples or embodiments (for example, "... may include") indicates that there is at least one example or embodiment that includes or implements such features, and all examples are not limited to this. Unless otherwise clearly defined, terms in the singular also include plural forms.

[0040] Recently, there are some methods for realizing image super-resolution in the related art, such as post-unsampling method, pre-unsampling method, residual-based SR method, etc. Specifically, the post-unsampling method increases the size of the input LR image to the size of the output HR image through convolution at the network output layer. Although this method can realize the mapping of LR image to HR image with less computation, the learning difficulty of the mapping of this method is high due to the inconsistency of the input image and output image sizes of the network. The pre-unsampling method upsamples the LR image to the size of the HR image through traditional image processing algorithms (such as Bicubic, etc.) before inputting the LR image into the network. Although this method can reduce the difficulty of network learning mapping, it will also introduce noise due to the upsampling of the LR image, thereby causing the quality of the SR result to decrease. In order to solve the noise problem caused by the upsampling of the Pre-unsampling method, the Residual-based SR method first uses a trained network to perform preliminary super-resolution on the LR image to obtain a blurred SR image, subtracts the blurred SR image from the LR image to obtain a residual image, and then trains another network to learn the mapping relationship between the SR image and the residual image. This method of learning the residual can reduce the difficulty of network learning, but this method requires time and resources to train two networks separately. In addition, some methods (such as the Diwa method and the Waveface method) use the frequency domain information extracted from the image to assist the network in learning the mapping from the LR image to the HR image to reduce the difficulty of network learning, but the networks of these methods still need to learn the super-resolution capability of the LR image to the low-frequency component.

[0041] Therefore, although some progress has been made in related image super-resolution work, related research work still models the image SR task as an implicit mapping from LR images to HR images. Although methods to achieve super-resolution from a frequency domain perspective may involve explicit processing or generation of high- and low-frequency information, the essence of these methods is still the super-resolution task from LR images to HR low-frequency images. Such an implicit mapping of low-resolution images to high-resolution images makes the model lack of interpretability, which leads to poor fidelity of super-resolution results.

[0042] In order to at least solve the problems in the above-mentioned related technologies, the present disclosure provides an image processing technology (or an image processing model framework - a Tranformer-based diffusion model (Multi-level Wavelet Diffusion Transformer, MWDT)) with better performance in both fidelity and quality, especially an image processing method, image processing device, electronic device and storage medium for image super-resolution.

[0043] In the present disclosure, image super-resolution processing for LR images is performed by using a framework combining a diffusion model and a Transformer to obtain super-resolution images with high quality and high fidelity. Specifically, by utilizing a multi-level wavelet transform, the traditional image super-resolution processing is split into a mapping of LR images to low-frequency components and the generation of high-frequency components under the guidance of the LR images, the low-frequency components and the high-frequency components are processed separately, and the processed low-frequency components are integrated with the high-frequency components to obtain HR images, thereby achieving the generation of HR images with higher quality and fidelity with a smaller amount of calculation. In the present disclosure, multi-level wavelet transform processing can separate high-frequency components level by level (or layer) and the image super-resolution process is a process of adding high-frequency components. Therefore, the embodiments of the present disclosure perform model modeling in a manner that aligns the super-resolution process. By implementing the embodiments of the present disclosure, an image super-resolution model with better interpretability can be obtained, and the image super-resolution model can obtain processing results with better fidelity and quality.

[0044] Before describing the present disclosure, some terms used in the present disclosure are first explained.

[0045] Diffusion model: a generative model, the principle of which is to add Gaussian noise to the original image in a forward process (or called diffusion process), and obtain the reverse process (or called inverse diffusion process, inference process or denoising process) through learning, wherein the image processing method in the present disclosure to be described later is a processing process iteratively performed in the reverse process of the diffusion model.

[0046] Diffusion Transformer (DiT): A novel diffusion model architecture in which the input data blocks (tokens) are denoised using DiT blocks to obtain denoised data blocks.

[0047] Image block (or patch): an image block obtained by dividing the image for processing, for example, performing convolution operations on each image block.

[0048] Token: A token for Transformer processing obtained by encoding an image block. In the present disclosure, a token is a token encoded using four-dimensional (4D) position coding as will be described below.

[0049] Spatial domain image: can be understood as a spatial domain image in the original space of the image. In the present disclosure, it can be understood as a red, green, and blue (RGB) image.

[0050] Spectrum: can be understood as a frequency domain or frequency domain image, and in the present disclosure, can be understood as a frequency domain image obtained by wavelet transform. Therefore, the wavelet spectrum in the present disclosure can be referred to as a spectrum or frequency domain image.

[0051] In addition, in the present disclosure, as an example, although the drawings are not shown in color, the spatial domain image shown as an example and the image corresponding to the low-frequency component in the spectrum diagram (i.e., the image containing the face shown in the drawings) can be understood as a color RGB image, and the image corresponding to the high-frequency component in the spectrum diagram (i.e., the image with gray shades shown in the drawings) can be understood as a gray image, and the present disclosure is not limited thereto. Figures 1 to 11 An image processing method, an image processing apparatus, an electronic device, and a storage medium for image super-resolution according to an embodiment of the present disclosure are described.

[0052] Figure 1 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure. Figure 2 is a diagram of a MWDT framework according to an embodiment of the present disclosure.

[0053] First, refer to Figure 2 (b) According to an embodiment of the present disclosure, the initial noise image x is transformed by using the Transformer-based diffusion model according to an embodiment of the present disclosure. T Iterative Execution Figure 2 The image processing method of (a) is used until the maximum number of iterations T is completed, thereby obtaining the final output super-resolution image x 0 ,in, Figure 2The image processing of (a) corresponds to Figure 2 (b) is an image processing at one time step. For example, at the tth time step, the noisy spatial domain image x t (Right now, Figure 2 The first spatial image in (a) ) is the noise image corresponding to the spatial domain image output at the t+1th time step, and the output spatial domain image x t-1 (Right now, Figure 2 The second spatial domain image in (a) ) will be added with noise and used for denoising in the t-1th time step, that is, the noisy first spatial domain image in the current iteration step is the noisy image of the initial low-resolution image or the noisy image of the second spatial domain image in the previous iteration step. Figure 1 right Figure 2 The image processing shown in (a) is described in detail.

[0054] Reference Figure 1 In operation S110, the low-resolution noisy first spatial domain image A multi-level discrete wavelet transform (MDWT) is performed to obtain a first multi-level wavelet spectrum. That is, by using MDWT, the spatial domain image as an RGB image is converted into a spectrum image step by step. Figure 3 and Figure 4 Describe this step in detail.

[0055] Figure 3 and Figure 4 2 is a diagram illustrating an example of an MDWT process and an inverse Multi-level discrete wavelet transform (iMDWT) process according to an embodiment of the present disclosure.

[0056] First, refer to Figure 4 (a), in the present disclosure, MDWT processing and iMDWT processing are equivalent inverse processing. Here, MDWT may refer to a process of performing wavelet transform on an image at multiple levels (or layers, hierarchical levels) to separate high-frequency information from low-frequency information. After MDWT, the spatial domain image can be transformed into a frequency domain image (i.e., Figure 4 The wavelet spectrum in (a) of Figure 1). The iMDWT process is similar to the SISR process, which is a process of adding high-frequency information from low-frequency information. After iMDWT, the frequency domain image (i.e., Figure 4The wavelet spectrum in (a) can be transformed back into a spatial domain image. In the present disclosure, the entire super-resolution processing performed on the multi-level wavelet spectrum is an equal symmetrical structure.

[0057] According to an embodiment of the present disclosure, the maximum number of levels of multi-level wavelet transform processing is determined based on the super-resolution enlargement scale for the first spatial domain image.

[0058] For example, the maximum number of levels l for multi-level wavelet transform processing can be determined by the following equation (1):

[0059] l = ceil(log 2 b / a) (1)

[0060] Among them, a can represent the size of the LR image, that is, the size of the LR image is a×a; b can represent the size of the HR image, that is, the size of the HR image is b×b; ceil() can represent a function for rounding up.

[0061] In addition, the magnification factor s=b / a can also be determined based on the sizes of the LR image and the HR image. Figure 4 (b), assuming that a=16 and upscaling factor s=8 of the LR image, the size of the HR image is 128×128 and the maximum number of levels l=3.

[0062] Return to reference Figure 2 According to an embodiment of the present disclosure, when the first spatial domain image input at the tth time step is When , the noise-added low-frequency component obtained after MDWT processing can be determined by the following equation (2): and one or more noisy high frequency components The noisy high-frequency component set X H :

[0063]

[0064] Among them, MDWT() can represent multi-level wavelet transform processing; l is the maximum level of multi-level wavelet transform; high-frequency component Wherein, j may represent the level, LH may represent the lower left, HL may represent the upper right, and HH may represent the lower right, for example, It can represent the high-frequency component in the lower left corner of the j-th wavelet spectrum; the high-frequency component set

[0065] The following uses the four-level wavelet transform as an example to explain this. Figure 3First, a wavelet transform (DWT) is performed on the 128×128 LR image to obtain the fourth-level wavelet spectrogram, wherein the 64×64 RGB image on the upper left of the fourth-level wavelet spectrogram corresponds to the low-frequency spectrogram (i.e., the spectrogram corresponding to the low-frequency component), and the shadow image of the remaining part corresponds to the high-frequency spectrogram (i.e., the spectrogram corresponding to the high-frequency component). Then, a DWT is performed on the 64×64 low-frequency spectrogram in the fourth-level wavelet spectrogram to obtain the third-level wavelet spectrogram, wherein the 32×32 RGB image on the upper left of the third-level wavelet spectrogram corresponds to the low-frequency spectrogram, and the shadow image of the remaining part corresponds to the high-frequency spectrogram. Then, a DWT is performed on the 32×32 low-frequency spectrogram in the third-level wavelet spectrogram to obtain the second-level wavelet spectrogram, wherein the 16×16 RGB image on the upper left of the second-level wavelet spectrogram corresponds to the low-frequency spectrogram, and the shadow image of the remaining part corresponds to the high-frequency spectrogram. Then, DWT is performed on the 16×16 low-frequency spectrum map in the second-level wavelet spectrum map to obtain the first-level wavelet spectrum map, wherein the 8×8 RGB image on the upper left of the first-level wavelet spectrum map corresponds to the low-frequency spectrum map, and the shadow image of the rest corresponds to the high-frequency spectrum map. Finally, the low-frequency spectrum map in the second-level wavelet spectrum map is covered by the first-level wavelet spectrum map, the low-frequency spectrum map in the third-level wavelet spectrum map is covered by the second-level wavelet spectrum map, and the low-frequency spectrum map in the fourth-level wavelet spectrum map is covered by the third-level wavelet spectrum map covered by the second-level wavelet spectrum map, and finally a multi-level wavelet spectrum map is obtained after four DWTs (i.e., MDWT). Here, the wavelet spectrum maps where different image blocks are located in the multi-level wavelet spectrum map have different levels in the multi-level wavelet spectrum map.

[0066] According to an embodiment of the present disclosure, the upper left area in the wavelet spectrum obtained by DWT may be a low-frequency component area corresponding to the low-frequency spectrum, and the upper right area, lower left area and lower right area may be high-frequency component areas corresponding to the high-frequency spectrum.

[0067] Here, although multi-level DWT is performed in a predetermined order and multi-level overlay of wavelet spectrum is performed after performing the multi-level DWT, the present disclosure does not limit the specific operation order of the multi-level DWT and the multi-level overlay, and the specific operation of MDWT processing can also be performed in any other applicable manner or order. For example, multi-level DWT can be performed in other orders, and for another example, each level of overlay can be performed after performing each level of DWT.

[0068] In the following, after performing subsequent processing (e.g., denoising) on ​​the multilevel wavelet spectrum graph obtained by MDWT (i.e., the first multilevel wavelet spectrum graph) to obtain the subsequently processed multilevel wavelet spectrum graph (i.e., the denoised second multilevel wavelet spectrum graph), the HR image is restored by performing iMDWT on the multilevel wavelet spectrum graph obtained by the subsequent processing (i.e., the denoised second multilevel wavelet spectrum graph).

[0069] For example, refer to Figure 3 First, by using the inverse wavelet transform (iWDT), a 16×16 image is obtained based on the 8×8 low-frequency spectrogram and the corresponding high-frequency spectrogram. Then, by using the iWDT, a 32×32 image is obtained based on the 16×16 low-frequency spectrogram (i.e., the 16×16 image obtained last time) and the corresponding high-frequency spectrogram. Then, by using the iWDT, a 64×64 image is obtained based on the 32×32 low-frequency spectrogram (i.e., the 32×32 image obtained last time) and the corresponding high-frequency spectrogram.

[0070] Here, the iMWDT process is similar to the SISR process, in that by integrating the high-frequency subband with the low-frequency subband, the image becomes sharp and the resolution is doubled, and in the present disclosure, modeling SISR in the wavelet spectrum can produce a higher resolution image by giving the LR input and learning the high-frequency wavelet spectrum complementary to the low-frequency subband.

[0071] Like this, by implementing the super-resolution task on the multi-level wavelet spectrum graph, the multi-level wavelet transform converts the spatial domain image into a spectrum image, which can explicitly separate the low-frequency data from the high-frequency data and hand it over to each model for processing separately, making the low-frequency part corresponding to the image generated by the model more direct and the result more consistent. In addition, the model can focus more on sparse high-frequency data and generate more refined image details.

[0072] Return to reference below Figure 1 and Figure 2 Image processing on the obtained multilevel wavelet spectrogram (ie, the first multilevel wavelet spectrogram) is described in detail.

[0073] In operation S120, a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components are obtained by performing block processing on the first multi-level wavelet spectrogram. Here, the noisy data blocks can be understood as data blocks for noisy images, and can also be referred to as noisy (or non-denoised) data blocks. Here, the noisy low-frequency components can also be referred to as low-frequency noisy components, and the noisy high-frequency components can also be referred to as high-frequency noisy components.

[0074] Specifically, the multi-level wavelet spectrum of the noisy image is processed into blocks to obtain noisy data blocks that can be used for Transformer processing, wherein the noisy data blocks include low-frequency noisy data blocks corresponding to the noisy low-frequency components and high-frequency noisy data blocks corresponding to the noisy high-frequency components. Figure 5 This is described in detail.

[0075] In the present disclosure, considering the high computational cost of Transformer, a pyramid tokenizer method is proposed to tokenize the wavelet spectrum graph so as to minimize the number of data blocks without affecting the quality of the result.

[0076] Specifically, considering the sparsity of high-frequency component information and the density of low-frequency component information, a pyramidal patch size is set for frequency subbands at different levels, a smaller patch size is used for frequency components with denser information, and a larger patch size is used for sparse high-frequency components. In the present disclosure, the blocks of the wavelet spectrogram are also referred to as image blocks. Here, in order to more effectively learn the relationship between different frequency components, the patch size should have the same receptive field.

[0077] According to an embodiment of the present disclosure, the block size set P can be determined by the following equation (3):

[0078] P = {p min ×2 j-1},j∈{1,l}, (3)

[0079] Among them, p min It can represent the minimum image block size of the low-frequency component in the multi-level wavelet spectrum graph, which is a hyperparameter that can be set to any positive integer (for example, it can usually be 2 or 4); j can represent the level of the wavelet spectrum graph where the image block is located in the multi-level wavelet spectrum graph; l can represent the maximum level of the multi-level wavelet spectrum graph.

[0080] Therefore, according to an embodiment of the present disclosure, the first-level wavelet spectrum graph in the multi-level wavelet spectrum graph can be divided into blocks with a minimum image block size p min That is to say, the low-frequency component area and the high-frequency component area in the first-level wavelet spectrum are all in accordance with the image block size p min The high-frequency component area of ​​the second-level wavelet spectrum in the multi-level wavelet spectrum can be divided into blocks with a minimum image block size p min ×2. The high-frequency component region in the third-level wavelet spectrum in the multi-level wavelet spectrum can be divided into regions with a minimum image block size p min×4. Therefore, the high-frequency component region in the j-th level wavelet spectrum in the multi-level wavelet spectrum can be divided into multiple image blocks with the minimum image block size p min ×2 j-1 Through such a division method, the multi-level wavelet spectrum diagram can make the image block sizes of the multiple image blocks gradually increase from the upper left to the lower right, thereby presenting a pyramid block shape.

[0081] Reference Figure 5 , assuming Figure 5 The multi-level wavelet spectrum in p min If 2=3, then the image block size set P of the three-level wavelet spectrum graph is {2, 4, 8}.

[0082] Specifically, in the 16×16 first-level wavelet spectrum map, the corresponding image block size is 2×2, so the low-frequency component area and the high-frequency component area of ​​the first-level wavelet spectrum map can all be divided into multiple 2×2 image blocks. In the 32×32 second-level wavelet spectrum map, the corresponding image block size is 4×4, and in the second-level wavelet spectrum map, the low-frequency component area in the upper left covered by the first-level wavelet spectrum map is divided into multiple 2×2 image blocks, and the image including only the three high-frequency component areas in the lower left, upper right and lower right can be understood as the second-level wavelet spectrum map in the multi-level wavelet spectrum map, and can be divided into multiple 4×4 image blocks. In the third-level wavelet spectrum map of 64×64, the corresponding image block size is 8×8. In the third-level wavelet spectrum map, the low-frequency component area in the upper left corner covered by the first-level wavelet spectrum map and the second-level wavelet spectrum map is divided into multiple 2×2 image blocks and multiple 4×4 image blocks. The image including only the three high-frequency component areas in the lower left, upper right and lower right can be understood as the third-level wavelet spectrum map in the multi-level wavelet spectrum map, and can be divided into multiple 8×8 image blocks. In this way, pyramid blocking is implemented for the entire multi-level wavelet spectrum map to obtain image blocks with different image block sizes.

[0083] By using such a pyramid block method for multi-level wavelet spectrograms, smaller image blocks can be image blocks corresponding to dense low-frequency information or low-frequency components, and larger image blocks can be image blocks corresponding to sparse high-frequency information or high-frequency components, so that the information distribution in the image blocks is more balanced. In addition, as the number of levels of each level of the wavelet spectrogram in the multi-level wavelet spectrogram increases, the image block size used to divide the high-frequency component area of ​​the wavelet spectrogram of the corresponding level also increases, that is, the image block size of the high-frequency component area in the low-level (e.g., j-th level) spectrogram is smaller than the image block size of the high-frequency component area in the high-level (e.g., j+1-th level) spectrogram.

[0084] Therefore, according to an embodiment of the present disclosure, the step of obtaining a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum graph comprises: dividing the first multi-level wavelet spectrum graph into a plurality of image blocks with different block sizes, wherein the block size of each image block in the plurality of image blocks is based on the level (j) of the wavelet spectrum graph where the image block is located in the first multi-level wavelet spectrum graph and the minimum block size (p) of the low-frequency components of the first multi-level wavelet spectrum graph. min ) is determined by Figure 5 , the block sizes of multiple image blocks in the first multi-level wavelet spectrum diagram gradually increase in a direction from the upper left to the lower right.

[0085] After obtaining the divided image blocks, in order to adapt the Transformer to the pyramid block mode, the present disclosure also proposes a four-dimensional (4D) position encoding method with spectral information. Figure 6 , the image blocks of the multi-level wavelet spectrum map on the left can be encoded into the 4D position coding information on the right.

[0086] For example, the level information j and component information in the spectrogram and the two-dimensional (2D) absolute position code can be integrated into a 4D position code [j, L|HL|HH|LH, x, y] (j∈[1, l], L=0, HL=1, HH=2, LH=3), and the 4D position code processed by the fully connected (FC) layer is added to the data block. Here, in order to avoid unnecessary details to obscure the present disclosure, the encoding process of converting the image block into the data block is not described in more detail, as long as it can be applied to the embodiments of the present disclosure.

[0087] Here, the value of the first dimension j can represent the level, the value of the second dimension can represent the position of the image block in the spectrum diagram of the current level, 0 represents the upper left L, 1 represents the upper right HL, 2 represents the lower right HH, 3 represents the lower left LH, and the x value of the third dimension can represent the horizontal axis position Pos of the image block h , the y value of the fourth dimension can represent the vertical position Pos of the image block w For example, refer to Figure 6 , the large image block in the upper right corner of the left spectrum The 4D position encoding information is [3,1,Pos h ,Pos w ], which means that the image block is located at the upper right corner of the third-level wavelet spectrum, and the two-dimensional absolute position coordinate is (Pos h ,Pos w ), here, in order to avoid unnecessary details obscuring the present disclosure, the specific values ​​of the absolute position coordinates are not shown.

[0088] Therefore, according to an embodiment of the present disclosure, the step of obtaining a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum graph also includes: obtaining a plurality of noisy data blocks with spectral information by performing four-dimensional position encoding on each of the plurality of image blocks.

[0089] For example, the wavelet spectrogram may be block-coded using a convolution operation (such as Conv2d()) to obtain a plurality of noisy blocks.

[0090] Here, since encoding is performed for each image block and the image block may correspond to a low-frequency component or a high-frequency component, the multiple noisy data blocks obtained may include one or more noisy data blocks corresponding to the noisy low-frequency component and one or more noisy data blocks corresponding to the noisy high-frequency component.

[0091] Like this, in response to the huge amount of computation caused by the self-attention mechanism in Transformer, the above-mentioned pyramid blocking method can significantly reduce the computational burden of Transformer.

[0092] In addition, according to the embodiment of the present disclosure, the way of dividing the multi-level wavelet spectrum graph is not limited to this, and the multi-level wavelet spectrum graph can also be divided into multiple image blocks with the same block size or divided according to other division methods. In addition, according to the embodiment of the present disclosure, the way of encoding the image block into the data block is not limited to this, and any other way can be used to convert between the image block and the data block, as long as the data block can be used for subsequent processing.

[0093] In the present disclosure, considering that the sizes of the input LR images are different in different usage scenarios of the image processing method, the number of data blocks of the noisy spatial domain image obtained based on the LR image may be different. In order to facilitate model learning, a multi-scale data block sampling (Multiple Scale Token Sampling, MSTS) method for different numbers of data blocks is also proposed to process SISR with multiple magnification scales.

[0094] Figure 7 is a diagram illustrating a multi-scale data block sampling process according to an embodiment of the present disclosure.

[0095] In the present disclosure, the multiple noise-added data blocks obtained through the block processing have a first number corresponding to the size of the first spatial domain image, that is, when the embodiments according to the present disclosure are applied, the first number may be any number.

[0096] Reference Figure 7, assuming that the number of multiple noisy data blocks obtained after the block processing is M, then based on the learnable parameters of shape (N, C), the N noisy data blocks of shape (N, C) can be obtained by performing a cross-attention operation on the M noisy data blocks of shape (M, C) of the input. Among them, N can represent the fixed number of data blocks that can be used for the denoising process of the Transformer, and C can represent the channel dimension of the data block.

[0097] Therefore, the step of obtaining multiple noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing denoising processing on the multiple noisy data blocks also includes: obtaining multiple noisy data blocks with a fixed number N for denoising processing by performing a cross-attention operation on multiple noisy data blocks with a first number M.

[0098] According to an embodiment of the present disclosure, the noise-added low-frequency component is obtained as described above and one or more noisy high frequency components The noisy high-frequency component set X H , then, the noisy low-frequency component and the noisy high-frequency component of a plurality of noisy data blocks with a fixed number N can be obtained by the following equations (4) and (5):

[0099]

[0100] in, It can represent the noisy low-frequency component after MSTS; It can represent the noisy high-frequency component of the first-level wavelet spectrum after MSTS. It can represent the noisy high-frequency component of the second-level wavelet spectrogram or higher-level wavelet spectrogram after MSTS; Conv2d() can represent convolution processing.

[0101] Like this, by performing a cross-attention operation on the noisy data blocks obtained by block processing, the number of noisy data blocks can be unified to a fixed number, so that regardless of the size of the input image, a fixed number of data blocks are guaranteed to be input into the subsequent model, which is conducive to the learning of the model and improves the convenience of model learning. In other words, the embodiments of the present disclosure are applicable to images of any scale. In addition, this image processing method that supports multi-scale image super-resolution can effectively process multi-scale tasks without changing the model by means of scale-related modules and parameter sharing main networks.

[0102] Return to reference Figure 1 After obtaining the noisy data block, the noisy data block can be denoised.

[0103] In operation S130, a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components are obtained by performing denoising processing on the plurality of noisy data blocks.

[0104] In the present disclosure, since wavelet transform is used to split image information into low-frequency components and high-frequency components, denoising processing can be performed on data blocks of low-frequency components and high-frequency components respectively under the guidance of LR images to achieve image super-resolution.

[0105] According to an embodiment of the present disclosure, the step of obtaining a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components by denoising the plurality of noisy data blocks comprises: obtaining a noise-free LR image (i.e., Figure 2 (a) ); Based on the LR image, a plurality of denoised data blocks are obtained by performing denoising processing on the low-frequency noisy data blocks and the high-frequency noisy data blocks respectively.

[0106] According to an embodiment of the present disclosure, in a conditional image generation model based on a diffusion model, the input image in each iteration step is a noisy first spatial domain image, and the noise-free low-resolution image here is the image of the first spatial domain image before noise is added.

[0107] Generally, in a conditional image generation model based on a diffusion model, the guiding condition is generally a simple data such as image classification or time step. In the present disclosure, it is proposed to use LR images as guiding conditions in image super-resolution tasks to achieve image super-resolution tasks. In this case, the model needs to have stronger conditional generation capabilities. Therefore, the embodiments of the present disclosure propose a guiding method for image super-resolution methods based on the DiT image generation model. Figure 8 and Fig. 9 is a diagram illustrating a denoising process according to an embodiment of the present disclosure.

[0108] Reference Figure 8 Through the above operations S110 to S120, a low-frequency noisy data block and a high-frequency noisy data block are obtained from the noisy image (ie, the first spatial domain image).

[0109] In addition, the LR data block can be obtained from the LR image without noise by block processing. Specifically, the LR data block can be obtained by directly block-coding and encoding the LR image. For example, the LR image can be divided into blocks with a minimum image block size p minFor example, the LR image can be divided into multiple image blocks and position encoding (for example, 4D position encoding) can be performed on the multiple image blocks. For example, the LR image can be divided into multiple image blocks in the same way as the multi-level wavelet spectrum diagram is divided, and position encoding (for example, 4D position encoding) can be performed on the multiple image blocks. However, the present disclosure is not limited to this. Any method of obtaining LR data blocks can be used as long as the data blocks can be obtained from the image. This will not be described in detail.

[0110] In the present disclosure, since the LR feature distributions required to generate the high-frequency subband and the low-frequency subband are different, in order to better utilize the LR information to guide the generation, the low-frequency subband and the high-frequency subband are processed respectively by a low-frequency decoder and a high-frequency decoder.

[0111] Therefore, according to an embodiment of the present disclosure, based on LR, the step of obtaining multiple denoised data blocks by denoising the low-frequency noisy data blocks and the high-frequency noisy data blocks respectively includes: under the guidance of the low-resolution image, denoising the low-frequency noisy data blocks by a low-frequency decoder to obtain low-frequency denoised data blocks corresponding to the denoised low-frequency components; under the guidance of the low-resolution image, denoising the high-frequency noisy data blocks by a high-frequency decoder to obtain high-frequency denoised data blocks corresponding to the denoised high-frequency components.

[0112] According to an embodiment of the present disclosure, the LR data block and the low-frequency noisy data block corresponding to the noisy low-frequency component are spliced ​​(Cat), and the spliced ​​data block is input into a low-frequency decoder to obtain an output result as a denoising processing result associated with the low-frequency component.

[0113] Here, a low-frequency denoised data block (ie, denoised data block) for denoising a low-frequency component and an LR data block may be separated from the output result.

[0114] Here, the low frequency decoder may include multiple DiT blocks and a layer normalization (Layer Norm) module, wherein, with reference to Fig. 9 , the DiT block may include multiple perception layers, and may obtain an output by performing at least one of a layer normalization operation, a scaling operation, a shift operation, a multi-head self-attention operation, and a multi-layer perceptron (MLP) operation on the input data block based on a time step (Timestep) using each perception layer. Here, in order to avoid unnecessary details obscuring the present disclosure, the denoising process of the DiT block in the related art is not described in detail.

[0115] According to an embodiment of the present disclosure, the LR data block output and separated from the low-frequency decoder and the high-frequency noisy data block corresponding to the noisy high-frequency component (i.e., the noisy data block) are spliced, and the spliced ​​data block related to the high-frequency component is input into the high-frequency decoder to obtain an output result as a denoising process result.

[0116] Here, a high-frequency denoised data block (i.e., a denoised data block) of a denoised high-frequency component and an LR data block for subsequent use can be separated from the output result. Here, the high-frequency decoder may include a plurality of DiT blocks and a layer normalization module, each of which has a similar function to that of the low-frequency decoder, and will not be described repeatedly here.

[0117] According to an embodiment of the present disclosure, the noise-added frequency component is obtained as described above. and Then, we can use the following equations (6) and (7) to calculate the LR image Information lr Guided to obtain denoised low-frequency components and denoised high frequency component G H :

[0118]

[0119] in, The noisy low-frequency component obtained by equation (4) can be expressed as: Can be expressed Figure 2 (a) The unnoised LR image; F H The noisy high-frequency component obtained by equation (4) can be expressed as and the noisy high frequency component obtained by equation (5) A collection of; Concat() can represent concatenation processing.

[0120] Although the present disclosure shows that the denoising processing of the low-frequency decoder and the high-frequency decoder is a serial process, the information of the LR image used by the high-frequency decoder here may be encoded and decoded during the processing of the low-frequency decoder, but the present disclosure is not limited to this, and the processing of the low-frequency decoder and the high-frequency decoder may be performed in parallel, or the processing of the high-frequency decoder may be performed first, and then the processing of the low-frequency decoder may be performed.

[0121] As such, by using a different conditioning strategy from that of simple conditioning data (i.e., LR images have more complex 2D information compared to simple conditioning data (e.g., timestep and class)), LR information can be more effectively extracted and utilized in SISR.

[0122] After the denoised data block including the denoised low-frequency component and the denoised high-frequency component is obtained in operation S130, an inverse blocking process may be performed on the denoised data block to restore the multi-level wavelet spectrogram.

[0123] return Figure 1In operation S140, a second multi-level wavelet spectrogram is obtained by performing inverse blocking processing on the plurality of denoised data blocks. Operation S140 may be understood as the inverse processing of operation S120, that is, operation S140 may include a pyramid inverse blocking operation opposite to the pyramid blocking operation and a multi-scale data block generation operation opposite to the multi-scale data block sampling operation.

[0124] According to an embodiment of the present disclosure, the step of obtaining a second multi-level wavelet spectrum diagram by inversely partitioning multiple denoised data blocks includes: obtaining a second multi-level wavelet spectrum diagram by fully connecting (FC) multiple denoised data blocks.

[0125] For example, refer to Figure 2 (a) By using multiple fully connected layers (e.g., the first fully connected layer FC 1 , the second fully connected layer FC 2 And the third fully connected layer FC 3 ) performs full connection processing on multiple denoised data blocks to obtain a second multi-level wavelet spectrum graph. Here, the fully connected layer The number of is related to the number of levels of multi-level wavelet transform, and each fully connected layer converts the data block of the corresponding level back to the wavelet spectrum of the corresponding level.

[0126] Here, the inverse blocking process of obtaining a wavelet spectrogram based on a plurality of data blocks is opposite to the blocking process, and will not be described in detail.

[0127] In addition, as described above, there may be a situation where the first number of multiple noisy data blocks obtained through block processing is not suitable for transformer processing. In this case, the first number of noisy data blocks are sampled as a fixed number of noisy data blocks. Therefore, according to an embodiment of the present disclosure, the step of obtaining a second multi-level wavelet spectrum diagram by performing inverse block processing on multiple denoised data blocks also includes: obtaining a first number of multiple denoised data blocks for full connection by performing a grid sampling operation on a fixed number of denoised data blocks obtained through inverse block processing. Fig.10 is a diagram illustrating a multi-scale data block generation process according to an embodiment of the present disclosure.

[0128] Reference Fig.10, the N data blocks provided by the low-frequency / high-frequency decoder can be reshaped, and the reshaped data blocks can be grid sampled to sample the 2D input to the required size, and the sampling results can be reshaped to obtain M data blocks. As an example, in the multi-scale data block generation process, the obtained M data blocks can also be fully connected through the FC layer to perform multiple data processing on the input data blocks again, so as to obtain the final M sampled data blocks. Here, the multi-scale data block generation process is opposite to the multi-scale data block sampling process, and is not described in detail. Such a first number of M denoised data blocks obtained by the multi-scale data block generation process can be input into the fully connected layer to obtain a second multi-level wavelet spectrogram.

[0129] According to an embodiment of the present disclosure, the denoised low-frequency component is obtained as described above. and the denoised high frequency component G H Then, the denoised low-frequency components included in the multiple denoised data blocks can be obtained by the following equation (8): and denoised high frequency component G H Convert to wavelet spectrum

[0130]

[0131] Among them, MSTG() can represent the multi-scale data block generation process; FC j ( ) can represent full connection processing; Concat() can represent concatenation processing.

[0132] Return to reference Figure 1 In operation S150, a high-resolution second spatial domain image is obtained by performing an inverse multilevel wavelet transform (iMWDT) on the second multilevel wavelet spectrum. Here, the relevant processing of iMWDT has been described above with reference to operation S110, and will not be described again here. After the second spatial domain image of the current time step is noised, the noisy second spatial domain image can be used as the first spatial domain image for iterative calculation at the next time step.

[0133] According to an embodiment of the present disclosure, the wavelet spectrum is obtained as described above. Then, based on the second multi-level wavelet spectrum, Obtain high-resolution second spatial domain image

[0134]

[0135] Among them, iMWDT() can represent inverse multi-level wavelet transform processing.

[0136] Operations S110 to S150 are iteratively performed using the diffusion model according to the embodiment until a maximum number of iterations is completed, wherein the calculation result obtained in each iteration is noisy and input into the next iteration, thereby restoring the LR image to a high-fidelity, high-quality HR image.

[0137] Like this, image super-resolution on wavelet spectrogram can have the following good technical effects: 1) Better fidelity: the mapping relationship from LR image to low-frequency component spectrogram is simpler and more direct, and the mapping will be explicitly supervised; 2) Better quality: the high-frequency component data can be directly and explicitly processed, generated and supervised, so that more details can be generated, thereby improving the quality of the processing results; 3) Better interpretability: the image information is split into high- and low-frequency parts, which is consistent with the process of image super-resolution.

[0138] In addition, in an embodiment according to the present disclosure, during training of a Transformer-based diffusion model, an adversarial loss function and a reconstruction loss function may be used to determine the loss function of the model, wherein the adversarial loss function is used to optimize the generator and the discriminator.

[0139] According to an embodiment of the present disclosure, the loss function of the discriminator can be determined by the following equations (10) and (11):

[0140]

[0141] in, It can represent the loss of the discriminator; It can represent the loss returned by the discriminator to the generator.

[0142] According to an embodiment of the present disclosure, the generation of frequency domain information can be constrained by the reconstruction loss function of the following equation (12):

[0143]

[0144] Therefore, the final loss function can be obtained as shown in the following equation (13):

[0145]

[0146] Here, λ is a hyperparameter, such as 0.05.

[0147] Through the loss function of the above equation, the diffusion model may be adjusted based on the loss of the image processing method according to the embodiment of the present disclosure during the training of the diffusion model.

[0148] Fig.11 is a block diagram illustrating an image processing apparatus according to an embodiment of the present disclosure.

[0149] Reference Fig.11 An image processing device 1100 includes: a wavelet transform unit 1110, configured to obtain a first multi-level wavelet spectrum diagram by performing multi-level wavelet transform processing on a low-resolution noisy first spatial domain image; a pre-processing unit 1120, configured to obtain a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum diagram; a denoising processing unit 1130, configured to obtain a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components by performing denoising processing on the plurality of noisy data blocks; an inverse blocking processing unit 1140, configured to obtain a second multi-level wavelet spectrum diagram by performing inverse blocking processing on the plurality of denoised data blocks; and an inverse wavelet transform unit 1150, configured to obtain a high-resolution second spatial domain image by performing inverse multi-level wavelet transform processing on the second multi-level wavelet spectrum diagram.

[0150] That is, the wavelet transform unit 1110 may be configured to perform operation S110, the preprocessing unit 1120 may be configured to perform operation S120, the denoising processing unit 1130 may be configured to perform operation S130, the inverse blocking processing unit 1140 may be configured to perform operation S140, and the inverse wavelet transform unit 1150 may be configured to perform operation S150.

[0151] According to an embodiment of the present disclosure, the maximum number of levels of multi-level wavelet transform processing is determined based on the super-resolution enlargement scale for the first spatial domain image.

[0152] According to an embodiment of the present disclosure, the block processing unit 1120 is configured to obtain a plurality of noisy data blocks including a noisy low-frequency component and a noisy high-frequency component through the following operations: dividing the first multi-level wavelet spectrum map into a plurality of image blocks, wherein the block size of each of the plurality of image blocks is determined based on the level of the wavelet spectrum map where the image block is located in the first multi-level wavelet spectrum map and the minimum block size of the low-frequency component of the first multi-level wavelet spectrum map; obtaining a plurality of noisy data blocks with spectral information by performing four-dimensional position encoding on each of the plurality of image blocks, wherein the plurality of noisy data blocks include a low-frequency noisy data block corresponding to the noisy low-frequency component and a high-frequency noisy data block corresponding to the noisy high-frequency component.

[0153] According to an embodiment of the present disclosure, the denoising processing unit 1130 is configured to obtain a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components through the following operations: obtaining a noise-free low-resolution image corresponding to the first spatial domain image; based on the low-resolution image, obtaining a plurality of denoised data blocks by performing denoising processing on low-frequency noisy data blocks and high-frequency noisy data blocks respectively, wherein the plurality of denoised data blocks include low-frequency denoised data blocks corresponding to the denoised low-frequency components and high-frequency denoised data blocks corresponding to the denoised high-frequency components.

[0154] According to an embodiment of the present disclosure, the denoising processing unit 1130 is configured to obtain multiple denoising data blocks through the following operations: under the guidance of the low-resolution image, the low-frequency noisy data block is denoised by a low-frequency decoder to obtain a low-frequency denoised data block corresponding to the denoised low-frequency component; under the guidance of the low-resolution image, the high-frequency noisy data block is denoised by a high-frequency decoder to obtain a high-frequency denoised data block corresponding to the denoised high-frequency component.

[0155] According to an embodiment of the present disclosure, the inverse blocking processing unit 1140 is configured to obtain a second multi-level wavelet spectrum diagram by performing the following operations: obtaining the second multi-level wavelet spectrum diagram by fully connecting multiple denoising data blocks.

[0156] According to an embodiment of the present disclosure, the block processing unit 1120 is also configured to obtain a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by the following operations: by performing a cross-attention operation on a plurality of noisy data blocks having a first number, a plurality of noisy data blocks having a fixed number for denoising processing are obtained.

[0157] According to an embodiment of the present disclosure, the inverse blocking processing unit 1140 is also configured to obtain a second multi-level wavelet spectrum diagram by the following operation: by performing a grid sampling operation on a plurality of denoised data blocks having a fixed number obtained through the inverse blocking processing, a plurality of denoised data blocks having a first number for full connection are obtained.

[0158] According to an embodiment of the present disclosure, the image processing method is iteratively executed by using a diffusion model until a maximum number of iterations is completed, wherein the noisy first spatial domain image in the current iteration step is a noisy image of the initial low-resolution image or a noisy image of the second spatial domain image in the previous iteration step.

[0159] Already in the above reference Figures 1 to 10 The specific operations of the various units of the image processing device 1100 are described in the description, so they will not be described in detail here.

[0160] In addition, it should be understood that the various modules or units in the image processing device 1100 according to the embodiment of the present disclosure may be implemented as hardware components and / or software components. Those skilled in the art may implement various modules or units, for example, using a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), according to the processing performed by the various modules or units defined.

[0161] In addition, according to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising: at least one processor; at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the image processing method as described above.

[0162] According to an embodiment of the present disclosure, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instruction (or instruction set) individually or jointly. The electronic device may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0163] In electronic devices, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0164] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.

[0165] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0166] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0167] In addition, according to an embodiment of the present disclosure, a computer-readable storage medium is also provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image processing method as described above.

[0168] According to an embodiment of the present disclosure, examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in electronic devices such as clients, hosts, proxy devices, servers, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0169] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0170] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims.

Claims

1. An image processing method, comprising: By performing multi-level wavelet transform processing on the low-resolution first spatial domain image containing noise, a first multi-level wavelet spectrum diagram is obtained; By performing block processing on the first multi-level wavelet spectrum graph, a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components are obtained; By performing denoising processing on the multiple noisy data blocks, a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components are obtained; By performing inverse block processing on a plurality of denoised data blocks, a second multi-level wavelet spectrum graph is obtained; By performing inverse multi-level wavelet transform processing on the second multi-level wavelet spectrum image, a high-resolution second spatial domain image is obtained.

2. The method according to claim 1, wherein: The maximum number of levels of multi-level wavelet transform processing is determined based on the super-resolution enlargement scale for the first spatial domain image.

3. The method according to claim 1, wherein: The step of obtaining a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum diagram comprises: Divide the first multi-level wavelet spectrum map into a plurality of image blocks, wherein the block size of each of the plurality of image blocks is determined based on the level of the wavelet spectrum map where the image block is located in the first multi-level wavelet spectrum map and the minimum block size of the low-frequency component of the first multi-level wavelet spectrum map; By performing four-dimensional position encoding on each of the multiple image blocks, multiple noise-added data blocks with spectrum information are obtained. The multiple noisy data blocks include low-frequency noisy data blocks corresponding to the noisy low-frequency components and high-frequency noisy data blocks corresponding to the noisy high-frequency components.

4. The method according to claim 3, wherein: The step of obtaining a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components by denoising the plurality of noisy data blocks comprises: Acquire a low-resolution image free of noise based on the first spatial domain image; Based on the low-resolution image, a plurality of denoised data blocks are obtained by denoising the low-frequency noisy data blocks and the high-frequency noisy data blocks respectively. The multiple denoising data blocks include low-frequency denoising data blocks corresponding to the denoised low-frequency components and high-frequency denoising data blocks corresponding to the denoised high-frequency components.

5. The method according to claim 4, wherein: Based on the low-resolution image, the step of obtaining a plurality of denoised data blocks by denoising the low-frequency noisy data blocks and the high-frequency noisy data blocks respectively comprises: Under the guidance of the low-resolution image, a low-frequency decoder is used to perform denoising on the low-frequency noisy data block to obtain a low-frequency denoised data block corresponding to the denoised low-frequency component; Under the guidance of the low-resolution image, a high-frequency decoder performs denoising processing on the high-frequency noisy data block to obtain a high-frequency denoised data block corresponding to the denoised high-frequency component.

6. The method according to claim 5, wherein: The step of obtaining a second multi-level wavelet spectrum diagram by performing inverse block processing on a plurality of denoised data blocks includes: By fully connecting multiple denoising data blocks, a second multi-level wavelet spectrum graph is obtained.

7. The method according to claim 6, in, The step of obtaining a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum graph further includes: By performing a cross attention operation on a first number of a plurality of noisy data blocks, a plurality of noisy data blocks having a fixed number for denoising processing are obtained, The step of obtaining a second multi-level wavelet spectrum diagram by performing inverse block processing on the plurality of denoised data blocks further includes: A first plurality of denoised data blocks for full connection is obtained by performing a grid sampling operation on a fixed number of denoised data blocks obtained through the inverse blocking process.

8. The method according to claim 1, wherein: The image processing method is iteratively performed by using a diffusion model until a maximum number of iterations is completed, wherein the noisy first spatial domain image in the current iteration step is a noisy image of the initial low-resolution image or a noisy image of the second spatial domain image in the previous iteration step.

9. An image processing device, comprising: A wavelet transform unit is configured to obtain a first multi-level wavelet spectrum diagram by performing a multi-level wavelet transform process on the low-resolution first spatial domain image containing noise; A block processing unit is configured to obtain a plurality of noisy data blocks including noisy low-frequency components and noisy high-frequency components by performing block processing on the first multi-level wavelet spectrum graph; A denoising processing unit configured to obtain a plurality of denoised data blocks including denoised low-frequency components and denoised high-frequency components by performing denoising processing on the plurality of noisy data blocks; an inverse block processing unit, configured to obtain a second multi-level wavelet spectrum graph by performing inverse block processing on a plurality of denoised data blocks; The inverse wavelet transform unit is configured to obtain a high-resolution second spatial domain image by performing an inverse multi-level wavelet transform process on the second multi-level wavelet spectrum map.

10. An electronic device comprising: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, wherein: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 8.

Images