Image denoising method and device, electronic equipment and storage medium

Abstract

The invention discloses an image denoising method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: gradually adding Gaussian noise into an original image in T steps, enabling the original distribution of the original image to be converted into Gaussian distribution, and obtaining T forward noise images; splicing the original image and the Tth noise image to obtain a spliced image, and inputting the spliced image into the diffusion model to obtain predicted noise; determining posterior distribution of the noise images according to Gaussian distribution of the T positive-sequence noise images; selecting the t0th noise image from the T noise images as an initialized image; fitting the posterior distribution with each noise image from the initialized image in the reverse direction of the forward direction so as to remove noise in each noise image and obtain reverse distribution; and removing the noise in the original image according to the original distribution, the prediction noise and the reverse distribution so as to reconstruct the original image and obtain a corresponding target image. According to the invention, the noise in the image can be removed.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image denoising method, device, electronic device and storage medium. Background Art

[0002] With the widespread application of digital images and videos in various fields, denoising technology has become a research field that has attracted much attention. In digital image and video processing, due to interference from sensors and transmission, there is often noise and distortion in images and videos. Noise not only affects the visual effects of pictures and videos, but also affects subsequent image analysis, computer vision and other tasks. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide an image denoising method, device, electronic device and storage medium to remove noise in an image.

[0004] To achieve the above object, an embodiment of the present application provides an image denoising method, which comprises the following steps:

[0005] Adding Gaussian noise gradually to the original image in T steps, so that the original distribution of the original image is converted into a Gaussian distribution, and obtaining T positive noise images;

[0006] splicing the original image and the T-th noise image to obtain a spliced ​​image, and inputting the spliced ​​image into a diffusion model to obtain predicted noise;

[0007] Determine the posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order;

[0008] Selecting the t0th noise image from the T noise images as an initialization image; fitting the posterior distribution with each noise image in the reverse direction of the forward direction starting from the initialization image to remove noise in each noise image, and obtaining a reverse distribution;

[0009] The noise in the original image is removed according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain a corresponding target image.

[0010] In some embodiments, the step of gradually adding Gaussian noise to the original image in T steps so that the original distribution of the original image is converted into a Gaussian distribution to obtain T noise images in a positive sequence includes the following steps:

[0011] Given the original distribution x0~q(x0) of the original image;

[0012] The Gaussian noise is gradually added to the original image x0 in T steps, so as to obtain latent variables x1....x T said Gaussian distribution being isotropic;

[0013] The expression of the Gaussian distribution includes:

[0014]

[0015]

[0016] in, is the Gaussian distribution expression, the mean and standard deviation of the Gaussian distribution are given by x t and β t Function determination; q(x t |x t-1 ) is a positive Gaussian distribution, q(x t-1 |x t ) is the inverse Gaussian distribution.

[0017] In some embodiments, inputting the stitched image into a diffusion model to obtain predicted noise comprises the following steps:

[0018] Using U-Net as the diffusion model, inputting the spliced ​​image into the U-Net, processing the spliced ​​image using a contraction path, an intermediate path, and an expansion path in the U-Net to obtain the predicted noise;

[0019] The U-Net includes 6 layers, each of which includes the contraction path, the intermediate path and the expansion path; each layer includes two residual blocks, each residual block includes two operations, including a group normalization, an activation function and a filter with a size of 3 and a stride of 1; a transformer sine position embedding is added between the two operations so that the residual block determines the time t; except for S6, other layers perform downsampling operations;

[0020] The intermediate path consists of two residual blocks and one attention module, the output of the attention module is the input of the expansion path; the expansion path includes three residual blocks for receiving feature maps of corresponding scales on the contraction path;

[0021] The expression of the U-Net includes:

[0022] M-S6(S5(S4(S3(S2(S1(x))))));

[0023] O k =E k (O k-1 +S N-k+1 );

[0024] Among them, x, S i , M represent the input of the U-Net, the operation of the i-th layer on the contraction path and the input of the intermediate path respectively; k ,E k ,S N-k+1 They respectively represent the output result of the kth layer, the operation of the kth layer of the expansion path, and the output of the corresponding layer of the contraction path; N is set to 6, indicating that the expansion path has 6 layers.

[0025] In some embodiments, the method further comprises the following steps:

[0026] Training the diffusion model using an objective loss function;

[0027] The expression of the objective loss function is:

[0028]

[0029] in,

[0030]

[0031] L new represents the target loss function, L represents the initial loss function, λ represents the hyperparameter, D(I,J) represents the L1 distance between I and J; x0 represents the original image, represents the predicted image of the original image; y represents the image with lower resolution of the original image; E represents the expected value; ∈ represents the real Gaussian noise, which obeys N(0,I); ∈ θ represents the prediction noise; T t represents the time step; Represents a dependency on T t The coefficient of is used to control the noise addition in the forward process; represents a normalization factor used to make the predicted noise and the true noise on the same scale.

[0032] In some embodiments, determining the posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order comprises the following steps:

[0033] Determine the posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order;

[0034] The expression of the posterior distribution is:

[0035]

[0036] in,

[0037]

[0038]

[0039] represents the conditional mean in the forward process; is the time step T t A related parameter that indicates the proportion of noise in the reverse process.

[0040] In some embodiments, selecting the t0th noise image from the T noise images as the initialization image comprises the following steps:

[0041] Determine the t0th noise image according to the following expression;

[0042]

[0043] where α t =1-β t , ∈ represents noise;

[0044] The step of fitting the posterior distribution with each of the noise images starting from the initialization image in the reverse direction of the forward direction to remove noise in each of the noise images to obtain a reverse distribution comprises the following steps:

[0045] The posterior distribution is fitted with each of the noise images in the reverse direction of the forward direction starting from the initialization image to remove the noise in each of the noise images, and the expression of the reverse distribution is obtained as follows:

[0046]

[0047] in,

[0048]

[0049] Represents the conditional probability of reconstructing the original image x0 starting from time step t0 given the low-resolution image y of the original image; represents the latent variable at time step t0 The probability distribution of θ (x t-1 |x t ,y) means that given the current latent variable x t When the condition y is given, the latent variable x of the previous time step is generated. t-1 The probability distribution of θ (x t ,y,t) represents the latent variable x based on the current time step t, the mean value calculated under condition y and time step t, the mean value is determined by the parameter θ of the diffusion model; ∑ θ (x t ,y,t) represents the value based on the current x t , y and t calculated covariance matrix.

[0050] In some embodiments, removing the noise in the original image according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain the corresponding target image comprises the following steps:

[0051] Determine a reconstruction expression for the original distribution, the predicted noise, and the inverse distribution;

[0052] Using the reconstruction expression to remove noise in the original image to reconstruct the original image to obtain a corresponding target image;

[0053] The reconstruction expression is:

[0054]

[0055] The reconstruction process includes the operation of the following expressions:

[0056]

[0057] in, Represents a linear low-pass filter.

[0058] To achieve the above object, another aspect of an embodiment of the present application provides an image denoising device, the device comprising:

[0059] A distribution conversion unit, used for gradually adding Gaussian noise into the original image in T steps, so that the original distribution of the original image is converted into a Gaussian distribution, and T positive noise images are obtained;

[0060] A noise prediction unit, configured to splice the original image and the T-th noise image to obtain a spliced ​​image, and input the spliced ​​image into a diffusion model to obtain predicted noise;

[0061] a posterior determination unit, configured to determine a posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order;

[0062] A reverse fitting unit is used to select the t0th noise image from the T noise images as an initialization image; fit the posterior distribution with each noise image in the reverse direction of the forward direction starting from the initialization image to remove noise in each noise image, and obtain a reverse distribution;

[0063] An image denoising unit is used to remove noise in the original image according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain a corresponding target image.

[0064] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0065] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0066] The embodiments of the present application include at least the following beneficial effects:

[0067] The present application can gradually add Gaussian noise to the original image in T steps, so that the original distribution of the original image is converted into Gaussian distribution, and T positive noise images are obtained; the original image and the T-th noise image are spliced ​​to obtain a spliced ​​image, and the spliced ​​image is input into a diffusion model to obtain predicted noise; the posterior distribution of the noise image is determined according to the Gaussian distribution of the T positive order noise images; the t0-th noise image is selected as the initialization image from the T noise images; the posterior distribution is fitted with each noise image in the forward reverse direction starting from the initialization image to remove the noise in each noise image to obtain a reverse distribution; the noise in the original image is removed according to the original distribution, the predicted noise and the reverse distribution to reconstruct the original image to obtain the corresponding target image. By learning the reverse process of adding noise to the low-resolution image, a high-resolution image is reconstructed from the low-resolution image to remove the noise in the low-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A schematic diagram of a process of an image denoising method provided in an embodiment of the present application;

[0070] Figure 2 An example flowchart of image denoising provided by an embodiment of the present application;

[0071] Figure 3 An example flow chart of positively increasing image noise provided by an embodiment of the present application;

[0072] Figure 4 An example flowchart of reverse image noise removal provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram of the structure of an image denoising device provided in an embodiment of the present application;

[0074] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0076] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0077] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0079] The embodiment of the present application provides an image denoising method, which relates to the field of image processing technology. The image denoising method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the image denoising method, etc., but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] Reference Figure 1 The embodiment of the present application provides an image denoising method, which may include but is not limited to S100 to S140, as follows:

[0082] S100: gradually adding Gaussian noise to the original image in T steps, so that the original distribution of the original image is converted into a Gaussian distribution, and T positive noise images are obtained.

[0083] Furthermore, S100 may include:

[0084] Given the original distribution x0~q(x0) of the original image;

[0085] The Gaussian noise is gradually added to the original image x0 in T steps, so as to obtain latent variables x1....x Tsaid Gaussian distribution being isotropic;

[0086] The expression of the Gaussian distribution includes:

[0087]

[0088]

[0089] in, is the Gaussian distribution expression, the mean and standard deviation of the Gaussian distribution are given by x t and β t Function determination; q(x t |x t-1 ) is a positive Gaussian distribution, q(x t-1 |x t ) is the inverse Gaussian distribution.

[0090] S110: splicing the original image and the T-th noise image to obtain a spliced ​​image, and inputting the spliced ​​image into a diffusion model to obtain predicted noise.

[0091] Furthermore, S110 may include:

[0092] Using U-Net as the diffusion model, inputting the spliced ​​image into the U-Net, processing the spliced ​​image using a contraction path, an intermediate path, and an expansion path in the U-Net to obtain the predicted noise;

[0093] The U-Net includes 6 layers, each of which includes the contraction path, the intermediate path and the expansion path; each layer includes two residual blocks, each residual block includes two operations, including a group normalization, an activation function and a filter with a size of 3 and a stride of 1; a transformer sine position embedding is added between the two operations so that the residual block determines the time t; except for S6, other layers perform downsampling operations;

[0094] The intermediate path consists of two residual blocks and one attention module, the output of the attention module is the input of the expansion path; the expansion path includes three residual blocks for receiving feature maps of corresponding scales on the contraction path;

[0095] The expression of the U-Net includes:

[0096]

[0097] O k =E k (O k-1 +S N-k+1 );

[0098] Among them, x, Si , M represent the input of the U-Net, the operation of the i-th layer on the contraction path and the input of the intermediate path respectively; k ,E k ,S N-k+1 They respectively represent the output result of the kth layer, the operation of the kth layer of the expansion path, and the output of the corresponding layer of the contraction path; N is set to 6, indicating that the expansion path has 6 layers.

[0099] Furthermore, the embodiment of the present application may also include a step of training a diffusion model.

[0100] In this embodiment, the target loss function may be used to train the diffusion model;

[0101] The expression of the objective loss function is:

[0102]

[0103] in,

[0104]

[0105] L new represents the target loss function, L represents the initial loss function, λ represents the hyperparameter, D(I,J) represents the L1 distance between I and J; x0 represents the original image, represents the predicted image of the original image; y represents the image with lower resolution of the original image; E represents the expected value; ∈ represents the real Gaussian noise, which obeys N(0,I); ∈ θ represents the prediction noise; T t represents the time step; Represents a dependency on T t The coefficient of is used to control the noise addition in the forward process; represents a normalization factor used to make the predicted noise and the true noise on the same scale.

[0106] S120: Determine a posterior distribution of the noise image according to the Gaussian distribution of the T noise images in positive order.

[0107] Further, S120 may include:

[0108] Determine the posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order;

[0109] The expression of the posterior distribution is:

[0110]

[0111] in,

[0112]

[0113]

[0114] represents the conditional mean in the forward process; is the time step T t A related parameter that indicates the proportion of noise in the reverse process.

[0115] S130: Selecting the t0th noise image from the T noise images as an initialization image; fitting the posterior distribution with each noise image in the reverse direction of the forward direction starting from the initialization image to remove noise in each noise image and obtain a reverse distribution.

[0116] Further, S130 may include:

[0117] Determine the t0th noise image according to the following expression;

[0118]

[0119] where α t =1-β t , ∈ represents noise;

[0120] The step of fitting the posterior distribution with each of the noise images starting from the initialization image in the reverse direction of the forward direction to remove noise in each of the noise images to obtain a reverse distribution comprises the following steps:

[0121] The posterior distribution is fitted with each of the noise images in the reverse direction of the forward direction starting from the initialization image to remove the noise in each of the noise images, and the expression of the reverse distribution is obtained as follows:

[0122]

[0123] in,

[0124]

[0125] Represents the conditional probability of reconstructing the original image x0 starting from time step t0 given the low-resolution image y of the original image; represents the latent variable at time step t0 The probability distribution of θ (x t-1 |x t ,y) means that given the current latent variable xt When the condition y is given, the latent variable x of the previous time step is generated. t-1 The probability distribution of θ (x t ,y,t) represents the latent variable x based on the current time step t , the mean value calculated under condition y and time step t, the mean value is determined by the parameter θ of the diffusion model; ∑ θ (x t ,y,t) represents the value based on the current x t , y and t calculated covariance matrix.

[0126] S140: removing noise in the original image according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain a corresponding target image.

[0127] Furthermore, S140 may include:

[0128] Determine a reconstruction expression for the original distribution, the predicted noise, and the inverse distribution;

[0129] Using the reconstruction expression to remove noise in the original image to reconstruct the original image to obtain a corresponding target image;

[0130] The reconstruction expression is:

[0131]

[0132] The reconstruction process includes the operation of the following expressions:

[0133]

[0134] in, Represents a linear low-pass filter.

[0135] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0136] De-noising methods in related technologies include average filtering, median filtering, etc. Related technologies often cause images to lose edge details and texture information, and the processed images become blurred and cannot meet the needs of practical applications. In order to make the image denoising effect more efficient and accurate, the diffusion model provides a new approach. This embodiment proposes an improved denoising diffusion probability model for high-resolution reconstruction of solar images. By learning the reverse process of adding noise to a low-resolution image, a high-resolution image is reconstructed from a low-resolution image. The diffusion model describes the evolution of the image through partial differential equations, slowly eliminates the noise in the image, and makes the image smoother. The diffusion model has the ability to handle various types of noise, and the processed image retains more detail information, which can better meet the needs of practical applications. Therefore, it is of great significance to apply the denoising diffusion model for image denoising.

[0137] The quality of solar images plays an important role in solar physics analysis. Therefore, it is very meaningful to improve image resolution based on high-resolution reconstruction technology. This embodiment proposes an improved conditional denoising diffusion probability model based on Markov chain, which can be used for solar image reconstruction. The method reconstructs a high-resolution image from a low-resolution image by learning the reverse process of adding noise to the low-resolution image.

[0138] Reference Figure 2 , an improved conditional denoising diffusion probability model based on Markov chain transforms the original distribution of the high-resolution solar image x0 into a Gaussian distribution by gradually adding Gaussian noise to the high-resolution image x0. This process is a forward process. Under the action of the low-resolution solar image y, the inverse process of the forward process is learned, so that the noisy image is denoised in multiple steps and a high-resolution image is reconstructed. y, x0 and ∈ θ Represent the HMI image, GST (gradient structure tensor) image and predicted noise respectively.

[0139] (1) Forward process:

[0140] Given a high-resolution solar image distribution x0~q(x0), Gaussian noise is gradually added to the solar image x0 in T steps, thereby obtaining latent variables x1....x T Approximately isotropic Gaussian distribution, see the example diagram Figure 3 The forward process is fixed on a Markov chain with T transitions, thus transforming the original data distribution into a known simple distribution.

[0141]

[0142]

[0143] in, is the Gaussian distribution formula. The mean and standard deviation parameters of the Gaussian distribution are given by x t and β t The function gives, β1...βT is an increasing sequence in the range of (0,1), which is used to control the noise addition rate of the forward process. Setting βt to a smaller value means adding noise with a smaller step size. This setting makes the forward q(x t |x t-1 ) is a Gaussian distribution, the reverse q(x t-1 |x t ) is also a Gaussian distribution.

[0144] The property of the forward process allows us to obtain a noisy solar image in closed form at any given t. This property avoids the need to iteratively solve for x from x0 t :

[0145]

[0146] where α t =1-β t , From formula (3), we can get the noisy solar image x t , the relationship between high-resolution solar image x0 and noise ∈:

[0147]

[0148] Noise∈obeys In image processing and generative models (such as diffusion models, generative adversarial networks, etc.), the distribution of noise is usually selected as the standard Gaussian distribution N(0,I) because of its mathematical convenience, universality in statistics, and rationality in processing high-dimensional data. According to equations (1) and (2), using Bayesian theorem, given x in the forward process t When x0 is equal to x, we can get x t-1 The reverse process fits the posterior distribution to complete the denoising operation. The principle of the inverse process will be explained later.

[0149]

[0150]

[0151]

[0152] represents the conditional mean in the forward process; is the time step T t A related parameter that indicates the proportion of noise in the reverse process.

[0153] (2) Reverse process (i.e. the reverse process of the forward process):

[0154] The model learns the inverse process of the forward process and reconstructs high-resolution solar images through the iterations of the inverse process. The inverse process is also fixed on a Markov chain. A hyperparameter t0 (t0 < T) is selected in the inverse process, and the iteration starts from t0. This has two advantages. First, it reduces the number of iterations from T to t0, saving the time for reconstructing solar images. Second, the result of the Gaussian perturbation of the low-resolution image y at time t0 is used as the initialization of the reconstruction. Compared with starting from a Gaussian distribution, it adds more information about the solar image. The improved initialization is expressed as:

[0155]

[0156] Under the condition of y, the reverse process learns p θ (x t-1 |x t ,y) to fit q(x t-1 |x t ,x0), thereby guiding the reconstructed image to approach the high-resolution image corresponding to the low-resolution image. For example, refer to the figure Figure 4 . Finally, the p θ (x0) obtained by the reverse iteration is used to fit the actual distribution q(x0) of the solar image:

[0157]

[0158] where p θ (xt-1|xt,y) can be expressed as a Gaussian transition with a learned mean and a fixed variance .

[0159]

[0160] According to equations (3) and (4), μ θ can be decomposed into a linear combination of x t and the predicted noise ∈ θ . The denoising diffusion implicit model deletes the random term by setting σt to 0. This operation can increase the determinacy of the reverse process. Since the high-resolution reconstruction task of the solar image requires reducing the uncertainty of the reconstruction process, the same setting is also adopted in this embodiment. This enables the reconstruction of high-resolution solar images from latent variables during the fixed process. That is, under the same initialization conditions, it avoids large differences in the detailed features of the results obtained from multiple reconstructions. The reconstruction process of the solar image is expressed as:

[0161]

[0162] In addition, a subsampling strategy in the Markov Chain Monte Carlo method is adopted. It can further accelerate the reconstruction process without affecting the quality of the reconstruction result. That is, during the process from 1 to t0, a subset of length S (S < t0) is extracted and the corresponding noise schedule is selected, where τ is an increasing subsequence of [1,..., t0]. In this way, from to after iterating S steps, the reconstructed image of the low-resolution image y can be obtained.

[0163] To make the reconstruction result clear and faithful to the low-resolution input, attention needs to be paid to the consistency between the input and the output. Therefore, the intermediate steps of the reverse process of the conditional DDPM (Denoising Diffusion Probabilistic Models) are improved. The idea of iterative latent variable refinement (ILVR) is introduced in this paper. After each iteration, the generated is refined by y and used as the input for the next iteration. This ensures that the downsampled image of the reconstructed image x0 is equal to That is, the target reconstructed image has the same low-frequency information as the low-resolution image.

[0164]

[0165] To sum up, the low-resolution solar image y has three purposes: one is to use y obtained from y t0 as the initialization x of the reverse process t0 , increasing the useful information of the initialization; second, it is the condition for guiding the reverse process; third, it is to refine each so that the reconstruction result is faithful to the corresponding low-resolution image. Therefore, through reasonable initialization, guiding the low-resolution image, and refining each reconstruction step, a reconstruction result with enhanced consistency with the input y can be obtained.

[0166] (3) Loss function:

[0167] To enable the diffusion model to better complete the mapping between the low-resolution and high-resolution solar images, the loss function of the model needs to be controlled. During the model training process, it is necessary to fit the true reverse distribution Therefore, the loss function is expressed as:

[0168]

[0169] According to the representation of the mean in Equation (8), the loss function can be simplified as:

[0170]

[0171] Where E represents the expected value; ∈ represents the real Gaussian noise, which obeys N(0,I); ∈ θ represents the prediction noise; T t represents the time step; Represents a dependency on T t The coefficient of is used to control the noise addition in the forward process; represents a normalization factor used to make the predicted noise and the true noise on the same scale.

[0172] In order to enhance the model's prediction ability for high-resolution images x0, this paper discusses a new loss function term on the basis of retaining the existing loss function. Contrast regularization can be well applied to the problem of improving image clarity. This idea is used in the high-resolution reconstruction task of solar images. t ∈ of the moment θ and known Combined with the above, we can get the prediction of x0 at this time according to formula (3): Only predictive The loss calculation of x0 cannot utilize the information of the low-resolution image y, so consider using The loss calculation is for details close to x0 but far from y. It can make full use of the known information of high-resolution and low-resolution images. The features extracted from each image are used to participate in the loss calculation:

[0173]

[0174] Where λ is a hyperparameter to prevent the additional auxiliary loss term from overwhelming L, and is set to 0.1. The value of n is 5, which represents the 5 layers of the pre-trained VGG-19 diffusion model. The hidden features of the image extracted by these five layers are used. ωi is the coefficient that controls the weight of each layer feature. D(I,J) represents the L1 distance between I and J. G i represents the Ith extracted feature.

[0175] LVGG helps improve some of the results. In order to further improve the results of multiple evaluation indicators, the image is directly used for loss calculation. The loss function (i.e., the target loss function) is further improved as follows:

[0176]

[0177] (4) Structure of the diffusion model:

[0178] The diffusion model structure of ICDDPM is a U-Net backbone network used to predict noise at different times. During training, the low-resolution image y is compared with its corresponding noisy high-resolution image spliced ​​as input, the pixel value range is normalized to [-1,1]. In order to facilitate the channel-based y and The low-resolution image is spliced ​​and four bilinear interpolations are performed on it. The predicted value ∈ of the noise added to the high-resolution image x0 θ The output is . During the test, the low-resolution image and its Gaussian perturbation result are connected and input into U-Net. By gradually removing the S prediction noise output by the diffusion model, the reconstructed high-resolution image is finally obtained. The diffusion model consists of three parts: contraction path, intermediate path and expansion path. The contraction path extracts the features of the input image, with a total of 6 layers:

[0179] M-S6(S5(S4(S3(S2(S1(x)))))); (14)

[0180] Among them, x, S i , M represent the input of the diffusion model, the operation of the i-th layer on the contraction path, and the input of the intermediate path, respectively. Each layer contains two residual blocks. Each residual block contains two operations, including a group normalization, an activation function, and a filter of size 3 and stride 1. The transformer sine position embedding is added between the two operations to make the residual block determine the time t. The residual block is used to alleviate the degradation of the diffusion model. Group normalization can reduce the impact of batch size. Except for S6, other layers perform downsampling operations. When the feature map sizes are 16×16 and 8×8, respectively, the contraction paths S5 and S6 add a SA mechanism with 4 attention heads. Multi-head attention helps the model focus on information in different representation subspaces at different positions. The number of channels gradually increases according to the specified channel multiplier.

[0181] The intermediate path consists of two residual blocks and an SA module (i.e., attention module), and the output of the SA module is the input of the expansion path. The expansion path is similar to the contraction path and also contains the SA mechanism. Each layer contains 3 residual blocks for receiving feature maps of the corresponding scale on the contraction path. This can maximize the use of information on the contraction path. Upsampling operations are performed in the first five layers, and the number of channels gradually decreases. As shown in the following formula:

[0182] O k =E k (O k-1 +S N-k+1 ); (15)

[0183] Among them, k ,E k ,S N-k+1They represent the output result of the kth layer, the operation of the kth layer of the expansion path, and the output of the corresponding layer of the contraction path. N is set to 6, indicating that the expansion path has 6 layers.

[0184] This embodiment has the following beneficial effects:

[0185] 1. Apply DDPM (Denoising Diffusion Probabilistic Models) to high-resolution reconstruction of solar images. Based on conditional DDPM, the initialization and intermediate links of the reconstruction process are improved, and the consistency between the reconstruction result and the input image is enhanced;

[0186] 2. Adding a new auxiliary term based on contrast regularization to the loss function improves the reconstruction ability of the model and improves the reconstruction results to a certain extent.

[0187] The present application may also provide another specific implementation as follows:

[0188] Step (1) Preprocessing: Preprocess the input noisy image, such as smoothing, gradient amplitude calculation, etc., to obtain some prior information of the input image. This prior information can provide more accurate basic data for subsequent denoising methods.

[0189] Step (2) uses a diffusion algorithm: The diffusion operator (also known as a thermodynamic evolution operator) is used to reduce the image noise intensity and improve the image clarity without excessive loss of image details. During the diffusion process, the color value and position of the image pixel will change dynamically at the same time.

[0190] Step (3) Design ICDDPM denoising diffusion probability model: ICDDPM uses probability modeling, diffusion process, MCMC sampling, log-likelihood function, SGD optimization and other methods to achieve efficient image denoising and enhancement.

[0191] Step (2) The diffusion algorithm is specifically as follows:

[0192] The basic principles of the diffusion algorithm are as follows:

[0193] 1. Diffusion operator: During the diffusion process, the color value and position of each pixel in the image will change dynamically at the same time. The diffusion process is achieved by applying the diffusion operator (also known as the thermodynamic evolution operator). The general form of the diffusion operator can be expressed as Where u is the grayscale function of the image, D is the diffusion coefficient, is the Laplacian of the image.

[0194] 2. Diffusion coefficient: The diffusion coefficient D controls the speed and degree of the diffusion process. In the denoising process, different denoising effects can be achieved by adjusting the diffusion coefficient. A larger diffusion coefficient can remove noise faster, but may cause loss of image details; a smaller diffusion coefficient can better preserve image details, but it may take longer to remove noise.

[0195] 3. Denoising process: The diffusion algorithm performs denoising based on the local properties of the image. In an image, the boundary of an object is usually represented by a strong gradient, while noise is usually represented by a weak gradient. The diffusion algorithm can achieve the denoising effect by simulating the heat conduction process and reducing the gradient of the boundary.

[0196] 4. Iteration step: Diffusion algorithms are usually performed in an iterative manner. In each iteration, the image pixel value is updated according to the difference between the current image and the neighboring pixels. The number of iterations depends on the required denoising level and the convergence of the algorithm.

[0197] Step (3) designs the ICDDPM denoising diffusion probability model as follows:

[0198] 1. Determine the probability model: DDPM describes the distribution of the image by establishing a probability model, which can be built in an infinite-dimensional function space and can extract high-dimensional features from the image. In this probability model, each pixel is considered to be sampled from a high-dimensional probability distribution.

[0199] 2. Establish the diffusion process: The diffusion process is the core part of DDPM, which provides a key tool for denoising and enhancement. In the diffusion process, the value of each pixel is affected by the values ​​of the surrounding pixels, and the probability distribution of the pixel is updated at each iteration.

[0200] 3. Sampling by MCMC: DDPM uses Markov Chain Monte Carlo (MCMC) sampling to solve the model. MCMC is a probability-based algorithm that can generate samples that conform to the original probability distribution. These samples can be used to estimate the parameters of the original probability distribution.

[0201] 4. Use the log-likelihood function: The log-likelihood function is a key part of DDPM, which is used to evaluate the performance of the model. The larger the log-likelihood function, the better the performance of the model. Therefore, DDPM uses the log-likelihood function to optimize the model parameters and minimize the measurement error.

[0202] 5. Optimize model parameters: DDPM uses the stochastic gradient descent (SGD) algorithm to optimize model parameters, which is an algorithm that can be optimized on large-scale data. Through continuous iteration, the model parameters can be gradually improved and show better performance.

[0203] 6. Prediction and Reconstruction: Finally, DDPM uses the model to predict and reconstruct the image. Through prediction, a clearer image can be obtained; through reconstruction, the damaged image can be restored to its original state.

[0204] Reference Figure 5 The embodiment of the present application further provides an image denoising device, which can implement the above-mentioned image denoising method, and the device includes:

[0205] A distribution conversion unit, used for gradually adding Gaussian noise into the original image in T steps, so that the original distribution of the original image is converted into a Gaussian distribution, and T positive noise images are obtained;

[0206] A noise prediction unit, configured to splice the original image and the T-th noise image to obtain a spliced ​​image, and input the spliced ​​image into a diffusion model to obtain predicted noise;

[0207] a posterior determination unit, configured to determine a posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order;

[0208] A reverse fitting unit is used to select the t0th noise image from the T noise images as an initialization image; fit the posterior distribution with each noise image in the reverse direction of the forward direction starting from the initialization image to remove noise in each noise image, and obtain a reverse distribution;

[0209] An image denoising unit is used to remove noise in the original image according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain a corresponding target image.

[0210] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0211] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image denoising method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0212] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0213] See also Figure 6, Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0214] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0215] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the image denoising method of the embodiment of the present application;

[0216] Input / output interface 603, used to implement information input and output;

[0217] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0218] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0219] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0220] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image denoising method is implemented.

[0221] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0222] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0223] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0224] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0225] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0227] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0228] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0230] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0233] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image denoising method, characterized in that: The method comprises the following steps: Adding Gaussian noise gradually to the original image in T steps, so that the original distribution of the original image is converted into a Gaussian distribution, and obtaining T positive noise images; splicing the original image and the T-th noise image to obtain a spliced ​​image, and inputting the spliced ​​image into a diffusion model to obtain predicted noise; Determine the posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order; Selecting the t0th noise image from the T noise images as an initialization image; fitting the posterior distribution with each noise image in the reverse direction of the forward direction starting from the initialization image to remove noise in each noise image, and obtaining a reverse distribution; The noise in the original image is removed according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain a corresponding target image.

2. The image denoising method according to claim 1, characterized in that: The step of gradually adding Gaussian noise to the original image in T steps so that the original distribution of the original image is converted into a Gaussian distribution, and obtaining T noise images in a positive sequence includes the following steps: Given the original distribution x0~q(x0) of the original image; The Gaussian noise is gradually added to the original image x0 in T steps, so as to obtain latent variables x1....x T said Gaussian distribution being isotropic; The expression of the Gaussian distribution includes: in, is the Gaussian distribution expression, the mean and standard deviation of the Gaussian distribution are given by x t and β t Function determination; q(x t |x t-1 ) is a positive Gaussian distribution, q(x t-1 |x t ) is the inverse Gaussian distribution.

3. The image denoising method according to claim 1, characterized in that: The step of inputting the spliced ​​image into a diffusion model to obtain predicted noise comprises the following steps: Using U-Net as the diffusion model, inputting the spliced ​​image into the U-Net, processing the spliced ​​image using a contraction path, an intermediate path, and an expansion path in the U-Net to obtain the predicted noise; The U-Net includes 6 layers, each of which includes the contraction path, the intermediate path and the expansion path; each layer includes two residual blocks, each residual block includes two operations, including a group normalization, an activation function and a filter with a size of 3 and a stride of 1; a transformer sine position embedding is added between the two operations so that the residual block determines the time t; except for S6, other layers perform downsampling operations; The intermediate path consists of two residual blocks and one attention module, the output of the attention module is the input of the expansion path; the expansion path includes three residual blocks for receiving feature maps of corresponding scales on the contraction path; The expression of the U-Net includes: M-S6(S5(S4(S3(S2(S1(x)))))); THE k =And k (THE k-1 +S N-k+1 ); Among them, x, S i , M represent the input of the U-Net, the operation of the i-th layer on the contraction path and the input of the intermediate path respectively; k ,E k ,S N-k+1 They respectively represent the output result of the kth layer, the operation of the kth layer of the expansion path, and the output of the corresponding layer of the contraction path; N is set to 6, indicating that the expansion path has 6 layers.

4. The image denoising method according to claim 3, characterized in that: The method further comprises the following steps: Training the diffusion model using an objective loss function; The expression of the objective loss function is: in, M new represents the target loss function, M represents the initial loss function, λ represents the hyperparameter, D(I,J) represents the L1 distance between I and J; x0 represents the original image, represents the predicted image of the original image; y represents the image with lower resolution of the original image; E represents the expected value; ∈ represents the real Gaussian noise, which obeys N(0,I); ∈ θ represents the prediction noise; T t represents the time step; Represents a dependency on T t The coefficient of is used to control the noise addition in the forward process; represents a normalization factor used to make the predicted noise and the true noise on the same scale.

5. The image denoising method according to claim 1, characterized in that: Determining the posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order comprises the following steps: Determine the posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order; The expression of the posterior distribution is: in, represents the conditional mean in the forward process; is the time step T t A related parameter that indicates the proportion of noise in the reverse process.

6. The image denoising method according to claim 1, characterized in that: The step of selecting the t0th noise image from the T noise images as the initialization image comprises the following steps: Determine the t0th noise image according to the following expression; where α t =1-β t , ∈ represents noise; The step of fitting the posterior distribution with each of the noise images starting from the initialization image in the reverse direction of the forward direction to remove noise in each of the noise images to obtain a reverse distribution comprises the following steps: The posterior distribution is fitted with each of the noise images starting from the initialization image in the reverse direction of the forward direction to remove the noise in each of the noise images, and the expression of the reverse distribution is obtained as follows: in, Represents the conditional probability of reconstructing the original image x0 starting from time step t0 given the low-resolution image y of the original image; represents the latent variable at time step t0 The probability distribution of θ (x t-1 |x t ,y) means that given the current latent variable x t When the condition y is given, the latent variable x of the previous time step is generated. t-1 The probability distribution of θ (x t ,y,t) represents the latent variable x based on the current time step t , the mean value calculated under condition y and time step t, the mean value is determined by the parameter θ of the diffusion model; ∑ θ (x t ,y,t) represents the value based on the current x t , y and t calculated covariance matrix.

7. The image denoising method according to claim 1, characterized in that: The method of removing the noise in the original image according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain a corresponding target image comprises the following steps: Determine a reconstruction expression for the original distribution, the predicted noise, and the inverse distribution; Using the reconstruction expression to remove noise in the original image to reconstruct the original image to obtain a corresponding target image; The reconstruction expression is: The reconstruction process includes the operation of the following expressions: in, Represents a linear low-pass filter.

8. An image denoising device, characterized in that: The device comprises: A distribution conversion unit, used for gradually adding Gaussian noise into the original image in T steps, so that the original distribution of the original image is converted into a Gaussian distribution, and T positive noise images are obtained; A noise prediction unit, configured to splice the original image and the T-th noise image to obtain a spliced ​​image, and input the spliced ​​image into a diffusion model to obtain predicted noise; a posterior determination unit, configured to determine a posterior distribution of the noise image according to the Gaussian distribution of the noise images in T positive order; A reverse fitting unit is used to select the t0th noise image from the T noise images as an initialization image; fit the posterior distribution with each noise image in the reverse direction of the forward direction starting from the initialization image to remove noise in each noise image, and obtain a reverse distribution; An image denoising unit is used to remove noise in the original image according to the original distribution, the predicted noise and the inverse distribution to reconstruct the original image to obtain a corresponding target image.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images