Image deblurring method, device, electronic device and storage medium

Abstract

The present invention relates to the field of image processing technology, and provides an image deblurring method, device, electronic device and storage medium. A denoising zero space diffusion model is used to deblur an image to be processed, and a target deblurred image of the image to be processed is obtained. A high-quality restored image can be estimated under a fuzzy observation value, laying a foundation for further processing in subsequent industries. Moreover, during the processing of the denoising zero space diffusion model, the general solution variables of the deblurring variables satisfy the consistency constraints and the authenticity constraints, and the clarity of the deblurred image can be guaranteed. By decomposing the general solution variables into range space terms and null space terms, the scale of the denoising zero space diffusion model can be reduced, the amount of calculation can be reduced, and the cost of deblurring can be reduced, and the efficiency of deblurring can be improved, providing the possibility of integrating the denoising zero space diffusion model into consumer-level devices.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image deblurring method, device, electronic equipment and storage medium. Background Art

[0002] Images are an important way for people to obtain information. However, in the imaging process, the images obtained are often blurred due to unfavorable factors such as camera shake, movement of objects in the scene, and camera defocus, which greatly hinders the normal use and subsequent processing of the images.

[0003] The process of restoring a blurred image, i.e., deblurring a blurred image, is a fundamental but challenging problem in computer vision and computational photography. Its goal is to restore a blurred image to a clear image. However, most existing image deblurring methods directly remove noise using prior knowledge, which causes the restored image to lack important structural information, resulting in poor restoration results. Summary of the invention

[0004] The present invention provides an image deblurring method, an apparatus, an electronic device and a storage medium, which are used to solve the defects in the prior art.

[0005] The present invention provides an image deblurring method, comprising:

[0006] Deblurring the image to be processed based on a denoising zero space diffusion model to obtain a target deblurred image of the image to be processed;

[0007] The processing process of the denoising null space diffusion model includes:

[0008] Based on the initial variance of the reverse diffusion distribution of the diffusion probability model, re-parameterize the initial mean and the initial variance of the reverse diffusion distribution to obtain a target mean and a target variance; the target mean is determined based on the real image variable and the current defuzzification variable;

[0009] Based on the marginal distribution of the diffusion probability model, the real image variable is estimated to obtain an initial estimation result, and based on the general solution variable of the defuzzification variable corresponding to the target defuzzified image, the initial estimation result is corrected to obtain a correction result; the general solution variable satisfies the consistency constraint and the authenticity constraint; the general solution variable includes a range space item and a null space item, the range space item is determined based on the to-be-processed variable corresponding to the to-be-processed image, and the null space item is determined based on the consistency solution variable of the defuzzification variable;

[0010] The correction result is used as an estimated value of the real image variable, and the previous deblurred variable is iteratively solved from the reverse diffusion distribution until the value of the real image variable is obtained, and the value of the real image variable is used as the target deblurred image.

[0011] According to an image deblurring method provided by the present invention, the general solution variable is expressed based on the following form:

[0012] ;

[0013] in, is the general solution variable, A is the linear operator of the consistency constraint, is the pseudo-inverse operator of the linear operator, y is the variable to be processed, is the identity matrix, is the consistent solution variable, is the range space item, is the null space term.

[0014] According to an image deblurring method provided by the present invention, the target mean and the target variance are expressed based on the following formula:

[0015] ;

[0016] ;

[0017] in, is the target mean, is the target variance, t is the step number of the Markov chain, is the current defuzzified variable, is the real image variable, is the initial variance, is the hyperparameter of the Markov chain at step t, is the hyperparameter of the Markov chain at step t-1.

[0018] According to an image deblurring method provided by the present invention, the mean and variance of the reverse diffusion distribution are obtained based on applying the Bayesian rule to the reverse diffusion distribution.

[0019] According to an image deblurring method provided by the present invention, the target deblurred image obeys the distribution obeyed by the real image corresponding to the image to be processed.

[0020] According to an image deblurring method provided by the present invention, the image to be processed is a color image;

[0021] The deblurring of the image to be processed based on the denoising zero space diffusion model to obtain a target deblurred image of the image to be processed includes:

[0022] Gray-scale the image to be processed to obtain a gray-scale image to be processed;

[0023] Based on the denoising zero space diffusion model, deblurring the grayscale image to be processed to obtain an initial deblurred image;

[0024] The initial deblurred image is colorized to obtain the target deblurred image.

[0025] According to an image deblurring method provided by the present invention, the initial deblurred image is subjected to colorization processing to obtain the target deblurred image, comprising:

[0026] Inputting the initial deblurred image into a colorization processing model to obtain color information output by the colorization processing model, and superimposing the color information onto the initial deblurred image to obtain the target deblurred image;

[0027] The colorization processing model includes an encoder, a pixel decoder, a color decoder and a fusion module;

[0028] The encoder is used to encode the initial deblurred image to obtain image features;

[0029] The pixel decoder is used to extract visual features of the initial deblurred image based on the image features;

[0030] The color decoder is used to extract semantically-aware color embedding features of the initial deblurred image based on the visual features and the color query features;

[0031] The fusion module is used to fuse the visual features with the semantically-perceived color embedding features to obtain the target deblurred image.

[0032] The present invention also provides an image deblurring device, comprising:

[0033] A deblurring module, used for deblurring the image to be processed based on a denoising zero space diffusion model to obtain a target deblurred image of the image to be processed;

[0034] The deblurring module comprises:

[0035] A parameterization unit, configured to re-parameterize an initial mean and an initial variance of the reverse diffusion distribution based on a standard deviation of the reverse diffusion distribution of the diffusion probability model to obtain a target mean and a target variance; the target mean is determined based on a real image variable and a current defuzzification variable;

[0036] A correction module, configured to estimate the real image variable based on the marginal distribution of the diffusion probability model to obtain an initial estimation result, and to correct the initial estimation result based on the general solution variable of the defuzzification variable corresponding to the target defuzzified image to obtain a correction result; the general solution variable satisfies the consistency constraint; the general solution variable includes a range space term and a null space term, the range space term is determined based on the to-be-processed variable corresponding to the to-be-processed image, and the null space term is determined based on the consistency solution variable of the defuzzification variable;

[0037] A solution module is used to use the correction result as an estimated value of the real image variable, and iteratively solve the previous deblurred variable from the reverse diffusion distribution until the value of the real image variable is obtained, and use the value of the real image variable as the target deblurred image.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image deblurring method as described above is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image deblurring method as described above is implemented.

[0040] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image deblurring method as described above is implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The image deblurring method, device, electronic device and storage medium provided by the present invention utilize a denoising zero space diffusion model to deblur the image to be processed, and obtain a target deblurred image of the image to be processed. A high-quality restored image can be estimated under a fuzzy observation value, laying a foundation for further processing in subsequent industries. Moreover, during the processing of the denoising zero space diffusion model, the general solution variables of the deblurring variables satisfy the consistency constraints and the authenticity constraints, and the clarity of the deblurred image can be guaranteed. By decomposing the general solution variables into range space terms and null space terms, the scale of the denoising zero space diffusion model can be reduced, the amount of calculation can be reduced, and the cost of deblurring can be reduced, and the efficiency of deblurring can be improved, providing the possibility of integrating the denoising zero space diffusion model into consumer-level devices. By re-parameterizing the initial mean and initial variance of the reverse diffusion distribution, a clean intermediate state of the range space and null space decomposition can be obtained, which is convenient for obtaining a final clean deblurred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without creative work.

[0044] Figure 1 It is a flowchart of the image deblurring method provided by the present invention;

[0045] Figure 2 It is a structural schematic diagram of a colorization processing model in the image deblurring method provided by the present invention;

[0046] Figure 3 is a structural schematic diagram of an image deblurring device provided by the present invention;

[0047] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Figure 1 is a flow chart of an image deblurring method provided in an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0050] S0, deblurring the image to be processed based on a denoising zero space diffusion model to obtain a target deblurred image of the image to be processed;

[0051] The processing process of the denoising null space diffusion model includes:

[0052] S1, based on the initial variance of the reverse diffusion distribution of the diffusion probability model, re-parameterize the initial mean and the initial variance of the reverse diffusion distribution to obtain a target mean and a target variance; the target mean is determined based on a real image variable and a current defuzzification variable;

[0053] S2, based on the marginal distribution of the diffusion probability model, the real image variable is estimated to obtain an initial estimation result, and based on the general solution variable of the defuzzification variable corresponding to the target defuzzified image, the initial estimation result is corrected to obtain a correction result; the general solution variable satisfies the consistency constraint and the authenticity constraint; the general solution variable includes a range space term and a null space term, the range space term is determined based on the to-be-processed variable corresponding to the to-be-processed image, and the null space term is determined based on the consistency solution variable of the defuzzification variable;

[0054] S3, using the correction result as the estimated value of the real image variable, and iteratively solving the previous deblurred variable from the reverse diffusion distribution until the value of the real image variable is obtained, and using the value of the real image variable as the target deblurred image.

[0055] Specifically, the image deblurring method provided in the embodiment of the present invention is executed by an image deblurring device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0056] Step S0 is executed to first obtain an image to be processed, which is a blurred image with motion blur and needs to be deblurred. It can be a large-size substrate image or an image in other fields, which is not specifically limited here. The image to be processed can be acquired by an image acquisition device. The image to be processed can be a color image or a grayscale image, which is not specifically limited here.

[0057] The image to be processed is input into the Denoising Diffusion Null-Space Model (DDNM), which can obtain and output the target deblurred image of the image to be processed. Among them, DDNM is a zero-shot framework for arbitrary linear IR problems. It only requires a pre-trained off-the-shelf diffusion model as a generative prior, without additional training or network modification. By refining only the null space content during the back diffusion process, diverse results that meet data consistency and authenticity can be generated.

[0058] In the denoising null space diffusion model, the following steps S1-S3 may be performed:

[0059] S1, using the initial variance of the reverse diffusion distribution of the diffusion probability model, reparameterize the initial mean and initial variance of the reverse diffusion distribution to obtain the target mean and target variance. Here, the diffusion probability model is a latent variable model specified by a t-step Markov chain called a diffusion process. The input of the diffusion probability model is a clean image sample, and according to the forward process It refers to the image in the forward t-1 step through the t-1 step, that is, the previous defuzzified variable Add Gaussian noise to get the image of step t, that is, the current deblurred variable , that is:

[0060] ;

[0061] in, is the initial variance of the diffusion probability model, and I is the unit matrix. It is a pre-set hyperparameter, called Noise schedule. It can be called a latent variable, and its dimension is the same as the real image variable The dimensions are the same. Represents a normal distribution.

[0062] Although this particular choice of diffusion process seems arbitrary, the marginal distributions of the diffusion probability model Diffusion distribution with reverse diffusion has the following closed form:

[0063] ;

[0064] ;

[0065] in, , , is the initial mean. and All with , and The relevant amount. is the variance of the diffusion probability model at the s-th step, and s is the s-th step of the diffusion probability model.

[0066] In the embodiment of the present invention, the Bayesian rule can be applied to the reverse diffusion distribution The mean and variance of the diffusion probability model are obtained, namely:

[0067] ;

[0068] .

[0069] Due to the reverse diffusion The process is completely random. is noisy. In order to obtain clean latent variables, the initial mean and initial variance of the back diffusion distribution need to be reparameterized to obtain the target mean and target variance respectively:

[0070] ;

[0071] ;

[0072] in, is the target mean, is the target variance, t is the step number of the Markov chain, is the current defuzzified variable, is the real image variable, is the initial variance, is the hyperparameter of the Markov chain at step t, is the hyperparameter of the Markov chain at step t-1.

[0073] Then, step S2 is performed, using the marginal distribution of the diffusion probability model , for the real image variable Make an estimate and get the initial estimate result . That is:

[0074] ;

[0075] in, Denoising network for diffusion probability model The output, For the denoising network network parameters.

[0076] The denoising network Try to Medium estimate Furthermore, we can Apply the reverse diffusion process in the unknown case , by estimating replace ,have:

[0077] ;

[0078] Through training make Close to true back-diffusion distribution This is achieved by optimizing To maximize the marginal log-likelihood This is achieved by using the variational lower bound of , where x is an arbitrary image variable.

[0079] For the inevitable estimate The conditional model of , y is the processed variable corresponding to the processed image, so that the denoising network Accepts y as a conditional input. In this way, the iterative denoising process depends on y:

[0080] ;

[0081] in, For the denoising network The objective function is is a constant. For expectations, is the L1 norm.

[0082] By running the back diffusion process to sample the image, given ,from ~ Start and repeat the reverse diffusion process to convert .

[0083] Thereafter, the initial estimation result is corrected using the general solution variable of the deblurring variable corresponding to the target deblurred image to obtain a corrected result.

[0084] The general solution variable of the defuzzified variable can be expressed as , the general solution variable satisfies the consistency constraint and the authenticity constraint. The consistency constraint can be expressed as: , A is the linear operator of the consistency constraint, and y is the variable to be processed corresponding to the image to be processed. The authenticity constraint can be expressed as: , is the distribution of the real image corresponding to the image to be processed.

[0085] For consistency constraints, we can use the range space-null space decomposition. Given a linear operator A with consistency constraints, its pseudo-inverse satisfy Solving the pseudo-inverse operator There are many ways to solve the matrix form. For example, singular value decomposition (SVD) can be used to solve , solve the convolution form by Fourier transform .

[0086] For any image variable x, its corresponding degraded image y0, and Ax=y0.

[0087] x can be decomposed into two parts, one in the range space of A and the other in the null space of A, namely:

[0088] ;

[0089] in, is the range space, which becomes y after being acted upon by A. is the null space, and becomes 0 after the operation of A, that is:

[0090] .

[0091] Therefore, for the processed variable y corresponding to the processed image, we can directly construct a The general solution variable ,Right now:

[0092] ;

[0093] Among them, the general solution variables include the range space term and the null space term , the range space item Based on the variable y to be processed corresponding to the image to be processed, the null space term Consistency solution variables based on defuzzified variables Sure.

[0094] Here, regardless of It will not affect the consistency. Decide Whether the authenticity constraint is satisfied .

[0095] Therefore, in the embodiment of the present invention, the goal of image deblurring is to find a suitable Make . Use the diffusion probability model to generate the space term with the range The compatible null space terms .

[0096] By general solution variables , the initial estimate can be Make a correction, that is, fix the range space item to , and keep the null space term unchanged, and get the corrected result for:

[0097] .

[0098] Finally, execute step S3 to correct the result. As Real image variables , thus allowing only the null space terms to participate in the back diffusion process.

[0099] From the reverse diffusion distribution Sampling from :

[0100] ;

[0101] Roughly speaking, yes The noise version of and the null space term Therefore, the previous defuzzified variable is iteratively solved from the reverse diffusion distribution until the true image variable is obtained. The value of can satisfy the authenticity constraint Therefore, the value of the real image variable is used as the target to deblur the image.

[0102] It should be noted that all the correction results All meet the consistency constraint because:

[0103] .

[0104] Considering , so the final real image variable The value of also satisfies the consistency constraint.

[0105] Thus, the denoising null space diffusion model used in the embodiment of the present invention can be used to fill the null space information. , DDNM can solve the deblurring task with any form of linear degradation operator A. It does not require task-specific training or optimization and forms a zero-shot solution for the deblurring task.

[0106] The image deblurring method provided in the embodiment of the present invention utilizes a denoising zero space diffusion model to deblur the image to be processed, and obtains a target deblurred image of the image to be processed. A high-quality restored image can be estimated under a blurred observation value, laying a foundation for further processing in the subsequent industry. Moreover, during the processing of the denoising zero space diffusion model, the general solution variables of the deblurring variables satisfy the consistency constraints and the authenticity constraints, and the clarity of the deblurred image can be guaranteed. By decomposing the general solution variables into the range space terms and the null space terms, the scale of the denoising zero space diffusion model can be reduced, the amount of calculation can be reduced, and the cost of deblurring can be reduced, and the efficiency of deblurring can be improved, providing the possibility of integrating the denoising zero space diffusion model into consumer-level devices. By re-parameterizing the initial mean and initial variance of the reverse diffusion distribution, a clean intermediate state of the range space and null space decomposition can be obtained, which is convenient for obtaining the final clean deblurred image.

[0107] On the basis of the above embodiment, the target deblurred image obeys the distribution obeyed by the real image corresponding to the image to be processed.

[0108] Specifically, since the general solution variables satisfy the authenticity constraint, the target deblurred image finally solved satisfies the authenticity constraint, that is, it obeys the distribution obeyed by the real image.

[0109] Based on the above embodiment, the image to be processed is a color image;

[0110] The deblurring of the image to be processed based on the denoising zero space diffusion model to obtain a target deblurred image of the image to be processed includes:

[0111] Gray-scale the image to be processed to obtain a gray-scale image;

[0112] Based on the denoising zero space diffusion model, deblurring the image to be processed to obtain an initial deblurred image;

[0113] The initial deblurred image is colorized to obtain the target deblurred image.

[0114] Specifically, when the image to be processed is a color image, it is necessary to first grayscale the image to be processed to obtain a grayscale image to be processed. Here, the red (R), green (G) and blue (B) channel values ​​of each pixel in the image to be processed can be weighted averaged to convert them into grayscale values, and finally obtain the grayscale image to be processed.

[0115] The process of image grayscale can help simplify image data, reduce the complexity of processing, and ensure that the model can focus more on the feature extraction of fuzzy details such as the structure of the image in subsequent operations. Moreover, converting a blurred color image to a blurred grayscale image can effectively reduce the number of parameters, speed up the training of the model, help improve the performance and accuracy of the model, improve the efficiency and quality of overall image processing, and make subsequent image analysis and recognition tasks more effective and reliable.

[0116] Afterwards, the denoising zero space diffusion model can be used to deblur the grayscale image to be processed to obtain an initial deblurred image, which is also a grayscale image.

[0117] Finally, the initial deblurred image needs to be colorized to obtain a target deblurred image, which is a color image.

[0118] Based on the above embodiment, the colorization process of the initial deblurred image to obtain the target deblurred image includes:

[0119] Inputting the initial deblurred image into a colorization processing model to obtain color information output by the colorization processing model, and superimposing the color information onto the initial deblurred image to obtain the target deblurred image;

[0120] The colorization processing model includes an encoder, a pixel decoder, a color decoder and a fusion module;

[0121] The encoder is used to encode the initial deblurred image to obtain image features;

[0122] The pixel decoder is used to extract visual features of the initial deblurred image based on the image features;

[0123] The color decoder is used to extract semantically-aware color embedding features of the initial deblurred image based on the visual features and the color query features;

[0124] The fusion module is used to fuse the visual features with the semantically-perceived color embedding features to obtain the target deblurred image.

[0125] Specifically, in an embodiment of the present invention, the initial deblurred image can be colorized by using a colorization processing model, that is, the initial deblurred image is input into the colorization processing model, and the colorization processing model obtains and outputs a target deblurred image.

[0126] like Figure 2 As shown, the colorization processing model may include an encoder 21 , a pixel decoder 22 , a color decoder 23 and a fusion module 24 .

[0127] The encoder 21 may encode the initial deblurred image to obtain image features;

[0128] The pixel decoder 22 can use the image features to extract the visual features of the initial deblurred image. The visual features output by the pixel decoder 22 have the same spatial resolution as the image features. The pixel decoder consists of four stages that gradually expand the image resolution. Each stage includes an upsampling layer and a shortcut layer. Pixel Shuffle is used as the upsampling layer, which rearranges the low-resolution feature map into a high-resolution feature map. The shortcut layer uses convolution to integrate the features of the corresponding stages of the encoder through shortcut connections.

[0129] The complete image feature pyramid is captured through a stepwise upsampling process, and the multi-scale visual features are further used as the input of the color decoder 23 to guide the optimization of the color query features.

[0130] The color decoder 23 can extract semantically-aware color embedding features of the initial deblurred image using visual features and color query features. Here, the color decoder 23 is composed of multiple color decoder blocks (CDBs), each of which receives visual features and color query features as input. Each CDB is built based on an improved transformer decoder.

[0131] The fusion module 24 may fuse the visual features with the semantically perceived color embedding features to obtain color information. The fusion module 24 may be a lightweight module.

[0132] Finally, the color information can be concatenated with the initial deblurred image along the channel dimension to obtain the target deblurred image.

[0133] In the embodiment of the present invention, the colorization processing model realizes realistic image colorization through dual decoders, the pixel decoder 22 can restore the spatial resolution of the initial deblurred image, and the color decoder 23 can use rich visual features to refine the color query, thereby avoiding hand-crafted priors. By making the two decoders work together, the correlation between color and multi-scale semantic representation is established through cross-attention, which can significantly alleviate the color bleeding effect.

[0134] like Figure 3 As shown, based on the above embodiment, an image deblurring device is provided in an embodiment of the present invention, comprising:

[0135] A deblurring module 31 is used to deblur the image to be processed based on a denoising zero space diffusion model to obtain a target deblurred image of the image to be processed;

[0136] The deblurring module 31 comprises:

[0137] A parameterization unit 311 is used to re-parameterize the initial mean and initial variance of the reverse diffusion distribution based on the standard deviation of the reverse diffusion distribution of the diffusion probability model to obtain a target mean and a target variance; the target mean is determined based on the real image variable and the current defuzzification variable;

[0138] A correction module 312 is used to estimate the real image variable based on the marginal distribution of the diffusion probability model to obtain an initial estimation result, and to correct the initial estimation result based on the general solution variable of the defuzzification variable corresponding to the target defuzzified image to obtain a correction result; the general solution variable satisfies the consistency constraint; the general solution variable includes a range space term and a null space term, the range space term is determined based on the to-be-processed variable corresponding to the to-be-processed image, and the null space term is determined based on the consistency solution variable of the defuzzification variable;

[0139] The solving module 313 is used to use the correction result as the estimated value of the real image variable, and iteratively solve the previous deblurred variable from the reverse diffusion distribution until the value of the real image variable is obtained, and use the value of the real image variable as the target deblurred image.

[0140] Based on the above embodiment, the general solution variables are expressed in the following form:

[0141] ;

[0142] in, is the general solution variable, A is the linear operator of the consistency constraint, is the pseudo-inverse operator of the linear operator, y is the variable to be processed, is the identity matrix, is the consistent solution variable, is the range space item, is the null space term.

[0143] Based on the above embodiment, the target mean and the target variance are expressed based on the following formula:

[0144] ;

[0145] ;

[0146] in, is the target mean, is the target variance, t is the step number of the Markov chain, is the current defuzzified variable, is the real image variable, is the initial variance, is the hyperparameter of the Markov chain at step t, is the hyperparameter of the Markov chain at step t-1.

[0147] Based on the above embodiment, the mean and variance of the reverse diffusion distribution are obtained based on applying the Bayesian rule to the reverse diffusion distribution.

[0148] On the basis of the above embodiment, the target deblurred image obeys the distribution obeyed by the real image corresponding to the image to be processed.

[0149] Based on the above embodiment, the image to be processed is a color image;

[0150] Also includes a grayscale module for:

[0151] Gray-scale the image to be processed to obtain a gray-scale image to be processed;

[0152] The deblurring module is specifically used for:

[0153] Based on the denoising zero space diffusion model, deblurring the grayscale image to be processed to obtain an initial deblurred image;

[0154] Also includes colorization modules for:

[0155] The initial deblurred image is colorized to obtain the target deblurred image.

[0156] Based on the above embodiment, the colorization module is specifically used for:

[0157] Inputting the initial deblurred image into a colorization processing model to obtain color information output by the colorization processing model, and superimposing the color information onto the initial deblurred image to obtain the target deblurred image;

[0158] The colorization processing model includes an encoder, a pixel decoder, a color decoder and a fusion module;

[0159] The encoder is used to encode the initial deblurred image to obtain image features;

[0160] The pixel decoder is used to extract visual features of the initial deblurred image based on the image features;

[0161] The color decoder is used to extract semantically-aware color embedding features of the initial deblurred image based on the visual features and the color query features;

[0162] The fusion module is used to fuse the visual features with the semantically-perceived color embedding features to obtain the target deblurred image.

[0163] Specifically, the functions of each module in the image deblurring device provided in the embodiment of the present invention correspond one-to-one to the operation flow of each step in the above method embodiment, and the effects achieved are also consistent. Please refer to the above embodiment for details, which will not be repeated in the embodiment of the present invention.

[0164] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (Processor) 410, a communication interface (Communications Interface) 420, a memory (Memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the image deblurring method provided in the above embodiments.

[0165] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image deblurring method provided in the above embodiments.

[0167] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the image deblurring method provided in the above embodiments.

[0168] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image deblurring method, characterized in that: include: Deblurring the image to be processed based on a denoising zero space diffusion model to obtain a target deblurred image of the image to be processed; The deblurring of the image to be processed based on the denoising zero space diffusion model to obtain a target deblurred image of the image to be processed specifically includes: Based on the initial variance of the reverse diffusion distribution of the diffusion probability model, the initial mean and the initial variance of the reverse diffusion distribution are reparameterized to obtain the target mean and the target variance; the target mean is determined based on the real image variable and the current defuzzification variable; the forward process of the diffusion probability model It is expressed as: ； in, is the initial variance of the diffusion probability model, I is the unit matrix; is a pre-set hyperparameter. represents normal distribution; is the current defuzzified variable, is the previous defuzzified variable; t is the step number of the Markov chain; The target mean and the target variance are expressed based on the following formula: ； ； in, is the target mean, is the target variance, is the real image variable, is the initial variance, is the hyperparameter of the Markov chain at step t, is the hyperparameter of the Markov chain at step t-1; Based on the marginal distribution of the diffusion probability model, the real image variable is estimated to obtain an initial estimation result, and based on the general solution variable of the defuzzification variable corresponding to the target defuzzified image, the initial estimation result is corrected to obtain a correction result; the general solution variable satisfies the consistency constraint and the authenticity constraint; the general solution variable includes a range space item and a null space item, the range space item is determined based on the to-be-processed variable corresponding to the to-be-processed image, and the null space item is determined based on the consistency solution variable of the defuzzification variable; the consistency constraint is expressed as: , the authenticity constraint is expressed as: , is the distribution obeyed by the real image corresponding to the image to be processed; is the general solution variable, A is the linear operator of the consistency constraint, and y is the variable to be processed; The step of estimating the real image variable based on the marginal distribution of the diffusion probability model to obtain an initial estimation result includes: ； in, Denoising network for diffusion probability model The output, For the denoising network The network parameters, is the marginal distribution of the diffusion probability model, is the real image variable, is the initial estimation result; The denoising network Try to Medium estimate ,exist Apply the reverse diffusion process in the unknown case , by estimating replace ,have: ； Through training make Close to true back-diffusion distribution ; This is achieved by optimizing To maximize the marginal log-likelihood This is achieved by the variational lower bound of , where x is an arbitrary image variable; For the inevitable estimate The conditional model, the denoising network Accepts y as a conditional input, and the iterative denoising process depends on y: ； in, For the denoising network The objective function is is a constant, For expectations, is the L1 norm; By running the back diffusion process to sample the image, given ,from ~ Start and repeat the reverse diffusion process to convert ; and reverse diffusion distribution has the following closed form: ； ； in, , , is the initial mean, and All with , and The relevant amount, is the variance of the diffusion probability model at the s-th step, s is the s-th step of the diffusion probability model; The general solution variables are expressed in the following form: ； in, is the pseudo-inverse operator of the linear operator, is the consistent solution variable, is the range space item, is the null space term; The initial estimation result is corrected based on the general solution variable of the deblurring variable corresponding to the target deblurred image to obtain a corrected result, including: fixing the range space term to , and keep the null space term unchanged, and get the corrected result : ； The correction result is used as an estimated value of the real image variable, and the previous deblurred variable is iteratively solved from the reverse diffusion distribution until the value of the real image variable is obtained, and the value of the real image variable is used as the target deblurred image; The image to be processed is a color image; The deblurring of the image to be processed based on the denoising zero space diffusion model to obtain a target deblurred image of the image to be processed includes: Gray-scale the image to be processed to obtain a gray-scale image to be processed; Based on the denoising zero space diffusion model, deblurring the grayscale image to be processed to obtain an initial deblurred image; The initial deblurred image is subjected to colorization processing to obtain the target deblurred image, comprising: Inputting the initial deblurred image into a colorization processing model to obtain color information output by the colorization processing model, and superimposing the color information onto the initial deblurred image to obtain the target deblurred image; The colorization processing model includes an encoder, a pixel decoder, a color decoder and a fusion module; The encoder is used to encode the initial deblurred image to obtain image features; The pixel decoder is used to extract visual features of the initial deblurred image based on the image features; The color decoder is used to extract semantically-aware color embedding features of the initial deblurred image based on the visual features and the color query features; The fusion module is used to fuse the visual features with the semantically-perceived color embedding features to obtain the target deblurred image.

2. The image deblurring method according to claim 1, characterized in that: The mean and variance of the reverse diffusion distribution are obtained based on applying the Bayesian rule to the reverse diffusion distribution.

3. The image deblurring method according to claim 1, characterized in that: The target deblurred image obeys the distribution obeyed by the real image corresponding to the image to be processed.

4. An image deblurring device, characterized in that: include: A deblurring module, used for deblurring the image to be processed based on a denoising zero space diffusion model to obtain a target deblurred image of the image to be processed; The deblurring module comprises: A parameterization unit is used to re-parameterize the initial mean and initial variance of the reverse diffusion distribution based on the standard deviation of the reverse diffusion distribution of the diffusion probability model to obtain a target mean and a target variance; the target mean is determined based on the real image variable and the current defuzzification variable; the forward process of the diffusion probability model It is expressed as: ； in, is the initial variance of the diffusion probability model, I is the unit matrix; is a pre-set hyperparameter. represents normal distribution; is the current defuzzified variable, is the previous defuzzified variable; t is the step number of the Markov chain; The target mean and the target variance are expressed based on the following formula: ； ； in, is the target mean, is the target variance, is the real image variable, is the initial variance, is the hyperparameter of the Markov chain at step t, is the hyperparameter of the Markov chain at step t-1; A correction module is used to estimate the real image variable based on the marginal distribution of the diffusion probability model to obtain an initial estimation result, and to correct the initial estimation result based on the general solution variable of the defuzzification variable corresponding to the target defuzzified image to obtain a correction result; the general solution variable satisfies the consistency constraint and the authenticity constraint; the general solution variable includes a range space item and a null space item, the range space item is determined based on the to-be-processed variable corresponding to the to-be-processed image, and the null space item is determined based on the consistency solution variable of the defuzzification variable; the consistency constraint is expressed as: , the authenticity constraint is expressed as: , is the distribution obeyed by the real image corresponding to the image to be processed; is the general solution variable, A is the linear operator of the consistency constraint, and y is the variable to be processed; The step of estimating the real image variable based on the marginal distribution of the diffusion probability model to obtain an initial estimation result includes: ； in, Denoising network for diffusion probability model The output, For the denoising network The network parameters, is the marginal distribution of the diffusion probability model, is the real image variable, is the initial estimation result; The denoising network Try to Medium estimate ,exist Apply the reverse diffusion process in the unknown case , by estimating replace ,have: ； Through training make Close to true back-diffusion distribution ; This is achieved by optimizing To maximize the marginal log-likelihood This is achieved by the variational lower bound of , where x is an arbitrary image variable; For the inevitable estimate The conditional model, the denoising network Accepts y as a conditional input, and the iterative denoising process depends on y: ； in, For the denoising network The objective function is is a constant, For expectations, is the L1 norm; By running the back diffusion process to sample the image, given ,from ~ Start and repeat the reverse diffusion process to convert ; and reverse diffusion distribution has the following closed form: ； ； in, , , is the initial mean, and All with , and The relevant amount, is the variance of the diffusion probability model at the s-th step, s is the s-th step of the diffusion probability model; The general solution variables are expressed in the following form: ； in, is the pseudo-inverse operator of the linear operator, is the consistent solution variable, is the range space item, is the null space term; The initial estimation result is corrected based on the general solution variable of the deblurring variable corresponding to the target deblurred image to obtain a corrected result, including: fixing the range space term to , and keep the null space term unchanged, and get the corrected result : ； A solving module, used for taking the correction result as an estimated value of the real image variable, and iteratively solving a previous deblurred variable from the reverse diffusion distribution until a value of the real image variable is obtained, and taking the value of the real image variable as the target deblurred image; The image to be processed is a color image; The deblurring of the image to be processed based on the denoising zero space diffusion model to obtain a target deblurred image of the image to be processed includes: Gray-scale the image to be processed to obtain a gray-scale image to be processed; Based on the denoising zero space diffusion model, deblurring the grayscale image to be processed to obtain an initial deblurred image; The initial deblurred image is subjected to colorization processing to obtain the target deblurred image, comprising: Inputting the initial deblurred image into a colorization processing model to obtain color information output by the colorization processing model, and superimposing the color information onto the initial deblurred image to obtain the target deblurred image; The colorization processing model includes an encoder, a pixel decoder, a color decoder and a fusion module; The encoder is used to encode the initial deblurred image to obtain image features; The pixel decoder is used to extract visual features of the initial deblurred image based on the image features; The color decoder is used to extract semantically-aware color embedding features of the initial deblurred image based on the visual features and the color query features; The fusion module is used to fuse the visual features with the semantically-perceived color embedding features to obtain the target deblurred image.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image deblurring method according to any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image deblurring method according to any one of claims 1 to 3 is implemented.

Images