Image deblurring method and device, electronic equipment and computer readable storage medium

By predicting clear images and autoregressive diffusion probability model processing, structure-guided distribution information is generated, and the problem of insufficient generalization ability of image defuzzing methods in the prior art is solved, and a clearer and more stable image defuzzing effect is achieved.

CN118918036APending Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411063905.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing image defuzzing methods are insufficient and unstable when processing images of different blur types or degrees of blur, and fail to make full use of the information of the blurred image itself, resulting in unsatisfactory processing results.

Method used

Using an image defuzzing method, by obtaining the images to be processed, multiple predicted clear images are predicted, and inputting them into the pre-trained conditional coding network and autoregressive diffusion probability model, image conditional coding vectors and autoregressive image distribution information are generated. Based on this information, the image structure guidance distribution information is calculated, and the defuzzy image is finally obtained through sampling.

Benefits of technology

This method can make full use of blurred image information, have strong generalization ability and stability, generate clearer images, solve the problems of insufficient generalization ability and instability, and preserve the details and texture of blurred image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918036A_ABST
    Figure CN118918036A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of medical science and technology, and provides an image deblurring method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a to-be-processed image; predicting a plurality of predicted clear images from the to-be-processed image; inputting the to-be-processed image into a pre-trained conditional coding network to generate an image conditional coding vector; inputting the image condition coding vector and the to-be-processed image into a pre-trained autoregression diffusion probability model, and predicting to obtain autoregression image distribution information; calculating to obtain image structure guide distribution information according to the autoregression image distribution information and the predicted clear image; and guiding distribution information sampling based on the image structure to obtain a deblurred image. According to the technical scheme, the information of the blurred image is fully utilized, the generalization ability and stability are high, and a clearer image can be generated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to but are not limited to the field of medical technology, and in particular to an image deblurring method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of social economy, people's living standards are constantly improving, science and technology are constantly advancing, and medical standards are also constantly improving. At present, for diseases inside the body, it is often necessary to use relevant medical equipment to take medical images, and then the disease can be diagnosed based on the medical images taken. However, in some cases, the medical images taken may appear blurred. In order to deblur the captured medical images, generation-based methods are generally used. However, generation-based methods do not make full use of the information of the blurred image itself, but only use it as a fixed condition input, which may lead to insufficient generalization or instability when processing images of different blur types or degrees. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] In order to solve the problems mentioned in the above background technology, the embodiments of the present application provide an image deblurring method, device, electronic device and computer-readable storage medium, which fully utilize the information of the blurred image itself, have strong generalization ability and stability, and can generate clearer images.

[0005] In a first aspect, an embodiment of the present application provides an image deblurring method, comprising:

[0006] Get the image to be processed;

[0007] Predicting and obtaining a plurality of predicted clear images from the image to be processed;

[0008] Inputting the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector;

[0009] Inputting the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict and obtain autoregressive image distribution information;

[0010] Calculate image structure guided distribution information according to the autoregressive image distribution information and the predicted clear image;

[0011] The deblurred image is obtained by sampling the distribution information guided by the image structure.

[0012] According to some embodiments of the present application, the step of predicting a plurality of predicted clear images from the image to be processed includes:

[0013] Determining convolution kernels of different scales from the pre-trained multi-scale convolutional neural network;

[0014] Based on the convolution kernels of different scales, feature extraction processing is performed on different areas of the image to be processed to obtain a plurality of predicted clear images.

[0015] According to some embodiments of the present application, inputting the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector includes:

[0016] determining a number of diffusion steps from the autoregressive diffusion probability model;

[0017] The image conditional coding vector is generated according to the conditional coding network using the diffusion steps and the image to be processed.

[0018] According to some embodiments of the present application, the step of inputting the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict the autoregressive image distribution information includes:

[0019] Performing Gaussian distribution sampling on the image to be processed to obtain a noise image;

[0020] Determine a diffusion image according to the noise image and the diffusion step number;

[0021] The autoregressive diffusion probability model is used to perform calculation and prediction processing on the diffusion image and the image to be processed according to the image conditional coding vector to obtain the autoregressive image distribution information.

[0022] According to some embodiments of the present application, the step of calculating the image structure guided distribution information based on the autoregressive image distribution information and the predicted clear image includes:

[0023] Using the predicted clear image as a target structure guide;

[0024] The autoregressive image distribution information is analyzed and processed according to the target structure guidance to obtain the image structure guidance distribution information.

[0025] According to some embodiments of the present application, the method of obtaining a deblurred image based on the image structure guided distribution information sampling includes:

[0026] Determining region information according to the image structure guided distribution information;

[0027] The deblurred image is obtained by extracting the deblurred image from the image structure guided distribution information according to the region information.

[0028] According to some embodiments of the present application, after obtaining the deblurred image by sampling the guided distribution information based on the image structure, the method further includes:

[0029] Determining the current diffusion step number according to the image structure guided distribution information;

[0030] Updating the current diffusion step number to obtain an adjusted diffusion step number;

[0031] The image conditional coding vector is updated according to the adjusted diffusion step number and the image to be processed.

[0032] In a second aspect, the embodiment of the present application further provides an image deblurring device, comprising:

[0033] A first processing module, used for acquiring an image to be processed;

[0034] A second processing module, used for predicting a plurality of predicted clear images from the image to be processed;

[0035] A third processing module is used to input the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector;

[0036] A fourth processing module, used for inputting the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict and obtain autoregressive image distribution information;

[0037] A fifth processing module, configured to calculate image structure guided distribution information according to the autoregressive image distribution information and the predicted clear image;

[0038] The sixth processing module is used to obtain a deblurred image by sampling the guided distribution information based on the image structure.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image deblurring method as described in the first aspect above is implemented.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the image deblurring method as described in the first aspect above.

[0041] According to the image deblurring method of the embodiment provided by the present application, at least the following beneficial effects are achieved: in the process of image deblurring, firstly, the image to be processed is obtained; then, multiple predicted clear images are predicted from the image to be processed; then, the image to be processed is input into the pre-trained conditional coding network, and then the image conditional coding vector is generated; then, the image conditional coding vector and the image to be processed are input into the pre-trained autoregressive diffusion probability model, and then the autoregressive image distribution information is predicted; then, the image structure guided distribution information is calculated based on the autoregressive image distribution information and the predicted clear image; finally, the corresponding deblurred image can be obtained by sampling based on the image structure guided distribution information. Through the above technical scheme, the information of the blurred image itself is fully utilized, it has strong generalization ability and stability, and can generate clearer images. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0043] Figure 1 is a flow chart of an image deblurring method provided by an embodiment of the present application;

[0044] Figure 2 is a specific flow chart of step S200 provided in one embodiment of the present application;

[0045] Figure 3 is a specific flow chart of step S300 provided in one embodiment of the present application;

[0046] Figure 4 is a specific flow chart of step S400 provided in one embodiment of the present application;

[0047] Figure 5 is a specific flow chart of step S500 provided in one embodiment of the present application;

[0048] Figure 6 is a specific flow chart of step S600 provided in one embodiment of the present application;

[0049] Figure 7 is a flow chart of an image deblurring method provided by another embodiment of the present application;

[0050] Figure 8 is a schematic diagram of an image deblurring device provided by an embodiment of the present application;

[0051] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0054] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0056] AI is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0057] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0058] Artificial intelligence is AI. AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] The servers involved in artificial intelligence technology can be independent servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0060] The present application provides an image deblurring method, device, electronic device and computer-readable storage medium. In the process of image deblurring, firstly, the image to be processed is obtained; then, multiple predicted clear images are predicted from the image to be processed; then, the image to be processed is input into a pre-trained conditional coding network, and then an image conditional coding vector is generated; then, the image conditional coding vector and the image to be processed are input into a pre-trained autoregressive diffusion probability model, and then the autoregressive image distribution information is predicted; then, the image structure guided distribution information is calculated based on the autoregressive image distribution information and the predicted clear image; finally, the corresponding deblurred image can be obtained by sampling based on the image structure guided distribution information. Through the above technical solution, the information of the blurred image itself is fully utilized, it has strong generalization ability and stability, and can generate clearer images.

[0061] The image deblurring method provided in the embodiment of the present application relates to the field of medical technology. The image deblurring method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image deblurring method, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0064] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0065] like Figure 1 As shown, Figure 1 Detailed description is a flowchart of an image deblurring method provided by an embodiment of the present application. The image deblurring method includes but is not limited to steps S100 to S600.

[0066] Step S100, obtaining an image to be processed;

[0067] Step S200, predicting and obtaining a plurality of predicted clear images from the image to be processed;

[0068] Step S300, inputting the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector;

[0069] Step S400, inputting the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict and obtain autoregressive image distribution information;

[0070] Step S500, calculating and obtaining image structure guided distribution information according to the autoregressive image distribution information and the predicted clear image;

[0071] Step S600: sampling distribution information based on the image structure to obtain a deblurred image.

[0072] It should be noted that in the process of image deblurring, firstly, the image to be processed is obtained; then, multiple predicted clear images are predicted from the image to be processed; then, the image to be processed is input into the pre-trained conditional coding network, and then the image conditional coding vector is generated; then, the image conditional coding vector and the image to be processed are input into the pre-trained autoregressive diffusion probability model, and then the autoregressive image distribution information is predicted; then, the image structure guided distribution information is calculated based on the autoregressive image distribution information and the predicted clear image; finally, the corresponding deblurred image can be obtained by sampling based on the image structure guided distribution information. Through the above technical solution, the information of the blurred image itself is fully utilized, it has strong generalization ability and stability, and can generate clearer images.

[0073] It is worth noting that the image to be processed may be a medical image, and in the process of obtaining the medical image, the user's permission is required. The user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. Only after the user's separate permission or consent is clearly obtained can the medical image be subsequently deblurred, and the user's relevant privacy information will be kept confidential, and relevant laws, regulations and standards will be respected.

[0074] It is worth noting that multiple predicted clear images can be predicted from the image to be processed; based on the predicted clear images, the subsequent structure-guided distribution can be prepared. And according to the pre-trained conditional coding network and the image to be processed, the image conditional coding vector can be generated; then, according to the pre-trained autoregressive diffusion probability model, the image conditional coding vector and the image to be processed are predicted to obtain the autoregressive image distribution information; according to the autoregressive image distribution information and the predicted clear image, the image structure-guided distribution information can be obtained by calculation; finally, the image structure-guided distribution information is sampled to obtain the corresponding deblurred image, thereby improving the clarity of the image to be processed.

[0075] It is worth noting that based on the above-mentioned technical solution, in the process of generating a deblurred image, the information of the blurred image itself can be fully utilized, which can solve the problem of insufficient generalization ability or instability when processing images of different blur types or degrees, and thus a clear image can be obtained. The obtained clear image retains the information of the blurred image itself, and will not cause distortion in the deblurred image.

[0076] It is worth noting that the embodiment of the present application uses a multi-scale structure guidance mechanism to assist the diffusion probability in restoring the image, which can provide more prior information and constraints to the diffusion probability model, which is conducive to restoring the details and texture of the image at different scales. The embodiment of the present application also uses a dynamic conditional coding mechanism to enhance the utilization of blurred images by the autoregressive diffusion probability model, so that the autoregressive diffusion probability model can better adapt to different types and degrees of blur, and at the same time enhance the capacity and expression ability of the model. The embodiment of the present application can generate more realistic and detailed deblurring results while maintaining a low distortion, and has strong generalization ability and stability.

[0077] In some embodiments, Figure 2 As shown, the above step S200 may include but is not limited to steps S210 to S220.

[0078] Step S210, determining convolution kernels of different scales from a pre-trained multi-scale convolutional neural network;

[0079] Step S220, performing feature extraction processing on different regions of the image to be processed based on convolution kernels of different scales to obtain multiple predicted clear images.

[0080] It should be noted that in the process of predicting multiple predicted clear images from the image to be processed, convolution kernels of different scales are first determined from the pre-trained multi-scale convolutional neural network; then, feature extraction processing is performed on different areas of the image to be processed based on the convolution kernels of different scales, so that multiple predicted clear images can be obtained, which is ready for the subsequent image structure-guided distribution; the multi-scale structure-guided mechanism is used to assist the diffusion probability model to restore the image, which can give the diffusion probability model more prior information and constraints, and is conducive to restoring the details and textures of the image at different scales.

[0081] In some embodiments, Figure 3 As shown, the above step S300 may include but is not limited to step S310 to step S320.

[0082] Step S310, determining the number of diffusion steps from the autoregressive diffusion probability model;

[0083] Step S320, generating an image conditional coding vector according to the conditional coding network using the diffusion steps and the image to be processed.

[0084] It should be noted that in the process of generating an image conditional coding vector based on the pre-trained conditional coding network and the image to be processed, the number of diffusion steps is first determined from the autoregressive diffusion probability model, and then the image conditional coding vector can be generated based on the conditional coding network using the diffusion steps and the image to be processed; based on the image conditional coding vector, the autoregressive diffusion probability model can be better adapted to different types and degrees of blur, while increasing the capacity and expression ability of the model.

[0085] It is worth noting that each autoregressive diffusion probability model corresponds to the number of diffusion steps. By inputting the image to be processed and different diffusion steps into the conditional coding network, different image conditional coding vectors can be generated, which can adapt to different types and degrees of blur and increase the capacity and expression ability of the model.

[0086] In some embodiments, Figure 4 As shown, the above step S400 may include but is not limited to step S410 to step S430.

[0087] Step S410, performing Gaussian distribution sampling on the image to be processed to obtain a noise image;

[0088] Step S420, determining a diffusion image according to the noise image and the diffusion step number;

[0089] Step S430: perform calculation and prediction processing on the diffusion image and the image to be processed using the autoregressive diffusion probability model according to the image conditional coding vector to obtain autoregressive image distribution information.

[0090] It should be noted that, in the process of predicting the image conditional coding vector and the image to be processed according to the pre-trained autoregressive diffusion probability model to obtain the autoregressive image distribution information, first, Gaussian distribution sampling is performed according to the image to be processed to obtain the noise image; then the diffusion image is determined according to the noise image and the number of diffusion steps; finally, the diffusion image and the image to be processed can be calculated and predicted by the autoregressive diffusion probability model according to the image conditional coding vector to obtain the autoregressive image distribution information; based on the autoregressive diffusion probability model, a more realistic and detail-rich deblurring result can be generated while maintaining a low distortion, and it has strong generalization ability and stability.

[0091] For example, a medical noise image can be obtained by sampling a medical image using a Gaussian distribution; then a medical diffusion image can be determined based on the medical noise image and the number of diffusion steps; finally, the medical diffusion image and the medical image are calculated and predicted using an autoregressive diffusion probability model based on the image conditional coding vector to obtain the autoregressive image distribution information of the medical image, in preparation for subsequent medical image deblurring.

[0092] In some embodiments, Figure 5 As shown, the above step S500 may include but is not limited to step S510 to step S520.

[0093] Step S510, using the predicted clear image as a target structure guide;

[0094] Step S520, analyzing and processing the autoregressive image distribution information according to the target structure guidance to obtain image structure guided distribution information.

[0095] It should be noted that, in the process of calculating the image structure guided distribution information based on the autoregressive image distribution information and the predicted clear image, the predicted clear image is first used as the target structure guidance, and then the autoregressive image distribution information is analyzed and processed according to the target structure guidance to obtain the image structure guided distribution information. Through the above technical scheme, the autoregressive image distribution information is analyzed and processed according to the target structure guidance to obtain the image structure guided distribution information, which can provide the diffusion probability model with more prior information and constraints, thereby facilitating the restoration of image details and textures at different scales.

[0096] In some embodiments, Figure 6 As shown, the above step S600 may include but is not limited to step S610 to step S620.

[0097] Step S610, determining region information according to image structure guided distribution information;

[0098] Step S620: extracting a deblurred image from the image structure guided distribution information according to the region information.

[0099] It should be noted that in the process of obtaining a deblurred image based on sampling of the image structure guided distribution information, the region information is first determined based on the image structure guided distribution information; then the deblurred image is extracted from the image structure guided distribution information based on the region information; multiple region information can be determined based on the image structure guided distribution information; and the deblurred image can be extracted from the image structure guided distribution information based on multiple region information, so as to facilitate the subsequent application of the deblurred image for medical purposes.

[0100] In some embodiments, Figure 7 As shown, after executing the above step S600, it may include but not be limited to steps S710 to S730.

[0101] Step S710, determining the current diffusion step number according to the image structure guided distribution information;

[0102] Step S720, updating the current diffusion step number to obtain an adjusted diffusion step number;

[0103] Step S730, updating the image conditional coding vector according to the adjusted diffusion step number and the image to be processed.

[0104] It should be noted that after sampling to obtain the deblurred image, the current diffusion step can be determined based on the image structure-guided distribution information; then the current diffusion step is updated to obtain the adjusted diffusion step; finally, the image conditional coding vector is updated based on the adjusted diffusion step and the image to be processed, so that the model can be updated, so that the deblurred image can be better generated subsequently, making the subsequent images clearer.

[0105] In order to more clearly explain the image deblurring method of the embodiment of the present application, a specific embodiment is proposed below for illustration.

[0106] The embodiment of the present application adopts an unsupervised approach to train the diffusion probability model, without paired data, and only requires a single clear image as a target. Specifically, the embodiment of the present application uses an autoregressive diffusion probability model (Autoregressive Diffus ion Probabilistic Model l, ADPM), and its generation process can be expressed as:

[0107] x0~p(x0)

[0108]

[0109] y=x T +n,n~N(0,σ 2 I)

[0110] Among them, x0 is the target clear image, xt is the image after the t-th step of diffusion, y is the final blurred image, βt is the diffusion coefficient of the t-th step, ∈t and n are the added Gaussian noise, T is the number of diffusion steps, and σ2 is the blur noise variance.

[0111] The embodiment of the present application trains ADPM by maximizing the likelihood function:

[0112]

[0113] Among them, θ is the parameter of ADPM, and pθ(xt|xt-1,y) is an autoregressive distribution conditioned on xt-1 and y, which can be approximated by a deep neural network. The embodiment of the present application can avoid relying on paired data by using a single clear image as the target, and can adaptively generate a blurred image matching the target image as a conditional input.

[0114] The embodiment of the present application adopts a multi-scale structure guidance mechanism to assist the diffusion probability model in restoring the image. Specifically, the embodiment of the present application uses a regression network (RN), which is used to predict multiple clear images of different resolutions from the blurred image as a structure guide. RN can be implemented using a multi-scale convolutional neural network, and its loss function can be expressed as:

[0115]

[0116] The embodiment of the present application adopts a dynamic conditional encoding mechanism to enhance the utilization of the diffusion probability model for blurred images. Specifically, the embodiment of the present application uses a conditional encoding network (CEN), which generates a dynamic conditional encoding vector as an additional input of ADPM based on the current diffusion steps and blurred images. CEN can be implemented with a recurrent neural network, and its output can be expressed as:

[0117] z y =g ψ (y,t)

[0118] Where ψ is a parameter of CEN, gψ(·) is a recurrent neural network, and zt is the conditional encoding vector of the tth step. The embodiment of the present application uses CEN to generate dynamic conditional encoding vectors, which allows ADPM to better adapt to different types and degrees of fuzziness and increase the capacity and expression ability of the model.

[0119] The algorithm flow is as follows:

[0120] Input: blurred image y;

[0121] Output: clear image x^0;

[0122] step:

[0123] Use RN to predict multiple scales of clear images fφ(y)s from y, s=1,…,S;

[0124] Sample a noise image xT~N(0,I) from Gaussian distribution;

[0125] For t = T,…,1, do the following:

[0126] Generate a conditional encoding vector zt=gψ(y,t) from y and t using CEN;

[0127] Use ADPM to predict an autoregressive distribution pθ(xt|xt-1,y) from xt-1 and y;

[0128] Use pθ(xt|xt-1,y) and fφ(y)s to calculate a structure-guided distribution q(xt|xt-1,y);

[0129] Sample a diffused image xt from q(xt|xt-1,y);

[0130] Output x0 as the final clear image x^0=x0.

[0131] Through the above technical solution, the embodiment of the present application uses an unsupervised method to train the diffusion probability model, without paired data, and only needs a single clear image as the target. This can avoid relying on paired data that is difficult to obtain or unrepresentative, and can adaptively generate a blurred image that matches the target image as a conditional input. The embodiment of the present application uses a multi-scale structure guidance mechanism to assist the diffusion probability model in restoring the image. This can provide more prior information and constraints to the diffusion probability model, which is conducive to restoring the details and texture of the image at different scales. The embodiment of the present application uses a dynamic conditional coding mechanism to enhance the utilization of the diffusion probability model for blurred images. This allows the diffusion probability model to better adapt to different types and degrees of blur, and at the same time increase the capacity and expression ability of the model. The embodiment of the present application combines the above three innovative points and proposes an image deblurring method and system based on a diffusion model, which can generate more realistic and detailed deblurring results while maintaining a low distortion, and has strong generalization ability and stability. The present invention has been experimented on multiple public image deblurring data sets, and compared with existing regression-based or generation-based methods, significant advantages and improvements have been achieved.

[0132] In addition, if Figure 8 As shown, an embodiment of the present application further provides an image deblurring device 10, comprising:

[0133] The first processing module 100 is used to obtain an image to be processed;

[0134] The second processing module 200 is used to predict and obtain multiple predicted clear images from the image to be processed;

[0135] The third processing module 300 is used to input the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector;

[0136] The fourth processing module 400 is used to input the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict and obtain autoregressive image distribution information;

[0137] A fifth processing module 500 is used to calculate image structure guided distribution information according to the autoregressive image distribution information and the predicted clear image;

[0138] The sixth processing module 600 is used to obtain a deblurred image based on the image structure guided distribution information sampling. It should be noted that in the process of image deblurring, firstly, the image to be processed is obtained; then, multiple predicted clear images are predicted from the image to be processed; then, the image to be processed is input into the pre-trained conditional coding network, and then the image conditional coding vector is generated; then, the image conditional coding vector and the image to be processed are input into the pre-trained autoregressive diffusion probability model, and then the autoregressive image distribution information is predicted; then, the image structure guided distribution information is calculated based on the autoregressive image distribution information and the predicted clear image; finally, the corresponding deblurred image can be obtained based on the image structure guided distribution information sampling. Through the above technical solution, the information of the blurred image itself is fully utilized, it has strong generalization ability and stability, and can generate clearer images.

[0139] It is worth noting that the image to be processed may be a medical image, and in the process of obtaining the medical image, the user's permission is required. The user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. Only after the user's separate permission or consent is clearly obtained can the medical image be subsequently deblurred, and the user's relevant privacy information will be kept confidential, and relevant laws, regulations and standards will be respected.

[0140] It is worth noting that multiple predicted clear images can be predicted from the image to be processed; based on the predicted clear images, the subsequent structure-guided distribution can be prepared. And according to the pre-trained conditional coding network and the image to be processed, the image conditional coding vector can be generated; then, according to the pre-trained autoregressive diffusion probability model, the image conditional coding vector and the image to be processed are predicted to obtain the autoregressive image distribution information; according to the autoregressive image distribution information and the predicted clear image, the image structure-guided distribution information can be obtained by calculation; finally, the image structure-guided distribution information is sampled to obtain the corresponding deblurred image, thereby improving the clarity of the image to be processed.

[0141] It is worth noting that based on the above-mentioned technical solution, in the process of generating a deblurred image, the information of the blurred image itself can be fully utilized, which can solve the problem of insufficient generalization ability or instability when processing images of different blur types or degrees, and thus a clear image can be obtained. The obtained clear image retains the information of the blurred image itself, and will not cause distortion in the deblurred image.

[0142] It is worth noting that the embodiment of the present application uses a multi-scale structure guidance mechanism to assist the diffusion probability in restoring the image, which can provide more prior information and constraints to the diffusion probability model, which is conducive to restoring the details and texture of the image at different scales. The embodiment of the present application also uses a dynamic conditional coding mechanism to enhance the utilization of blurred images by the autoregressive diffusion probability model, so that the autoregressive diffusion probability model can better adapt to different types and degrees of blur, and at the same time enhance the capacity and expression ability of the model. The embodiment of the present application can generate more realistic and detailed deblurring results while maintaining a low distortion, and has strong generalization ability and stability.

[0143] The specific implementation of the image deblurring device 10 is substantially the same as the specific implementation of the above-mentioned image deblurring method, and will not be described in detail herein.

[0144] In addition, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 700 , which includes: a memory 720 , a processor 710 , and a computer program stored in the memory 720 and executable on the processor 710 .

[0145] The processor 710 and the memory 720 may be connected via a bus or in other ways.

[0146] The non-transient software program and instructions required to implement the image deblurring method of the above embodiments are stored in the memory 720. When executed by the processor 710, the image deblurring method of the above embodiments is executed, for example, the image deblurring method described above is executed. Figure 1 The method steps S100 to S600, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S520 of the method, Figure 6 Method steps S610 to S620 and Figure 7 Method steps S710 to S730 in.

[0147] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor 710 or a controller, for example, by a processor 710 in the above-mentioned device embodiment, so that the processor 710 can execute the image deblurring method in the above-mentioned embodiment, for example, execute the above-mentioned Figure 1 The method steps S100 to S600, Figure 2 Steps S210 to S220 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S520 of the method, Figure 6 Method steps S610 to S620 and Figure 7 Method steps S710 to S730 in.

[0149] The above embodiments may be used in combination, and modules with the same name in different embodiments may be the same or different.

[0150] The above describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0152] The apparatus, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be repeated here.

[0153] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and produce dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs. The original code before compilation must also be written in a specific programming language, which is called Hardware Description Language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Basic Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCall, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used one is VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language). Those skilled in the art should also be aware that by programming the method flow in the above-mentioned hardware description languages ​​and programming it into an integrated circuit, a hardware circuit for implementing the logic method flow can be easily obtained.

[0154] The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0155] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0156] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0158] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0162] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0163] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (Transitory Med ia), such as modulated data signals and carrier waves.

[0164] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0165] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0166] Embodiments of the present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0167] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0168] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image deblurring method, characterized in that: The method comprises: Get the image to be processed; Predicting and obtaining a plurality of predicted clear images from the image to be processed; Inputting the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector; Inputting the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict and obtain autoregressive image distribution information; Calculate image structure guided distribution information according to the autoregressive image distribution information and the predicted clear image; The deblurred image is obtained by sampling the distribution information guided by the image structure.

2. The image deblurring method according to claim 1, characterized in that: The step of predicting a plurality of predicted clear images from the image to be processed comprises: Determining convolution kernels of different scales from the pre-trained multi-scale convolutional neural network; Based on the convolution kernels of different scales, feature extraction processing is performed on different areas of the image to be processed to obtain a plurality of predicted clear images.

3. The image deblurring method according to claim 1, characterized in that: The step of inputting the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector comprises: determining a number of diffusion steps from the autoregressive diffusion probability model; The image conditional coding vector is generated according to the conditional coding network using the diffusion steps and the image to be processed.

4. The image deblurring method according to claim 3, characterized in that: The step of inputting the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict and obtain autoregressive image distribution information includes: Performing Gaussian distribution sampling on the image to be processed to obtain a noise image; Determine a diffusion image according to the noise image and the diffusion step number; The autoregressive diffusion probability model is used to perform calculation and prediction processing on the diffusion image and the image to be processed according to the image conditional coding vector to obtain the autoregressive image distribution information.

5. The image deblurring method according to claim 1, characterized in that: The step of calculating the image structure guided distribution information according to the autoregressive image distribution information and the predicted clear image includes: Using the predicted clear image as a target structure guide; The autoregressive image distribution information is analyzed and processed according to the target structure guidance to obtain the image structure guidance distribution information.

6. The image deblurring method according to claim 1, characterized in that: The method of obtaining a deblurred image by sampling the distribution information guided by the image structure comprises: Determining region information according to the image structure guided distribution information; The deblurred image is obtained by extracting the deblurred image from the image structure guided distribution information according to the region information.

7. The image deblurring method according to claim 1, characterized in that: After obtaining the deblurred image by sampling the distribution information guided by the image structure, the method further includes: Determining the current diffusion step number according to the image structure guided distribution information; Updating the current diffusion step number to obtain an adjusted diffusion step number; The image conditional coding vector is updated according to the adjusted diffusion step number and the image to be processed.

8. An image deblurring device, characterized in that: include: A first processing module, used for acquiring an image to be processed; A second processing module, used for predicting a plurality of predicted clear images from the image to be processed; A third processing module is used to input the image to be processed into a pre-trained conditional coding network to generate an image conditional coding vector; A fourth processing module, used for inputting the image conditional coding vector and the image to be processed into a pre-trained autoregressive diffusion probability model to predict and obtain autoregressive image distribution information; A fifth processing module, configured to calculate image structure guided distribution information according to the autoregressive image distribution information and the predicted clear image; The sixth processing module is used to obtain a deblurred image by sampling the guided distribution information based on the image structure.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image deblurring method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the image deblurring method according to any one of claims 1 to 7.