Image deblurring method, device, equipment and readable storage medium

Abstract

The present invention provides an image deblurring method, device, equipment and readable storage medium. The image deblurring method includes: uniformly modeling the removal process of complex blur kernels in a multi-scale generative network, and gradually generating a restored image corresponding to the blurred image from coarse to fine. At the same time, the generative network of each scale in the multi-scale generative network adopts a global jump connection, so that the trained multi-scale generative network is more sensitive to the details that should be learned, and the details are the details that the multi-scale generative network should add to the input image. Then, an end-to-end adversarial deep learning network is constructed, and the adversarial loss, content loss and structural similarity loss are combined to obtain a joint loss to guide the training of the multi-scale generative network. When the joint loss of the multi-scale generative network converges, a trained multi-scale generative network is obtained. Based on the trained multi-scale generative network, the quality of the restored images of different scales can be improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and image processing, and in particular to an image deblurring method, device, equipment and readable storage medium. Background Art

[0002] Image processing has always been an important topic in the computer field, and is closely related to daily life, military, aerospace, biology and many other fields. Among them, image deblurring is a technology that restores blurred images formed due to various reasons into clear images. In 1964, NASA used image deblurring technology to process photos of the moon, which was a successful application of this technology. Since then, image deblurring has been widely used in various fields. For example: in the field of transportation, it is necessary to take pictures of license plates, but due to excessive speed or long exposure time, blurriness is caused. Through image deblurring technology, a clear license plate image can be obtained to improve the recognition accuracy; in the field of aerospace, the relative speed and long distance between the spacecraft and the ground may cause blurriness, and image deblurring technology can improve the accuracy of satellite positioning; in addition, such as fingerprint and face recognition in the field of criminal investigation, target recognition in the military field and scanning imaging in the medical field, image deblurring technology plays an important role.

[0003] Image blur can be abstracted as the result of convolution operation between the original image and the blur kernel operator, while deblurring is the inverse process. Depending on the blurring method, the blur kernel is also different. Depending on whether the blur kernel is known, the mainstream image deblurring methods can be divided into two categories: non-blind deblurring and blind deblurring. Non-blind deblurring performs inverse convolution on the blurred image according to the known blur kernel, which can achieve very good results in certain cases. Since the blur kernel is generally unknown in actual situations, this type of method has great limitations and is not suitable for image deblurring tasks in various complex scenes. Blind deblurring requires a model to estimate the blur kernel, but since the blur kernel in natural scenes is not single, different positions in the same scene may have different types of blur, such as local blur and overall blur, motion blur and defocus blur, and various blurs may also overlap. It is difficult for general blur kernel estimation methods to correctly simulate the real situation. In addition, due to the complex calculation process, many constraints, and difficulty in determining hyperparameters, the reliability of traditional blind deblurring methods in various complex scenes is difficult to guarantee. With the development of deep learning technology, blind deblurring algorithms based on convolutional neural networks have gradually emerged and achieved excellent results. However, using convolutional neural networks to estimate blur kernels also brings the problem of too many parameters, resulting in low efficiency and difficulty in applying to fields with real-time processing requirements. Summary of the invention

[0004] The main purpose of the present invention is to provide an image deblurring method, device, equipment and readable storage medium, aiming to solve the technical problem in the prior art that the image deblurring method based on traditional convolutional neural network generates too many parameters when removing complex blur kernels, is inefficient and difficult to apply in fields with real-time processing requirements.

[0005] In a first aspect, the present invention provides an image deblurring method, the image deblurring method comprising the following steps:

[0006] Inputting the original blurred image into a trained multi-scale generative network to obtain a restored image corresponding to the original blurred image, wherein the multi-scale generative network includes a preset number of generative networks of different scales;

[0007] The step of inputting the original blurred image into the trained multi-scale generation network to obtain a restored image corresponding to the original blurred image comprises:

[0008] Downsampling the original blurred image to the Kth power of a preset multiple to obtain a blurred image of the Kth scale, where the Kth scale is the coarsest scale in the multi-scale generation network;

[0009] Inputting the blurred image of the Kth scale into a generation network of the Kth scale to obtain a restored image of the Kth scale;

[0010] The restored image of the Kth scale is upsampled by a preset multiple and input into the generation network of the K-1th scale to obtain a restored image of the K-1th scale, and so on, until a restored image of the first scale is obtained, where the first scale is the finest scale in the multi-scale generation network, and the restored image of the first scale is used as the restored image corresponding to the original blurred image.

[0011] Optionally, each scale of the generative network in the multi-scale generative network includes a preset number of convolutional layers, and each scale of the generative network adopts a global skip connection, that is, the output image of the last convolutional layer of the generative network of each scale plus the input image of the generative network of the scale is the restored image output by the generative network of the scale.

[0012] Optionally, before the step of inputting the original blurred image into the trained multi-scale generative network and outputting the restored image corresponding to the original blurred image, the step further includes:

[0013] Constructing a paired training data set, wherein the training data set includes a plurality of blurred images and clear images corresponding to the blurred images;

[0014] Inputting the blurred image to be restored in the training data set into the multi-scale generation network to be trained to obtain a preset number of restored images of different scales;

[0015] Downsampling the clear image corresponding to the blurred image to be restored by a preset multiple to obtain a preset number of clear images of different scales;

[0016] Based on a preset number of clear images of different scales and a preset number of restored images of different scales, a structural similarity loss is calculated;

[0017] Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the multi-scale discriminant network to be trained respectively, to obtain a preset number of discriminant matrices for restored images and a preset number of discriminant matrices for clear images;

[0018] Based on the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images, the adversarial loss is calculated;

[0019] Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the perception network respectively, to obtain a preset number of depth feature maps for the restored images and a preset number of depth feature maps for the clear images;

[0020] Calculating content loss based on the preset number of depth feature maps for restored images and the preset number of depth feature maps for clear images;

[0021] A multi-scale joint loss function is obtained based on the structural similarity loss, adversarial loss and content loss, and the joint loss function is solved by maximum-minimum optimization;

[0022] Back-propagating the joint loss function to the multi-scale generation network to be trained and the multi-scale discrimination network to be trained, and alternately optimizing and adjusting the parameters of the multi-scale generation network and the multi-scale discrimination network;

[0023] Check whether the joint loss function of the multi-scale generative network converges;

[0024] If the joint loss function of the multi-scale generative network has not converged, a new blurred image in the training data set is used as the blurred image to be restored, and the step of returning to the step of inputting the blurred image to be restored in the training data set into the multi-scale generative network to be trained to obtain a preset number of restored images of different scales;

[0025] If the joint loss function of the multi-scale generative network converges, the latest multi-scale generative network is used as the trained multi-scale generative network.

[0026] Optionally, the step of calculating the structural similarity loss based on a preset number of clear images of different scales and a preset number of restored images of different scales includes:

[0027] Substituting the pixel value mean, pixel value variance and pixel value covariance of the preset number of different scale clear images and the preset number of different scale restored images into the structural similarity loss formula, the structural similarity loss is calculated, wherein the structural similarity loss formula is:

[0028]

[0029] The calculation method of structural similarity is:

[0030]

[0031] in, is the structural similarity loss for generating the network at the k-th scale, represents the restored image at the kth scale, x (k) Represents the clear image of the kth scale, c1 = 0.01 2 , c2=0.03 2 , express The mean pixel value of Represents x (k) The mean pixel value of ; express The pixel value variance is Represents x (k) The pixel value variance of yes With x (k) The pixel value covariance.

[0032] Optionally, the step of calculating the adversarial loss based on the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images includes:

[0033] Substitute the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images into the adversarial loss formula to calculate the adversarial loss, where the adversarial loss formula is:

[0034]

[0035]

[0036] in, is the adversarial loss between the k-th scale generator and discriminator, Denotes the discriminant matrix of the k-th scale discriminant network output for the k-th scale restored image, D (k) (x (k) ) represents the discriminant matrix of the k-th scale discriminant network output for the k-th scale clear image, E represents the expectation, P g To restore the statistical distribution of the image, P r is the statistical distribution of the clear image; the critic loss part is the training target of the adversarial loss, based on the Wasserstein distance; the gradient penalty part adopts the gradient clipping strategy to ensure that the gradient of the discriminant network can be automatically constrained during training, where represents the calculation of gradient, λ is the weight of gradient penalty, Represents the clear image x (k) and the resulting restored image Random interpolation sampling between for The statistical distribution of x (k) represents the clear image of the kth scale, represents the restored image at the kth scale.

[0037] Optionally, the step of calculating the content loss based on the preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image includes:

[0038] Substituting the preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image into the content loss formula, the content loss is calculated, wherein the content loss formula is:

[0039]

[0040] in, is the content loss for the k-th scale generation network, φ i，j (x (k) ) represents the deep feature map obtained by the k-th scale clear image after the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. W represents the depth feature map obtained after the k-th scale restored image passes through the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. i，j is the depth feature map φ i，j (x (k) )or Width, H i，j is the depth feature map φ i，j (x (k) )or height.

[0041] Optionally, the step of obtaining a multi-scale joint loss function based on the structural similarity loss, the adversarial loss and the content loss, and solving the joint loss function by maximum-minimum optimization includes:

[0042] Substitute the structural similarity loss, adversarial loss and content loss into the multi-scale joint loss function formula to obtain a multi-scale joint loss function, where the multi-scale joint loss function formula is:

[0043]

[0044] The joint loss function is solved by maximum-minimum optimization, where the calculation formula for solving the joint loss function by maximum-minimum optimization is:

[0045]

[0046]

[0047] Among them, G is the multi-scale generative network to be trained, D is the multi-scale discriminative network to be trained, L(G, D) is the joint loss function, is the adversarial loss between the k-th scale generator and discriminator, is the content loss for the k-th scale generation network, is the structural similarity loss for the k-th scale generated network, α and κ are hyperparameters.

[0048] In a second aspect, the present invention further provides an image deblurring device, the image deblurring device comprising:

[0049] A processing module, used for inputting the original blurred image into a trained multi-scale generation network to obtain a restored image corresponding to the original blurred image, wherein the multi-scale generation network includes a preset number of generation networks of different scales;

[0050] The processing module is specifically used for:

[0051] Downsampling the original blurred image to the Kth power of a preset multiple to obtain a blurred image of the Kth scale, where the Kth scale is the coarsest scale in the multi-scale generation network;

[0052] Inputting the blurred image of the Kth scale into a generation network of the Kth scale to obtain a restored image of the Kth scale;

[0053] The restored image of the Kth scale is upsampled by a preset multiple and input into the generation network of the K-1th scale to obtain a restored image of the K-1th scale, and so on, until a restored image of the first scale is obtained, where the first scale is the finest scale in the multi-scale generation network, and the restored image of the first scale is used as the restored image corresponding to the original blurred image.

[0054] Optionally, each scale of the generative network in the multi-scale generative network includes a preset number of convolutional layers, and each scale of the generative network adopts a global skip connection, that is, the output image of the last convolutional layer of the generative network of each scale plus the input image of the generative network of the scale is the restored image output by the generative network of the scale.

[0055] Optionally, the image deblurring device further includes a training module, which is used to:

[0056] Constructing a paired training data set, wherein the training data set includes a plurality of blurred images and clear images corresponding to the blurred images;

[0057] Inputting the blurred image to be restored in the training data set into the multi-scale generation network to be trained to obtain a preset number of restored images of different scales;

[0058] Downsampling the clear image corresponding to the blurred image to be restored by a preset multiple to obtain a preset number of clear images of different scales;

[0059] Based on a preset number of clear images of different scales and a preset number of restored images of different scales, a structural similarity loss is calculated;

[0060] Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the multi-scale discriminant network to be trained respectively, to obtain a preset number of discriminant matrices for restored images and a preset number of discriminant matrices for clear images;

[0061] Based on the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images, the adversarial loss is calculated;

[0062] Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the perception network respectively, to obtain a preset number of depth feature maps for the restored images and a preset number of depth feature maps for the clear images;

[0063] Calculating content loss based on the preset number of depth feature maps for restored images and the preset number of depth feature maps for clear images;

[0064] A multi-scale joint loss function is obtained based on the structural similarity loss, adversarial loss and content loss, and the joint loss function is solved by maximum-minimum optimization;

[0065] Back-propagating the joint loss function to the multi-scale generation network to be trained and the multi-scale discrimination network to be trained, and alternately optimizing and adjusting the parameters of the multi-scale generation network and the multi-scale discrimination network;

[0066] Check whether the joint loss function of the multi-scale generative network converges;

[0067] If the joint loss function of the multi-scale generative network has not converged, a new blurred image in the training data set is used as the blurred image to be restored, and the step of returning to the step of inputting the blurred image to be restored in the training data set into the multi-scale generative network to be trained to obtain a preset number of restored images of different scales;

[0068] If the joint loss function of the multi-scale generative network converges, the latest multi-scale generative network is used as the trained multi-scale generative network.

[0069] Optionally, the image deblurring device further includes a first calculation module, configured to:

[0070] Substituting the pixel value mean, pixel value variance and pixel value covariance of the preset number of different scale clear images and the preset number of different scale restored images into the structural similarity loss formula, the structural similarity loss is calculated, wherein the structural similarity loss formula is:

[0071]

[0072] The calculation method of structural similarity is:

[0073]

[0074] in, is the structural similarity loss for generating the network at the k-th scale, represents the restored image at the kth scale, x (k) Represents the clear image of the kth scale, c1 = 0.01 2 , c2=0.03 2 , express The mean pixel value of Represents x (k) The mean pixel value of ; express The pixel value variance is Represents x (k) The pixel value variance of yes With x (k) The pixel value covariance.

[0075] Optionally, the image deblurring device further includes a second calculation module, configured to:

[0076] Substitute the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images into the adversarial loss formula to calculate the adversarial loss, where the adversarial loss formula is:

[0077]

[0078]

[0079] in, is the adversarial loss between the k-th scale generator and discriminator, Denotes the discriminant matrix of the k-th scale discriminant network output for the k-th scale restored image, D (k) (x (k) ) represents the discriminant matrix of the k-th scale discriminant network output for the k-th scale clear image, E represents the expectation, P g To restore the statistical distribution of the image, P r is the statistical distribution of the clear image; the critic loss part is the training target of the adversarial loss, based on the Wasserstein distance; the gradient penalty part adopts the gradient clipping strategy to ensure that the gradient of the discriminant network can be automatically constrained during training, where represents the calculation of gradient, λ is the weight of gradient penalty, Represents the clear image x (k) and the resulting restored image Random interpolation sampling between for The statistical distribution of x (k) represents the clear image of the kth scale, represents the restored image at the kth scale.

[0080] Optionally, the image deblurring device further includes a third calculation module, configured to:

[0081] Substituting the preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image into the content loss formula, the content loss is calculated, wherein the content loss formula is:

[0082]

[0083] in, is the content loss for the k-th scale generation network, φ i，j (x (k) ) represents the deep feature map obtained by the k-th scale clear image after the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. W represents the depth feature map obtained after the k-th scale restored image passes through the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. i，j is the depth feature map φ i，j (x (k) )or Width, H i，j is the depth feature map φ i，j (x (k) )or height.

[0084] Optionally, the image deblurring device further includes a fourth calculation module, configured to:

[0085] Substitute the structural similarity loss, adversarial loss and content loss into the multi-scale joint loss function formula to obtain a multi-scale joint loss function, where the multi-scale joint loss function formula is:

[0086]

[0087] The joint loss function is solved by maximum-minimum optimization, where the calculation formula for solving the joint loss function by maximum-minimum optimization is:

[0088]

[0089]

[0090] Among them, G is the multi-scale generative network to be trained, D is the multi-scale discriminative network to be trained, L(G, D) is the joint loss function, is the adversarial loss between the k-th scale generator and discriminator, is the content loss for the k-th scale generation network, is the structural similarity loss for the k-th scale generated network, and α and κ are hyperparameters.

[0091] In a third aspect, the present invention also provides an image deblurring device, comprising a processor, a memory, and an image deblurring program stored in the memory and executable by the processor, wherein when the image deblurring program is executed by the processor, the steps of the image deblurring method described above are implemented.

[0092] In a fourth aspect, the present invention further provides a readable storage medium, on which an image deblurring program is stored, wherein when the image deblurring program is executed by a processor, the steps of the image deblurring method as described above are implemented.

[0093] The present invention aims at the technical problem that the image deblurring method of the traditional convolutional neural network generates too many parameters and is inefficient when removing complex blur kernels, making it difficult to apply in fields with real-time processing requirements. The removal process of the complex blur kernel is uniformly modeled in a multi-scale generative network, and the restored image corresponding to the blurred image is gradually generated from coarse to fine. At the same time, the generative network of each scale in the multi-scale generative network adopts a global jump connection, so that the trained multi-scale generative network is more sensitive to the details that should be learned, and the details are the details that the multi-scale generative network should add to the input image. Then, an end-to-end adversarial deep learning network is constructed, and the training of the multi-scale generative network is guided by combining adversarial loss, content loss and structural similarity loss. Among them, by minimizing the structural similarity loss, the structure of the original blurred image input can be maintained when the blurred image is input into the multi-scale generative network, avoiding local deformation and distortion of the restored images of different scales output. By minimizing the content loss, the multi-scale generative network can be made to tend to restore the overall content and shape of the blurred image rather than the local pixels, so that the restored image obtained based on the multi-scale generative network has a closer contour and visual features to the corresponding clear image of the same scale. And the discriminant network and the generative network can be alternately optimized through adversarial loss, so that the multi-scale discriminant network can more reasonably measure the statistical distribution of the data, and the multi-scale generative network can reduce the difference in statistical analysis between the restored images of different scales and the clear images of different scales corresponding to the original blurred image. When the joint loss of the multi-scale generative network converges, the trained multi-scale generative network is obtained, and the quality of the restored images of different scales can be improved based on the trained multi-scale generative network. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 A schematic diagram of the hardware structure of an image deblurring device involved in an embodiment of the present invention;

[0095] Figure 2 A schematic diagram of a flow chart of an embodiment of an image deblurring method of the present invention;

[0096] Figure 3 FIG. 4 is a flow chart of another embodiment of the image deblurring method of the present invention.

[0097] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0098] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0099] In a first aspect, an embodiment of the present invention provides an image deblurring device.

[0100] Reference Figure 1 , Figure 1 Schematic diagram of the hardware structure of the image deblurring device involved in the embodiment of the present invention. In the embodiment of the present invention, the image deblurring device may include a processor 1001 (e.g., a central processing unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a wireless fidelity WIreless-FIdelity, WI-FI interface); the memory 1005 may be a high-speed random access memory (random access memory, RAM), or a stable memory (non-volatile memory), such as a disk storage, and the memory 1005 may optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation of the present invention, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0101] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an image deblurring program. The processor 1001 may call the image deblurring program stored in the memory 1005 and execute the image deblurring method provided in the embodiment of the present invention.

[0102] In a second aspect, an embodiment of the present invention provides an image deblurring method.

[0103] Reference Figure 2 , Figure 2 FIG. 4 is a flow chart of an embodiment of an image deblurring method according to the present invention.

[0104] In an embodiment of the image deblurring method of the present invention, the image deblurring method includes:

[0105] Step S10, inputting the original blurred image into a trained multi-scale generative network to obtain a restored image corresponding to the original blurred image, wherein the multi-scale generative network includes a preset number of generative networks of different scales;

[0106] The step S10 comprises:

[0107] Step S101, downsampling the original blurred image to a Kth power of a preset multiple to obtain a blurred image of a Kth scale, where the Kth scale is the coarsest scale in the multi-scale generation network;

[0108] Step S102, inputting the blurred image of the Kth scale into a generation network of the Kth scale to obtain a restored image of the Kth scale;

[0109] Step S103, upsampling the restored image of the Kth scale by a preset multiple, and inputting the restored image into the generation network of the K-1th scale to obtain a restored image of the K-1th scale;

[0110] Step S104, determining whether the K-1th scale is the first scale, the first scale being the finest scale in the multi-scale generation network;

[0111] Step S105, if the K-1th scale is not the 1st scale, the K-1th scale is used as the new Kth scale, and the process returns to the step of upsampling the restored image of the Kth scale by a preset multiple and inputting the restored image into the generation network of the K-1th scale to obtain the restored image of the K-1th scale;

[0112] Step S106: If the K-1th scale is the first scale, the restored image at the K-1th scale is used as the restored image corresponding to the original blurred image.

[0113] In this embodiment, the original blurred image is input into a trained multi-scale generative network to obtain a restored image corresponding to the original blurred image, wherein the multi-scale generative network includes a preset number of generative networks of different scales. When the multi-scale generative network performs deblurring processing on the original blurred image, a restored image corresponding to the original blurred image is generated step by step from coarse to fine. Specifically, the step of inputting the original blurred image into the trained multi-scale generative network to obtain a restored image corresponding to the original blurred image includes:

[0114] The original blurred image is downsampled to the Kth power of a preset multiple to obtain a blurred image of the Kth scale, where the Kth scale is the coarsest scale in the multi-scale generation network. The blurred image of the Kth scale is input into the generation network of the Kth scale to obtain a restored image of the Kth scale. The restored image of the Kth scale is the restored image of the coarsest scale generated by the multi-scale generation network, and it is necessary to continue to perform a more refined deblurring process on the restored image of the coarsest scale, that is, it is necessary to upsample the restored image of the Kth scale by a preset multiple, and input the upsampled image into the generation network of the K-1th scale to obtain a restored image of the K-1th scale, where the K-1th scale is the second coarsest scale in the multi-scale generation network. Determine whether the K-1th scale is the 1st scale. If the K-1 scale is not the 1st scale, the K-1 scale is used as the new Kth scale, and the process returns to the step of upsampling the restored image of the K-1 scale by a preset multiple and inputting it into the generative network of the K-1 scale to obtain the restored image of the K-1 scale. If the K-1 scale is the 1st scale, the restored image of the K-1 scale is used as the restored image corresponding to the original blurred image. That is, by analogy, the restored image obtained at this scale is upsampled by a preset multiple and inputted into the generative network of the next scale, and the restored image of the next scale is inputted until the restored image of the 1st scale is obtained. The 1st scale is the finest scale in the multi-scale generative network, and the restored image of the 1st scale is used as the restored image corresponding to the original blurred image.

[0115] The multi-scale generative network used in the image deblurring method uniformly models the complex blur kernel removal process, and gradually generates a clear image from coarse to fine. The trained multi-scale generative network has learned the mapping from blurry images to clear images in complex scenes, and can restore the input blurry images in complex scenes, solving the problem that the traditional convolutional neural network image deblurring method generates too many parameters when removing complex blur kernels, is inefficient and difficult to apply in fields with real-time processing requirements, thereby realizing end-to-end image deblurring in a variety of complex scenes.

[0116] Furthermore, in one embodiment, each scale of the generative network in the multi-scale generative network includes a preset number of convolutional layers, and each scale of the generative network adopts a global skip connection, that is, the output of the last convolutional layer of the generative network of each scale plus the input image of the generative network of the scale is the restored image output by the generative network of the scale.

[0117] In this embodiment, the generative network of each scale in the multi-scale generative network includes a preset number of convolutional layers, and the generative network of each scale adopts global skip connection, that is, the output of the last convolutional layer of the generative network of each scale plus the input image of the generative network of the scale is the restored image output by the generative network of the scale. The generative network of each scale in the multi-scale generative network adopts global skip connection, so that when training the generative network, what is trained is not the direct mapping between the output and the input image, but the residual between the output target restored image and the input blurred image. Therefore, the use of global skip connection in the generative network of each scale in the multi-scale generative network can make the trained multi-scale generative network more sensitive to the details that should be learned, and the details are the details that the multi-scale generative network should add to the input image.

[0118] Further, in one embodiment, referring to Figure 3 , Figure 3 FIG. 4 is a flow chart of another embodiment of the image deblurring method of the present invention.

[0119] In an embodiment of the image deblurring method of the present invention, before step S10, the method further includes:

[0120] Step S001, constructing a paired training data set, wherein the training data set includes a plurality of blurred images and clear images corresponding to the blurred images;

[0121] In this embodiment, a paired training data set is constructed, and the training data set includes a number of blurred images and clear images corresponding to the blurred images. Specifically, a number of clear images are collected and cropped to obtain images of the same size, for example, 512*512 pixels. The clear images are generated into a number of corresponding blurred images through a blur algorithm, and the obtained pairs of blurred images and clear images corresponding to the blurred images are used as training data sets.

[0122] Step S002, inputting the blurred image to be restored in the training data set into the multi-scale generation network to be trained to obtain a preset number of restored images of different scales;

[0123] In this embodiment, the blurred image to be restored in the training data set is input into the multi-scale generation network to be trained to obtain a preset number of restored images of different scales. Specifically, if the multi-scale generation network corresponds to K scales, K restored images of the 1st, 2nd, ..., K-1, and K scales can be obtained.

[0124] Step S003, downsampling the clear image corresponding to the blurred image to be restored by a preset multiple to obtain a preset number of clear images of different scales;

[0125] In this embodiment, the clear image corresponding to the blurred image to be restored is downsampled by a preset multiple to obtain a preset number of clear images of different scales. Specifically, if the multi-scale generation network corresponds to K scales, the original clear image can be downsampled by a preset multiple of 1 to K-1 to obtain K clear images of the 1st, 2nd, ..., K-1, K scales.

[0126] Step S004, calculating the structural similarity loss based on a preset number of clear images of different scales and a preset number of restored images of different scales;

[0127] In this embodiment, based on the preset number of different-scale clear images obtained in step S002 and the preset number of different-scale restored images obtained in step S003, the structural similarity loss can be calculated. In the training process of the multi-scale generative network, the multi-scale generative network can be constrained by minimizing the structural similarity between the different-scale restored images generated by the multi-scale generative network and the corresponding different-scale clear images. Therefore, by minimizing the structural similarity loss, the structure of the original blurred image input can be maintained when the blurred image is input into the multi-scale generative network, avoiding local deformation and distortion of the restored images of different scales output.

[0128] Further, in one embodiment, the step S004 includes:

[0129] The pixel value means, pixel value variances, and pixel value covariances of the preset number of clear images of different scales and the preset number of restored images of different scales are input into the structural similarity loss formula to calculate the structural similarity loss, wherein the structural similarity loss formula is:

[0130]

[0131] The calculation method of structural similarity is:

[0132]

[0133] in, is the structural similarity loss for generating the network at the k-th scale, represents the restored image at the kth scale, x (k) Represents the clear image of the kth scale; c1 = 0.01 2 , c2=0.03 2 ; express The mean pixel value of Represents x (k) The mean pixel value of ; express The pixel value variance is Represents x (k) The pixel value variance of yes With x (k) The pixel value covariance.

[0134] In this embodiment, the pixel value means, pixel value variances, and pixel value covariances of the preset number of clear images of different scales and the preset number of restored images of different scales are input into the structural similarity loss formula to calculate the structural similarity loss, wherein the structural similarity loss formula is:

[0135]

[0136] The calculation method of structural similarity is:

[0137]

[0138] in, is the structural similarity loss for generating the network at the k-th scale, represents the restored image at the kth scale, x (k) Represents the clear image of the kth scale; c1 = 0.01 2 , c2=0.03 2 ; express The mean pixel value of Represents x (k) The mean pixel value of ; express The pixel value variance is Represents x (k) The pixel value variance of yes With x (k) The pixel value covariance.

[0139] Step S005, inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the multi-scale discriminant network to be trained respectively, to obtain a preset number of discriminant matrices for restored images and a preset number of discriminant matrices for clear images;

[0140] In this embodiment, an adversarial learning strategy is adopted when training a multi-scale generative network, and the obtained preset number of restored images of different scales and the preset number of clear images of different scales are respectively input into the multi-scale discriminant network to be trained, and a preset number of discriminant matrices for restored images and a preset number of discriminant matrices for clear images are obtained. Each element in the discriminant matrix is ​​a discriminant result of a local area of ​​the input image.

[0141] Step S006, calculating the adversarial loss based on the preset number of discriminant matrices for the restored image and the preset number of discriminant matrices for the clear image;

[0142] In this embodiment, the adversarial loss can be calculated based on the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images obtained by the multi-scale discriminant network. The adversarial loss can be used to measure the difference in statistical distribution between clear images of different scales corresponding to the original blurred image and restored images of different scales obtained based on the multi-scale generative network. Based on the adversarial loss, the discriminant network and the generative network can be alternately optimized, so that the multi-scale discriminant network can more reasonably measure the statistical distribution of the data, and prompt the multi-scale generative network to reduce the difference in statistical analysis between the restored images of different scales outputted and the clear images of different scales corresponding to the original blurred image, thereby improving the quality of the restored images of different scales outputted.

[0143] Further, in one embodiment, step S006 includes:

[0144] The preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images are input into the adversarial loss formula to calculate the adversarial loss, wherein the adversarial loss formula is:

[0145]

[0146]

[0147] in, is the adversarial loss between the k-th scale generator and discriminator, Denotes the discriminant matrix of the k-th scale discriminant network output for the k-th scale restored image, D (k) (x (k) ) represents the discriminant matrix of the k-th scale discriminant network output for the k-th scale clear image, E represents the expectation, P g To restore the statistical distribution of the image, P r is the statistical distribution of clear images; the critic loss part is the training target of adversarial loss, based on Wasserstein distance; the gradient penalty part adopts the gradient clipping strategy to ensure that the gradient of the multi-scale discriminant network can be automatically constrained during training, where represents the calculation of gradient, λ is the weight of gradient penalty, Represents the clear image x (k) and the resulting restored image Random interpolation sampling between for The statistical distribution of x(k) represents the clear image of the kth scale, represents the restored image at the kth scale.

[0148] In this embodiment, the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images are input into the adversarial loss formula to calculate the adversarial loss, wherein the adversarial loss formula is:

[0149]

[0150]

[0151] in, is the adversarial loss between the k-th scale generator and discriminator, Denotes the discriminant matrix of the k-th scale discriminant network output for the k-th scale restored image, D (k) (x (k) ) represents the discriminant matrix of the k-th scale discriminant network output for the k-th scale clear image, E represents the expectation, P g To restore the statistical distribution of the image, P r is the statistical distribution of clear images; the critic loss part is the training target of adversarial loss, based on Wasserstein distance; the gradient penalty part adopts the gradient clipping strategy to ensure that the gradient of the multi-scale discriminant network can be automatically constrained during training, where represents the calculation of gradient, and input is the weight of gradient penalty. Represents the clear image x (k) and the resulting restored image Random interpolation sampling between for The statistical distribution of x (k) represents the clear image of the kth scale, represents the restored image at the kth scale.

[0152] Step S007, inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the perception network respectively, to obtain a preset number of depth feature maps for the restored images and a preset number of depth feature maps for the clear images;

[0153] In this embodiment, the perception network focuses on the visual content of the image, can extract the depth features of the image, and input the preset number of restored images of different scales and the preset number of clear images of different scales into the perception network respectively. The preset number of restored images of different scales and the preset number of clear images of different scales can be mapped to the content space through the perception network to obtain a preset number of depth feature maps for the restored images and a preset number of depth feature maps for the clear images.

[0154] Step S008, calculating the content loss based on the preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image;

[0155] In this embodiment, the content loss can be calculated based on a preset number of deep feature maps for the restored image and a preset number of deep feature maps for the clear image obtained by the perception network. Content loss is used to measure the distance between the restored image and its corresponding clear image of the same scale in the content space, that is, the difference in visual content. Minimizing the content loss can make the multi-scale generation network tend to restore the overall content and shape of the blurred image rather than local pixels, thereby causing the restored image obtained based on the multi-scale generation network to have a closer contour and visual features to the corresponding clear image of the same scale.

[0156] Further, in one embodiment, the step S008 includes:

[0157] The preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image are input into a content loss formula to calculate the content loss, wherein the content loss formula is:

[0158]

[0159] in, is the content loss for the k-th scale generation network, φ i，j (x (k) ) represents the deep feature map obtained by the k-th scale clear image after the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. W represents the depth feature map obtained after the k-th scale restored image passes through the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. i，j is the depth feature map φ i，j (x (k) )or Width, H i，j is the depth feature map φ i，j (x (k) )or height.

[0160] In this embodiment, the preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image are input into the content loss formula to calculate the content loss, wherein the content loss formula is:

[0161]

[0162] in, is the content loss for the k-th scale generation network, φ i，j (x (k) ) represents the deep feature map obtained by the k-th scale clear image after the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. W represents the depth feature map obtained after the k-th scale restored image passes through the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. i，j is the depth feature map φ i，j (x (k) )or Width, H i，j is the depth feature map φ i，j (x (k) )or height.

[0163] Step S009, obtaining a multi-scale joint loss function based on the structural similarity loss, the adversarial loss and the content loss, and solving the joint loss function by maximum-minimum optimization;

[0164] In this embodiment, a multi-scale joint loss function is obtained based on the structural similarity loss, adversarial loss and content loss, and the joint loss function is solved by maximum-minimum optimization.

[0165] Further, in one embodiment, the step S009 includes:

[0166] Substitute the structural similarity loss, adversarial loss and content loss into the multi-scale joint loss function formula to obtain a multi-scale joint loss function, where the multi-scale joint loss function formula is:

[0167]

[0168] The joint loss function is solved by maximum-minimum optimization, where the calculation formula for solving the joint loss function by maximum-minimum optimization is:

[0169]

[0170]

[0171] Among them, G is the multi-scale generative network to be trained, D is the multi-scale discriminative network to be trained, L(G, D) is the joint loss function, is the adversarial loss between the k-th scale generator and discriminator, is the content loss for the k-th scale generation network, is the structural similarity loss for the k-th scale generated network, α and κ are hyperparameters.

[0172] In this embodiment, the structural similarity loss, adversarial loss and content loss are substituted into the multi-scale joint loss function formula to obtain a multi-scale joint loss function, wherein the multi-scale joint loss function formula is:

[0173]

[0174] In this embodiment, an end-to-end adversarial deep learning network is used to train the multi-scale generative network to be trained. Therefore, it is necessary to solve the joint loss function by maximum-minimum optimization, that is, to solve the multi-scale discriminant network so that the loss corresponding to the joint loss function is the maximum value and to solve the multi-scale generative network so that the loss corresponding to the joint loss function is the minimum value. The calculation formula for solving the joint loss function by maximum-minimum optimization is:

[0175]

[0176]

[0177] Among them, G is the multi-scale generative network to be trained, D is the multi-scale discriminative network to be trained, L(G, D) is the joint loss function, is the adversarial loss between the k-th scale generator and discriminator, is the content loss for the k-th scale generation network, is the structural similarity loss for the k-th scale generated network, α and κ are hyperparameters.

[0178] Step S010, back-propagating the joint loss function to the multi-scale generative network to be trained and the multi-scale discriminative network to be trained, and alternately optimizing and adjusting the parameters of the multi-scale generative network and the multi-scale discriminative network;

[0179] Step S011, detecting whether the joint loss function of the multi-scale generation network converges;

[0180] Step S012, if the joint loss function of the multi-scale generative network has not converged, the new blurred image in the training data set is used as the blurred image to be restored, and the process returns to the step of inputting the blurred image to be restored in the training data set into the multi-scale generative network to be trained to obtain a preset number of restored images of different scales;

[0181] Step S013: if the joint loss function of the multi-scale generative network converges, the latest multi-scale generative network is used as the trained multi-scale generative network.

[0182] In this embodiment, the multi-scale generative network and the multi-scale discriminative network are alternately trained using a back propagation algorithm until the joint loss function of the multi-scale generative network converges, then the training is stopped, and the latest image deblurring model is used as the trained multi-scale generative network. Specifically, the joint loss function is back-propagated to the multi-scale generative network to be trained and the multi-scale discriminative network to be trained, and the parameters of the multi-scale generative network and the multi-scale discriminative network are alternately optimized and adjusted. It is detected whether the joint loss function of the multi-scale generative network converges. If the joint loss function of the multi-scale generative network does not converge, the new blurred image in the training data set is used as the blurred image to be restored, and the step of inputting the blurred image to be restored in the training data set into the multi-scale generative network to be trained is returned to obtain a preset number of restored images of different scales. If the joint loss function of the multi-scale generative network converges, the latest multi-scale generative network is used as the trained multi-scale generative network.

[0183] In this embodiment, in view of the technical problem that the image deblurring method of the traditional convolutional neural network generates too many parameters and is inefficient when removing complex blur kernels, making it difficult to apply it in fields with real-time processing requirements, the removal process of the complex blur kernel is uniformly modeled in a multi-scale generative network, and the restored image corresponding to the blurred image is gradually generated from coarse to fine. At the same time, the generative network of each scale in the multi-scale generative network adopts a global jump connection, so that the trained multi-scale generative network is more sensitive to the details that should be learned, and the details are the details that the multi-scale generative network should add to the input image. Then, an end-to-end adversarial deep learning network is constructed, and the training of the multi-scale generative network is guided by combining adversarial loss, content loss and structural similarity loss. Among them, by minimizing the structural similarity loss, the structure of the original blurred image input can be maintained when the blurred image is input into the multi-scale generative network, avoiding local deformation and distortion of the restored images of different scales output. By minimizing the content loss, the multi-scale generative network can be made to tend to restore the overall content and shape of the blurred image rather than the local pixels, so that the restored image obtained based on the multi-scale generative network has a closer contour and visual features to the corresponding clear image of the same scale. And the discriminant network and the generative network can be alternately optimized through adversarial loss, so that the multi-scale discriminant network can more reasonably measure the statistical distribution of the data, and the multi-scale generative network can reduce the difference in statistical analysis between the restored images of different scales and the clear images of different scales corresponding to the original blurred image. When the joint loss of the multi-scale generative network converges, the trained multi-scale generative network is obtained, and the quality of the restored images of different scales can be improved based on the trained multi-scale generative network.

[0184] In a third aspect, an embodiment of the present invention further provides an image deblurring device.

[0185] In this embodiment, the image deblurring device includes:

[0186] A processing module, used for inputting the original blurred image into a trained multi-scale generation network to obtain a restored image corresponding to the original blurred image, wherein the multi-scale generation network includes a preset number of generation networks of different scales;

[0187] The processing module is specifically used for:

[0188] Downsampling the original blurred image to the Kth power of a preset multiple to obtain a blurred image of the Kth scale, where the Kth scale is the coarsest scale in the multi-scale generation network;

[0189] Inputting the blurred image of the Kth scale into a generation network of the Kth scale to obtain a restored image of the Kth scale;

[0190] The restored image of the Kth scale is upsampled by a preset multiple and input into the generation network of the K-1th scale to obtain a restored image of the K-1th scale, and so on, until a restored image of the first scale is obtained, where the first scale is the finest scale in the multi-scale generation network, and the restored image of the first scale is used as the restored image corresponding to the original blurred image.

[0191] Optionally, each scale of the generative network in the multi-scale generative network includes a preset number of convolutional layers, and each scale of the generative network adopts a global skip connection, that is, the output image of the last convolutional layer of the generative network of each scale plus the input image of the generative network of the scale is the restored image output by the generative network of the scale.

[0192] Optionally, the image deblurring device further includes a training module, which is used to:

[0193] Constructing a paired training data set, wherein the training data set includes a plurality of blurred images and clear images corresponding to the blurred images;

[0194] Inputting the blurred image to be restored in the training data set into the multi-scale generation network to be trained to obtain a preset number of restored images of different scales;

[0195] Downsampling the clear image corresponding to the blurred image to be restored by a preset multiple to obtain a preset number of clear images of different scales;

[0196] Based on a preset number of clear images of different scales and a preset number of restored images of different scales, a structural similarity loss is calculated;

[0197] Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the multi-scale discriminant network to be trained respectively, to obtain a preset number of discriminant matrices for restored images and a preset number of discriminant matrices for clear images;

[0198] Based on the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images, the adversarial loss is calculated;

[0199] Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the perception network respectively, to obtain a preset number of depth feature maps for the restored images and a preset number of depth feature maps for the clear images;

[0200] Calculating content loss based on the preset number of depth feature maps for restored images and the preset number of depth feature maps for clear images;

[0201] A multi-scale joint loss function is obtained based on the structural similarity loss, adversarial loss and content loss, and the joint loss function is solved by maximum-minimum optimization;

[0202] Back-propagating the joint loss function to the multi-scale generation network to be trained and the multi-scale discrimination network to be trained, and alternately optimizing and adjusting the parameters of the multi-scale generation network and the multi-scale discrimination network;

[0203] Check whether the joint loss function of the multi-scale generative network converges;

[0204] If the joint loss function of the multi-scale generative network has not converged, a new blurred image in the training data set is used as the blurred image to be restored, and the step of returning to the step of inputting the blurred image to be restored in the training data set into the multi-scale generative network to be trained to obtain a preset number of restored images of different scales;

[0205] If the joint loss function of the multi-scale generative network converges, the latest multi-scale generative network is used as the trained multi-scale generative network.

[0206] Optionally, the image deblurring device further includes a first calculation module, configured to:

[0207] The pixel value mean, pixel value variance and pixel value mean square error of the preset number of clear images of different scales and the preset number of restored images of different scales are input into the structural similarity loss formula to calculate the structural similarity loss, wherein the structural similarity loss formula is:

[0208]

[0209] The calculation method of structural similarity is:

[0210]

[0211] in, is the structural similarity loss for the k-th scale generated network, represents the restored image at the kth scale, x (k) Represents the clear image of the kth scale; c1 = 0.01 2 , c2=0.03 2 , express The mean pixel value of Represents x (k) The mean pixel value of ; express The pixel value variance is Represents x (k) The pixel value variance of yes With x (k) The pixel value covariance.

[0212] Optionally, the image deblurring device further includes a second calculation module, configured to:

[0213] The preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images are input into the adversarial loss formula to calculate the adversarial loss, wherein the adversarial loss formula is:

[0214]

[0215]

[0216] in, is the adversarial loss between the k-th scale generator and discriminator, Denotes the discriminant matrix of the k-th scale discriminant network output for the k-th scale restored image, D (k) (x (k) ) represents the discriminant matrix of the k-th scale discriminant network output for the k-th scale clear image, E represents the expectation, P g To restore the statistical distribution of the image, P r is the statistical distribution of the clear image; the critic loss part is the training target of the adversarial loss, based on the Wasserstein distance; the gradient penalty part adopts the gradient clipping strategy to ensure that the gradient of the discriminant network can be automatically constrained during training, where represents the calculation of gradient, and input is the weight of gradient penalty. Represents the clear image x (k) and the resulting restored image Random interpolation sampling between for The statistical distribution of x (k) represents the clear image of the kth scale, represents the restored image at the kth scale.

[0217] Optionally, the image deblurring device further includes a third calculation module, configured to:

[0218] The preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image are input into a content loss formula to calculate the content loss, wherein the content loss formula is:

[0219]

[0220] in, is the content loss for the k-th scale generation network, φ i，j (x (k) ) represents the deep feature map obtained by the k-th scale clear image after the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. W represents the depth feature map obtained after the k-th scale restored image passes through the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. i，j is the depth feature map φ i，j (x (k) )or Width, H i，j is the depth feature map φ i，j (x (k) )or height.

[0221] Optionally, the image deblurring device further includes a fourth calculation module, configured to:

[0222] Substitute the structural similarity loss, adversarial loss and content loss into the multi-scale joint loss function formula to obtain a multi-scale joint loss function, where the multi-scale joint loss function formula is:

[0223]

[0224] The joint loss function is solved by maximum-minimum optimization, where the calculation formula for solving the joint loss function by maximum-minimum optimization is:

[0225]

[0226]

[0227] Among them, G is the multi-scale generative network to be trained, D is the multi-scale discriminative network to be trained, L(G, D) is the joint loss function, is the adversarial loss between the k-th scale generator and discriminator, is the content loss for the k-th scale generation network, is the structural similarity loss for the k-th scale generated network, α and κ are hyperparameters.

[0228] Among them, the functional implementation of each module in the above-mentioned image deblurring device corresponds to each step in the above-mentioned image deblurring method embodiment, and its functions and implementation processes are no longer repeated here.

[0229] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium.

[0230] The readable storage medium of the present invention stores an image deblurring program, wherein when the image deblurring program is executed by a processor, the steps of the above-mentioned image deblurring method are implemented.

[0231] The method implemented when the image deblurring program is executed may refer to the various embodiments of the image deblurring method of the present invention, and will not be described in detail here.

[0232] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0233] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0234] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present invention.

[0235] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image deblurring method, characterized in that: The image deblurring method comprises: Inputting the original blurred image into a trained multi-scale generative network to obtain a restored image corresponding to the original blurred image, wherein the multi-scale generative network includes a preset number of generative networks of different scales; The step of inputting the original blurred image into the trained multi-scale generation network to obtain a restored image corresponding to the original blurred image comprises: Downsampling the original blurred image to the Kth power of a preset multiple to obtain a blurred image of the Kth scale, where the Kth scale is the coarsest scale in the multi-scale generation network; Inputting the blurred image of the Kth scale into a generation network of the Kth scale to obtain a restored image of the Kth scale; The restored image of the Kth scale is upsampled by a preset multiple and input into the generation network of the K-1th scale to obtain a restored image of the K-1th scale, and so on, until a restored image of the first scale is obtained, wherein the first scale is the finest scale in the multi-scale generation network, and the restored image of the first scale is used as the restored image corresponding to the original blurred image; Each scale of the generative network in the multi-scale generative network includes a preset number of convolutional layers, and each scale of the generative network adopts a global skip connection, that is, the output image of the last convolutional layer of the generative network of each scale plus the input image of the generative network of the scale is the restored image output by the generative network of the scale; The step of inputting the original blurred image into the trained multi-scale generative network and outputting the restored image corresponding to the original blurred image also includes: Constructing a paired training data set, wherein the training data set includes a plurality of blurred images and clear images corresponding to the blurred images; Inputting the blurred image to be restored in the training data set into the multi-scale generation network to be trained to obtain a preset number of restored images of different scales; Downsampling the clear image corresponding to the blurred image to be restored by a preset multiple to obtain a preset number of clear images of different scales; Based on a preset number of clear images of different scales and a preset number of restored images of different scales, a structural similarity loss is calculated; Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the multi-scale discriminant network to be trained respectively, to obtain a preset number of discriminant matrices for restored images and a preset number of discriminant matrices for clear images; Based on the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images, the adversarial loss is calculated; Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the perception network respectively, to obtain a preset number of depth feature maps for the restored images and a preset number of depth feature maps for the clear images; Calculating content loss based on the preset number of depth feature maps for restored images and the preset number of depth feature maps for clear images; A multi-scale joint loss function is obtained based on the structural similarity loss, adversarial loss and content loss, and the joint loss function is solved by maximum-minimum optimization; Back-propagating the joint loss function to the multi-scale generation network to be trained and the multi-scale discrimination network to be trained, and alternately optimizing and adjusting the parameters of the multi-scale generation network and the multi-scale discrimination network; Check whether the joint loss function of the multi-scale generative network converges; If the joint loss function of the multi-scale generative network has not converged, a new blurred image in the training data set is used as the blurred image to be restored, and the step of returning to the step of inputting the blurred image to be restored in the training data set into the multi-scale generative network to be trained to obtain a preset number of restored images of different scales; If the joint loss function of the multi-scale generative network converges, the latest multi-scale generative network is used as the trained multi-scale generative network.

2. The image deblurring method according to claim 1, wherein: The step of calculating the structural similarity loss based on a preset number of clear images of different scales and a preset number of restored images of different scales comprises: Substituting the pixel value mean, pixel value variance and pixel value covariance of the preset number of different scale clear images and the preset number of different scale restored images into the structural similarity loss formula, the structural similarity loss is calculated, wherein the structural similarity loss formula is: The calculation method of structural similarity is: in, is the structural similarity loss for the k-th scale generated network, represents the restored image at the kth scale, x (k) Represents the clear image of the kth scale, c1 = 0.01 2 , c2=0.03 2 , express The mean pixel value of Represents x (k) The mean pixel value of ; express The pixel value variance is Represents x (k) The pixel value variance of yes With x (k) The pixel value covariance.

3. The image deblurring method according to claim 1, wherein: The step of calculating the adversarial loss based on the preset number of discriminant matrices for the restored image and the preset number of discriminant matrices for the clear image comprises: Substitute the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images into the adversarial loss formula to calculate the adversarial loss, where the adversarial loss formula is: in, is the adversarial loss between the k-th scale generator and discriminator, Denotes the discriminant matrix of the k-th scale restored image output by the discriminant network of the k-th scale, D (k) (x (k) ) represents the discriminant matrix of the k-th scale discriminant network output for the k-th scale clear image, E represents the expectation, P g To restore the statistical distribution of the image, P r is the statistical distribution of the clear image; the critic loss part is the training target of the adversarial loss, based on the Wasserstein distance; the gradient penalty part adopts the gradient clipping strategy to ensure that the gradient of the discriminant network can be automatically constrained during training, where represents the calculation of gradient, λ is the weight of gradient penalty, Represents the clear image x (k) and the resulting restored image Random interpolation sampling between for The statistical distribution of x (k) represents the clear image of the kth scale, represents the restored image at the kth scale.

4. The image deblurring method according to claim 1, wherein: The step of calculating the content loss based on the preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image comprises: Substituting the preset number of depth feature maps for the restored image and the preset number of depth feature maps for the clear image into the content loss formula, the content loss is calculated, wherein the content loss formula is: in, is the content loss for the k-th scale generation network, φ i,j (x (k) ) represents the deep feature map obtained by the k-th scale clear image after the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. W represents the depth feature map obtained after the k-th scale restored image passes through the activation function of the j-th convolutional layer before the i-th maximum pooling layer of the perception network. i,j is the depth feature map φ i,j (x (k) )or Width, H i,j is the depth feature map φ i,j (x (k) )or height.

5. The image deblurring method according to claim 1, wherein: The step of obtaining a multi-scale joint loss function based on the structural similarity loss, the adversarial loss and the content loss, and solving the joint loss function by maximum-minimum optimization includes: Substitute the structural similarity loss, adversarial loss and content loss into the multi-scale joint loss function formula to obtain a multi-scale joint loss function, where the multi-scale joint loss function formula is: The joint loss function is solved by maximum-minimum optimization, where the calculation formula for solving the joint loss function by maximum-minimum optimization is: Among them, G is the multi-scale generative network to be trained, D is the multi-scale discriminative network to be trained, L(G,D) is the joint loss function, is the adversarial loss between the k-th scale generator and discriminator, is the content loss for the k-th scale generation network, is the structural similarity loss for the k-th scale generated network, α and κ are hyperparameters.

6. An image deblurring device, characterized in that: The image deblurring device comprises: A processing module, used for inputting the original blurred image into a trained multi-scale generation network to obtain a restored image corresponding to the original blurred image, wherein the multi-scale generation network includes a preset number of generation networks of different scales; The processing module is specifically used for: Downsampling the original blurred image to the Kth power of a preset multiple to obtain a blurred image of the Kth scale, where the Kth scale is the coarsest scale in the multi-scale generation network; Inputting the blurred image of the Kth scale into a generation network of the Kth scale to obtain a restored image of the Kth scale; The restored image of the Kth scale is upsampled by a preset multiple and input into the generation network of the K-1th scale to obtain a restored image of the K-1th scale, and so on, until a restored image of the first scale is obtained, wherein the first scale is the finest scale in the multi-scale generation network, and the restored image of the first scale is used as the restored image corresponding to the original blurred image; The processing module is also used for: Each scale of the generative network in the multi-scale generative network includes a preset number of convolutional layers, and each scale of the generative network adopts a global skip connection, that is, the output image of the last convolutional layer of the generative network of each scale plus the input image of the generative network of the scale is the restored image output by the generative network of the scale; Also includes training modules for: Constructing a paired training data set, wherein the training data set includes a plurality of blurred images and clear images corresponding to the blurred images; Inputting the blurred image to be restored in the training data set into the multi-scale generation network to be trained to obtain a preset number of restored images of different scales; Downsampling the clear image corresponding to the blurred image to be restored by a preset multiple to obtain a preset number of clear images of different scales; Based on a preset number of clear images of different scales and a preset number of restored images of different scales, a structural similarity loss is calculated; The preset number of restored images of different scales and the preset number of clear images of different scales are combined The images are respectively input into a multi-scale discriminant network to be trained to obtain a preset number of discriminant matrices for restored images and a preset number of discriminant matrices for clear images; Based on the preset number of discriminant matrices for restored images and the preset number of discriminant matrices for clear images, the adversarial loss is calculated; Inputting the preset number of restored images of different scales and the preset number of clear images of different scales into the perception network respectively, to obtain a preset number of depth feature maps for the restored images and a preset number of depth feature maps for the clear images; Calculating content loss based on the preset number of depth feature maps for restored images and the preset number of depth feature maps for clear images; A multi-scale joint loss function is obtained based on the structural similarity loss, adversarial loss and content loss, and the joint loss function is solved by maximum-minimum optimization; Back-propagating the joint loss function to the multi-scale generation network to be trained and the multi-scale discrimination network to be trained, and alternately optimizing and adjusting the parameters of the multi-scale generation network and the multi-scale discrimination network; Check whether the joint loss function of the multi-scale generative network converges; If the joint loss function of the multi-scale generative network has not converged, a new blurred image in the training data set is used as the blurred image to be restored, and the step of returning to the step of inputting the blurred image to be restored in the training data set into the multi-scale generative network to be trained to obtain a preset number of restored images of different scales; If the joint loss function of the multi-scale generative network converges, the latest multi-scale generative network is used as the trained multi-scale generative network.

7. An image deblurring device, characterized in that: The image deblurring device comprises a processor, a memory, and an image deblurring program stored in the memory and executable by the processor, wherein when the image deblurring program is executed by the processor, the steps of the image deblurring method according to any one of claims 1 to 5 are implemented.

8. A readable storage medium, characterized in that: An image deblurring program is stored on the readable storage medium, wherein when the image deblurring program is executed by a processor, the steps of the image deblurring method according to any one of claims 1 to 5 are implemented.

Images