Image deblurring method based on improved generative adversarial network

Abstract

The present invention discloses an image deblurring method based on an improved generative adversarial network, comprising the following steps: 1. inputting an image to be deblurred into a pre-trained improved generative adversarial network; 2. the improved generative adversarial network deblurring the image to be deblurred to obtain a deblurred image. Aiming at the ill-posed problem of non-blind deblurring methods estimating blur kernels, the present invention adopts a deep learning method, and proposes an image deblurring method based on an improved generative adversarial network on the basis of analyzing the theory of generative adversarial networks. On the basis of the generative adversarial network, an improved generative adversarial network is constructed, and a new objective function is constructed by introducing the idea of ​​weighting. By adjusting the weight coefficient to balance the proportion of forward KL divergence and reverse KL divergence, a clear image can be better restored.

Description

Technical Field

[0001] The present invention belongs to the technical field of image deblurring processing, and in particular relates to an image deblurring method based on an improved generative adversarial network. Background Art

[0002] With the popularization of intelligent photography equipment, images play an increasingly important role in daily life, transportation, medical treatment, criminal justice, aerospace, etc. However, due to camera shake, object movement speed, or sensor noise, atmospheric turbulence effects and other factors, the resulting image is generated by superimposing images at different times during the exposure period, making the generated image blurred and losing its own use value.

[0003] Image deblurring can be studied mainly from two aspects: one is to improve the hardware equipment and build a control system for stable shooting. This method can control the generation of blurred images from the source, but the cost is high and the hardware conditions are high. The second is to restore the corresponding clear image of the blurred image through a certain algorithm. This method focuses on how to estimate the blur kernel and then restore the clear image based on the blur kernel. However, the estimation of the blur kernel is an ill-posed problem. The shooting equipment will shake, and the change of the scene makes the blur kernel of each type of picture different. This traditional deblurring method has great limitations and is difficult to use in complex environments. Therefore, image deblurring methods need further research.

[0004] With the development of deep learning, scholars began to use the Convolutional Neural Network (CNN) method for image deblurring. This type of method applies neural networks to image deblurring, laying the foundation for the subsequent development of image deblurring. However, because it still uses the idea of ​​non-blind deblurring to estimate the blur kernel, it runs slowly and has poor generalization ability. In most cases, we do not know the blur kernel of the blurred image and it is difficult to estimate the blur kernel based on the blurred image. After continuous in-depth research by researchers, image deblurring began to adopt an end-to-end learning method. There is no need to estimate the blur kernel. The network directly learns the mapping relationship between input and output. The introduction of the Generative Adversarial Network (GAN) in 2014, with its ingenious ideas, brought image deblurring technology to another new stage. However, the objective function defined by the generator in GAN is equivalent to minimizing the real data P data (x) and the generated sample P g (z) JS divergence (Jensen-Shannon) JS (P data ||P G), the optimization process will cause gradient vanishing and mode collapse problems, and the deblurring effect is not good. In response to the mode collapse problem of GAN, some people have proposed the dual discriminator generative adversarial network (D2GAN) algorithm in the prior art. D2GAN introduces dual discriminators to try to solve the mode collapse problem of GAN, but does not give clear guidance on hyperparameters. Its generator can learn most of the distribution, but some distributions are still forgotten. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an image deblurring method based on an improved generative adversarial network in response to the deficiencies in the above-mentioned prior art. On the basis of the generative adversarial network, an improved generative adversarial network is constructed, and the idea of ​​weighting is introduced to construct a new objective function. By adjusting the weight coefficient to balance the proportion of forward KL divergence and reverse KL divergence, a clear image can be better restored.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: an image deblurring method based on an improved generative adversarial network, the method comprising the following steps:

[0007] Step 1: Input the image to be deblurred into a pre-trained improved generative adversarial network; wherein the training process of the improved generative adversarial network is:

[0008] Step 101: Combining the KL divergence and the reverse KL divergence into an objective function by using a weighted method to construct an improved generative adversarial network; the improved generative adversarial network includes a generator G, and a discriminator D1 and a discriminator D2 connected in parallel to the output end of the generator G;

[0009] Step 102: construct the loss functions of the generator G, the discriminator D1 and the discriminator D2 from the two aspects of image texture details and image content;

[0010] Step 103: Select training samples, and use the training samples and the loss function constructed in step 102 to train the improved generative adversarial network constructed in step 101 to obtain a trained improved generative adversarial network;

[0011] Step 2: The improved generative adversarial network deblurs the image to be deblurred to obtain a deblurred image.

[0012] In the above-mentioned image deblurring method based on the improved generative adversarial network, the expression of the KL divergence in step 101 is:

[0013]

[0014] The KL divergence has the following three properties:

[0015] Feature 1: KL(P data ||P G )≥0;

[0016] Characteristic 2: It does not satisfy the triangle inequality;

[0017] Feature 3: KL(P data ||P G )≠KL(P G ||P data );

[0018] The expression of the reverse KL divergence in step 101 is:

[0019]

[0020] The reverse KL divergence has the following three properties:

[0021] Feature 1: KL(P G ||P data )≤0;

[0022] Feature 2: Satisfies the triangle inequality;

[0023] Feature 3: KL(P G ||P data )≠KL(P data ||P G );

[0024] Among them, P G represents the generative model, P data represents the true distribution, and x represents the variable;

[0025] In the above-mentioned image deblurring method based on an improved generative adversarial network, the network structure of the generator G is:

[0026] G1, one convolution, one instance normalization operation and ReLU activation;

[0027] G2, two downsampling modules, each of which includes a convolution, an instance normalization operation and a ReLU activation;

[0028] G3, 9 residual networks, each residual block uses a ResNetV2 network, after an IN normalization layer, a Relu activation, a convolution layer and a Dropout layer;

[0029] G4, two upsampling modules, using the scaled convolution method, first use interpolation to enlarge the image, then use convolution operation to extract deep features, and finally perform an instance normalization operation and ReLU activation;

[0030] G5, one convolution, using Tanh activation.

[0031] The above-mentioned image deblurring method based on an improved generative adversarial network,

[0032] The convolution kernel size of the convolution described in G1 is 7×7, the number of convolution kernels is 64, and the step size is 1;

[0033] The convolution kernel size of the two downsampling modules in G2 is 3×3, the number of convolution kernels is 128 and 256 respectively, and the step size is 2;

[0034] The size of the convolution kernel of the convolution layer in G3 is 3×3, the number of convolution kernels is 256, and the step size is 1;

[0035] The convolution kernel size of the two upsampling modules in G4 is 3×3, the number of convolution kernels is 128 and 64 respectively, and the step size is 1;

[0036] The size of the convolution kernel of the convolution layer in G5 is 7×7, the number of convolution kernels is 3, and the stride is 1.

[0037] In the above-mentioned image deblurring method based on an improved generative adversarial network, the network structure of the generator G uses a global residual connection to directly connect the blurred image with the clear image, so that the network only learns the residual information between the two; it is expressed as:

[0038] I S =I B +I R (F7)

[0039] Among them, I S Indicates a clear image; I B I represents the input image to be deblurred; R represents the residual image.

[0040] In the above-mentioned image deblurring method based on an improved generative adversarial network, the network structures of the discriminator D1 and the discriminator D2 are the same and both include:

[0041] D01, one convolution, using LeakyRelu activation;

[0042] D02, one convolution, one instance normalization operation and LeakyRelu activation;

[0043] D03, one convolution, one instance normalization operation and LeakyRelu activation;

[0044] D04, one convolution, one instance normalization operation and LeakyRelu activation;

[0045] D05, one convolution, using Softplus activation.

[0046] The above-mentioned image deblurring method based on an improved generative adversarial network,

[0047] The convolution kernel size of the convolution described in D01 is 4×4, the number of convolution kernels is 64, and the step size is 2;

[0048] The convolution kernel size of the convolution described in D02 is 4×4, the number of convolution kernels is 128, and the step size is 2;

[0049] The convolution kernel size of the convolution described in D03 is 4×4, the number of convolution kernels is 256, and the step size is 2;

[0050] The convolution kernel size of the convolution described in D04 is 4×4, the number of convolution kernels is 512, and the step size is 1;

[0051] The convolution kernel size of the convolution described in D05 is 4×4, the number of convolution kernels is 1, and the step size is 1.

[0052] In the above-mentioned image deblurring method based on the improved generative adversarial network, the loss function of the generator in step 102 is expressed as:

[0053] L=L GAN +λ*L X (F2)

[0054] Among them, L GAN is the adversarial loss function starting from the texture details of the image and L GAN The expression is:

[0055]

[0056] L X is the content loss function starting from the content of the image and L X The expression is:

[0057]

[0058] The loss function expression of the first discriminator in step 102 is:

[0059]

[0060] The loss function expression of the second discriminator in step 102 is:

[0061]

[0062] Among them, λ is the weight parameter of the content loss function, and ρ is the forward objective function The weight parameter, and the value range of ρ is 0≤ρ≤1; ω is the reverse objective function The weight parameter of ω is 0≤ω≤1; ρ+ω=1; D1(G(z)) represents the discrimination result of the discriminator D1 on the image G(z) generated by the generator G, and D2(G(z)) represents the discrimination result of the discriminator D2 on the image G(z) generated by the generator G. [·] indicates that the blurred image z follows the generated data P z The mathematical expectation of the distribution [·], [·] indicates that the clear image x follows the real data P data The mathematical expectation of the distribution [·], is the feature map generated by the jth convolution before the i-th maximum pooling layer in the VGG19 network; i is the number of maximum pooling layers and its value is (0, 256), j is the number of convolutions and its value is (0, 256), w i,j is the width of the feature map, H i,j is the height of the feature map, I B is the input image to be deblurred, I S For clear images; For clear image I S The feature map generated after the (x, y)th pixel value of passes through the VGG19 network and the jth convolution before the i-th maximum pooling layer; G θG (I B ) is the clear image generated by inputting the blurred image into the generator G, The image I to be deblurred B Input the clear image G generated by the generator G θG (I B ) passes through the VGG19 network and the jth convolution before the i-th maximum pooling layer to generate the feature map.

[0063] The above-mentioned image deblurring method based on an improved generative adversarial network, in step 103, a training sample is selected, and the improved generative adversarial network constructed in step 101 is trained using the training sample and the loss function constructed in step 102. When the trained improved generative adversarial network is obtained, the GOPRO data set is selected as the training sample set, the blurred image is used as the input of the generator, and the data generated by the generator and the clear image are used as the input of the two discriminators.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] 1. Aiming at the ill-posed problem of non-blind deblurring method in estimating blur kernel, the present invention adopts deep learning method, and proposes an image deblurring method based on improved generative adversarial network on the basis of analyzing the theory of generative adversarial network. On the basis of generative adversarial network, an improved generative adversarial network is constructed, and the weighted idea is introduced to construct a new objective function. By adjusting the weight coefficient to balance the proportion of forward KL divergence and reverse KL divergence, a clear image can be better restored.

[0066] 2. The present invention utilizes the complementary characteristics of forward KL divergence and reverse KL divergence, and balances the proportion of forward and reverse KL divergence by adjusting the weight coefficient, so as to diversify the generated patterns, avoid the generation of other bad data on the basis of learning real data, and can better restore a clear image.

[0067] 3. In order to achieve image deblurring, the generator of the present invention uses 9 residual blocks, and the downsampling uses scaled convolution. The image is first interpolated and enlarged, and then convolution is used to extract features. The global residual connection is also used to learn only the residual information of the blurred image and the clear image. The two discriminators use the same structure and different loss functions. The network training speed is fast, the image deblurring efficiency is high, and the clear image can be better restored.

[0068] 4. The present invention constructs the loss functions of generator G, discriminator D1 and discriminator D2 from the two aspects of image texture details and image content. The generator loss function consists of two parts: adversarial loss and content loss. The adversarial loss mainly restores the texture details of the image, and the content loss focuses on restoring the content part of the image, which can better restore a clear image.

[0069] 5. The method of the present invention was simulated and tested, using PSNR and SSIM as evaluation criteria to judge the effect of image deblurring. The deblurring effect under two extreme cases of weight parameters ρ and ω was also discussed, proving that the present invention can better restore a clear image.

[0070] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a flowchart of the method of the present invention;

[0072] Figure 2 The structure diagram of the improved generative adversarial network of the present invention;

[0073] Figure 3 This is a model structure diagram of the generator G of the present invention;

[0074] Figure 4 : is a model structure diagram of the discriminator D1 and the discriminator D2 of the present invention;

[0075] Figure 5 The combined training data set provided by the embodiment of the present invention;

[0076] Figure 6 Deblurring effect and detail image after 4 times magnification under two extreme conditions provided by the embodiment of the present invention (a) deblurring effect and detail image after 4 times magnification when ρ=1, ω=0 (b) deblurring effect and detail image after 4 times magnification when ρ=0, ω=1;

[0077] Figure 7 A blurred image provided by an embodiment of the present invention;

[0078] Figure 8 A GAN network deblurring effect diagram provided by an embodiment of the present invention;

[0079] Fig. 9 A diagram showing the deblurring effect of the D2GAN network provided in an embodiment of the present invention;

[0080] Fig.10 A deblurring effect diagram of the DeblurGAN network provided in an embodiment of the present invention;

[0081] Fig.11 A diagram showing the deblurring effect of the D2WGAN network provided in an embodiment of the present invention;

[0082] Fig.12 A clear image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] like Figure 1 to Figure 4 As shown, the image deblurring method based on the improved generative adversarial network of the present invention comprises the following steps:

[0084] Step 1: Input the image to be deblurred into a pre-trained improved generative adversarial network; wherein the training process of the improved generative adversarial network is:

[0085] Step 101: Combining the KL divergence and the reverse KL divergence into an objective function by using a weighted method to construct an improved generative adversarial network; the improved generative adversarial network includes a generator G, and a discriminator D1 and a discriminator D2 connected in parallel to the output end of the generator G;

[0086] In this embodiment, in the case of a continuous random variable, the expression of the KL divergence in step 101 is:

[0087]

[0088] The KL divergence has the following three properties:

[0089] Feature 1: KL(P data ||P G )≥0; i.e. non-negative;

[0090] Characteristic 2: It does not satisfy the triangle inequality;

[0091] Feature 3: KL(P data ||P G )≠KL(P G ||P data ); that is, asymmetry, forward KL divergence KL(P data ||P G ) is not equal to the reverse KL divergence KL(P G ||P data );

[0092] In the case of a continuous random variable, the expression of the reverse KL divergence in step 101 is:

[0093]

[0094] The reverse KL divergence has the following three properties:

[0095] Feature 1: KL(P G ||P data )≤0; i.e. non-positive;

[0096] Feature 2: Satisfies the triangle inequality;

[0097] Feature 3: KL(P G ||P data )≠KL(P data ||P G ); that is, asymmetry, reverse KL divergence KL(P G ||P data ) is not equal to the forward KL divergence KL(P data ||P G );

[0098] Among them, P G represents the generative model, P data represents the true distribution, and x represents the variable;

[0099] In specific implementation, in the forward KL divergence, when P data →0 and P G >0, KL(P data ||P G )→0; when P data >0,P G →0, KL(P data ||P G )→∞. It can be seen that the positive KL divergence will encourage the generation of model PG Cover all real distributions P as much as possible data >0, without avoiding P G ≈0, that is, when using the forward KL divergence to measure the difference between the two distributions, the data generated by the generator will cover multiple modes of the real data, however, it will also generate some samples that do not exist in the real data;

[0100] In the reverse KL divergence, when P data →0 and P G >0, KL(P data ||P G )→∞; when P data >0 and P G →0, KL(P data ||P G )→0. It can be seen that the reverse KL divergence encourages the generation of model P G Avoid the true distribution P as much as possible data ≈0, regardless of whether the generated model covers the true distribution P data >0, as long as the true distribution does not tend to 0, the generated data will tend to be more like a single mode of the true data, ignoring other modes and causing the problem of mode collapse.

[0101] By utilizing the complementary characteristics of forward KL divergence and reverse KL divergence, the weight coefficient is adjusted to balance the proportion of forward and reverse KL divergence, so as to diversify the generated patterns and avoid the generation of other bad data on the basis of learning real data.

[0102] In this embodiment, the network structure of the generator G is:

[0103] G1, one convolution, one instance normalization operation and ReLU activation;

[0104] The convolution kernel size of the convolution described in G1 is 7×7, the number of convolution kernels is 64, and the step size is 1;

[0105] In order to extract the main features of the image and reduce the amount of calculation, a convolution is performed on the input image to be deblurred. In order to improve the accuracy of the model, an instance normalization operation and a modified linear unit Relu activation function are performed.

[0106] G2, two downsampling modules, each of which includes a convolution, an instance normalization operation and a ReLU activation;

[0107] The convolution kernel size of the two downsampling modules in G2 is 3×3, the number of convolution kernels is 128 and 256 respectively, and the step size is 2;

[0108] G3, 9 residual networks, each residual block (ResBlock) uses a ResNetV2 network, after an IN normalization layer, a Relu activation, a convolution layer and a Dropout layer; the random inactivation ratio is 0.5 to prevent overfitting during training;

[0109] The size of the convolution kernel of the convolution layer in G3 is 3×3, the number of convolution kernels is 256, and the step size is 1;

[0110] Using residual blocks can deepen the depth of the network, extract deeper features, and effectively alleviate the phenomenon of gradient disappearance.

[0111] G4, two upsampling modules, using the scaled convolution method, first use interpolation to enlarge the image, then use convolution operation to extract deep features, and finally perform an instance normalization operation and ReLU activation;

[0112] The convolution kernel size of the two upsampling modules in G4 is 3×3, the number of convolution kernels is 128 and 64 respectively, and the step size is 1;

[0113] The size of the output image of the input image to be deblurred after passing through the convolution layer, the downsampling module, and the residual network is only 1 / 4 of the original size. The purpose of upsampling is to restore the input image to be deblurred to its original size. The present invention uses a scaled convolution method, which can avoid the problem of checkerboard effect caused by deconvolution during the restoration process.

[0114] G5, one convolution, using Tanh activation.

[0115] The size of the convolution kernel of the convolution layer in G5 is 7×7, the number of convolution kernels is 3, and the stride is 1.

[0116] The network structure of the generator G of the present invention is a fully convolutional network, which does not use fully connected layers and pooling layers, and can effectively improve the computing speed of image deblurring.

[0117] When using deep learning to restore blurred images, a low-resolution image is often filled with details to obtain a high-resolution image, and deconvolution is generally used. Deconvolution involves a one-to-many mapping relationship. When the size of the convolution kernel cannot be divided by the step size, uneven overlap is likely to occur, and the image distribution is uneven. Some pixels are calculated multiple times, while some pixels are forgotten, resulting in an uneven image. We expect to obtain a smooth image, but what we actually get are some small squares of varying depths, and a checkerboard effect is likely to occur when generating an image. The present invention chooses to use a scaled convolution method when designing sampling on the network, which is divided into two steps: interpolation and ordinary convolution. Interpolation is used to enlarge the image, and ordinary convolution is used thereafter to extract deep features to better repair the image, without the problem of uneven overlap in deconvolution.

[0118] In this embodiment, the network structure of the generator G uses a global residual connection to directly connect the blurred image with the clear image, so that the network only learns the residual information between the two; it is expressed as:

[0119] I S =I B +I R (F7)

[0120] Among them, I S Indicates a clear image; I B I represents the input image to be deblurred; R represents the residual image.

[0121] Blurry images and clear images are very similar in content, color, style, etc., and are also very close in pixel value, except for the large difference in the details of the high-frequency part. Using global residual connections to let the generator only learn the residual information between the two images can reduce the amount of network calculations, speed up network convergence, prevent the generated image from having color difference with the clear image, and improve the generalization ability of the model.

[0122] In this embodiment, the network structures of the discriminator D1 and the discriminator D2 are the same and both include:

[0123] D01, one convolution, using LeakyRelu activation;

[0124] The convolution kernel size of the convolution described in D01 is 4×4, the number of convolution kernels is 64, and the step size is 2;

[0125] D02, one convolution, one instance normalization operation and LeakyRelu activation;

[0126] The convolution kernel size of the convolution described in D02 is 4×4, the number of convolution kernels is 128, and the step size is 2;

[0127] D03, one convolution, one instance normalization operation and LeakyRelu activation;

[0128] The convolution kernel size of the convolution described in D03 is 4×4, the number of convolution kernels is 256, and the step size is 2;

[0129] D04, one convolution, one instance normalization operation and LeakyRelu activation;

[0130] The convolution kernel size of the convolution described in D04 is 4×4, the number of convolution kernels is 512, and the step size is 1;

[0131] D05, one convolution, using Softplus activation.

[0132] The convolution kernel size of the convolution described in D05 is 4×4, the number of convolution kernels is 1, and the step size is 1.

[0133] In specific implementation, the slope of the LeakyRelu activation function is 0.2.

[0134] In the prior art, the last layer of a common discriminator is a fully connected layer, and the output is the probability that the input sample comes from real data, that is, the result is a real number. The present invention improves the common discriminator and adopts a fully convolutional network, which can better restore image details during training.

[0135] Step 102: construct the loss functions of the generator G, the discriminator D1 and the discriminator D2 from the two aspects of image texture details and image content;

[0136] In this embodiment, the loss function of the generator in step 102 is expressed as:

[0137] L=L GAN +λ*L X (F2)

[0138] That is, the generator loss function consists of two parts: adversarial loss and content loss; adversarial loss mainly restores the texture details of the image, and content loss focuses on restoring the content part of the image.

[0139] Among them, L GAN is the adversarial loss function starting from the texture details of the image and L GAN The expression is:

[0140]

[0141] L X is the content loss function starting from the content of the image and L X The expression is:

[0142]

[0143] The loss function expression of the first discriminator in step 102 is:

[0144]

[0145] Discriminator D1 mainly focuses on real data;

[0146] The loss function expression of the second discriminator in step 102 is:

[0147]

[0148] The discriminator D2 mainly focuses on the data generated by the generator;

[0149] Among them, λ is the weight parameter of the content loss function, and ρ is the forward objective function The weight parameter, and the value range of ρ is 0≤ρ≤1; ω is the reverse objective function The weight parameter of ω is 0≤ω≤1; ρ+ω=1; D1(G(z)) represents the discrimination result of the discriminator D1 on the image G(z) generated by the generator G, and D2(G(z)) represents the discrimination result of the discriminator D2 on the image G(z) generated by the generator G. Denotes that the blurred image z is subject to the generated data P z The mathematical expectation of the distribution [·], Indicates that the clear image x obeys the real data P data The mathematical expectation of the distribution [·], is the feature map generated by the jth convolution before the i-th maximum pooling layer in the VGG19 network; i is the number of maximum pooling layers and its value is (0, 256), j is the number of convolutions and its value is (0, 256), w i,j is the width of the feature map, H i,j is the height of the feature map, I B is the input image to be deblurred, I S For clear images; For clear image I S The feature map generated after the (x, y)th pixel value of passes through the VGG19 network and the jth convolution before the i-th maximum pooling layer; G θG (I B ) is the clear image generated by inputting the blurred image into the generator G, The image I to be deblurred B Input the clear image G generated by the generator G θG (I B )'s (x,y)th pixel value passes through the VGG19 network and the feature map generated after the jth convolution before the i-th maximum pooling layer;

[0150] During specific implementation, the value of λ is 100.

[0151] Adversarial loss function: The generator hopes that the data it generates can deceive the discriminator D1 and make it believe it is true; it also hopes that the "fake" data it generates can be identified by the discriminator D2. Therefore, it hopes that the discriminator D1 will get a high score for the generated data and that the discriminator D2 will get a low score for the generated data.

[0152] Content loss function: The present invention uses the perceptual loss commonly used in the image conversion process, and sends the deblurred image and the clear image to the pre-trained VGG-19 network respectively, and calculates the error sum between the feature map pixels obtained in each layer.

[0153] The main purpose of the discriminator is to determine whether the input data comes from the real distribution or the distribution generated by the generator. The network structure of discriminator D1 and discriminator D2 is the same, but because of their different loss functions during training, the two discriminators will optimize in different directions. Discriminator D1 scores high for real data and low for data generated by the generator, while discriminator D2 scores high for data generated by the generator and low for real data.

[0154] Step 103: Select training samples, and use the training samples and the loss function constructed in step 102 to train the improved generative adversarial network constructed in step 101 to obtain a trained improved generative adversarial network;

[0155] In this embodiment, the training samples are selected in step 103, and the improved generative adversarial network constructed in step 101 is trained using the training samples and the loss function constructed in step 102. When the trained improved generative adversarial network is obtained, the GOPRO data set is selected as the training sample set, the blurred image is used as the input of the generator, and the data generated by the generator and the clear image are used as the input of the two discriminators.

[0156] Step 2: The improved generative adversarial network deblurs the image to be deblurred to obtain a deblurred image.

[0157] In the improved generative adversarial network, the generator G, the discriminator D1 and the discriminator D2 are all multi-layer perceptrons. After the image to be deblurred is input, the generator G will generate a relatively clear image G(z) learned by itself based on the input, and then the image G(z) generated by the generator and the clear image x corresponding to the blurred image are input into the discriminator D1 and the discriminator D2. The discriminator D1 scores a high score for the clear image x and obtains D1(x), and scores a low score for the image G(z) generated by the generator G and obtains D1(G(z)); the discriminator D2 scores a high score for the image G(z) generated by the generator G and obtains D2(G(z)), and scores a low score for the clear image x and obtains D2(x).

[0158] The purpose of the generator G is to make the generated G(z) image deceive both the discriminator D1 and the discriminator D2 at the same time, and the three will compete with each other until a balance is reached among them.

[0159] In order to verify the technical effect that the present invention can produce, a simulation verification is carried out, and the specific simulation process is as follows:

[0160] 1. Experimental environment:

[0161] The simulation experiment environment of the present invention is: the processor is Inter Xeon E5-2620 v4, the operating system is 64-bit Windows 10, the graphics card is NVIDIA GeForce RTX 2080Ti, the pycharm editor is used, and the PyTorch deep learning framework is used. Each training is iterated 300 times in total, the learning rate is 0.0001 in the first 150 iterations, and the learning rate gradually decays to 0 in the last 150 iterations. The Adam optimization algorithm is selected, and the batch size is set to 8. The GOPRO data set is used, in which there are 2103 pairs of training data and 1111 pairs of test data.

[0162] 2. Simulation experiment content:

[0163] The simulation experiment uses the GOPRO training set as the training sample. The GOPRO dataset was proposed by Nah et al. and is collected by a GOPRO camera in high-speed motion. The main scenes include buildings, parks, vehicles, pedestrians, etc. The GOPRO dataset uses a high-definition camera to shoot 33 clear videos at 240fps, of which 22 are training datasets and 11 are test datasets. In each video image, the adjacent 7-13 frames are averaged to obtain a blurred image. The blurred image obtained in this way takes into account the influence of factors such as sensors and light in real scenes. A total of 3214 pairs of blurred-clear images are generated, with about 100 images for each type of scene, and the resolution is 1280×720. There are 2103 pairs of training data and 1111 pairs of test data. Before inputting the data set, the blurred image and the clear image are merged into one image; secondly, the merged input image is cropped to a resolution of 1280×360; finally, a 256×256 blurred image and the corresponding clear image are randomly cropped from the cropped image for training, and the input image pixel values ​​are normalized to [-1, 1] to speed up the convergence of the network. During training, a 256×256 image is randomly cropped from the data set each time, so that the data set input to the network each time is different, which is equivalent to expanding the data set. Figure 5 This is a partially merged training data set, where the red boxes are blurry and clear images that were randomly cropped.

[0164] The present invention introduces two weights ρ and ω, and uses experiments to find the best parameters to obtain a better deblurring model. The weight parameters ρ and ω are introduced in D2WGAN, where ρ+ω=1 and 0≤ρ,ω≤1. When the ρ value is large, the network focuses on the forward KL divergence; when the ω value is large, the network focuses on the reverse KL divergence. The present invention first verifies the deblurring effect in two extreme cases: ρ=1, ω=0 and ρ=0, ω=1, to determine which direction the weights should focus on for better results. Figure 6The deblurring effect pictures in two extreme cases and the 4x magnified detail pictures; the PSNR and SSIM indicators in Table 1 are the average evaluation indicators of all pictures in the test set.

[0165] Table 1 Evaluation quality of D2WGAN in two extreme cases

[0166]

[0167] from Figure 6 It can be seen that when the weight ρ=1, ω=0, the network focuses on the forward KL divergence, and the deblurred image has a chessboard artifact, which is composed of deep and shallow blocks, affecting people's intuitive experience, and the deblurring effect is poor; while ρ=0, ω=1, the network focuses on the reverse KL divergence, and the generated image is smoother and the details are better processed. And from the indicators in Table 1, it can be seen that when focusing on the reverse KL divergence, whether it is PSNR or SSIM, the results are better than focusing on the forward KL divergence, which provides a basis for subsequent training. In the subsequent training process, the weight will focus more on the reverse KL divergence.

[0168] In order to find the optimal weights of the D2WGAN algorithm when combining the forward KL divergence and the reverse KL divergence, the evaluation index data in Tables 2 and 3 are used to determine the optimal weights.

[0169] Table 2 Evaluation quality under different weights

[0170]

[0171] Table 3 Evaluation quality under different weights

[0172]

[0173] It can be seen from Table 2 that the evaluation index is higher when the weight ρ=0.1, ω=0.9. The present invention continues to determine the search range and continues to train the four weights near ρ=0.1, ω=0.9 in Table 3 to find better weight parameters for better training models.

[0174] According to the interval approximation method, combined with Table 1, Table 2 and Table 3, the present invention finds the currently optimal parameters ρ=0.1, ω=0.9, and the present invention will conduct experiments based on this weight.

[0175] The deblurring effects of the D2WGAN algorithm are compared with those of the original GAN, D2GAN, and DeblurGAN methods. To facilitate viewing of the deblurred results, the displayed image is halved from the original image. Figure 7 For blurred images. Figure 8 For the GAN deblurring effect, Fig. 9 For the D2GAN deblurring effect, Fig.10 For the deblurGAN deblurring effect, Fig.11 For the deblurring effect of the present invention, Fig.12 For clear images. When testing PSNR and SSIM, the test set resolution is 256×256. Table 4 shows the average peak signal-to-noise ratio (PSNR) and image structure similarity (SSIM) of different methods in all test sets.

[0176] Table 4 Average quality evaluation results of different methods on all test sets

[0177]

[0178] 3. Analysis of simulation results:

[0179] By comparison Figure 7-12 The deblurred image of the present invention has a better visual experience. The reconstructed image is real and natural, and is basically consistent with the original image in structure, color characteristics, and content. It can effectively restore the details of the motion blurred image, and there are no checkerboard artifacts, sharpening and other phenomena, which improves the image visual effect.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0184] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. An image deblurring method based on improved generative adversarial network, It is characterized in that The method comprises the following steps: Step 1: Input the image to be deblurred into a pre-trained improved generative adversarial network; wherein the training process of the improved generative adversarial network is: Step 101: Use a weighted method to combine KL divergence and reverse KL divergence into an objective function to construct an improved generative adversarial network; the improved generative adversarial network includes a generator G and a discriminator D connected in parallel to the output end of the generator G. 1 and the discriminator D 2 ; The network structure of the generator G is: G1, one convolution, one instance normalization operation and ReLU activation; G2, two downsampling modules, each of which includes a convolution, an instance normalization operation and a ReLU activation; G3, 9 residual networks, each residual block uses a ResNetV2 network, after an IN normalization layer, a Relu activation, a convolution layer and a Dropout layer; G4, two upsampling modules, using the scaled convolution method, first use interpolation to enlarge the image, then use convolution operation to extract deep features, and finally perform an instance normalization operation and ReLU activation; G5, one convolution, using Tanh activation; The discriminator D 1 and the discriminator D 2 The network structures are the same and include: D01, one convolution, using LeakyRelu activation; D02, one convolution, one instance normalization operation and LeakyRelu activation; D03, one convolution, one instance normalization operation and LeakyRelu activation; D04, one convolution, one instance normalization operation and LeakyRelu activation; D05, one convolution, using Softplus activation; Step 102: Based on the image texture details and image content, construct the generator G and the discriminator D 1 and the discriminator D 2 The loss function of Step 103, select training samples, use the training samples and the loss function constructed in step 102 to train the improved generative adversarial network constructed in step 101, and obtain a trained improved generative adversarial network; Step 2, the improved generative adversarial network deblurs the image to be deblurred to obtain a deblurred image.

2. The image deblurring method based on an improved generative adversarial network according to claim 1, Features: The expression of KL divergence in step 101 is: The KL divergence has the following three properties: Feature 1: KL(P data ||P G )≥0; Characteristic 2: It does not satisfy the triangle inequality; Feature 3: KL(P data ||P G )≠KL(P G ||P data ); The expression of the reverse KL divergence in step 101 is: The reverse KL divergence has the following three properties: Feature 1: KL(P G ||P data )≤0; Feature 2: Satisfies the triangle inequality; Feature 3: KL(P G ||P data )≠KL(P data ||P G ); Among them, P G represents the generative model, P data represents the true distribution and x represents the variable.

3. According to the image deblurring method based on improved generative adversarial network according to claim 1, the network structure of the generator G in step 1 is: Features: The convolution kernel size of the convolution described in G1 is 7×7, the number of convolution kernels is 64, and the step size is 1; The convolution kernel size of the two downsampling modules in G2 is 3×3, the number of convolution kernels is 128 and 256 respectively, and the step size is 2; The size of the convolution kernel of the convolution layer in G3 is 3×3, the number of convolution kernels is 256, and the step size is 1; The convolution kernel size of the two upsampling modules in G4 is 3×3, the number of convolution kernels is 128 and 64 respectively, and the step size is 1; The size of the convolution kernel of the convolution layer in G5 is 7×7, the number of convolution kernels is 3, and the stride is 1.

4. An image deblurring method based on an improved generative adversarial network according to claim 1 or 3, Features: The network structure of the generator G uses a global residual connection to directly connect the blurred image with the clear image, so that the network only learns the residual information between the two; it is expressed as: I S =I B +I R (F7) Among them, I S Indicates a clear image; I B I represents the input image to be deblurred; R represents the residual image.

5. According to the image deblurring method based on improved generative adversarial network as claimed in claim 1, the network structure of the discriminator D1 and the discriminator D2 is: Features: The convolution kernel size of the convolution described in D01 is 4×4, the number of convolution kernels is 64, and the step size is 2; The convolution kernel size of the convolution described in D02 is 4×4, the number of convolution kernels is 128, and the step size is 2; The convolution kernel size of the convolution described in D03 is 4×4, the number of convolution kernels is 256, and the step size is 2; The convolution kernel size of the convolution described in D04 is 4×4, the number of convolution kernels is 512, and the step size is 1; The convolution kernel size of the convolution described in D05 is 4×4, the number of convolution kernels is 1, and the step size is 1.

6. The image deblurring method based on an improved generative adversarial network according to claim 1, Features: The loss function of the generator in step 102 is expressed as: L=L GAN +λ*L X (F2) Among them, L GAN is the adversarial loss function starting from the texture details of the image and L GAN The expression is: L X is the content loss function starting from the content of the image and L X The expression is: The loss function expression of the first discriminator in step 102 is: The loss function expression of the second discriminator in step 102 is: Among them, λ is the weight parameter of the content loss function, and ρ is the forward objective function The weight parameter, and the value range of ρ is 0≤ρ≤1; ω is the reverse objective function The weight parameter of ω is 0≤ω≤1; ρ+ω=1; D 1 (G(z)) represents the discriminator D 1 The discrimination result of the image G(z) generated by the generator G, D 2 (G(z)) represents the discriminator D 2 The judgment result of the image G(z) generated by the generator G, Denotes that the blurred image z is subject to the generated data P z The mathematical expectation of the distribution [·], Indicates that the clear image x obeys the real data P data The mathematical expectation of the distribution [·], is the feature map generated by the jth convolution before the i-th maximum pooling layer in the VGG19 network; i is the number of maximum pooling layers and its value is (0, 256), j is the number of convolutions and its value is (0, 256), w i,j is the width of the feature map, H i,j is the height of the feature map, I B is the input image to be deblurred, I S For clear images; For clear image I S The feature map generated after the (x, y)th pixel value of passes through the VGG19 network and the jth convolution before the i-th maximum pooling layer; G θG (I B ) is the clear image generated by inputting the blurred image into the generator G, The image I to be deblurred B Input the clear image G generated by the generator G θG (I B ) passes through the VGG19 network and the jth convolution before the i-th maximum pooling layer to generate the feature map.

7. The image deblurring method based on an improved generative adversarial network according to claim 1, Features: In step 103, training samples are selected, and the improved generative adversarial network constructed in step 101 is trained using the training samples and the loss function constructed in step 102. When the trained improved generative adversarial network is obtained, the GOPRO data set is selected as the training sample set, the blurred image is used as the input of the generator, and the data generated by the generator and the clear image are used as the input of the two discriminators.

Images