A method for image defocusing based on fuzzy conditional residual network

Abstract

The present invention relates to an image defocusing and blurring method based on a fuzzy conditional residual network. The method comprises: step 1, preparing a training set of an image defocusing and blurring model; step 2, constructing a fuzzy conditional network; step 3, constructing a conditional residual restoration network using a conditional residual module; and step 4, adopting a two-stage training strategy for image defocusing and blurring model training. The present invention can realize image defocusing and blurring more quickly and obtain a clear full-focus image of higher quality.

Description

Technical Field

[0001] The present invention belongs to the field of image deblurring, and in particular relates to an image defocusing and blurring method based on a fuzzy conditional residual network. Background Art

[0002] Due to hardware limitations, out-of-focus blur is inevitable when the depth range of the scene area is large. Out-of-focus blur is generally caused by shallow depth of field imaging under large aperture, that is, the camera can only focus on a certain range of shadow depth of field, resulting in blur in other scene areas. In out-of-focus blurred images, the spatial variance of blur size is large, while the spatial variance of blur shape is small, so it is difficult to eliminate this blur and restore the details of the out-of-focus image. Recently, some studies have addressed this problem through dual-pixel (DP) sensors found on most modern cameras. Although DP sensors are widely used in many fields, such as depth estimation, defocus blur removal, reflection removal, and synthesis of degrees of freedom, etc. However, in real scenes, we can only obtain a single out-of-focus blur image, not a binocular image, which makes it difficult to apply the current binocular defocus blur algorithm to practical engineering. Therefore, compared with the binocular image, which can apply more information, defocus deblurring of a single image is still a challenging problem.

[0003] According to the deblurring process, defocus blur removal techniques can be classified into two categories: 1) One is a two-stage cascade method, which consists of two steps: the first step is defocus blur detection, and the second step is non-blind deblurring through deconvolution. 2) Directly remove defocus blur through end-to-end deep learning methods.

[0004] In the two-stage method, the out-of-focus blur map is usually estimated first, and then the blurred area of ​​the out-of-focus blur map knowledge is restored by using deconvolution. Out-of-focus blur map estimation is a relatively important stage in this method. Representative works include: Karaali[1] et al. use identifiable hand-crafted features such as image gradient to calculate the difference between blurred edges and original image edges. Similarly, Shi[2] et al. use edge representation and Yi[3] et al. use local binary patterns to measure blur maps. Recently, some studies have used deep learning-based methods to estimate out-of-focus blur maps. For example, Park[4] et al. combine deep features and hand-crafted features to estimate the amount of edge blur. Zhao[5] et al. proposed a fully convolutional network to effectively estimate blur maps. Although these methods have achieved good results in estimating blur maps, their common disadvantage is that the out-of-focus blur map must be converted into a binary mask before the second stage, so the information of the estimated out-of-focus blur map will not be fully utilized.

[0005] In the end-to-end learning method, Aboulaim et al. [6] proposed an end-to-end deep learning defocus blurring method DPDNet, which does not rely on a specific blur model, but directly restores a clear image through a neural network. Due to the powerful nonlinear expression ability of the neural network, DPDNet is significantly better than previous methods in defocus blurring in real scenes. This method also provides a binocular defocus blur dataset (DPDD). However, since this method uses a simple UNet structure, the model cannot handle blur with large spatial variations, resulting in poor restoration effect. Therefore, in order to solve this problem, many methods have been proposed, such as Hyeongseok et al. [7] who used the spatial variation reversible kernel characteristics of defocus blur to design a kernel shared parallel atrous convolution (KPAC) network. In order to better handle spatial variations and defocus blur in defocus blur images, Junyong et al. [8] proposed an adaptive iterative filtering network using iterative adaptive convolution and achieved excellent results.

[0006] Although relevant research has achieved good image defocusing and blurring effects, sometimes the reconstruction effect is not satisfactory due to the complexity of the blur kernel of the real scene, and even image texture errors and artifacts may occur. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide an image defocusing method based on a fuzzy conditional residual network, which can remove defocusing blur more quickly and obtain an image with better texture details.

[0008] To achieve the above object, the technical solution of the present invention is: a method for removing defocus blur of an image based on a fuzzy conditional residual network, comprising the following steps:

[0009] Step 1: Create a training set for the image defocusing model;

[0010] Step 2: construct a fuzzy conditional network;

[0011] Step 3: Use the conditional residual module to construct a conditional residual restoration network;

[0012] Step 4: The image defocus blur model training adopts a two-stage training strategy.

[0013] In one embodiment of the present invention, step 1 is specifically implemented as follows:

[0014] Step 1.1, select the image pairs of out-of-focus blurred images and high-definition images in the DPDD training set as training;

[0015] Step 1.2, crop the image into 480x480 size tiles and build an image dataset;

[0016] Step 1.3: For the input high-definition image I target and out-of-focus blurred image I blur ,The image dataset is augmented by flipping, rotating, and randomly cropping the image into blocks of 256 size as the final training set.

[0017] In one embodiment of the present invention, step 2 is specifically implemented as follows:

[0018] Step 2.1, using the LBP algorithm module to build a fuzzy conditional network, extract features from the fuzzy image estimated by the input LBP, and obtain the corresponding conditional features, which are used to add the out-of-focus fuzzy image information to the conditional residual restoration network, thereby effectively restoring the texture details of the fuzzy area;

[0019] The blur condition network uses the LBP algorithm to estimate the out-of-focus blur map, and its calculation formula is:

[0020] F LBP =H LBP (I blur )

[0021] H LBP represents the LBP algorithm module in the fuzzy conditional network, F LBP represents the estimated out-of-focus blur map;

[0022] Step 2.2: Use a shallow feature extraction convolution and three downsampling convolutions to match the features of the out-of-focus blur image to the conditional residual restoration network; the calculation formula can be expressed as:

[0023] F cond1 =Conv down (Conv 1 (F LBP ))

[0024] F cond2 =Conv down ((F cond1 ))

[0025] F cond3 =Conv down (F cond2 )

[0026] F cond1 ,F cond2 ,F cond3 Respectively represent the three conditional features input into the conditional residual recovery network; Conv down Represents 2 times downsampling convolution, Conv 1 Represents a shallow feature extraction layer.

[0027] In one embodiment of the present invention, step 3 is specifically implemented as follows:

[0028] Step 3.1, conditional residual module, its formula is:

[0029] F res =Conv(LReLU(Conv(F input ))

[0030] F c =Conv(F cond )

[0031] F ouput =F cond *F res +F input

[0032] Where Conv represents the convolution operation, LReLU represents the LeakyReLU activation function, and F cond Represents the conditional features of the input;

[0033] Step 3.2: Obtain the conditional features F corresponding to different scales cond1 ,F cond2 ,F cond3 Finally, a conditional residual restoration network is constructed, in which the last layer of downsampling uses 4 conditional residual modules, and the other layers use 2 conditional residual modules, and two skip connection layers are added in the middle to achieve the best network performance; the specific formula is as follows:

[0034] Downsampling:

[0035] F 1 =H CRB×2 (Conv(I blur ),F cond1 )

[0036] F 2 =H CRB×2 (Conv down (F 1 ),F cond2 )

[0037] F 3 =H CRB×4 (Conv down (F 2 ),F cond2 )

[0038] Upsampling:

[0039] f up1 =H CRB×2 (Conv c (cat(F 3↑ 2 ,Conv skip (F 2 ))),F cond2 )

[0040] F up2 =H CRB×2 (Conv c (cat(F up1 ↑ 2 ,Conv skip (F 1 ) )) ,F cond1 )

[0041] F output =Conv Re (F up2 )

[0042] Among them, H CRB×2 Indicates the use of two conditional residual modules, H CRB×4 Indicates the use of 4 conditional residual modules; Conv,Conv c ,Conv skip ,cat and Conv Re They represent the initial feature extraction convolution layer, channel adjustment convolution layer, skip connection convolution layer, channel connection operation, and reconstruction convolution layer, ↑ 2 Indicates a bilinear upsampling 2x operation.

[0043] In one embodiment of the present invention, step 4 is specifically implemented as follows:

[0044] Step 4.1: Input the input image into the first-stage network, that is, input it into the blur conditional network and the first-stage conditional residual recovery network to train and restore a relatively clear image I output1 , and then use the output of the first-stage conditional residual recovery network as the input of the second-stage network, and input it into the fuzzy conditional network again to obtain new conditional features, and further use the second-order conditional residual recovery network to recover more refined features I output2 , thus obtaining a clearer image; the specific expression is as follows:

[0045] I output1 =H Deblur1 (I blur ,H cond (I blur ))

[0046] I output2 =H Deblur2 (I output1 ,H cond (I blur ))

[0047] Among them, H Deblur represents the conditional residual recovery network, H cond represents a fuzzy conditional network;

[0048] Step 4.2: The output of the two-stage network is L 1 The loss function calculates the point-to-point loss function of the image, and the model loss function is expressed as:

[0049] Loss = L 1 (I target ,I output1 )+L 1 (I target ,I output2 )

[0050] L 1 (x,y)=||xy|| 1

[0051] Step 4.3: The model training adopts a progressive training strategy, which divides the training process into multiple sub-training cycles. The sub-training cycles are trained in sequence. The training starts with a large learning rate and the learning rate is gradually reduced after each sub-training cycle until the iteration is completed.

[0052] Compared with the prior art, the present invention has the following beneficial effects: the present invention can more quickly remove defocus blur from an image and obtain a clear full-focus image of higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the principle of the image defocusing method based on the fuzzy conditional residual network of the present invention;

[0054] Figure 2 It is a schematic diagram of the principle of the conditional residual module in the fuzzy conditional residual network of the present invention.

[0055] Figure 3 It is a defocus blur image effect diagram generated by LBP in the image defocus blur method based on the fuzzy conditional residual network of the present invention.

[0056] Figure 4 This is a rendering of the image defocusing blurring method based on the fuzzy conditional residual network of the present invention.

[0057] Figure 5 This is a comparison of the image defocusing blurring method based on the fuzzy conditional residual network of the present invention. Figure 1 .

[0058] Figure 6 This is a comparison of the image defocusing blurring method based on the fuzzy conditional residual network of the present invention. Figure 2 . DETAILED DESCRIPTION

[0059] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0060] The present invention provides an image defocusing blurring method based on a fuzzy conditional residual network, comprising the following steps:

[0061] Step 1: Create a training set for the image defocusing model;

[0062] Step 1.1, select the image pairs of out-of-focus blurred images and high-definition images in the DPDD training set as training;

[0063] Step 1.2, crop the image into 480x480 size tiles and build an image dataset;

[0064] Step 1.3: For the input high-definition image I target and out-of-focus blurred image I blur The image dataset is augmented by flipping, rotating, and randomly cropping the image into blocks of 256 size as the final training set, which is then input into the image defocusing model for training.

[0065] Step 2: Construct a blur condition network. This paper finds that the LBP algorithm can effectively estimate the out-of-focus blur image.

[0066] Step 2.1, use the LBP algorithm module to build a Figure 1 The fuzzy conditional network of the network structure extracts features from the fuzzy image estimated by the input LBP to obtain corresponding conditional features, which are used to add out-of-focus fuzzy image information to the conditional residual restoration network, thereby effectively restoring texture details for the fuzzy area;

[0067] The blur condition network uses the LBP algorithm to estimate the out-of-focus blur map, and its calculation formula is:

[0068] F LBP =H LBP (I blur )

[0069] H LBP represents the LBP algorithm module in the fuzzy conditional network, F LBP represents the estimated out-of-focus blur map;

[0070] Step 2.2: Use a shallow feature extraction convolution and three downsampling convolutions to match the features of the out-of-focus blur image to the conditional residual restoration network; the calculation formula can be expressed as:

[0071] F cond1 =Conv down(Conv 1 (F LBP ))

[0072] F cond2 =Conv down ((F cond1 ))

[0073] F cond3 =Conv down (F cond2 )

[0074] F cond1 ,F cond2 ,F cond3 Respectively represent the three conditional features input into the conditional residual recovery network; Conv down Represents 2 times downsampling convolution, Conv 1 Represents a shallow feature extraction layer.

[0075] Step 3: Use the conditional residual module to construct a conditional residual restoration network;

[0076] Step 3.1, conditional residual module such as Figure 2 As shown, the formula is:

[0077] F res =Conv(LReLU(Conv(F input ))

[0078] F c =Conv(F cond )

[0079] F output =F cond *F res +F input

[0080] Where Conv represents the convolution operation, LReLU represents the LeakyReLU activation function, and F cond Represents the conditional features of the input;

[0081] Step 3.2: Obtain the conditional features F corresponding to different scales cond1 ,F cond2 ,F cond3 After that, the conditional residual recovery network is constructed. The specific structure is as follows Figure 2 As shown in the figure, the last layer of downsampling uses 4 conditional residual modules, and the other layers use 2 conditional residual modules. Two skip connection layers are added in the middle to achieve the best network performance. The specific formula is as follows:

[0082] Downsampling:

[0083] F1 =H CRB×2 (Conv(I blur ),F cond1 )

[0084] F 2 =H CRB×2 (Conv down (F 1 ),F cond2 )

[0085] F 3 =H CRB×4 (Conv down (F 2 ),F cond2 )

[0086] Upsampling:

[0087] F up1 =H CRB×2 (Conv c (cat(F 3 ↑ 2 ,Conv skip (F 2 ))),F cond2 )

[0088] F up2 =H CRB×2 (Conv c (cat(F up1 ↑ 2 ,Conv skip (F 1 ))),F cond1 )

[0089] F output =Conv Re (F up2 )

[0090] Among them, H CRB×2 Indicates the use of two conditional residual modules, H CRB×4 Indicates the use of 4 conditional residual modules; Conv,Conv c ,Conv skip ,cat and Conv Re They represent the initial feature extraction convolution layer, channel adjustment convolution layer, skip connection convolution layer, channel connection operation, and reconstruction convolution layer, ↑ 2 Indicates a bilinear upsampling 2x operation.

[0091] Step 4: The image defocus blur model training adopts a two-stage training strategy;

[0092] Step 4.1: Input the input image into the first-stage network, that is, input it into the blur conditional network and the first-stage conditional residual recovery network to train and restore a relatively clear image I output1 , and then use the output of the first-stage conditional residual recovery network as the input of the second-stage network, and input it into the fuzzy conditional network again to obtain new conditional features, and further use the second-order conditional residual recovery network to recover more refined features I output2 , thus obtaining a clearer image; the specific expression is as follows:

[0093] I output1 =H Deblur1 (I blur ,H cond (I blur ))

[0094] I output2 =H Deblur2 (I output1 ,H cond (I blur ))

[0095] Among them, H Deblur represents the conditional residual recovery network, H cond represents a fuzzy conditional network;

[0096] Step 4.2: The output of the two-stage network is L 1 The loss function calculates the point-to-point loss function of the image, and the model loss function is expressed as:

[0097] Loss = L 1 (I target ,I output1 )+L 1 (I target ,I output2 )

[0098] L 1 (x,y)=||xy|| 1

[0099] Step 4.3: The model training adopts a progressive training strategy, which divides the training process into multiple sub-training cycles. The sub-training cycles are trained in sequence. The training starts with a large learning rate and the learning rate is gradually reduced after each sub-training cycle until the iteration is completed.

[0100] The present invention uses Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to measure the image defocusing blur performance.

[0101] Table 1 Average values ​​of PSNR and SSIM of the present invention and the prior art for the DPDD test set

[0102] Defocus blur method DPDNet KPAC MDP IFAN The present invention PSNR 24.34 25.24 25.3 25.33 25.38 SSIM 0.747 0.842 0.763 0.789 0.849

[0103] As can be seen from Table 1, under the DPDD test set, the PSNR value and SSIM value of the present invention are respectively improved by 0.05dB, 0.08dB, 0.14dB and 0.06, 0.086, 0.007 compared with the best single-image defocus blurring algorithm in the prior art. It can be seen that the present invention adopts the above technical solution to obtain a higher quality clear image compared with other prior arts.

[0104] Figure 3 It is a defocus blur image effect diagram generated by LBP in the image defocus blur method based on the fuzzy conditional residual network of the present invention. Figure 4 This is the effect diagram of the deep learning image defocusing method based on the conditional residual network of the present invention. Figure 5 This is a comparison of the effects of the deep learning image defocusing method based on the conditional residual network of the present invention. Figure 1 , Figure 6 This is a comparison of the effects of the deep learning image defocusing method based on the conditional residual network of the present invention. Figure 2 .

[0105] The present invention is mainly based on the conditional residual network technology, and its method mainly uses the estimated out-of-focus blur map as the input condition of the conditional residual restoration network, so as to help the conditional residual restoration network reconstruct a clear full-focus image from the out-of-focus blur image. The innovation of the image defocus blur algorithm proposed in the present invention is mainly reflected in two aspects: first, the present invention uses the out-of-focus blur map estimated by the LBP algorithm to construct a conditional network, and effectively adds the out-of-focus blur map information to the conditional residual restoration network to help the network focus on the blurry area. Second, compared with the latest single-image image defocus blur method, we construct a two-stage U-Net structure conditional residual restoration network, effectively using the information obtained by the conditional network as the control condition of the conditional residual module, further promoting the entire network to more accurately process the image blur area, thereby restoring a clear texture and obtaining a higher quality image. The purpose of the present invention is to provide a single-image defocus blur algorithm based on a conditional residual network, combined with the traditional LBP defocus blur estimation algorithm, so that the network focuses on the out-of-focus blur area, thereby reducing the problem of artifacts and erroneous textures in the clear area of ​​the image restored by the current algorithm.

[0106] References:

[0107] [1]A.Karaali and C.R.Jung,“Edge-based defocus blur estimation withadaptive scale selection,”IEEE Transactions on Image Processing,vol.27,no.3,pp.1126–1137,2017.

[0108] [2]J.Shi,L.Xu,and J.Jia,“Discriminative blur detection features,”inProceedings ofthe IEEE Conference on ComputerVision andPattern Recognition,2014,pp.2965–2972.

[0109] [3]X.Yi and M.Eramian,“Lbp-based segmentation ofdefocus blur,”IEEEtransactions on image processing,vol.25,no.4,pp.1626–1638,2016.

[0110] [4]J.Park,Y.-W.Tai,D.Cho,and I.So Kweon,“A unified approach ofmulti-scaledeep and hand-crafted features for defocus estimation,”in Proceedingsofthe IEEE Conference on Computer Vision and Pattern Recognition,2017,pp.1736–1745.

[0111] [5]W.Zhao,F.Zhao,D.Wang,and H.Lu,“Defocus blur detection via multi-stream bottom-top-bottom fully convolutional network,”in Proceedings oftheIEEE conference on computer vision andpattern recognition,2018,pp.3080–3088.

[0112] [6] A.Abuolaim and MSBrown, "Defocus deblurring using dual-pixeldata," in European Conference on Computer Vision. Springer, 2020, pp.111–126.

[0113] [7] Lee, J., Son, H., Rim, J., Cho, S., & Lee, S. (2021). Iterative FilterAdaptive Network for Single Image Defocus Deblurring. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 2034-2042).

[0114] [8]Son,H.,Lee,J.,Cho,S.,&Lee,S.(2021).Single Image Defocus DeblurringUsing Kernel-Sharing Parallel Atrous Convolutions.arXiv preprint arXiv:2108.09108.

[0115] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A method for removing defocused blur of an image based on a fuzzy conditional residual network, characterized in that: The steps include: Step 1: Create a training set for the image defocusing model; Step 2: construct a fuzzy conditional network; Step 3: Use the conditional residual module to construct a conditional residual restoration network; Step 4: The image defocus blur model training adopts a two-stage training strategy; The step 1 is specifically implemented as follows: Step 1.1, select the image pairs of out-of-focus blurred images and high-definition images in the DPDD training set as training; Step 1.2, crop the image into 480x480 size tiles and build an image dataset; Step 1.3: For the input high-definition image I target and out-of-focus blurred image I blur , the image dataset is augmented by flipping, rotating, and randomly cropping the image into blocks of 256 size as the final training set; The step 2 is specifically implemented as follows: Step 2.1, use the LBP algorithm module to build a fuzzy conditional network, extract features from the fuzzy image estimated by the input LBP, and obtain the corresponding conditional features, which are used to add the out-of-focus fuzzy image information to the conditional residual restoration network, thereby effectively restoring the texture details of the fuzzy area; The blur condition network uses the LBP algorithm to estimate the out-of-focus blur map, and its calculation formula is: F LBP =H LBP (I blur ) H LBP represents the LBP algorithm module in the fuzzy conditional network, F LBP represents the estimated out-of-focus blur map; Step 2.2: Use a shallow feature extraction convolution and three downsampling convolutions to match the features of the out-of-focus blur image to the conditional residual restoration network; the calculation formula is expressed as: F cond1 =Conv down (Conv1(F LBP )) F cond2 =Conv down ((F cond1 )) F cond3 =Conv down (F cond2 ) F cond1 ,F cond2 ,F cond3 Respectively represent the three conditional features input into the conditional residual recovery network; Conv down represents 2 times downsampling convolution, Conv1 represents the shallow feature extraction layer; The step 3 is specifically implemented as follows: Step 3.1, conditional residual module, its formula is: F rec =Conv(LReLU(Conv(F input )) F c =Conv(F cond ) F ouput =F cond *F res +F input Where Conv represents the convolution operation, LReLU represents the LeakyReLU activation function, and F cond Represents the conditional features of the input; Step 3.2: Obtain the conditional features F corresponding to different scales cond1 ,F cond2 ,F cond3 Finally, a conditional residual restoration network is constructed, in which the last layer of downsampling uses 4 conditional residual modules, and the other layers use 2 conditional residual modules, and two skip connection layers are added in the middle to achieve the best network performance; the specific formula is as follows: Downsampling: F1=H CRB×2 (Conv(I blur ),F cond1 ) F2=H CRB×2 (Conv down (F1),F cond2 ) F3=H CRB×4 (Conv down (F2),F cond2 ) Upsampling: F up1 =H CRB×2 (Conv c (cat(F3↑2,Conv skip (F2))),F cond2 ) F up2 =H CRB×2 (Conv c (cat(F up1 ↑2,Conv skip (F1))),F cond1 ) F output =Conv Re (F up2 ) Among them, H CRB×2 Indicates the use of two conditional residual modules, H CRB×4 Indicates the use of 4 conditional residual modules; Conv,Conv c ,Conv skip ,cat and Conv Re They represent the initial feature extraction convolution layer, channel adjustment convolution layer, skip connection convolution layer, channel connection operation, and reconstruction convolution layer respectively. ↑2 represents the bilinear upsampling 2 times operation. The step 4 is specifically implemented as follows: Step 4.1: Input the input image into the first-stage network, that is, input it into the blur conditional network and the first-stage conditional residual recovery network to train and restore a relatively clear image I output1 , and then use the output of the first-stage conditional residual recovery network as the input of the second-stage network, and input it into the fuzzy conditional network again to obtain new conditional features, and further use the second-order conditional residual recovery network to recover more refined features I output2 , thus obtaining a clearer image; the specific expression is as follows: I output1 =H Deblur1 (I blur ,H cond (I blur )) I output2 =H Deblur2 (I output1 ,H cond (I blar )) Among them, H Deblur represents the conditional residual recovery network, H cond represents a fuzzy conditional network; Step 4.2: The output of the two-stage network uses the L1 loss function to calculate the point-to-point loss function of the image. The model loss function is expressed as: Loss=L1(I target ,I output1 )+L1(I target ,I output2 ) L1(x,y)=||xy||1 Step 4.3: Model training adopts a progressive training strategy, dividing the training process into multiple sub-training cycles, and training the sub-training cycles in sequence. Training starts with a large learning rate, and gradually reduces the learning rate after each sub-training cycle until the iteration is completed.

Images