A dynamic scene blind deblurring method based on asymmetric U-Net network

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a dynamic scene blind deblurring method based on an asymmetric U-Net network. The method comprises: obtaining a blurred image and inputting the image into an encoder module to obtain a first feature map; processing the first feature map using a blur kernel inverse kernel estimation module to obtain a second feature map; inputting the second feature map into a decoder module to obtain three inferred clear images; selecting an image with the highest image quality evaluation index from the three inferred clear images as a final clear image; calculating a total loss based on the three inferred clear images, a true clear image, and a true blurred image; and adjusting model parameters based on the total loss to obtain a trained dynamic scene blind deblurring model based on an asymmetric U-Net network. The present invention achieves high-quality blind restoration of spatially varying blurred images in dynamic scenes while reducing the number of model parameters.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a dynamic scene blind deblurring method based on an asymmetric U-Net network. Background Art

[0002] Image blur caused by multiple object motion or camera shake in dynamic scene imaging is a highly ill-posed inverse problem, representing a spatially varying blur. To obtain clear images from dynamic scenes, a dynamic scene blind deblurring method is required to remove the blur.

[0003] Existing blind deblurring methods for dynamic scenes can be divided into two main categories: optimization-based methods and learning-based methods. Numerous optimization-based methods exist. For example, one method expresses the constraints on the noise Gaussian distribution as derivatives of different orders to define the likelihood function. Furthermore, a smoothness constraint is introduced to suppress ringing artifacts, and an alternating optimization strategy is used to estimate the blur kernel and the clear image. Another method first uses variational Bayesian estimation to obtain the blur kernel, adopts a coarse-to-fine strategy to avoid falling into a local optimal solution, and then uses the Lucy-Richardson method to reconstruct the clear image. Another method, based on the edge sparsity of natural images, combines a super-Laplacian prior to attempt to address the image deblurring problems caused by camera shake and saturated or overexposed pixels. There are also many learning-based methods in the existing technology. For example, a method based on Convolutional Neural Networks (CNN) first estimates the blur kernel and then uses the estimated blur kernel to reconstruct a clear image; a multi-scale convolutional neural network is proposed, which is inspired by the coarse-to-fine strategy in the optimization-based method. It removes various blurs in dynamic scenes in an end-to-end manner and proposes the GOPRO deblurring dataset; a scaled recurrent neural network (SRN), which includes a codec network built based on residual blocks (Residual Block: ResBlock) and can gradually restore clear images of different resolutions in the pyramid.

[0004] Due to the spatial diversity (i.e., spatially varying) of blur kernels in dynamic scenes, existing optimization-based methods cannot accurately estimate the spatially varying blur kernels in dynamic scenes, which in turn affects the restoration of clear images. Existing learning-based methods almost all use an end-to-end approach to directly restore the corresponding clear image from a blurred image, while ignoring the estimation of the spatially varying blur kernel. This results in a large number of network parameters required to achieve a sufficient receptive field.

[0005] In summary, there is an urgent need for a method that can solve the problem of accurately estimating the spatially varying blur kernel in dynamic scenes and the problem of excessive number of model parameters. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention proposes a dynamic scene blind deblurring method based on an asymmetric U-Net network, the method comprising: obtaining a blurred image to be deblurred, inputting the blurred image into a trained dynamic scene blind deblurring model based on an asymmetric U-Net network, and obtaining a clear image;

[0007] The training process of the dynamic scene blind deblurring model based on the asymmetric U-Net network includes:

[0008] S1: Get the blurred image and input it into the encoder module to get the first feature map;

[0009] S2: Process the first feature map using a fuzzy kernel inverse kernel estimation module to obtain a second feature map;

[0010] S3: Input the second feature map into the decoder module to obtain three inference clear images; select the image with the highest image quality evaluation index from the three inference clear images as the final clear image;

[0011] S4: Calculate the total loss based on the three inferred clear images, the real clear image, and the real blurred image; adjust the model parameters based on the total loss to obtain a trained dynamic scene blind deblurring model based on the asymmetric U-Net network.

[0012] Preferably, the encoder module includes a first encoder, a second encoder and a third encoder; wherein the first encoder includes a layer of 3×3 convolution with 3 input channels, 32 output channels and 3 NAFBlocks; the second encoder includes a layer of 3×3 convolution with 32 input channels, 64 output channels, a stride of 2 and 3 NAFBlocks; the third encoder includes a layer of 3×3 convolution with 64 input channels, 128 output channels, a stride of 2 and 30 NAFBlocks.

[0013] Preferably, the process of processing the first feature map using the blur kernel inverse kernel estimation module includes:

[0014] The first feature map is processed by a 3×3 convolution with 128 input and output channels, two NAFBlocks, and a 3×3 convolution with 128 input channels and 25 output channels to obtain the blur kernel.

[0015] The blur kernel is processed by a 3×3 convolution with 25 input and output channels, two NAFBlocks, and a 3×3 convolution with 25 input channels and 49 output channels to obtain the inverse kernel.

[0016] The first feature map and the inverse kernel are processed using an adaptive filter convolution layer to obtain a second feature map.

[0017] Preferably, the decoder module includes a first decoder, a second decoder and a third decoder; wherein, the first decoder includes a layer of 1×1 convolution with an input channel of 128 and an output channel of 256, 3 NAFBlocks and 1 pixelShuffle layer; the second decoder includes a layer of 1×1 convolution with an input channel of 64 and an output channel of 128, 3 NAFBlocks and 1 pixelShuffle layer; the third decoder includes a layer of 3×3 convolution with an input channel of 32 and an output channel of 9 and 3 NAFBlocks.

[0018] Preferably, the process of calculating the total loss includes:

[0019] Adaptive filter convolution layer is used to process the real clear image and the blur kernel obtained in the process of blur kernel inverse kernel estimation module to obtain the inferred blurred image;

[0020] Calculate the reblurring loss based on the inferred blurred image and the true blurred image;

[0021] Calculate the multi-output fusion loss based on the real clear image and three inference clear images;

[0022] The total loss is calculated based on the reblurring loss and the multi-output fusion loss.

[0023] Furthermore, the formula for calculating the reblurring loss is:

[0024] L ReBlur =PSNR(B ReBlur ,↓4B GT )

[0025] Among them, L ReBlur represents the reblurring loss, B ReBlur represents the inference fuzzy image, B GT represents a real blurred image, and ↓4 represents four times bilinear interpolation downsampling of the image.

[0026] Furthermore, the formula for calculating the multi-output fusion loss is:

[0027] L MORL =min(PSNR(S i , S GT ))

[0028] Among them, L MORL represents the multi-output fusion loss, S i represents the i-th reasoning clear image, S GT Represents a real clear image, and PSNR() represents the PSNR loss function.

[0029] Furthermore, the formula for calculating the total loss is:

[0030] L=L ReBlur +0.01L MORL

[0031] Among them, L represents the total loss, L ReBlur represents the reblurring loss, L MORL represents the multi-output fusion loss.

[0032] The beneficial effects of the present invention are as follows: the present invention proposes a blind deblurring method for dynamic scenes based on an asymmetric U-Net network, which adopts an asymmetric U-shaped network and can achieve stronger feature extraction capabilities with fewer parameters, solving the problems of accurate estimation of spatially varying blur kernels in dynamic scenes and excessive number of model parameters. In addition, the present invention adopts a blur kernel inverse kernel estimation module and a novel blur kernel self-supervised reblurring loss function to accurately estimate the blur kernel for each pixel in the dynamic scene blurred image, thereby achieving accurate estimation of the spatially varying blur kernel in the dynamic scene. By using a multi-output fusion loss, the deblurring performance of the network can be further improved without increasing the network parameters, thereby achieving high-quality blind restoration of spatially varying blurred images in dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the structure of the dynamic scene blind deblurring model based on the asymmetric U-Net network in the present invention;

[0034] Figure 2 This is a comparison chart of the blurred image deblurring results of the present invention and the comparison method. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] This paper proposes a dynamic scene blind deblurring method based on an asymmetric U-Net network. Figure 1 As shown, the method includes the following contents:

[0037] Obtain the blurred image to be deblurred, input the blurred image into the trained dynamic scene blind deblurring model based on the asymmetric U-Net network, and obtain a clear image; the dynamic scene blind deblurring model based on the asymmetric U-Net network includes an encoder module, a blur kernel inverse kernel estimation module (Blur-kemel Inverse-kernel Estimation Module: BIEM) and a decoder module; use B to represent the input blurred image, EB i represents the i-th encoder, represents the input of the i-th encoder, represents the output of the i-th encoder. Represents the input of BIEM, Represents the output of BIEM. DB i represents the i-th decoder, represents the input of the i-th decoder, represents the output of the i-th decoder, S i Represents the i-th reasoning clear image output, where i∈{1, 2, 3}.

[0038] The training process of the dynamic scene blind deblurring model based on the asymmetric U-Net network includes:

[0039] S1: Get the blurred image and input it into the encoder module to get the first feature map.

[0040] Obtain a blurred image for training the model, which is a real blurred image, and obtain a real clear image corresponding to the real blurred image.

[0041] The encoder module includes a first encoder EB1, a second encoder EB2, and a third encoder EB3. The first encoder includes a 3×3 convolution layer with 3 input channels and 32 output channels, as well as three nonlinear activation free blocks (NAFBlocks). The second encoder includes a 3×3 convolution layer with 32 input channels and 64 output channels, a stride of 2, and three NAFBlocks. The third encoder includes a 3×3 convolution layer with 64 input channels and 128 output channels, a stride of 2, and 30 NAFBlocks. All convolutions with unspecified strides have a stride of 1.

[0042] The blurred image is input into the encoder module for processing. The blurred image is processed in the first encoder EB1, the second encoder EB2 and the third encoder EB3 in turn and outputs and The input of the next encoder is the output of the previous encoder, and the output of the final encoder module is That is the first feature map.

[0043] S2: Use the blur kernel inverse kernel estimation module to process the first feature map to obtain a second feature map.

[0044] Input of the fuzzy kernel inverse estimation module BIEM In the blur kernel inverse kernel estimation module, the first feature map is processed in sequence by a layer of 3×3 convolution with both input and output channels of 128, two NAFBlocks, and a layer of 3×3 convolution with input channels of 128 and output channels of 25 to obtain the blur kernel (Blur Kerel: BK), whose estimated size is 5×5.

[0045] The blur kernel is processed by a 3×3 convolution with a layer of 25 input channels and 25 output channels, two NAFBlocks, and a 3×3 convolution with a layer of 25 input channels and 49 output channels to obtain the inverse kernel (IK), whose estimated size is 7×7.

[0046] The first feature map and the inverse kernel are processed using the Filter Adaptive Convolutional Layer (FAC Layer) to obtain the output BIEM of the BIEM module. out That is the second feature map, expressed as:

[0047]

[0048] Among them, repeat 128 (IK) represents 128 times replication in the channel dimension of IK, and FAC() represents the adaptive filter convolution layer.

[0049] S3: Input the second feature map into the decoder module to obtain three inference clear images; select the image with the highest image quality evaluation index from the three as the final clear image.

[0050] The decoder module includes a first decoder DB1, a second decoder DB2 and a third decoder DB3; among them, the first decoder includes a layer of 1×1 convolution with an input channel of 128 and an output channel of 256, 3 NAFBlocks and a pixelShuffle layer (pixel reorganization layer) for improving the resolution; the second decoder includes a layer of 1×1 convolution with an input channel of 64 and an output channel of 128, 3 NAFBlocks and a pixelShuffle layer; the third decoder includes a layer of 3×3 convolution with an input channel of 32 and an output channel of 9, and 3 NAFBlocks.

[0051] DB1 input After the second feature map is processed by the first decoder, the output DB2 Input After being processed by the second decoder, the output DB3 Input After being processed by the third decoder, the output DB3 outputs 9 channels to obtain 3 inference clear images instead of 3 channels to obtain 1 inference clear image like other methods; the blurred image and DB3 output are processed to finally output three inference clear images, which are expressed as

[0052] The image with the highest image quality evaluation index is selected from the three inference clear images as the final clear image. Preferably, the image quality evaluation index can be a PSNR index.

[0053] In the asymmetric U-shaped network proposed in this paper, NAFBlock replaces the commonly used ResBlock to reduce the overall network parameter count. By controlling the number of NAFBlocks in the encoder and decoder, the network parameters are concentrated in EB3. The decoder uses 1×1 convolution and pixelShuffie layers instead of transposed convolution to improve the decoder's performance in restoring clear images. To facilitate the calculation of the Multiple Output Reuse Loss (MORL), DB3 outputs three clear images instead of one.

[0054] S4: Calculate the total loss based on the three inferred clear images, the real clear image, and the real blurred image; adjust the model parameters based on the total loss to obtain a trained dynamic scene blind deblurring model based on the asymmetric U-Net network.

[0055] The present invention shares a blur kernel or inverse kernel for all channels at each point on the feature map, reducing the difficulty of estimating the blur kernel and inverse kernel. The present invention uses an adaptive filter convolution layer to convolve the estimated blur kernel or inverse kernel with the feature map. To self-supervise the estimated BK, the present invention proposes a reblur loss (ReBlur Loss). The process of calculating the reblur loss includes:

[0056] The adaptive filter convolution layer is used to process the real clear image and the blur kernel obtained in the process of the blur kernel inverse kernel estimation module to obtain the inferred blurred image, which is expressed as:

[0057] B ReBlur =FAC(repeat3(BK),↓4S GT )

[0058] Among them, repeat3(BK) means to copy 3 times in the channel dimension of BK, S GT Indicates a true and clear image, B ReBlur represents the inference fuzzy image, B GT Represents a real blurred image, PSNR() represents the PSNR loss function, and ↓4 represents four times bilinear interpolation downsampling of the image.

[0059] Calculate the reblurring loss L based on the inference blurred image and the real blurred image ReBlur , expressed as:

[0060] L ReBlur =PSNR(B ReBlur ,↓4B GT )

[0061] In order to fully explore the performance of the model, inspired by the feature reuse in reparameterization, this paper proposes a multi-output fusion loss to calculate the multi-output fusion loss L MoRL The formula is:

[0062] L MoRL =min(PSNR(S i , S GT ))

[0063] Among them, S i , i∈{1, 2, 3} represents the i-th reasoning clear image, and min() represents taking the minimum of the three loss values.

[0064] The final total loss L of the present invention is:

[0065] L=L ReBlur +0.01L MoRL

[0066] Adjusting the model parameters based on the total loss yields a trained dynamic scene blind deblurring model based on an asymmetric U-Net network. The total loss is calculated by taking the weighted sum of the reblurring loss and the multi-output fusion loss, with a weight of 0.01 achieving the best deblurring effect. Using the trained dynamic scene blind deblurring model based on an asymmetric U-Net network to deblur the blurred image to be deblurred yields the desired clear image.

[0067] Evaluation of the present invention:

[0068] Ablation experiments were conducted on this paper; specifically, training was performed on a PC equipped with eight NVIDIA GeForce RTX3090 GPUs, using PyTorch version 1.11.0. Peak signal-to-noise ratio and structural similarity were used as quantitative metrics to demonstrate the effectiveness of the proposed BIEM and MORL methods on the GoPro dataset. The number of parameters is measured in millions.

[0069] Table 1 Ablation experiments of various parts proposed in this invention

[0070] BIEM-NoReBlur BIEM MORL PSNR(dB) Params.(M) 32.68 4.2 √ 32.73 4.6 √ 32.83 4.6 √ 32.82 4.2 √ √ √ 32.92 4.6

[0071] BIEM-NoReBlur means that the blur kernel estimated by BIEM is not constrained by the ReBlur loss function, PSNR represents the peak signal-to-noise ratio, and Params represents the number of parameters. As shown in Table 1, we can see that:

[0072] 1) The proposed asymmetric U-shaped network achieves a PSNR of 32.68 with only 4.2 parameters, which is highly competitive with existing state-of-the-art methods. When BIEM and MORL are added to the asymmetric U-shaped network, the proposed method achieves a PSNR of 32.92 with only 4.6 parameters, surpassing existing state-of-the-art methods.

[0073] 2) When BIEM and MORL are added to the asymmetric U-shaped network, PSNR increases by 0.15 and 0.14, respectively. Notably, adding BIEM only increases the number of parameters by 0.1, while adding MORL does not. These two ablation experiments strongly demonstrate that the blur kernel and inverse kernel estimated by BIEM effectively contribute to the restoration of clear images, while MORL fully exploits the model's potential by constraining the reuse of model inference results.

[0074] 3) Compared to adding BIEM to an asymmetric U-shaped network, BIEM-NoReBlur does not apply the ReBlur constraint to the blur kernel estimated by BIEM. This results in a 0.1 decrease in PSNR for BIEM-NoReBlur compared to BIEM, without reducing the number of parameters. This demonstrates that the proposed ReBlur loss function is highly effective in constraining the blur kernel estimated by BIEM. It can make the BIEM-estimated blur kernel more accurate without adding additional network parameters, thereby improving the model's restoration performance.

[0075] To demonstrate the superiority of our proposed method in terms of quantitative metrics, we compared it with six existing methods on the GoPro and HIDE datasets. Specifically, all methods were trained on the GoPro training set and then tested on the GoPro and HIDE test sets, respectively.

[0076] Table 2 Comparison of the present invention with other existing methods

[0077]

[0078] Table 2 shows the average PSNR and average SSIM (structural similarity) of all methods on the GoPro and HIDE test sets, as well as the number of parameters used by these methods. As shown in Table 2, the proposed method not only outperforms existing methods on the GoPro dataset, but also has the fewest parameters among all methods.

[0079] In order to demonstrate the superiority of the method proposed in this invention in terms of subjective visual effects, the deblurring effects of the method proposed in this invention are compared with those of seven existing methods. Figure 2 As shown, Figure 2 In the figure, from left to right and from top to bottom are the blurred image, the images deblurred by the existing methods DeepDeblur, SRN, DMPHN, PSSNSC, DBCPENet, MIMO-UNet+ and MPRNet, the image deblurred by the method proposed in the present invention, and the true clear image; the pictures deblurred by the existing methods have different degrees of distortion, blur and artifacts, while the present invention can not only obtain richer details and sharp edges, but also achieve the highest PSNR, indicating that the deblurred image of the present invention is clearer and the deblurring effect is better.

[0080] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A dynamic scene blind deblurring method based on an asymmetric U-Net network, characterized in that: include: Obtain the blurred image to be deblurred and input it into the trained dynamic scene blind deblurring model based on the asymmetric U-Net network to obtain a clear image; The training process of the dynamic scene blind deblurring model based on the asymmetric U-Net network includes: S1: Get the blurred image and input it into the encoder module to get the first feature map; S2: Processing the first feature map using a fuzzy kernel inverse kernel estimation module to obtain a second feature map. The process of processing the first feature map using the fuzzy kernel inverse kernel estimation module includes: The first feature map is processed by a 3×3 convolution with 128 input and output channels, two NAFBlocks, and a 3×3 convolution with 128 input channels and 25 output channels to obtain the blur kernel. The blur kernel is processed by a 3×3 convolution with 25 input and output channels, two NAFBlocks, and a 3×3 convolution with 25 input channels and 49 output channels to obtain the inverse kernel. The first feature map and the inverse kernel are processed using an adaptive filter convolution layer to obtain a second feature map; S3: Input the second feature map into the decoder module to obtain three inference clear images; select the image with the highest image quality evaluation index from the three inference clear images as the final clear image; S4: Calculate the total loss based on the three inferred clear images, the true clear image, and the true blurred image; adjust the model parameters based on the total loss to obtain a trained dynamic scene blind deblurring model based on the asymmetric U-Net network; the total loss is composed of the reblurring loss and the multi-output fusion loss; The formula for calculating the reblurring loss is: L ReBlur =PSNR(B ReBlur ,↓4B GT ) Among them, L ReBlur represents the reblurring loss, B ReBlur represents the inference fuzzy image, B GT Represents a real blurred image, ↓4 represents four times bilinear interpolation downsampling of the image; The formula for calculating the multi-output fusion loss is: L MORL =min(PSNR(S i ,S GT )) Among them, L MORL represents the multi-output fusion loss, S i represents the i-th reasoning clear image, S GT Represents a real clear image, and PSNR() represents the PSNR loss function.

2. The method for blind deblurring of dynamic scenes based on an asymmetric U-Net network according to claim 1, characterized in that: The encoder module includes a first encoder, a second encoder and a third encoder; wherein the first encoder includes a layer of 3×3 convolution with 3 input channels, 32 output channels and 3 NAFBlocks; the second encoder includes a layer of 3×3 convolution with 32 input channels, 64 output channels and a stride of 2 and 3 NAFBlocks; the third encoder includes a layer of 3×3 convolution with 64 input channels, 128 output channels, a stride of 2 and 30 NAFBlocks.

3. The method for blind deblurring of dynamic scenes based on an asymmetric U-Net network according to claim 1, wherein: The decoder module includes a first decoder, a second decoder and a third decoder; wherein the first decoder includes a 1×1 convolution layer with an input channel of 128 and an output channel of 256, 3 NAFBlocks and a pixelShuffle layer; the second decoder includes a 1×1 convolution layer with an input channel of 64 and an output channel of 128, 3 NAFBlocks and a pixelShuffle layer; the third decoder includes a 3×3 convolution layer with an input channel of 32 and an output channel of 9, and 3 NAFBlocks.

4. The method for blind deblurring of dynamic scenes based on an asymmetric U-Net network according to claim 1, wherein: The process of calculating total loss includes: Adaptive filter convolution layer is used to process the real clear image and the blur kernel obtained in the process of blur kernel inverse kernel estimation module to obtain the inferred blurred image; Calculate the reblurring loss based on the inferred blurred image and the true blurred image; Calculate the multi-output fusion loss based on the real clear image and three inference clear images; The total loss is calculated based on the reblurring loss and the multi-output fusion loss.

5. The method for blind deblurring of dynamic scenes based on an asymmetric U-Net network according to claim 4, characterized in that: The formula for calculating the total loss is: L=L Reblur +0.01L MORL Among them, L represents the total loss, L ReBlur represents the reblurring loss, L MORL represents the multi-output fusion loss.

Images