Image deblurring system and method based on dual control network

Abstract

The present invention proposes an image deblurring system and method based on a dual control network, the system comprising: a coding module for extracting image features from an input image; a data module for performing degradation processing and further feature extraction processing based on the extracted image features; a control module for controlling the processing process of the data module; a decoding module for decoding the processed data to obtain reconstructed features; wherein a cyclic skip connection is established between the coding module and the decoding module, and the output of the coding module is added to the input of the decoding module via the cyclic skip connection, and the output of the decoding module is added to the input of the coding module. The present invention improves the effectiveness problem based on the deep learning network and solves the flexibility problem based on the model through dual control, greatly improves the SSIM and PSNR indicators of image deblurring, and enhances the visual effect.

Description

Technical Field

[0001] The present invention relates to image processing, and in particular to an image deblurring system and method. Background Art

[0002] Traditional image deblurring uses filtering methods to divide the image into different components, filter out the blurred components that affect the image quality, and synthesize a clear image. With the development of artificial intelligence, deep learning end-to-end processing of image restoration tasks has been widely studied. By building a deep learning network architecture, the features that make up a clear image are continuously learned, and then the feature map is formed by blurring the image. Finally, the feature fusion restores the clear image.

[0003] Most of the existing deep learning-based deblurring methods are based on the codec structure, adding an attention module at the output end to enhance the output features. Such image deblurring methods have complex calculation processes, large number of parameters, severe ringing effects, and much room for improvement in restoration effects. They also have insufficient capabilities in identifying blur kernels. Summary of the invention

[0004] Purpose of the invention: In view of the shortcomings of the prior art, the present invention provides an image deblurring system and method based on a dual control network, which saves computing costs while maintaining good image restoration effects.

[0005] Technical solution: In the first aspect, an image deblurring system based on a dual control network includes: an encoding module for extracting image features from an input image; a data module for performing degradation processing and further feature extraction processing based on the extracted image features; a control module for controlling the processing process of the data module; a decoding module for decoding the processed data to obtain reconstructed features; wherein a cyclic skip connection is established between the encoding module and the decoding module, and the output of the encoding module is added to the input of the decoding module via the cyclic skip connection, and the output of the decoding module is added to the input of the encoding module.

[0006] Furthermore, the data module includes a degradation branch and a processing branch, wherein the degradation branch generates a blurred image using a blur kernel according to the image features extracted by the encoding module, and the generated blurred image is added to the input image; the processing branch further extracts the image features using a residual network module to obtain features that meet the clarity conditions; when the degradation branch generates a degraded image, the control module controls the blur kernel to generate a blurred image, and uses the degraded image as input for the processing branch to process when the processing branch performs processing.

[0007] Furthermore, the encoding module and the decoding module are stacked structures. The encoding module extracts image features including three stages with different channels. Each stage extracts image features through a layer of feature extraction blocks, and then mathematically transforms the extracted image features through two layers of nonlinear transformation blocks; the decoding module includes three stages corresponding to the encoding module, and each stage completes decoding through two nonlinear transformation layers and one feature reconstruction layer.

[0008] Furthermore, the features output by the feature extraction layer of the first stage of the encoding module are forward jump connected to the feature reconstruction layer of the corresponding stage of the decoding module, and the features output by the second nonlinear transformation layer of the first stage and the second stage of the encoding module are forward jump connected to the feature reconstruction layer of the next stage corresponding to the decoding module.

[0009] Furthermore, the residual network module is ResUNet which is a residual module combined with U-Net.

[0010] In a second aspect, an image deblurring method based on a dual control network comprises the following steps:

[0011] The encoding module extracts image features based on the input image;

[0012] The control module applies a blur kernel to the extracted image features, generates a degraded blurred image after being processed by the data module, and the control module adds the degraded blurred image to the input image;

[0013] The encoding module extracts features according to the updated input image. The extracted features are processed by the data module to obtain features that meet the clarity conditions. At the same time, the features extracted by the encoding module are added to the input end of the decoding module through a loop jump connection.

[0014] The decoding module completes feature reconstruction and restores a clear image based on the features that meet the clarity conditions and the features transmitted by the cyclic skip connection. The output of the decoding module is added to the input of the encoding module via the cyclic skip connection.

[0015] Furthermore, the encoding module extracts image features including three stages with different channels. Each stage extracts image features through a layer of feature extraction blocks, and then mathematically transforms the extracted image features through two layers of nonlinear transformation blocks; the decoding module includes three stages corresponding to the encoding module, and each stage completes decoding through two nonlinear transformation layers and one feature reconstruction layer.

[0016] Furthermore, the features output by the feature extraction layer of the first stage of the encoding module are forward jump connected to the feature reconstruction layer of the corresponding stage of the decoding module, and the features output by the second nonlinear transformation layer of the first stage and the second stage of the encoding module are forward jump connected to the feature reconstruction layer of the next stage corresponding to the decoding module.

[0017] According to a third aspect of the present invention, a computer device is provided, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the one or more programs are executed by the one or more processors, the image deblurring method based on a dual control network according to the second aspect of the present invention is implemented.

[0018] Beneficial effects: The present invention provides an image deblurring system and method based on a dual control network, which uses a dual control module to respectively control the degradation branch and the processing branch, can identify 12 different blur kernels, and simplifies a large number of network parameters. This network structure has good flexibility and effectiveness, enhances the clarity of image deblurring, and has an outstanding restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the overall structure of an image deblurring system based on a dual control network according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a codec structure according to an embodiment of the present invention;

[0021] Figure 3 Deblurring comparison of different methods on two test sets according to an embodiment of the present invention;

[0022] Figure 4 Comparison of real blurred images before and after DCNet network processing according to an embodiment of the present invention;

[0023] Figure 5 This is the restoration effect of a real blurred image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0025] The present invention provides an image deblurring system based on a dual control network, the overall structure of which is as follows: Figure 1 The image deblurring system includes: an input module for inputting an image; an encoding module for encoding the input image and extracting image features; a data module for performing degradation processing and feature extraction processing on the extracted image feature data; a control module for controlling the processing process of the feature data; a decoding module for decoding the processed data and restoring the image; and a loop-jump connection module for performing loop-jump processing between the encoding module and the decoding module. Figure 1 In this system structure, X i , X dRepresent the input and output images respectively; the data module contains a processing branch and a degradation branch, where D n represents a degenerate branch, S n Denotes the processing branch. The degradation branch is used to perform degradation processing on the image, and the processing branch is used to extract features from the image. The control module controls the degradation branch and the processing branch through parameters. image It represents the blurred image obtained by the degenerate branch and then input into the system. The loop skip connection refers to putting the output features into the input end to further extract features to enhance the output features.

[0026] Based on this image deblurring system, a large number of training data sets are learned through the network, features are extracted through the encoder, degraded space data sets are generated through the degradation branch, features are processed through the processing branch, and finally features are reconstructed through the decoder to obtain a clear image.

[0027] The encoding module and the decoding module in the present invention are referred to as the codec structure below. Figure 2 As shown in the figure, the codec structure adopts a stacked structure, including a feature extraction layer, a nonlinear transformation layer, and a feature reconstruction layer. The stacked codec structure helps to extract, learn, and reconstruct features more deeply. The codec structure adopts a cyclic skip connection, which strengthens the feature mapping relationship between the bottom layer and the top layer.

[0028] Specifically, refer to Figure 2 In the encoder stage, it includes feature extraction blocks and nonlinear transformation blocks, because image deblurring requires the extraction of blurry features and clear background layers, which is also an image feature. Therefore, in order to extract these features, a feature extraction block is set; nonlinear transformation blocks are commonly used in image processing. The grayscale values ​​or some features of the image are transformed using mathematical methods (such as logarithms, exponents, powers, etc.) to make the image more consistent with human visual characteristics, and in order to better reconstruct features, the greater the contrast between the image background layer and the blur layer, the more obvious the features. In the decoder stage, it includes nonlinear transformation layers and feature reconstruction layers. Feature reconstruction refers to the reorganization of features to form clear features required for the new restored image.

[0029] The loop skip connection includes forward skip connection and reverse skip connection. Forward skip connection refers to adding the input to the output, and reverse skip connection refers to returning the output to the input and re-outputting it to form a loop. Skip connection is also called jump connection, which can be used interchangeably in the present invention. The purpose of jump connection is to eliminate gradients, and it can also strengthen the gradient back propagation, passing the convolution layer feature map to the deconvolution layer, and obtaining more image information. Figure 2, the loop skip connection refers to the formation of a loop through forward skip connection and reverse skip connection. The first forward skip connection adds the input directly to the output. This is the most direct skip connection method. The following two forward skip connections connect the nonlinear transformation feature block of the encoding stage to the feature reconstruction block of the decoder stage. The use of forward skip connection can solve the gradient disappearance problem, strengthen back propagation, and speed up the training process. It will not affect the original process of the network. The reverse skip connection is set with only one jump, considering that if each decoder stage block is set with a corresponding reverse skip connection, it will increase the complexity of network training. Therefore, it is only necessary to reversely jump to the feature extraction block at the encoder end before the final feature output end to fully perform feature extraction-transformation. The loop can enhance the reconstruction of image details.

[0030] As shown in the figure, the encoding module extracts image features in three stages with different numbers of channels. Each stage extracts image features through a layer of feature extraction blocks, and then mathematically transforms the extracted image features through two layers of nonlinear transformation blocks in turn; the decoding module includes three stages corresponding to the encoding module, and each stage completes decoding through two nonlinear transformation layers and a feature reconstruction layer. First, the features output by the feature extraction layer of the first stage of the encoding module are forwardly jumped and connected to the feature reconstruction layer of the corresponding stage of the decoding module to strengthen the output feature representation. Secondly, a cyclic skip connection structure is established between the nonlinear transformation layer of the encoding module and the feature reconstruction layer of the decoding module, that is, the nonlinear transformation layers of the first and second stages of the encoding module are connected to the reconstruction layers of the second and third stages of the decoding module respectively. Finally, the linear transformation layer features of the first stage of the decoding module are returned to the feature extraction layer of the first stage of the encoding module to form a cycle. The cyclic skip connection first strengthens the feature representation of the output end, and secondly solves the gradient disappearance problem to a large extent, speeding up the network training process.

[0031] The control module controls the degradation branch and the processing branch in the form of controlling the blur kernel. The control of the degradation branch means that the control module controls the degradation branch to add different blur kernels to the input image to generate a series of blurred images. In an embodiment of the present invention, the blur kernel includes four isotropic Gaussian kernels K1-K4 with different widths (0.7, 1.2, 1.6, 2.0), four anisotropic Gaussian kernels K5-K8, and four specific blur kernels K9-K12, wherein the specific forms of the four anisotropic Gaussian kernels can be found in “Zhang K, Zuo W, Zhang L. Learning a single convolutional super-resolution network for multiple degradations [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 3262-3271.”, and the specific forms of the four specific blur kernels can be found in “Boracchi G, Foi A. Modeling the performance of image restoration from motion blur [J]. IEEE Transactions on Image Processing, 2012, 21 (8): 3502-3517.” and “Levin A, Weiss Y, Durand F, et al. Understanding and evaluating blind deconvolution algorithms[C] / / 2009IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:1964-1971. ". The degradation branch uses Fourier transform and inverse Fourier transform to obtain a closed-form solution, that is, to generate a degraded image.

[0032] The commonly used technique for generating blurred images is to use interpolation methods, which introduce different variables (noise, blur kernel, weight, etc.) to perform fast Fourier transform and then perform inverse transform to obtain the solution to obtain a degraded image. For details on image interpolation, see "ZHANG K, GOOL LV, TIMOFTE R. Deep unfolding network for image super-resolution [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 3217-3226.". In the present invention, generating blurred images is simplified to the execution of formula 1.

[0033]

[0034] Where k represents the fuzzy kernel factor, n represents the noise, represents the two-dimensional convolution of x and k. When the degenerate branch generates a degraded image, Formula 1 is executed with the participation of the control module to generate degraded images with different blur levels. The control module controls the fuzzy kernel factor k, which is the 12 different fuzzy kernel factors mentioned above. Figure 3 Shown are examples of degraded images generated under different blurs.

[0035] The processing branch is intended to obtain a clear image. In the present invention, the processing branch adopts a residual network module (ResUNet), which is to combine the residual module with U-Net. The introduction of the residual network module can accelerate training and quickly process large-capacity modules. The combination of residual modules and U-Net is also a commonly used method in recent years. For U-Net, four scales are commonly designed, and the number of channels at each scale from the first scale to the fourth scale is 64, 128, 256, and 512 respectively. The scale change is completed by upsampling and downsampling. Downsampling uses three pooling methods, namely average pooling, maximum pooling, and interleaved convolution. The pooling operation is equivalent to an operation of the human visual system to reduce and reduce the dimension of the object. Upsampling uses upconvolution, pixel offset, and transposition convolution. The function of these three methods is to enlarge the image. Finally, interleaved convolution and transposition convolution are used for downsampling and upsampling operations. Two residual modules are used in the upsampling operation and downsampling operation, each residual module consists of 2 3×3 convolution layers, and the Relu activation function is used between the residual modules.

[0036] When the processing branch processes the features, the degraded image is processed as input. At this time, the control module will supply different blurred images or clear image features that have been degraded as input to the processing branch.

[0037] By controlling the control module to participate in the degeneration branch and the processing branch, the number of operation parameters is reduced, and the running time and efficiency are improved. Therefore, the network structure can greatly save the computing cost. The blur kernels in the real world are various and complicated, but most of them are mainly motion blur. The degeneration branch of the present invention uses a fuzzy space composed of 12 different blur kernels. This fuzzy space covers most of the natural blur situations in the real world. Through the degeneration branch, fuzzy images with various blur degrees are obtained. It provides data support for testing the network structure.

[0038] The entire implementation plan for image deblurring based on the network structure includes a training process and a testing process, specifically including:

[0039] 1. In the training code, a large amount of training data sets that have been stored are put in, and the stacked encoder structure works first. The encoder extracts features from the images that pass through.

[0040] 2. After the encoder structure extracts features, the degenerate branch generates a degraded blurred image under the control of the control module. The control module adds the degraded blurred image to the input image. The encoder extracts features and the feature data is sent to the processing branch to learn the extracted features. The deep learning network needs to repeat learning continuously until the loss function that best suits the network structure is obtained and the best convergence is achieved. This is also to ensure that the restored image is the clearest.

[0041] 3. After processing the branch learning features, the decoder reconstructs these features according to the learned image rules, and synthesizes a clear image after multiple reconstructions. At this point, the entire training process comes to an end.

[0042] 4. The completion of the training code indicates that the network has learned certain rules for restoring clear image features. When running the test code, a blurry image obtained by the degenerate branch is randomly inserted, and a clear image is obtained after the network processing. The entire single image deblurring task is completed.

[0043] The present invention adopts a deep learning framework and proposes a dual control network for image deblurring. A training set consisting of a large number of pictures is placed in the network for training. By learning the clear or blurred features of the images in the training set, the learned features are fused through the processing branch, and the image is restored through the decoder structure.

[0044] In order to verify the performance of the method of the present invention, a verification example is used to compare with other methods under different blur kernels. The verification results are shown in Tables 1 and 2. Table 1 is a comparison of PSNR with other methods under different blur kernels, and Table 2 is a comparison of SSIM with other methods under different blur kernels.

[0045] Table 1 PSNR comparison of different blur kernels and other methods

[0046]

[0047]

[0048] Table 2 Comparison of SSI M with different blur kernels and other methods

[0049]

[0050] Among them, the method involved by Gao[1] can be found in “Gao H, Tao X, Shen X, et al. Dynamic scene deblurring with parameter selective sharing and nested skip connections[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 3848-3856.”; the method involved by Nah[2] can be found in “Nah S, Hyun Kim T, Mu Lee K. Deep multi-scale convolutional neural network for dynamic scene deblurring[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 3883-3891.”; the method involved by Tao[3] can be found in “Tao X, Shen C, Yang Y B. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[J]. Advances in neural information processingsystems,2016,29:2802-2810."; DCNet is the image deblurring method proposed by the present invention. HIDE and GoPro are the existing image datasets used in the experiment.

[0051] A more intuitive effect can be obtained from Figure 4 and Figure 5 It can be seen that Figure 4The figure shows the deblurring effect for a synthetic image. The so-called synthetic image refers to a non-real image synthesized by a computer. For real image deblurring, the fuzzy space of the present invention includes most blurry situations. The restoration effect for a real blurred image is shown in FIG. Figure 5 shown.

[0052] Compared with other methods, the method of the present invention improves the effectiveness problem based on deep learning network and solves the flexibility problem based on model, greatly improves SSIM and PSNR indicators, and enhances visual effects. The present invention has good application prospects. In the civilian field, it is used in the transportation system to facilitate timely detection of traffic violations. In the military field, it is used for drone detection to make observations clearer and more accurate.

[0053] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An image deblurring system based on a dual control network, characterized in that: include: An encoding module for extracting image features from an input image; A data module for performing degradation processing and further feature extraction processing according to the extracted image features; a control module for controlling the processing process of the data module; a decoding module for decoding the processed data to obtain reconstructed features; A loop-jump connection is established between the encoding module and the decoding module, via which the output of the encoding module is added to the input of the decoding module, and the output of the decoding module is added to the input of the encoding module; The data module includes a degradation branch and a processing branch. The degradation branch generates a blurred image using a blur kernel according to the image features extracted by the encoding module, and the generated blurred image is added to the input image. The processing branch further extracts the image features using a residual network module to obtain features that meet the clarity conditions. When the degradation branch generates a degraded image, the control module controls the blur kernel to generate a blurred image, and when the processing branch performs processing, the degraded image is used as input for the processing branch to process. The formula for generating blurred images by the degenerate branch is expressed as: Where k represents the fuzzy kernel factor, which is controlled by the control module, and n represents the noise. represents the two-dimensional convolution of x and k; The encoding module and the decoding module are stacked structures. The encoding module extracts image features including three stages with different channels. Each stage extracts image features through a layer of feature extraction blocks, and then mathematically transforms the extracted image features through two layers of nonlinear transformation blocks; the decoding module includes three stages corresponding to the encoding module, and each stage completes decoding through two nonlinear transformation layers and one feature reconstruction layer; the features output by the feature extraction layer of the first stage of the encoding module are forward jumped and connected to the feature reconstruction layer of the corresponding stage of the decoding module, and the features output by the second nonlinear transformation layer of the first and second stages of the encoding module are respectively forward jumped and connected to the feature reconstruction layer of the next stage corresponding to the decoding module; the linear transformation layer features of the first stage of the decoding module are returned to the feature extraction layer of the first stage of the encoding module to form a loop.

2. The image deblurring system based on a dual control network according to claim 1, characterized in that: The residual network module is ResUNet, which is a residual module combined with U-Net.

3. An image deblurring method based on a dual control network, characterized in that: The following steps are involved: The encoding module extracts image features based on the input image; The control module applies a blur kernel to the extracted image features, generates a degraded blurred image after being processed by the data module, and the control module adds the degraded blurred image to the input image; The encoding module extracts features according to the updated input image. The extracted features are processed by the data module to obtain features that meet the clarity conditions. At the same time, the features extracted by the encoding module are added to the input end of the decoding module through a loop jump connection. The decoding module completes the reconstruction of features according to the features that meet the clear conditions and the features transmitted by the cyclic skipping connection, and restores a clear image, wherein the output of the decoding module is added to the input of the encoding module via the cyclic skipping connection; Among them, the data module includes a degradation branch and a processing branch. The degradation branch generates a blurred image using a blur kernel according to the image features extracted by the encoding module, and the generated blurred image is added to the input image; the processing branch further extracts the image features using the residual network module to obtain features that meet the clear conditions; when the degradation branch generates a degraded image, the control module controls the blur kernel to generate a blurred image, and when the processing branch performs processing, the degraded image is used as input for the processing branch to process; the formula for the degradation branch to generate a blurred image is expressed as: Where k represents the fuzzy kernel factor, which is controlled by the control module, and n represents the noise. represents the two-dimensional convolution of x and k; The encoding module extracts image features in three stages with different channels. In each stage, image features are extracted through a layer of feature extraction blocks, and then the extracted image features are mathematically transformed through two layers of nonlinear transformation blocks. The decoding module includes three stages corresponding to the encoding module, and each stage completes decoding through two nonlinear transformation layers and one feature reconstruction layer. The features output by the feature extraction layer of the first stage of the encoding module are forward jumped to the feature reconstruction layer of the corresponding stage of the decoding module, and the features output by the second nonlinear transformation layer of the first and second stages of the encoding module are respectively forward jumped to the feature reconstruction layer of the next stage corresponding to the decoding module. The linear transformation layer features of the first stage of the decoding module are returned to the feature extraction layer of the first stage of the encoding module to form a loop.

4. A computer device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the one or more programs are executed by the one or more processors, the image deblurring method based on the dual control network of claim 3 is implemented.

Images