A Deep Learning-Based Image Deblurring Method and System

Abstract

The present invention relates to a deep learning-based image deblurring method and system, comprising obtaining a target image, a first-scale image, and a second-scale image; the first-scale image and the second-scale image are images obtained by reducing the target image by the first scale and the second scale, respectively; the target image, the first-scale image, and the second-scale image are input into a trained image deblurring model to obtain a deblurred image corresponding to the target image; wherein the image deblurring model adopts an improved MIMO-Unet model, and the improved MIMO-Unet model is obtained by replacing the convolution block in the MIMO-Unet model with a depthwise separable convolution block. The image deblurring model of the present invention adopts a depthwise separable convolution block, which reduces the number of model parameters and enables the image deblurring model to be deployed on a terminal.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image deblurring method and system based on deep learning. Background Art

[0002] In recent years, with the popularization of various digital imaging devices, how to effectively deal with blurred images caused by these imaging devices has become a hot topic in current research.

[0003] The goal of image deblurring is to efficiently utilize the limited information in a blurred image to produce a high-quality, clear image. Addressing image blur not only yields clearer images, improving the viewing experience and facilitating post-editing, but also restores lost information.

[0004] Image deblurring involves removing blur from a blurred input image to reconstruct a high-quality, sharp image. Due to the complex causes of blur, this is a challenging problem, often requiring strong prior knowledge of the image for effective restoration. Deep learning excels at learning generalized prior knowledge from large-scale data, making it a superior alternative to traditional restoration methods. With the rise of deep learning, deep learning-based image deblurring technology is developing rapidly, and many researchers have proposed numerous models and algorithms with excellent deblurring effects. However, existing models require a large number of parameters, making them difficult to deploy on mobile devices such as mobile phones. Summary of the Invention

[0005] The purpose of the present invention is to provide an image deblurring method and system based on deep learning, which uses depthwise separable convolution to reduce the number of model parameters and solves the problem that the model parameters are too large to be deployed on the terminal.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A deep learning-based image deblurring method comprising:

[0008] Acquire a target image, a first-scale image, and a second-scale image; wherein the first-scale image and the second-scale image are images obtained by reducing the target image by the first scale and the second scale respectively;

[0009] Inputting the target image, the first-scale image, and the second-scale image into a trained image deblurring model to obtain a deblurred image corresponding to the target image; the trained image deblurring model is a model trained with a sample image, a first-scale sample image obtained by reducing the sample image by a first scale, and a second-scale sample image obtained by reducing the sample image by a second scale as inputs, and with the deblurred image of the sample image as a label;

[0010] The trained image deblurring model adopts an improved MIMO-Unet model, and the improved MIMO-Unet model is obtained by replacing the convolution block in the MIMO-Unet model with a depth-separable convolution block.

[0011] Optionally, the improved MIMO-Unet model further replaces the residual module in the MIMO-Unet model with a wavelet transform module.

[0012] Optionally, the wavelet transform module includes a feature extraction unit and a wavelet transform unit; the feature extraction unit includes a plurality of sequentially connected depthwise separable convolution blocks.

[0013] Optionally, the feature extraction module in the improved MIMO-Unet model includes 5 depth-separable convolution blocks connected in sequence.

[0014] Optionally, before inputting the target image, the first scale image, and the second scale image into an image deblurring model to obtain a deblurred image corresponding to the target image, the image deblurring model is trained to obtain a trained image deblurring model. The training process is as follows:

[0015] Acquire a sample data set, the sample data set including a plurality of sample images and a first-scale sample image, a second-scale sample image, and a deblurred sample image corresponding to each of the sample images; the first-scale sample image and the second-scale sample image are respectively images obtained by reducing the sample image by the first scale and the second scale;

[0016] The sample image, the first scale sample image and the second scale sample image corresponding to the sample image are used as inputs, and the deblurred sample image corresponding to the sample image is used as a label to train an image deblurring model to obtain a trained image deblurring model.

[0017] Optionally, when training the image deblurring model, the loss function used includes a multi-scale content loss function and a multi-scale frequency loss function.

[0018] Optionally, the loss function is calculated as follows:

[0019]

[0020] Among them, loss represents the loss function value, k represents the scale level, K is the total number of scale levels, t k represents the regularization coefficient, and S kare the image generated in the kth stage and the corresponding clear image, represents the fast Fourier transform, is the multi-scale content loss function, is the multi-scale frequency loss function, and λ is the weighting coefficient.

[0021] Optionally, when training the image deblurring model, an Adam optimizer is used to optimize the network parameters of the image deblurring model.

[0022] A deep learning-based image deblurring system, comprising:

[0023] An image acquisition module is configured to acquire a target image, a first-scale image, and a second-scale image; the first-scale image and the second-scale image are images obtained by reducing the target image by the first scale and the second scale, respectively;

[0024] a deblurring module, configured to input the target image, the first scale image, and the second scale image into a trained image deblurring model to obtain a deblurred image corresponding to the target image; the trained image deblurring model is a model trained with a sample image, a first scale sample image obtained by reducing the sample image by a first scale, and a second scale sample image obtained by reducing the sample image by a second scale as inputs, and with the deblurred image of the sample image as a label

[0025] The trained image deblurring model adopts an improved MIMO-Unet model, and the improved MIMO-Unet model is obtained by replacing the convolution block in the MIMO-Unet model with a depth-separable convolution block.

[0026] Optionally, the improved MIMO-Unet model further replaces the residual module in the MIMO-Unet model with a wavelet transform module.

[0027] According to the specific embodiment provided by the present invention, the present invention discloses the following technical effects: the present invention provides an image deblurring method and system based on deep learning, including obtaining a target image, a first-scale image, and a second-scale image; the first-scale image and the second-scale image are images obtained by reducing the target image by a first scale and a second scale respectively; the target image, the first-scale image, and the second-scale image are input into a trained image deblurring model to obtain a deblurred image corresponding to the target image; the trained image deblurring model is a model trained with a sample image, a first-scale sample image obtained by reducing the sample image by a first scale, and a second-scale sample image obtained by reducing the sample image by a second scale as input, and a deblurred image of the sample image as a label; wherein the trained image deblurring model adopts an improved MIMO-Unet model, and the improved MIMO-Unet model is obtained by replacing the convolution block in the MIMO-Unet model with a depthwise separable convolution block. The image deblurring model of the present invention adopts a depthwise separable convolution block, which reduces the number of model parameters, so that the image deblurring model can be deployed on the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 Flowchart of the image deblurring method based on deep learning provided in Example 1 of the present invention;

[0030] Figure 2 A schematic diagram of the image deblurring model structure provided in Example 1 of the present invention;

[0031] Figure 3 A schematic diagram of the structure of a feature extraction module provided in Example 1 of the present invention;

[0032] Figure 4 A schematic diagram of the structure of a wavelet transform module provided in Example 1 of the present invention;

[0033] Figure 5 A block diagram of an image deblurring system based on deep learning provided in Example 2 of the present invention;

[0034] Figure 6 This is the defuzzification system interface provided by Example 2 of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] The purpose of the present invention is to provide an image deblurring method and system based on deep learning, which uses depthwise separable convolution to reduce the number of model parameters and floating-point operations, solving the problem that the model parameters are large and difficult to deploy on the terminal.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Example 1

[0039] This embodiment provides an image deblurring method based on deep learning. Figure 1 , the method comprising:

[0040] Step S1: Acquire a target image, a first-scale image, and a second-scale image; the first-scale image and the second-scale image are images obtained by reducing the target image by a first scale and a second scale, respectively.

[0041] Specifically, a blurred image to be deblurred is first obtained. In this embodiment, the blurred image to be deblurred can be images from the following fields: in the medical field, medical images obtained using medical instruments such as CT and MRI; in the aerospace field, astronomers need to use images taken by satellites or astronomical telescopes; in the military field, real-time battlefield images taken by various cameras are required to enable military strategists to complete remote deployment based on real-time battlefield information. During the above-mentioned image formation, storage, and transfer processes, various noise interferences often occur, resulting in significant image quality degradation and information loss. These low-quality images not only cannot meet the requirements of high-definition image display devices that have long been popular in people's lives, but also bring great difficulties to subsequent image information analysis. Therefore, the present invention can solve the problem of image quality degradation caused by image blur and restore a clear image from a blurred image.

[0042] The blurred image to be deblurred is used as the target image, and the target image is resized to obtain a first-ratio image and a second-ratio image. In this embodiment, the first ratio can be selected as 1 / 2, and the second ratio can be selected as 1 / 4.

[0043] Step S2: Input the target image, the first-scale image, and the second-scale image into a trained image deblurring model to obtain a deblurred image corresponding to the target image; the trained image deblurring model is a model trained using a sample image, a first-scale sample image obtained by reducing the sample image by a first scale, and a second-scale sample image obtained by reducing the sample image by a second scale as input, and using the deblurred image of the sample image as a label. The trained image deblurring model uses an improved MIMO-Unet model, which is obtained by replacing the convolution block in the MIMO-Unet model with a depthwise separable convolution block. The improved MIMO-Unet model also replaces the residual module in the MIMO-Unet model with a wavelet transform module.

[0044] Specifically, if Figure 2 As shown, the image deblurring model is divided into three stages, through which a clear image is obtained from a blurred image. The image deblurring model includes an encoder and a decoder; the encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; the decoder includes a first decoding stage, a second decoding stage, and a third decoding stage. Specifically:

[0045] The first encoding stage uses the feature extraction module (SCM) to extract features from the second scale image. The second encoding stage uses the feature extraction module (SCM) to extract features from the first scale image. The third encoding stage uses only one depthwise separable convolution module (DWConw) to extract features from the target image. The structure of the feature extraction module SCM is as follows: Figure 3 As shown, the feature extraction module includes five sequentially connected depthwise separable convolution blocks, with convolution kernel sizes of 3x3, 1x1, 3x3, 1x1, and 1x1, respectively. The feature extraction module (SCM) concatenates the output features of the four convolutions with the original input and uses an additional 1x1 convolution layer to further refine the resulting features to obtain rich image information. Each encoding stage is not simply cascaded, but rather a feature attention module (FAM) is added between each two encoding stages. The FAM effectively fuses the feature maps between two adjacent stages, strengthening information transfer between stages.

[0046] The three encoding stages all use the wavelet transform module (Wavelet_transform) to extract the high-frequency information in the feature map. The structure of the wavelet transform module is as follows: Figure 4As shown, the wavelet transform module includes a feature extraction unit and a wavelet transform unit. The feature extraction unit includes multiple sequentially connected depthwise separable convolution blocks. In this implementation, eight depthwise separable convolution blocks are sequentially connected to form the feature extraction unit. The module input is passed to the wavelet transform unit and the feature extraction unit respectively, and the processed results of the two parts are added and output. The use of wavelet transform can improve the deblurring ability of the model.

[0047] In addition, the model effectively connects the intermediate feature maps of different stages through asymmetric feature fusion (AFF) to improve the deblurring ability of the model.

[0048] Each decoding stage includes a depthwise separable convolution module, the output of which is added to the features of the corresponding initial input image. The first decoding stage and the second decoding stage directly output the second-scale deblurred image and the first-scale deblurred image, respectively, through an addition operation. The first decoding stage includes a wavelet transform module, a deconvolution module, and a depthwise separable convolution module. The feature map output by the first encoding stage is input into the wavelet transform module. The high-frequency information of the feature map extracted by the wavelet transform module is then input into the deconvolution module and the depthwise separable convolution module, respectively. The output of the deconvolution module is fused with the asymmetric feature fusion feature before being input into the second decoding stage. The second decoding stage includes a depthwise separable convolution module and a wavelet transform module connected in sequence. The output of the wavelet transform module is input into a deconvolution module and a depthwise separable convolution module, respectively. The output of the deconvolution module is fused with the asymmetric feature fusion feature before being input into the third decoding stage. The third decoding stage includes a depthwise separable convolution module, a wavelet transform module, and a depthwise separable convolution module connected in sequence. After the output of the last depthwise separable convolution module is added to the features of the target image, a deblurred image corresponding to the target image is obtained.

[0049] In this embodiment, before inputting the target image, the first scale image, and the second scale image into the image deblurring model to obtain a deblurred image corresponding to the target image, the image deblurring model is further trained to obtain a trained image deblurring model. The training process is as follows:

[0050] Obtain a sample dataset, the sample dataset comprising a plurality of sample images and a first-scale sample image, a second-scale sample image, and a deblurred sample image corresponding to each of the sample images; the first-scale sample image and the second-scale sample image are images obtained by reducing the sample image by the first scale and the second scale, respectively. The deblurred sample image comprises a deblurred image corresponding to the sample image, a deblurred image corresponding to the first-scale sample image, and a deblurred image corresponding to the second-scale sample image.

[0051] In this example, the REDS (Realistic and Dynamic Scenes) dataset is used for training and testing. Compared to existing datasets, REDS increases content diversity and provides more realistic image degradation, particularly motion blur. The REDS dataset contains 300 video sequences, each with 100 images at a resolution of 720 x 1280. These 300 videos are divided proportionally: 240 videos are used to train the image deblurring model, 30 videos are used to test the model, and 30 videos are used to validate the model.

[0052] The sample image, the first scale sample image and the second scale sample image corresponding to the sample image are used as inputs, and the deblurred sample image corresponding to the sample image is used as a label to train an image deblurring model to obtain a trained image deblurring model.

[0053] When training the image deblurring model, the loss functions used include a multi-scale content loss function and a multi-scale frequency loss function. L1 loss is adopted as the multi-scale content loss function, and an auxiliary loss function is added to effectively improve the performance of the network. Since the purpose of image deblurring is to restore the lost high-frequency components as much as possible, the difference in frequency space must be reduced. To this end, the present invention adopts a multi-scale frequency loss function as an auxiliary loss function, which calculates the L1 distance between the clear image and the blurred image in the frequency domain.

[0054] The calculation formula of the loss function is:

[0055]

[0056] Among them, loss represents the loss function value, k represents the scale level, K is the total number of scale levels, t k represents the regularization coefficient, and S k are the image generated in the kth stage and the corresponding clear image, for example, the clear image corresponding to the first stage is the deblurred image corresponding to the second scale sample image, the clear image corresponding to the second stage is the deblurred image corresponding to the first scale sample image, and the clear image corresponding to the third stage is the deblurred image corresponding to the sample image. represents the fast Fourier transform, is the multi-scale content loss function, is a multi-scale frequency loss function, and λ is a weighting coefficient. In this embodiment, λ is set to 0.1.

[0057] During training of the image deblurring model, the Adam (Adaptive Moment Estimation) optimizer is used to optimize the network parameters. The learning rate of each parameter is dynamically adjusted using the first- and second-order moments of the gradient. Its main advantage is that after bias correction, the learning rate for each iteration is within a defined range, resulting in relatively stable parameters. This approach is suitable for most non-sudden optimization scenarios, large datasets, and high-dimensional spaces, while also requiring minimal memory.

[0058] Example 2

[0059] This embodiment provides an image deblurring system based on deep learning. Figure 5 , the system comprising:

[0060] An image acquisition module T1 is configured to acquire a target image, a first-scale image, and a second-scale image; the first-scale image and the second-scale image are images obtained by reducing the target image by the first scale and the second scale, respectively;

[0061] The deblurring module T2 is configured to input the target image, the first scale image, and the second scale image into a trained image deblurring model to obtain a deblurred image corresponding to the target image; the trained image deblurring model is a model trained with a sample image, a first scale sample image obtained by reducing the sample image by a first scale, and a second scale sample image obtained by reducing the sample image by a second scale as inputs, and with the deblurred image of the sample image as a label.

[0062] The image deblurring model adopts an improved MIMO-Unet model, and the improved MIMO-Unet model is obtained by replacing the convolution block in the MIMO-Unet model with a depth-separable convolution block.

[0063] In this embodiment, the improved MIMO-Unet model further replaces the residual module in the MIMO-Unet model with a wavelet transform module.

[0064] In order to better demonstrate the deblurring performance of the model, the present invention uses the Open Neural Network Exchange (ONNX) to convert the model so that the image deblurring task can be performed in an environment where the model dependency library is not installed, and uses PyQt to build the deblurring system interface. Figure 6 The deblurring system interface is displayed. The system interface is simple and easy to use. The left side of the system displays the selected blurred image, and the right side displays the deblurred image effect. Click Open Image to select the blurred image, and click Deblur Image to obtain the corresponding clear image.

[0065] As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0066] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A deep learning-based image deblurring method, characterized in that: include: Acquire a target image, a first-scale image, and a second-scale image; wherein the first-scale image and the second-scale image are images obtained by reducing the target image by the first scale and the second scale respectively; Inputting the target image, the first-scale image, and the second-scale image into a trained image deblurring model to obtain a deblurred image corresponding to the target image; the trained image deblurring model is a model trained with a sample image, a first-scale sample image obtained by reducing the sample image by a first scale, and a second-scale sample image obtained by reducing the sample image by a second scale as inputs, and with the deblurred image of the sample image as a label; The trained image deblurring model adopts an improved MIMO-Unet model, and the improved MIMO-Unet model is obtained by replacing the convolution block in the MIMO-Unet model with a depth-separable convolution block.

2. The image deblurring method based on deep learning according to claim 1, characterized in that The improved MIMO-Unet model also replaces the residual module in the MIMO-Unet model with a wavelet transform module.

3. The image deblurring method based on deep learning according to claim 2, characterized in that: The wavelet transform module includes a feature extraction unit and a wavelet transform unit; the feature extraction unit includes a plurality of sequentially connected depth-wise separable convolution blocks.

4. The image deblurring method based on deep learning according to claim 1, characterized in that The feature extraction module in the improved MIMO-Unet model includes five depth-wise separable convolution blocks connected in sequence.

5. The image deblurring method based on deep learning according to claim 1, characterized in that: Before inputting the target image, the first scale image, and the second scale image into an image deblurring model to obtain a deblurred image corresponding to the target image, the image deblurring model is trained to obtain a trained image deblurring model. The training process is as follows: Acquire a sample data set, the sample data set including a plurality of sample images and a first-scale sample image, a second-scale sample image, and a deblurred sample image corresponding to each of the sample images; the first-scale sample image and the second-scale sample image are respectively images obtained by reducing the sample image by the first scale and the second scale; The sample image, the first scale sample image and the second scale sample image corresponding to the sample image are used as inputs, and the deblurred sample image corresponding to the sample image is used as a label to train an image deblurring model to obtain a trained image deblurring model.

6. The image deblurring method based on deep learning according to claim 5, characterized in that: When training the image deblurring model, the loss functions used include a multi-scale content loss function and a multi-scale frequency loss function.

7. The image deblurring method based on deep learning according to claim 6, characterized in that: The calculation formula of the loss function is: Among them, loss represents the loss function value, k represents the scale level, K is the total number of scale levels, t k represents the regularization coefficient, and S k are the image generated in the kth stage and the corresponding clear image, represents the fast Fourier transform, is the multi-scale content loss function, is the multi-scale frequency loss function, and λ is the weighting coefficient.

8. The image deblurring method based on deep learning according to claim 5, characterized in that: When training the image deblurring model, an Adam optimizer is used to optimize the network parameters of the image deblurring model.

9. An image deblurring system based on deep learning, characterized in that: include: An image acquisition module, configured to acquire a target image, a first-scale image, and a second-scale image; The first scale image and the second scale image are images obtained by reducing the target image by a first scale and a second scale respectively; a deblurring module, configured to input the target image, the first scale image, and the second scale image into a trained image deblurring model to obtain a deblurred image corresponding to the target image; the trained image deblurring model is a model trained with a sample image, a first scale sample image obtained by reducing the sample image by a first scale, and a second scale sample image obtained by reducing the sample image by a second scale as inputs, and with the deblurred image of the sample image as a label The trained image deblurring model adopts an improved MIMO-Unet model, and the improved MIMO-Unet model is obtained by replacing the convolution block in the MIMO-Unet model with a depth-separable convolution block.

10. The deep learning-based image deblurring system according to claim 9, characterized in that: The improved MIMO-Unet model also replaces the residual module in the MIMO-Unet model with a wavelet transform module.

Images