Image deblurring method and system based on UNet structure

Abstract

The invention relates to the technical field of image processing, discloses an image deblurring method and system based on a UNet structure, and aims to solve the problems of many model parameters and poorer deblurring effect of an existing method. The scheme mainly comprises the following steps: inputting a blurred image into a feature extraction module for down-sampling and feature extraction to obtain blurred image features; inputting the blurred image features into an encoder for encoding operation; a feature attention module is inserted between the coding blocks, and the feature attention module adaptively adjusts the attention on different image features by using a Ghost module and a convolutional layer; the encoded blurred image features are input into a feature fusion module for adaptive feature fusion; the self-adaptive feature fusion result is input into a decoder for up-sampling, and a decoding result is obtained; and carrying out pixel-by-pixel superposition fusion on the blurred image features and the decoding result to obtain a deblurred image. The deblurring effect is improved, the model parameters are reduced, and the method is particularly suitable for industrial scenes with limited resources.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image deblurring method and system based on a UNet structure. Background Art

[0002] Motion blur caused by camera shake and rapid object movement often occurs in daily life and industrial scenarios. Severe motion blur can lead to image degradation, affecting target detection tasks and causing detection tasks to fail. However, image blur at different depths of field in industrial environments is also a major problem. In industrial inspections, the depth of field of the objects being inspected is too different, resulting in blurred images and failure of the inspection task.

[0003] In recent years, deep learning-based deblurring methods have been shown to effectively improve target detection performance when used with target detection systems. With the development of deep learning, significant success has been achieved in the field of image deblurring. However, most methods generally have insufficient extraction of detailed features of blurred images and serious loss of edge features, resulting in blurred details of restored images. In addition, due to the cost-effectiveness of hardware equipment on industrial production lines, algorithm models with large parameters have high requirements for hardware equipment and are difficult to implement. Summary of the invention

[0004] This application aims to solve the problems of existing image deblurring methods having many model parameters and poor deblurring effect, and proposes an image deblurring method and system based on UNet structure.

[0005] The technical solution adopted by this application to solve the above technical problems is:

[0006] In a first aspect, the present application provides an image deblurring method based on a UNet structure, the method comprising:

[0007] Inputting the blurred image into a feature extraction module for downsampling and feature extraction to obtain blurred image features, and inputting the blurred image features into an encoder for encoding to obtain encoded blurred image features;

[0008] Inserting a feature attention module between the coding blocks, wherein the feature attention module uses a Ghost module and a convolutional layer to adaptively adjust the attention to different image features;

[0009] Inputting the encoded blurred image features into a feature fusion module for adaptive feature fusion to obtain an adaptive feature fusion result;

[0010] Inputting the adaptive feature fusion result into a decoder for upsampling to obtain a decoding result;

[0011] The blurred image features and the decoding results are superimposed and fused pixel by pixel to obtain a deblurred image.

[0012] Furthermore, the blurred image is downsampled and features are extracted, specifically including:

[0013] The blurred image is downsampled twice to obtain blurred images of three different sizes. Feature extraction is performed on each blurred image of different sizes to obtain blurred image features corresponding to the three blurred images of different sizes.

[0014] Furthermore, the three blurred images of different sizes include: a blurred image of the original size, a blurred image of half the original size, and a blurred image of one quarter of the original size.

[0015] Furthermore, the encoder includes a first encoding block, a second encoding block and a third encoding block, and the blurred image feature is input into the encoding block for encoding operation, specifically including:

[0016] The blurred image features corresponding to the blurred images of three different sizes are input into the corresponding encoding blocks for encoding operations, so as to obtain the encoded blurred image features corresponding to the blurred images of three different sizes.

[0017] Furthermore, the feature attention module uses the Ghost module and the convolutional layer to adaptively adjust the attention to different image features, specifically including:

[0018] The feature attention module uses 3x3 and 1x1 Ghost modules to extract features from blurred images of half the original size and blurred images of one quarter the original size, and crosses and connects the blurred images to perform pixel-by-pixel superposition and fusion, and introduces the SimAM attention mechanism in the feature extraction process;

[0019] The 1x1 and 3x3 convolutional layers are used to adjust the number of channels and combine the output of the encoding block for feature fusion to adapt to the input of the next encoding block or decoder.

[0020] Furthermore, the encoded blurred image features are input into a feature fusion module for adaptive feature fusion, which specifically includes:

[0021] The feature fusion module resizes the encoded blurred image features output by the three encoding blocks and concatenates them on the corresponding channel dimensions, and then processes them using the 1x1 and 3x3 Ghost modules to obtain the adaptive feature fusion results.

[0022] Furthermore, the feature fusion module includes a first feature fusion module and a second feature fusion module, the first feature fusion module outputs a first adaptive feature fusion result, and the second feature fusion module outputs a second adaptive feature fusion result.

[0023] Furthermore, the decoder includes a first decoding block, a second decoding block and a third decoding block, and inputting the adaptive feature fusion result into the decoder for upsampling, specifically including:

[0024] The third decoding block upsamples the blurred image features output by the third encoding block to obtain a third decoding result, the second decoding block upsamples the spliced ​​second adaptive feature fusion result and the third decoding result to obtain a second decoding result, and the first decoding block upsamples the spliced ​​first adaptive feature fusion result and the second decoding result to obtain a first decoding result.

[0025] Furthermore, the blurred image features and the decoding results are superimposed and fused pixel by pixel, specifically including:

[0026] The blurred image features corresponding to the blurred image of the original size are superimposed and fused with the first decoding result pixel by pixel to obtain a deblurred image of the original size; the blurred image features corresponding to the blurred image of half the original size are superimposed and fused with the second decoding result pixel by pixel to obtain a deblurred image of half the original size; the blurred image features corresponding to the blurred image of one quarter of the original size are superimposed and fused with the third decoding result pixel by pixel to obtain a deblurred image of one quarter of the original size.

[0027] In a second aspect, the present application provides an image deblurring system based on a UNet structure, wherein the system is used to implement the image deblurring method based on a UNet structure as described in the first aspect.

[0028] The beneficial effects of the present application are as follows: the image deblurring method and system based on the UNet structure provided in the present application, for motion blurred and blurred images with different depths of field, utilizes multi-scale input images, combines the U-shaped structure of the UNet network to construct a codec, inserts an adaptive feature fusion module at an appropriate position in combination with the feature extraction module, utilizes the Ghost module to reduce model parameters without reducing the model effect, and utilizes the feature attention mechanism and the parameter-free SimAM attention module to enhance the network representation capability, thereby improving the deblurring effect of the model while reducing the model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of an image deblurring method based on a UNet structure provided in an embodiment of the present application;

[0030] Figure 2 A flowchart of another image deblurring method based on a UNet structure provided in an embodiment of the present application;

[0031] Figure 3An overall framework diagram of a UNet structure provided in an embodiment of the present application;

[0032] Figure 4 A structural diagram of a feature attention module provided in an embodiment of the present application;

[0033] Figure 5 A structural diagram of a feature fusion module provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the structure of an image deblurring system based on a UNet structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0036] The technical solution of the embodiment of the present application is applicable to industrial scenarios where image deblurring is required with limited resources. Since the deblurring effect of the current image deblurring method is poor, if a better deblurring effect is to be achieved, the number of parameters of the corresponding model is large, and the hardware equipment is difficult to support the application of the model.

[0037] Based on this, the technical solution of the present application is proposed. In an embodiment of the present application, the blurred image is input into a feature extraction module for downsampling and feature extraction to obtain blurred image features, and the blurred image features are input into an encoder for encoding operation to obtain encoded blurred image features; a feature attention module is inserted between the encoding blocks, and the feature attention module uses a Ghost module and a convolutional layer to adaptively adjust the attention to different image features; the encoded blurred image features are input into a feature fusion module for adaptive feature fusion to obtain adaptive feature fusion results; the adaptive feature fusion results are input into a decoder for upsampling to obtain a decoding result; the blurred image features and the decoding results are superimposed and fused pixel by pixel to obtain a deblurred image.

[0038] Specifically, for motion blur and defocused blurred images with different depths of field, the embodiment of the present application uses the U-shaped structure of the UNet network to build an encoder and decoder, and inserts an adaptive feature fusion module and a feature attention mechanism and a parameter-free SimAM attention module at appropriate locations in combination with the feature extraction module to enhance the network representation capability, and uses the Ghost module to reduce model parameters without reducing the model effect. The embodiment of the present application reduces model parameters without reducing model accuracy, enhances feature interaction and information transfer, improves the deblurring effect, and avoids the problem that the large number of parameters makes it difficult to use in industrial environments.

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0040] The image deblurring method based on the UNet structure provided in the embodiment of the present application can, in practical applications, collect motion clear-blurred image pairs or use different depth of field lenses to collect defocused blur-clear image pairs, use Albumentations to expand the images to enrich the image information, and use the expanded image data set to train the image deblurring model based on the UNet structure. The blurred image is input into the trained image deblurring model to obtain the deblurred image.

[0041] It can be understood that UNet is a deep learning network structure that uses an encoder-decoder structure, in which the encoder is responsible for gradually downsampling the input image into a feature map, while the decoder is responsible for gradually upsampling the feature map into a segmentation result. The encoder-decoder structure of UNet can help the network better extract image features while retaining more detailed information, thereby achieving better performance in image segmentation tasks.

[0042] See also Figure 1 and Figure 2 The image deblurring method based on the UNet structure provided in the embodiment of the present application includes the following steps:

[0043] S1. Input the blurred image into a feature extraction module for downsampling and feature extraction to obtain blurred image features, and input the blurred image features into an encoder for encoding to obtain encoded blurred image features.

[0044] In the embodiment of the present application, downsampling and feature extraction are performed on the blurred image, specifically including:

[0045] The blurred image is downsampled twice to obtain blurred images of three different sizes. Feature extraction is performed on each blurred image of different sizes to obtain blurred image features corresponding to the three blurred images of different sizes.

[0046] In practical applications, in actual production and manufacturing scenarios, sample images are collected using image acquisition equipment. If a blurred image is generated, the blurred image is input into the image deblurring model of the UNet structure. In the image deblurring model, the feature extraction module first downsamples the blurred image twice to obtain three blurred images of different sizes, namely, a blurred image of the original size, a blurred image of half the original size, and a blurred image of one quarter of the original size. The feature extraction module is formed by stacking multiple convolutional layers of Sequential. Then, feature extraction is performed on the blurred images of the three sizes respectively to obtain blurred image features corresponding to the blurred images of the three different sizes.

[0047] See also Figure 3 In the embodiment of the present application, the encoder includes a first encoding block EBlock1, a second encoding block Eblock2 and a third encoding block Eblock3. The encoding block is connected by multiple residual blocks through Sequential to form a sequential execution structure. The image features are input from the first residual block and are processed by all residual blocks one by one to obtain the final output.

[0048] The blurred image features are input into the encoding block for encoding operation, specifically including:

[0049] The blurred image features corresponding to the blurred images of three different sizes are input into the corresponding encoding blocks for encoding operations, so as to obtain the encoded blurred image features corresponding to the blurred images of three different sizes.

[0050] See also Figure 3 In the embodiment of the present application, the blurred image features corresponding to the blurred image of the original size are input into the first coding block EBlock1 for coding operation, the blurred image features corresponding to the blurred image of half size are input into the second coding block Eblock2 for coding operation, and the blurred image features corresponding to the blurred image of one quarter size are input into the third coding block Eblock3 for coding operation.

[0051] S2. Insert a feature attention module between the coding blocks, and the feature attention module uses the Ghost module and the convolution layer to adaptively adjust the attention to different image features.

[0052] In the embodiment of the present application, the feature attention module uses the Ghost module and the convolutional layer to adaptively adjust the attention to different image features, specifically including:

[0053] The feature attention module uses 3x3 and 1x1 Ghost modules to extract features from blurred images of half the original size and blurred images of one quarter the original size, and crosses the blurred images for pixel-by-pixel superposition and fusion, and introduces the SimAM attention mechanism in the feature extraction process; uses 1x1 and 3x3 convolutional layers to adjust the number of channels and combines the output of the encoding block for feature fusion to adapt to the input of the next encoding block or decoder.

[0054] See also Figure 3 In the embodiment of the present application, a feature attention module FAMBlock is inserted between the first encoding block EBlock1 and the second encoding block Eblock2, and between the second encoding block Eblock2 and the third encoding block Eblock3.

[0055] See also Figure 4 The feature attention module FAMBlock uses two 3x3 Ghost modules and two 1x1 Ghost modules to connect sequentially in pairs, then splices the blurred image features of different sizes, and then performs 1x1 convolution and element-wise fusion with the encoded blurred image features corresponding to the blurred image of the original size output by the first encoding block EBlock1. Finally, they are combined and added through a basic convolution layer to obtain the output features of the feature attention module FAMBlock.

[0056] In practical applications, the two feature attention modules FAMBlock use 3x3 and 1x1 Ghost modules to extract features from the 1 / 2 blurred image and the 1 / 4 blurred image respectively, and cross-connect the blurred images for pixel-by-pixel superposition and fusion. The 1x1 and 3x3 convolutional layers are used to adjust the number of channels and combine the output of the encoding block for feature fusion to adapt to the input of the next encoder or decoder. The SimAM module is used in the feature extraction process to enhance the expressiveness of the feature map without increasing the number of parameters. Each convolution layer in the Ghost module consists of two superimposed parts. The first part is a normal convolution, in which the number of image features will be strictly controlled to reduce the amount of calculation. The second part generates image features through simple linear transformation.

[0057] The feature attention module FAMBlock dynamically adjusts the weights of different features, allowing the model to better focus on features that are more important to the task while reducing reliance on irrelevant features. At the same time, the Ghost module is a lightweight network module that generates more feature maps through a series of low-cost linear transformations, thereby reducing parameters and computation while ensuring accuracy.

[0058] The embodiment of the present application can improve the efficiency and accuracy of the model by inserting the feature attention module FAMBlock, and can better adapt the model to various tasks and data by adaptively adjusting the degree of attention of the model to different features. And the Ghost module is used to reduce the model parameters without reducing the model effect.

[0059] S3. Input the encoded blurred image features into a feature fusion module for adaptive feature fusion to obtain an adaptive feature fusion result.

[0060] In the embodiment of the present application, the feature fusion module includes a first feature fusion module and a second feature fusion module, and the difference is that each adaptive feature fusion module outputs a decoded feature for subsequent fusion splicing, the purpose of which is to enhance feature interaction and consolidate information transmission. The first feature fusion module outputs a first adaptive feature fusion result, and the second feature fusion module outputs a second adaptive feature fusion result.

[0061] Inputting the encoded blurred image features into a feature fusion module for adaptive feature fusion, specifically comprising:

[0062] The feature fusion module resizes the encoded blurred image features output by the three encoding blocks and concatenates them on the corresponding channel dimensions, and then processes them using the 1x1 and 3x3 Ghost modules to obtain the adaptive feature fusion results.

[0063] See also Figure 5 The first feature fusion module and the second feature fusion module resize the output results of the first coding block EBlock1, the second coding block Eblock2 and the third coding block Eblock3 and splice them on the channel dimension (dim=1), and then process them through the 1x1 and 3x3 Ghost modules to obtain the final adaptive feature fusion results. At the same time, the SimAM attention mechanism is combined to further reduce the model parameters.

[0064] S4. Input the adaptive feature fusion result into a decoder for upsampling to obtain a decoding result.

[0065] See also Figure 3 In the embodiment of the present application, the decoder includes a first decoding block DBlock1, a second decoding block DBlock2 and a third decoding block DBlock3. The decoding block is connected by multiple residual blocks through Sequential to form a sequential execution structure. The image features are input from the first residual block and are processed by all residual blocks one by one to obtain the final output.

[0066] Inputting the adaptive feature fusion result into a decoder for upsampling, specifically comprising:

[0067] The third decoding block DBlock3 upsamples the blurred image features output by the third encoding block EBlock to obtain a third decoding result, the second decoding block DBlock2 upsamples the spliced ​​second adaptive feature fusion result and the third decoding result to obtain a second decoding result, and the first decoding block DBlock1 upsamples the spliced ​​first adaptive feature fusion result and the second decoding result to obtain a first decoding result.

[0068] S5. Superimpose and fuse the blurred image features and the decoding results pixel by pixel to obtain a deblurred image.

[0069] See also Figure 3 In the embodiment of the present application, the blurred image features and the decoding results are superimposed and fused pixel by pixel, specifically including:

[0070] The blurred image features corresponding to the blurred image of the original size are superimposed and fused with the first decoding result pixel by pixel to obtain a deblurred image of the original size; the blurred image features corresponding to the blurred image of half the original size are superimposed and fused with the second decoding result pixel by pixel to obtain a deblurred image of half the original size; the blurred image features corresponding to the blurred image of one quarter of the original size are superimposed and fused with the third decoding result pixel by pixel to obtain a deblurred image of one quarter of the original size.

[0071] In summary, the image deblurring method based on the UNet structure provided in the embodiment of the present application, for motion blurred and blurred images with different depths of field, utilizes multi-scale input images, combines the U-shaped structure of the UNet network to construct a codec, inserts an adaptive feature fusion module at an appropriate position in combination with the feature extraction module, and utilizes the Ghost module to reduce model parameters without reducing the model effect. The feature attention mechanism and the parameter-free SimAM attention module are used to enhance the network representation capability, thereby improving the deblurring effect of the model while reducing the model parameters.

[0072] Based on the above technical solution, the embodiment of the present application also proposes an image deblurring system based on a UNet structure, and the system is used to implement the image deblurring method based on a UNet structure as described in the embodiment of the present application.

[0073] See also Figure 6 The image deblurring system based on the UNet structure provided in the embodiment of the present application includes a feature extraction module, a feature attention module, an encoder module, a feature fusion module, a decoder module and a deblurred image generation module.

[0074] It can be understood that since the image deblurring system based on the UNet structure described in the embodiment of the present application is a system for implementing the image deblurring method based on the UNet structure described in the embodiment, for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For relevant matters, please refer to the partial description of the method, which will not be repeated here.

Claims

1. An image deblurring method based on UNet structure, characterized in that: The method comprises: Inputting the blurred image into a feature extraction module for downsampling and feature extraction to obtain blurred image features, and inputting the blurred image features into an encoder for encoding to obtain encoded blurred image features; Inserting a feature attention module between the coding blocks, wherein the feature attention module uses a Ghost module and a convolutional layer to adaptively adjust the attention to different image features; Inputting the encoded blurred image features into a feature fusion module for adaptive feature fusion to obtain an adaptive feature fusion result; Inputting the adaptive feature fusion result into a decoder for upsampling to obtain a decoding result; The blurred image features and the decoding results are superimposed and fused pixel by pixel to obtain a deblurred image.

2. The image deblurring method based on UNet structure according to claim 1, characterized in that: Downsampling and feature extraction of blurred images, including: The blurred image is downsampled twice to obtain blurred images of three different sizes. Feature extraction is performed on each blurred image of different sizes to obtain blurred image features corresponding to the three blurred images of different sizes.

3. The image deblurring method based on UNet structure according to claim 2, characterized in that: The three blurred images of different sizes include: a blurred image of the original size, a blurred image of half the original size, and a blurred image of one quarter of the original size.

4. The image deblurring method based on UNet structure according to claim 3, characterized in that: The encoder includes a first encoding block, a second encoding block and a third encoding block, and the blurred image feature is input into the encoding block for encoding operation, specifically including: The blurred image features corresponding to the blurred images of three different sizes are input into the corresponding encoding blocks for encoding operations, so as to obtain the encoded blurred image features corresponding to the blurred images of three different sizes.

5. The image deblurring method based on UNet structure according to claim 4, characterized in that: The feature attention module uses the Ghost module and the convolutional layer to adaptively adjust the attention to different image features, specifically including: The feature attention module uses 3x3 and 1x1 Ghost modules to extract features from blurred images of half the original size and blurred images of one quarter the original size, and crosses and connects the blurred images to perform pixel-by-pixel superposition and fusion, and introduces the SimAM attention mechanism in the feature extraction process; The 1x1 and 3x3 convolutional layers are used to adjust the number of channels and combine the output of the encoding block for feature fusion to adapt to the input of the next encoding block or decoder.

6. The image deblurring method based on UNet structure according to claim 4, characterized in that: Inputting the encoded blurred image features into a feature fusion module for adaptive feature fusion, specifically comprising: The feature fusion module resizes the encoded blurred image features output by the three encoding blocks and concatenates them on the corresponding channel dimensions, and then processes them using the 1x1 and 3x3 Ghost modules to obtain the adaptive feature fusion results.

7. The image deblurring method based on UNet structure according to claim 6, characterized in that: The feature fusion module includes a first feature fusion module and a second feature fusion module. The first feature fusion module outputs a first adaptive feature fusion result, and the second feature fusion module outputs a second adaptive feature fusion result.

8. The image deblurring method based on UNet structure according to claim 7, characterized in that: The decoder includes a first decoding block, a second decoding block and a third decoding block, and inputting the adaptive feature fusion result into the decoder for upsampling, specifically including: The third decoding block upsamples the blurred image features output by the third encoding block to obtain a third decoding result, the second decoding block upsamples the spliced ​​second adaptive feature fusion result and the third decoding result to obtain a second decoding result, and the first decoding block upsamples the spliced ​​first adaptive feature fusion result and the second decoding result to obtain a first decoding result.

9. The image deblurring method based on UNet structure according to claim 8, characterized in that: The blurred image features and the decoding results are superimposed and fused pixel by pixel, specifically including: The blurred image features corresponding to the blurred image of the original size are superimposed and fused with the first decoding result pixel by pixel to obtain a deblurred image of the original size; the blurred image features corresponding to the blurred image of half the original size are superimposed and fused with the second decoding result pixel by pixel to obtain a deblurred image of half the original size; the blurred image features corresponding to the blurred image of one quarter of the original size are superimposed and fused with the third decoding result pixel by pixel to obtain a deblurred image of one quarter of the original size.

10. An image deblurring system based on UNet structure, characterized in that: The system is used to implement the image deblurring method based on the UNet structure as described in any one of claims 1 to 9.

Images