Image deblurring method based on multi-domain feature enhanced U-Net

Abstract

The invention discloses an image deblurring method based on multi-domain feature enhanced U-Net. The method comprises the following steps: using an improved U-Net as a backbone network for image deblurring; fourier transform and Haar wavelet transform are carried out on an input image to obtain a spectrogram and a wavelet sub-band graph of the image, the original input image, the spectrogram and the wavelet sub-band graph are respectively used as inputs of a multi-domain feature enhancement convolution module, the multi-domain feature enhancement convolution module is used to combine and enhance features of multiple domains, and the multi-domain feature enhancement convolution module is used to enhance the features of multiple domains. The acquisition capability of the model on the image information is improved; for the loss function of the model, a weighted Fourier frequency domain loss function is added on the basis of an MSE loss function, the perception ability of the model for the frequency domain difference of the image is enhanced, the repaired image is closer to a clear image in frequency distribution, and experiments prove that the method effectively improves the quality of the repaired image.

Description

Technical Field

[0001] The present invention relates to the field of motion blurred image restoration based on deep learning, and in particular to a motion image deblurring method based on U-Net. Background Art

[0002] With the widespread popularity of photographic devices and imaging technology, images, as an efficient medium for information transmission, have been integrated into people's daily lives, greatly promoting the convenient exchange of information. Compared with other forms of information carriers, images have become the dominant way to exchange information due to their unique and irreplaceable advantages. However, with the widespread application of imaging technology, its limitations have gradually emerged, and image blur has frequently appeared in various scenarios, especially in mobile device photography, surveillance camera recording, and high-speed imaging.

[0003] When shooting a moving object, if there is a relative displacement between the camera and the object, the image will often produce obvious blur and smearing. This type of image blur caused by the relative motion between the object and the camera is called motion blur. Motion blur not only seriously weakens the structural clarity and edge detail expression of the object, but also greatly reduces the overall quality of the image and the accuracy of information transmission. Specifically, blurred images have a significant lack of detail information retention, resulting in a significant negative impact on the performance and effect of the task model in subsequent image analysis, object recognition, feature extraction and other image tasks. Deblurring images through deep learning technology can restore certain image features and detail information, provide clearer images, and reduce its adverse effects on subsequent tasks.

[0004] Xie Yizhen proposed a deblurring method using pyramid convolution and attention mechanism in reference 1 (Xie Yizhen. Research on image deblurring algorithm of multi-scale cross module [J]. Information Technology and Informatization, 2024, (10): 16-19.). This method convolves the image through three groups of convolution modules of different sizes to perceive image features at multiple scales, and uses the attention mechanism to guide the model to focus on relatively important information. Although this method can improve the model's ability to perceive image features, the use of more convolution kernels will greatly increase the difficulty of model training, and the attention module also has the problem of huge computational complexity. Liu et al. applied Haar wavelet transform to the network structure of U-Net in reference 2 (Liu P, Zhang H, Zhang K, et al. Multi-level wavelet-CNN for image restoration [C] / / 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops. 2018: 88600-88609.). This method can separate the high-frequency and low-frequency components of the image and enhance the ability to restore edge details. However, by directly replacing the pooling operation with the wavelet transform, it also loses the advantages of pooling in reducing blur and ghosting, retaining significant features, and reducing the amount of calculation. In reference 3 (Xing Yan, Chen Xiaolu, Xu Qiao, et al. Image deblurring by integrating spatial domain and frequency domain information [J]. Computer Systems Applications, 2024, 33(02): 1-12.), Xing Yan et al. used two sub-networks with the same structure to extract features in the spatial domain and frequency domain respectively, and then fused the features through the fusion network. Although the separation design adopted by this method can make the feature extraction sub-network pay more attention to the features of the corresponding domain, the spatial domain and frequency domain information only interact in the fusion network, and the fusion network is easy to become the bottleneck of the network. Mao et al. used Fourier transform to analyze the image in the frequency domain in the literature 4 (Mao X, Liu Y, Liu F, et al. Intriguing Findings of Frequency Selection for Image Deblurring [J]. Proceedings of the AAAIConference on Artificial Intelligence, 2023, 37 (2): 1905-1913.), and fused the image spatial domain and frequency domain information through the Fourier residual convolution module. However, its Fourier transform branch only used 1×1 convolution, which lacked the use of information on each frequency component of the image.

[0005] Sun Rui et al. proposed a multi-scale dynamic convolution image deblurring method in patent 1 (Sun Rui, Li Xuming, Liu Gang, et al. A crack image deblurring method, device, electronic device and storage medium based on deep learning [P]. Chongqing: CN202410623414.3, 2024-09-17.), which uses DMNet as the backbone network and a multi-scale dynamic convolution module. The convolution module can adaptively select feature scales according to the input image, so as to better handle the complex structure in the image. However, this method does not take into account the extraction of frequency domain features, and the dynamic convolution module has high requirements for the distribution and diversity of input data. Feng Jing et al. proposed an image deblurring method based on frequency domain features in patent 2 (Feng Jing, Sun Yu, Liu Juan, et al. A pathological image deblurring method based on frequency domain features [P]. Hubei Province: CN202410261584.1, 2024-06-28.). This method uses Fourier transform to extract frequency domain features, but only uses simple 1×1 convolution and Gelu activation function for the spectrum obtained after Fourier transform, which cannot fully mine the information in the frequency domain. Niu Yuzhen et al. proposed an image deblurring method based on a void double residual network in Patent 3 (Niu Yuzhen, Lin Huangwei, Chen Youkun. Image deblurring method and system based on void double residual multi-scale deep network [P]. Fujian Province: CN202210335774.4, 2024-09-06.). This method uses a special double residual connection to integrate shallow features into the deep layer, and uses a wavelet transform module at the end of the network to better restore details. However, the wavelet transform module processes the feature map output by the previous network, and there is a problem that the front part of the network model is prone to losing detail information.

[0006] The above image deblurring methods do not make good use of frequency domain information and spatial domain information for network model training at the same time. There are defects such as insufficient feature information extraction, and when using the frequency domain information extracted by Fourier transform and wavelet transform, it is not well integrated with the spatial domain information. Spatial domain information reflects the position and structure of objects in the image, and frequency domain information reflects the speed of change of image pixel values. The two have different characteristics and require different processing methods to better complement each other. For the frequency domain loss function of the model, some of the above methods directly calculate the difference by using the spectrum after Fourier transform. Although it reflects the difference in frequency distribution between the restored image and the clear image to a certain extent, the frequency distribution of the image is not uniform, and its low-frequency components often occupy a larger proportion in the spectrum. However, for blurred images, their high-frequency components are more seriously damaged. Directly using the spectrum to calculate the loss function is likely to make the model pay more attention to low-frequency information, affecting the restoration effect of image contour details. The present invention adopts U-Net network as the backbone network, and retains the main features of the blurred image through downsampling and upsampling of the network to suppress ghosting. For the jump connection of the network, a multi-scale mixed feature jump connection is adopted to enhance the information interaction between different layers of the network, and a deep and shallow layer feature fusion convolution module is used at the end of the model to fuse the information of different layers. In the network, a multi-domain feature enhancement convolution module including a Fourier transform branch, a wavelet transform branch and the original spatial convolution branch is used to perform feature enhancement and information fusion in the spatial domain and frequency domain. For the Fourier frequency domain loss function, a new frequency domain loss function is weighted to construct it, so that the model focuses on high-frequency components that are easily damaged. By introducing frequency domain information into the model, the model can better restore the blurred image. Summary of the invention

[0007] In order to improve the performance of the model in the image deblurring task, enhance the restoration effect of motion blurred images, and alleviate the ghosting problem of images, the present invention proposes an image deblurring method based on multi-domain feature enhanced U-Net. First, the present invention improves the jump connection of U-Net, uses multi-scale mixed feature jump connection to replace the original jump connection, and uses deep and shallow layer feature fusion convolution module to fuse the features of different stages in the model. Secondly, in the Fourier frequency domain loss function, the spectrum graph is weighted to enhance the proportion of high-frequency components and balance the model's attention to frequency components. In addition, a multi-domain feature enhanced convolution module is designed to fuse information from multiple domains. Among them, the spatial convolution branch uses ordinary convolution and hole convolution to perceive the main structure of the image; the Fourier transform branch uses spatial attention to the spectrum graph to make the model pay more attention to important frequencies; the wavelet transform branch performs two layers of wavelet transform, expands the receptive field of the convolution kernel, and fuses features at multiple scales. Through the above method, the restoration effect of the model on blurred images can be effectively improved.

[0008] 1. An image deblurring method based on multi-domain feature enhanced U-Net, characterized in that it comprises the following steps:

[0009] S1. Data preprocessing: cropping the motion blurred image dataset containing blurred and clear image pairs, and cropping the blurred and clear image pairs contained in the dataset into multiple blurred and clear image pairs of 256×256 pixels;

[0010] S2. Deblurring network model construction: Using the improved U-Net as the basic architecture of the network model, adopting multi-scale mixed feature jump connection, and combining the multi-domain feature enhancement convolution module and the deep and shallow feature fusion convolution module with the network, the image deblurring network model is constructed;

[0011] S3, model training and image restoration: input the blurred image into the constructed network, suppress the ghosting of the blurred image, enhance the features of the blurred image to achieve the effect of image deblurring, and output the restored image;

[0012] S4, calculate the loss value: compare the repaired image with the clear image, calculate the loss value of the two through the loss function, and continuously update the model parameters until the model converges;

[0013] S5. The final network model can complete the image deblurring task, perform feature restoration on the input blurred image, and output a clearer image.

[0014] 2. The image deblurring method based on multi-domain feature enhanced U-Net according to claim 1, characterized in that step S2 comprises the following steps:

[0015] S21. Build the basic network framework. The model uses the improved U-Net as the basic structure. The difference from the original U-Net is that it uses multi-scale mixed feature jump connections. For feature maps of different sizes, they are uniformly scaled to the same size as the input image, and fused between channels through 1×1 convolution. Each batch of feature maps of different sizes is adjusted to the same number of channels and superimposed together, and then scaled to the original different sizes through downsampling and superimposed on the corresponding layer;

[0016] S22, constructing the overall network structure, the model designed by the present invention is composed of a convolution block composed of downsampling alternately composed of 4 groups of multi-domain feature enhancement convolution modules and a maximum pooling layer, a convolution block composed of upsampling alternately composed of 1×1 convolution, 4 groups of multi-domain feature enhancement convolution modules and bilinear interpolation, and a deep and shallow feature fusion convolution module;

[0017] S23, construct a multi-domain feature enhancement convolution module, which includes three branches: Fourier branch, spatial convolution branch, and wavelet transform branch;

[0018] A spatial attention module (SAM) is introduced in the Fourier branch to enhance the frequency domain features. This branch can be expressed as:

[0019] x f =IFFT(SAM(FFT(x)))

[0020] Among them, FFT() and IFFT() represent fast Fourier transform and inverse transform, SAM() represents spatial attention module, and x f and x are the output and input of the Fourier branch;

[0021] In the spatial convolution branch, ordinary convolution and dilation rate 2 dilated convolution are used to convolve the feature map at the same time, and the feature maps after the convolution are added as the output of the spatial convolution branch. This branch can be expressed as:

[0022] x s =Conv(x)+DConv(x)

[0023] Among them, x s and x represent the output and input of the spatial convolution branch, Conv() represents ordinary convolution, and DConv() represents dilated convolution;

[0024] The wavelet transform branch uses a double-layer Haar wavelet transform, shrinks the subband map of the first layer to the same size as the subband map of the second layer through average pooling, and splices it to the subband map of the second layer in the channel dimension; this branch performs 3×3 ordinary convolution on the subband map of the first layer and the subband map of the second layer, performs inverse wavelet transform on the subband of the second layer and superimposes it on the mean subband of the first layer, and then performs inverse wavelet transform on the subband map of the first layer as the output of the wavelet branch. This branch can be expressed as:

[0025] x LL1 ,x LH1 ,x HL1 ,x HH1 =HWT(x)

[0026] x LL2* =IHWT(Conv(HWT(x LL1 )+Avgpool(HWT(x))))

[0027] x' LL =x LL1 +x LL2*

[0028] x wt =IHWT(Conv(x LL ', x LH1 ,x HL1 ,x HH1 ))

[0029] Among them, x wt and x represent the output table and input of the spatial convolution branch, x LL2* Represents the output of the second layer wavelet transform, HWT() and IHWT() represent Haar wavelet transform and inverse transform, Avgpool() represents average pooling;

[0030] The outputs of the three branches are added together and ordinary convolution is performed to obtain the output of the multi-domain feature enhancement convolution module, which can be expressed as:

[0031] x FE =Conv(x s +x f +x wt )

[0032] S24, build a deep and shallow feature fusion convolution module, fuse the image features at different stages in the model, and splice the original feature map, the feature map obtained by multi-scale mixed feature jump connection, and the feature map finally output by the multi-domain feature enhancement convolution module together in the channel dimension. The process can be expressed as:

[0033] x fusion =Cat channel (x in ,x mix ,x out )

[0034] Among them, x fusion Represents the feature map obtained after splicing, x in represents the original feature map, x out The feature map representing the final output of the module, Cat channel () indicates splicing in the channel dimension;

[0035] For the mixed feature map, use 1×1 convolution to fuse and interact information between channels, then perform double-branch convolution including ordinary convolution and hole convolution and splice in the channel dimension, and finally perform 1×1 convolution and two layers of ordinary convolution to adjust the fused features to obtain the final repaired image. The process can be expressed as:

[0036] x0=Conv 1×1 (x fusion )

[0037] x c =Conv(x0)

[0038] x dc =DConv(x0)

[0039] x final =Conv(Conv(Conv1×1 (Cat channel (x c ,x dc ))))

[0040] Among them, x0 represents the feature map after 1×1 convolution, x c and x dc They represent the feature maps of the output of ordinary convolution and dilated convolution respectively, x final Represents the final repaired image.

[0041] 3. The image deblurring method based on multi-domain feature enhanced U-Net according to claim 1, characterized in that step S3 comprises the following steps:

[0042] S31. After the input image passes through the model, a restored image is obtained. The loss values ​​of the restored image and the clear image are calculated through a loss function. The loss function is composed of an MSE loss function and a weighted Fourier frequency domain loss function. The MSE loss function can be expressed as:

[0043]

[0044] Where M and N are the width and height of the image, respectively, and I'(i,j) and I(i,j) represent the pixel values ​​of the restored image and the clear image;

[0045] The weighted Fourier frequency domain loss function calculates the loss value of the spectrum obtained after the image is Fourier transformed. The two-dimensional discrete Fourier transform used can be expressed as:

[0046]

[0047] Among them, f(x,y) is the image in the spatial domain, and F(u,v) is the spectrum in the frequency domain;

[0048] The low-frequency part of the spectrum obtained by two-dimensional discrete Fourier transform is distributed in the four corners. The frequency can be positively correlated with the distance from the pixel to the center of the image by centering the spectrum through spectrum shift. The high-frequency and low-frequency components are not evenly distributed in the spectrum. The model usually pays more attention to the low-frequency components. In order to improve the model's attention to high-frequency components, a weight function based on the center point distance is designed to weight the spectrum. The weighting process of the spectrum can be expressed as:

[0049]

[0050] W(u,v)=α Sigmoid(d)

[0051] F w (u,v)=F c (u,v)×W(u,v)

[0052] Wherein, W(u,v) represents the weight function, Sigmoid() represents the normalization function, and α represents the range parameter of the weight, which is used to limit the value range of the weight to [1,α]. In the present invention, α is taken as e, and F c (u,v) and F w (u,v) represents the centered spectrum graph and the weighted spectrum graph;

[0053] When calculating the loss function, since the calculated value is a complex number rather than a real number, it is necessary to calculate the loss values ​​of amplitude and phase separately;

[0054] The amplitude can be expressed as:

[0055]

[0056] The phase can be expressed as:

[0057] P(F w )=atan(Im(F w (u,v)) / Re(F w (u,v)))

[0058] The weighted Fourier frequency domain loss function can be expressed as:

[0059]

[0060] Among them, F' w (u, v) and F(u, v) represent the restored image and the clear image, λ is a weight used to adjust the ratio between the two, and in the present invention, the value is 1;

[0061] The complete loss function can be expressed as:

[0062] L total =L MSE +αL FT

[0063] Wherein, α is a weight used to adjust the ratio of the two loss functions, and in the present invention, the value is 1;

[0064] S32, according to the obtained loss value, continuously update the model parameters to reduce the loss value until the model converges, and finally obtain a network model capable of performing the image deblurring task, the size of the image processed by the model is 256×256 pixels;

[0065] S33. For an image of any size, use mirror padding to fill its width and height to an integer multiple of 256, then divide it into multiple image sub-blocks of 256×256 pixels, input these image sub-blocks into the image deblurring network model respectively, obtain the repaired sub-blocks, then splice these sub-blocks together in the original order, crop them to the size of the original image, and output the repaired image.

[0066] Beneficial effects of the present invention:

[0067] 1. The present invention proposes an image deblurring method based on multi-domain feature enhanced U-Net. The multi-domain feature enhanced convolution module is integrated into the U-Net network, Fourier transform is used to obtain frequency domain information, and wavelet transform is used to separate the subject structure and edge contour of the image. The addition of multiple types of image information enhances the network model's perception of the image and improves the quality of the restored image.

[0068] 2. The present invention modifies the jump connection of the traditional U-Net. Different from the jump connection of the single-scale feature used in the traditional U-Net, the multi-scale mixed feature jump connection is not limited to the feature transfer between the same spatial scale levels, but through the scaling of the feature map, the feature maps of different scales obtained by the first 4 groups of convolution and downsampling blocks are mixed and scaled to the same size, and then resampled to different scales after mixing and reorganization, and sent to the last 4 groups of convolution and upsampling blocks. This design enables the jump connection to transmit multi-scale feature information, enhances the diversity of feature extraction of the model, and improves the stability and generalization ability of the model.

[0069] 3. In terms of the design of the loss function, the present invention adds the Fourier loss function in the frequency domain, and enhances the learning of high-frequency components by focusing on high-frequency components and low-frequency components through the weighted equalization model of the Fourier loss function. Combined with the MSE loss function, the model's reconstruction performance in the main structure of the image and edge contour details is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The overall structure diagram of the network model of the image deblurring method based on multi-domain feature enhanced U-Net used in the embodiment of the present invention;

[0071] Figure 2 A structural diagram of a multi-domain feature enhancement convolution module described in an image deblurring method based on multi-domain feature enhancement U-Net used in an embodiment of the present invention;

[0072] Figure 3 A flow chart for implementing the weighted Fourier loss function described in the image deblurring method based on multi-domain feature enhanced U-Net used in an embodiment of the present invention;

[0073] Figure 4A diagram showing an implementation of the training method for the image deblurring method based on multi-domain feature enhanced U-Net used in an embodiment of the present invention;

[0074] Figure 5 This is a diagram showing the effect of implementing the image deblurring method based on multi-domain feature enhanced U-Net used in an embodiment of the present invention. The figure compares the deblurring effects of the present invention, DeepDeblur, and DeblurGAN-V2. DETAILED DESCRIPTION

[0075] In order to understand the technical features, purposes and beneficial effects of the present invention more clearly, an embodiment of the present invention is further described in conjunction with the accompanying drawings. The embodiment is only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Some non-essential improvements and adjustments made by technicians in this field based on the content of the present invention also fall within the scope of protection of the present invention.

[0076] 1. An image deblurring method based on multi-domain feature enhanced U-Net, characterized in that it comprises the following steps:

[0077] S1. Data preprocessing: cropping the motion blurred image dataset containing blurred and clear image pairs, and cropping the blurred and clear image pairs contained in the dataset into multiple blurred and clear image pairs of 256×256 pixels;

[0078] Furthermore, the present invention uses the HIDE dataset to train and test the model. The HIDE dataset covers a variety of blur situations caused by the relative motion between the imaging device and the scene, and includes 8422 pairs of clear-blurred images;

[0079] S2. Deblurring network model construction: The improved U-Net is used as the basic architecture of the network model. Multi-scale mixed feature jump connection is adopted. The multi-domain feature enhancement convolution module and the deep and shallow feature fusion convolution module are combined with the network to build an image deblurring network model. The overall structure of the network model is shown in the figure below. Figure 1 As shown;

[0080] S3, model training and image restoration: input the blurred image into the constructed network, suppress the ghosting of the blurred image, enhance the features of the blurred image to achieve the effect of image deblurring, and output the restored image;

[0081] Furthermore, the optimizer of the network model uses the Adam optimizer, the learning rate is set to 0.0001, and the number of training rounds is set to 200 rounds;

[0082] S4, calculate the loss value: compare the repaired image with the clear image, calculate the loss value of the two through the loss function, and continuously update the model parameters until the model converges;

[0083] S5. The final network model can complete the image deblurring task, perform feature restoration on the input blurred image, and output a clearer image.

[0084] 2. The image deblurring method based on multi-domain feature enhanced U-Net according to claim 1, characterized in that step S2 comprises the following steps:

[0085] S21. Build the basic network framework. The model uses the improved U-Net as the basic structure. The difference from the original U-Net is that it uses multi-scale mixed feature jump connections. For feature maps of different sizes, they are uniformly scaled to the same size as the input image, and fused between channels through 1×1 convolution. Each batch of feature maps of different sizes is adjusted to the same number of channels and superimposed together, and then scaled to the original different sizes through downsampling and superimposed on the corresponding layer;

[0086] S22, constructing the overall network structure, the model designed by the present invention is composed of a convolution block composed of downsampling alternately composed of 4 groups of multi-domain feature enhancement convolution modules and a maximum pooling layer, a convolution block composed of upsampling alternately composed of 1×1 convolution, 4 groups of multi-domain feature enhancement convolution modules and bilinear interpolation, and a deep and shallow feature fusion convolution module;

[0087] S23, construct a multi-domain feature enhancement convolution module, which contains three branches: Fourier branch, spatial convolution branch, and wavelet transform branch. The module structure diagram is shown in Figure 2 As shown;

[0088] A spatial attention module (SAM) is introduced in the Fourier branch to enhance the frequency domain features. This branch can be expressed as:

[0089] x f =IFFT(SAM(FFT(x)))

[0090] Among them, FFT() and IFFT() represent fast Fourier transform and inverse transform, SAM() represents spatial attention module, and x f and x are the output and input of the Fourier branch;

[0091] In the spatial convolution branch, ordinary convolution and dilation rate 2 dilated convolution are used to convolve the feature map at the same time, and the feature maps after the convolution are added as the output of the spatial convolution branch. This branch can be expressed as:

[0092] x s =Conv(x)+DConv(x)

[0093] Among them, x sand x represent the output and input of the spatial convolution branch, Conv() represents ordinary convolution, and DConv() represents dilated convolution;

[0094] The wavelet transform branch uses a double-layer Haar wavelet transform, shrinks the subband map of the first layer to the same size as the subband map of the second layer through average pooling, and splices it to the subband map of the second layer in the channel dimension; this branch performs 3×3 ordinary convolution on the subband map of the first layer and the subband map of the second layer, performs inverse wavelet transform on the subband of the second layer and superimposes it on the mean subband of the first layer, and then performs inverse wavelet transform on the subband map of the first layer as the output of the wavelet branch. This branch can be expressed as:

[0095] x LL1 ,x LH1 ,x HL1 ,x HH1 =HWT(x)

[0096] x LL2* =IHWT(Conv(HWT(x LL1 )+Avgpool(HWT(x))))

[0097] x' LL =x LL1 +x LL2*

[0098] x wt =IHWT(Conv(x LL ', x LH1 ,x HL1 ,x HH1 ))

[0099] Among them, x wt and x represent the output table and input of the spatial convolution branch, x LL2* Represents the output of the second layer wavelet transform, HWT() and IHWT() represent Haar wavelet transform and inverse transform, Avgpool() represents average pooling;

[0100] The outputs of the three branches are added together and ordinary convolution is performed to obtain the output of the multi-domain feature enhancement convolution module, which can be expressed as:

[0101] x FE =Conv(x s +x f +x wt )

[0102] S24, build a deep and shallow feature fusion convolution module, fuse the image features at different stages in the model, and splice the original feature map, the feature map obtained by multi-scale mixed feature jump connection, and the feature map finally output by the multi-domain feature enhancement convolution module together in the channel dimension. The process can be expressed as:

[0103] x fusion =Cat channel (x in ,x mix ,x out )

[0104] Among them, x fusion Represents the feature map obtained after splicing, x in represents the original feature map, x out The feature map representing the final output of the module, Cat channel () indicates splicing in the channel dimension;

[0105] For the mixed feature map, use 1×1 convolution to fuse and interact information between channels, then perform double-branch convolution including ordinary convolution and hole convolution and splice in the channel dimension, and finally perform 1×1 convolution and two layers of ordinary convolution to adjust the fused features to obtain the final repaired image. The process can be expressed as:

[0106] x0=Conv 1×1 (x fusion )

[0107] x c =Conv(x0)

[0108] x dc =DConv(x0)

[0109] x final =Conv(Conv(Conv 1×1 (Cat channel (x c ,x dc ))))

[0110] Among them, x0 represents the feature map after 1×1 convolution, x c and x dc They represent the feature maps of the output of ordinary convolution and dilated convolution respectively, x final Represents the final repaired image.

[0111] 3. The image deblurring method based on multi-domain feature enhanced U-Net according to claim 1, characterized in that step S3 comprises the following steps:

[0112] S31. After the input image passes through the deblurring network model, a restored image is obtained. The loss values ​​of the restored image and the clear image are calculated through a loss function. The loss function is composed of an MSE loss function and a weighted Fourier frequency domain loss function. The MSE loss function can be expressed as:

[0113]

[0114] Where M and N are the width and height of the image, respectively, and I'(i,j) and I(i,j) represent the pixel values ​​of the restored image and the clear image;

[0115] The weighted Fourier frequency domain loss function calculates the loss value of the spectrum obtained after the image is Fourier transformed. The implementation flowchart is as follows: Figure 3 As shown, the two-dimensional discrete Fourier transform used can be expressed as:

[0116]

[0117] Among them, f(x,y) is the image in the spatial domain, and F(u,v) is the spectrum in the frequency domain;

[0118] The low-frequency part of the spectrum obtained by two-dimensional discrete Fourier transform is distributed in the four corners. The frequency can be positively correlated with the distance from the pixel to the center of the image by centering the spectrum through spectrum shift. The high-frequency and low-frequency components are not evenly distributed in the spectrum. The model usually pays more attention to the low-frequency components. In order to improve the model's attention to high-frequency components, a weight function based on the center point distance is designed to weight the spectrum. The weighting process of the spectrum can be expressed as:

[0119]

[0120] W(u,v)=α Sigmoid(d)

[0121] F w (u,v)=F c (u,v)×W(u,v)

[0122] Wherein, W(u,v) represents the weight function, Sigmoid() represents the normalization function, and α represents the range parameter of the weight, which is used to limit the value range of the weight to [1,α]. In the present invention, α is taken as e, and F c (u,v) and F w (u,v) represents the centered spectrum graph and the weighted spectrum graph;

[0123] When calculating the loss function, since the calculated value is a complex number rather than a real number, it is necessary to calculate the loss values ​​of amplitude and phase separately;

[0124] The amplitude can be expressed as:

[0125]

[0126] The phase can be expressed as:

[0127] P(F w )=atan(Im(F w (u,v)) / Re(F w (u,v)))

[0128] The weighted Fourier frequency domain loss function can be expressed as:

[0129]

[0130] Among them, F' w (u, v) and F(u, v) represent the restored image and the clear image, λ is a weight used to adjust the ratio between the two, and in the present invention, the value is 1;

[0131] The complete loss function can be expressed as:

[0132] L total =L MSE +αL FT

[0133] Wherein, α is a weight used to adjust the ratio of the two loss functions, and in the present invention, the value is 1;

[0134] S32. Based on the obtained loss value, the model parameters are continuously updated to reduce the loss value until the model converges, and finally a network model capable of performing the image deblurring task is obtained. The training method is implemented as shown in the figure. Figure 4 As shown; the size of the image processed by the model is 256×256 pixels;

[0135] S33. For an image of any size, use mirror padding to fill its width and height to an integer multiple of 256, then divide it into multiple image sub-blocks of 256×256 pixels, input these image sub-blocks into the image deblurring network model respectively, obtain the repaired sub-blocks, then splice these sub-blocks together in the original order, crop them to the size of the original image, and output the repaired image.

[0136] Simulation experiment

[0137] Figure 5 is a diagram showing the effect of implementing the image deblurring method based on multi-domain feature enhanced U-Net used in an embodiment of the present invention, Figure 5It can be seen that the present invention can effectively complete the image deblurring task and can adapt to a variety of environments. In order to evaluate the deblurring effect of the present invention, the peak signal-to-noise ratio (PSNR) index is used to compare with other mainstream methods, and the relevant results are shown in Table 1.

[0138] Table 1 Comparative experimental evaluation index statistics

[0139] method PSNR DeepDeblur 25.73 DeblurGAN-V2 26.61 The present invention 27.13

[0140] All methods are experimented with the HIDE dataset. The above experimental results show that the present invention achieves better results in the image deblurring task and effectively improves the quality of the restored image.

[0141] The above is a description of the method of the present invention, and those skilled in the art can implement the method of the present invention based on the description of the content. Based on the above content of the present invention, other embodiments obtained by those skilled in the art without creative work should all fall within the scope of protection of the present invention.

Claims

1. An image deblurring method based on multi-domain feature enhanced U-Net, characterized in that: The following steps are involved: S1. Data preprocessing: cropping the motion blurred image dataset containing blurred and clear image pairs, and cropping the blurred and clear image pairs contained in the dataset into multiple blurred and clear image pairs of 256×256 pixels; Furthermore, the present invention uses the HIDE dataset for model training and testing. The HIDE dataset covers a variety of blur situations caused by the relative motion between the imaging device and the scene, and includes 8422 pairs of clear-blurred images; S2. Deblurring network model construction: Using the improved U-Net as the basic architecture of the network model, adopting multi-scale mixed feature jump connection, and combining the multi-domain feature enhancement convolution module and the deep and shallow feature fusion convolution module with the network, the image deblurring network model is constructed. The overall structure of the network model is shown in Figure 1. S3, model training and image restoration: input the blurred image into the constructed network, suppress the ghosting of the blurred image, enhance the features of the blurred image to achieve the effect of image deblurring, and output the restored image; Furthermore, the optimizer of the network model uses the Adam optimizer, the learning rate is set to 0.0001, and the number of training rounds is set to 200 rounds; S4, calculate the loss value: compare the repaired image with the clear image, calculate the loss value of the two through the loss function, and continuously update the model parameters until the model converges; S5. The final network model can complete the image deblurring task, perform feature restoration on the input blurred image, and output a clearer image.

2. The image deblurring method based on multi-domain feature enhanced U-Net according to claim 1, characterized in that: The step S2 comprises the following steps: S21. Build the basic network framework. The model uses the improved U-Net as the basic structure. The difference from the original U-Net is that it uses multi-scale mixed feature jump connections. For feature maps of different sizes, they are uniformly scaled to the same size as the input image, and fused between channels through 1×1 convolution. Each batch of feature maps of different sizes is adjusted to the same number of channels and superimposed together, and then scaled to the original different sizes through downsampling and superimposed on the corresponding layer; S22, constructing the overall network structure, the model designed by the present invention is composed of a convolution block composed of downsampling alternately composed of 4 groups of multi-domain feature enhancement convolution modules and a maximum pooling layer, a convolution block composed of upsampling alternately composed of 1×1 convolution, 4 groups of multi-domain feature enhancement convolution modules and bilinear interpolation, and a deep and shallow feature fusion convolution module; S23, construct a multi-domain feature enhancement convolution module, which includes three branches: Fourier branch, spatial convolution branch, and wavelet transform branch. The module structure diagram is shown in Figure 2; A spatial attention module (SAM) is introduced in the Fourier branch to enhance the frequency domain features. This branch can be expressed as: x f =IFFT(SAM(FFT(x))) Among them, FFT() and IFFT() represent fast Fourier transform and inverse transform, SAM() represents spatial attention module, and x f and x are the output and input of the Fourier branch; In the spatial convolution branch, ordinary convolution and dilation rate 2 dilated convolution are used to convolve the feature map at the same time, and the feature maps after the convolution are added as the output of the spatial convolution branch. This branch can be expressed as: x s =Conv(x)+DConv(x) Among them, x s and x represent the output and input of the spatial convolution branch, Conv() represents ordinary convolution, and DConv() represents dilated convolution; The wavelet transform branch uses a double-layer Haar wavelet transform, shrinks the subband map of the first layer to the same size as the subband map of the second layer through average pooling, and splices it to the subband map of the second layer in the channel dimension; this branch performs 3×3 ordinary convolution on the subband map of the first layer and the subband map of the second layer, performs inverse wavelet transform on the subband of the second layer and superimposes it on the mean subband of the first layer, and then performs inverse wavelet transform on the subband map of the first layer as the output of the wavelet branch. This branch can be expressed as: x LL1 ,x LH1 ,x HL1 ,x HH1 =HWT(x) x LL2* =IHWT(Conv(HWT(x LL1 )+Avgpool(HWT(x)))) x LL '=x LL1 +x LL2* x wt =IHWT(Conv(x LL ',x LH1 ,x HL1 ,x HH1 )) Among them, x wt and x represent the output table and input of the spatial convolution branch, x LL2* Represents the output of the second layer wavelet transform, HWT() and IHWT() represent Haar wavelet transform and inverse transform, Avgpool() represents average pooling; The outputs of the three branches are added together and ordinary convolution is performed to obtain the output of the multi-domain feature enhancement convolution module, which can be expressed as: x FE =Conv(x s +x f +x wt ) S24, build a deep and shallow feature fusion convolution module, fuse the image features at different stages in the model, and splice the original feature map, the feature map obtained by multi-scale mixed feature jump connection, and the feature map finally output by the multi-domain feature enhancement convolution module together in the channel dimension. The process can be expressed as: x fusion =Cat channel (x in ,x mix ,x out ) Among them, x fusion Represents the feature map obtained after splicing, x in represents the original feature map, x out The feature map representing the final output of the module, Cat channel () indicates splicing in the channel dimension; For the mixed feature map, use 1×1 convolution to fuse and interact information between channels, then perform double-branch convolution including ordinary convolution and hole convolution and splice in the channel dimension, and finally perform 1×1 convolution and two layers of ordinary convolution to adjust the fused features to obtain the final repaired image. The process can be expressed as: x0=Conv 1×1 (x fusion ) x c =Conv(x0) x dc =DConv(x0) x final =Conv(Conv(Conv 1×1 (Cat channel (x c ,x dc )))) Among them, x0 represents the feature map after 1×1 convolution, x c and x dc They represent the feature maps of the output of ordinary convolution and dilated convolution respectively, x final Represents the final repaired image.

3. The image deblurring method based on multi-domain feature enhanced U-Net according to claim 1, characterized in that: The step S3 comprises the following steps: S31. After the input image passes through the deblurring network model, a restored image is obtained. The loss values ​​of the restored image and the clear image are calculated through a loss function. The loss function is composed of an MSE loss function and a weighted Fourier frequency domain loss function. The MSE loss function can be expressed as: Where M and N are the width and height of the image, respectively, and I'(i,j) and I(i,j) represent the pixel values ​​of the restored image and the clear image; The weighted Fourier frequency domain loss function calculates the loss value of the spectrum obtained after the image is Fourier transformed. Its implementation flowchart is shown in Figure 3. The two-dimensional discrete Fourier transform used can be expressed as: Among them, f(x,y) is the image in the spatial domain, and F(u,v) is the spectrum in the frequency domain; The low-frequency part of the spectrum obtained by two-dimensional discrete Fourier transform is distributed in the four corners. The frequency can be positively correlated with the distance from the pixel to the center of the image by centering the spectrum through spectrum shift. The high-frequency and low-frequency components are not evenly distributed in the spectrum. The model usually pays more attention to the low-frequency components. In order to improve the model's attention to high-frequency components, a weight function based on the center point distance is designed to weight the spectrum. The weighting process of the spectrum can be expressed as: W(u,v)=a Sigmoid(d) F w (u,v)=F c (u,v)×W(u,v) Wherein, W(u,v) represents the weight function, Sigmoid() represents the normalization function, and α represents the range parameter of the weight, which is used to limit the value range of the weight to [1,α]. In the present invention, α is taken as e, and F c (u,v) and F w (u,v) represents the centered spectrum graph and the weighted spectrum graph; When calculating the loss function, since the calculated value is a complex number rather than a real number, it is necessary to calculate the loss values ​​of amplitude and phase separately; The amplitude can be expressed as: The phase can be expressed as: P(F w )=atan(Im(F w (u,v)) / Re(F w (u,v))) The weighted Fourier frequency domain loss function can be expressed as: Among them, F' w (u, v) and F(u, v) represent the restored image and the clear image, λ is a weight used to adjust the ratio between the two, and the value is 1 in the present invention; The complete loss function can be expressed as: L total =L MSE +αL FT Wherein, α is a weight used to adjust the ratio of the two loss functions, and in the present invention, the value is 1; S32, according to the obtained loss value, the model parameters are continuously updated to reduce the loss value until the model converges, and finally a network model capable of performing the image deblurring task is obtained. The implementation diagram of the training method is shown in FIG4; the size of the image processed by the model is 256×256 pixels; S33. For an image of any size, use mirror padding to fill its width and height to an integer multiple of 256, then divide it into multiple image sub-blocks of 256×256 pixels, input these image sub-blocks into the image deblurring network model respectively, obtain the repaired sub-blocks, then splice these sub-blocks together in the original order, crop them to the size of the original image, and output the repaired image.

Images