A dual-branch image deblurring method and device based on frequency domain perception

Abstract

The application provides a dual-branch image deblurring method and device based on frequency domain perception, relates to the field of computer vision, and comprises the following steps: S1, acquiring a sample image set, performing image preprocessing on the sample image set, and obtaining a training image set; S2, constructing a deblurring network, training the deblurring network through the training image set, and obtaining a trained deblurring network; and S3, performing deblurring operation on the image through the trained deblurring network.The deblurring network of the application not only maintains the characteristics of simplicity and efficiency when processing frequency domain and spatial domain information, but also takes into account detail recovery and calculation efficiency, and the dual branches do not interfere with each other, thereby effectively improving the deblurring effect of the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and particularly relates to a dual-branch image deblurring method and device based on frequency domain perception. BACKGROUND

[0002] In the field of computer vision, image deblurring is one of the important image restoration tasks in computer vision, aiming to restore blurred images to clear images. Image blur is usually caused by device imaging systems or external factors, mainly including motion blur, defocus blur and Gaussian blur. In particular, image motion blur caused by camera or object motion is the most concerned and challenging problem.

[0003] In the problem of single motion image deblurring, most traditional deblurring methods are based on Richardson-Lucy method and Wiener deconvolution method, such as two-step deblurring method for preserving texture (see Chen F, Huang X, Chen W. Texture-preserving image deblurring [J]. IEEE Signal processing letters, 2010, 17(12): 1018-1021.) and blind image deblurring method based on local maximum gradient prior (see Chen L, Fang F, Wang T, et al. Blind image deblurring with local maximum gradient prior [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 1742-1750.) and the like. The improved version of these traditional methods usually relies on the estimation of blur kernel and the prior knowledge of image, and restores the image through deconvolution or regularization. However, these traditional methods highly depend on the accurate estimation of blur kernel, and in practical applications, it is often difficult to accurately estimate due to the existence of complex blur kernel, and it is difficult to cope with complex situations in real scenarios. With the continuous progress of deep learning technology, more and more deep neural network (DNN) based methods are applied to the task of single motion image deblurring. Unlike traditional methods, DNN can automatically learn the features of blurred images through a large amount of training data, without relying on accurate blur kernel estimation as traditional methods do. The powerful nonlinear characteristics of DNN enable it to handle complex blur patterns, including non-uniform or spatially varying blur phenomena, greatly enhancing the robustness and generalization ability of deblurring algorithms.

[0004] Although DNNs have made significant progress in single motion image deblurring, there are still some limitations in existing research. At present, most DNN-based single motion image deblurring algorithms either focus on processing spatial domain information or frequency domain information, and it is difficult to consider both. In the spatial domain processing method, the network extracts image features through the convolution layer, and restores the clear image by using the deconvolution or up-sampling layer, focusing on processing the pixel-level features, effectively enhancing the local details and edge features of the image (see Wang X, Girshick R, Gupta A, et al. Non-local neural networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7794-7803. Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module [C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 3-19.). However, this method ignores the frequency information of the image, making it difficult to fully restore the high-frequency details.

[0005] In the frequency domain processing method, the model converts the image to the frequency space, extracts and enhances the features, which is particularly helpful for restoring high-frequency information and removing blur components. However, this method usually relies on complex operations such as Fourier transform or wavelet transform. Over-reliance on frequency domain processing not only greatly increases the complexity of the model, but also easily ignores the spatial context information of the image. The MLWNet method proposed by Gao et al. introduces a learnable wavelet transform module and a multi-scale loss function, and uses the SIMO multi-scale architecture to improve the image deblurring effect and detail restoration ability, and also better adapt to real scenes. However, due to the over-reliance on frequency domain processing, although the performance is good, the model complexity is high and the computing resource demand is large (see Gao X, Qiu T, Zhang X, et al. Efficient multi-scale network with learned discrete wavelet transform for blind motion deblurring [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 2733-2742.). SUMMARY

[0006] Therefore, the present application aims to provide a dual-branch image deblurring method and device based on frequency domain perception, which solves the technical problems of complex structure and poor deblurring effect of existing deblurring networks.

[0007] The present application provides a dual-branch image deblurring method based on frequency domain perception, comprising the steps of:

[0008] S1: obtaining a sample image set, performing image preprocessing on the sample image set to obtain a training image set;

[0009] S2: constructing a deblurring network, training the deblurring network with the training image set to obtain a trained deblurring network;

[0010] S3: performing image deblurring operation through the trained deblurring network.

[0011] Preferably, step S1 is specifically:

[0012] S11: loading sample images, and sequentially performing cropping operation, flipping operation, rotation operation and format conversion operation on the sample images to obtain training images;

[0013] S12: repeating step S11 until all sample images are traversed to obtain a training image set.

[0014] Preferably,

[0015] The deblurring network comprises a first FEM module, an encoder, a double-branch structure network, a decoder and a second FEM module;

[0016] The first FEM module is connected with the encoder, the encoder is connected with the decoder, and the decoder is connected with the second FEM module.

[0017] The encoder, the double-branch structure network and the decoder are connected in sequence.

[0018] Preferably,

[0019] The double-branch structure network comprises a first double-branch module and a second double-branch module in parallel;

[0020] The first double-branch module and the second double-branch module have the same structure;

[0021] The first double-branch module comprises a NAF module, an LFA module and a D-FM module;

[0022] The encoder is connected with the NAF module and the LFA module;

[0023] The NAF module and the LFA module are connected with the D-FM module;

[0024] The D-FM module is connected with the decoder;

[0025] The LFA module comprises a wavelet decomposition module, a convolution network, a high-frequency feature enhancement module, a wavelet fusion module and a feature distillation module;

[0026] The encoder is connected with the wavelet decomposition module;

[0027] The wavelet decomposition module is connected with the convolution network and the high-frequency feature enhancement module;

[0028] The convolution network and the high-frequency feature enhancement module are connected with the wavelet fusion module;

[0029] The wavelet fusion module, the feature distillation module and the D-FM module are connected in sequence.

[0030] Preferably,

[0031] The encoder comprises a first ENAF module, a second ENAF module, a third ENAF module and a fourth ENAF module connected in sequence;

[0032] The decoder comprises a first DNAF module, a second DNAF module, a third DNAF module and a fourth DNAF module connected in sequence;

[0033] The first ENAF module is connected with the first DNAF module;

[0034] The second ENAF module is connected with the second DNAF module;

[0035] The third ENAF module, the first double-branch module and the third DNAF module are sequentially connected;

[0036] The fourth ENAF module, the second double-branch module and the fourth DNAF module are sequentially connected;

[0037] The fourth ENAF module comprises 28 NAF modules, and the first ENAF module, the second ENAF module, the third ENAF module, the first DNAF module, the second DNAF module, the third DNAF module and the fourth DNAF module each comprise one NAF module.

[0038] Preferably, the step S2 specifically comprises:

[0039] S21: inputting the training image into the first FEM module to obtain an initial feature image;

[0040] S22: inputting the initial feature image into the first ENAF module to obtain a first feature image; inputting the first feature image into the second ENAF module after down-sampling operation to obtain a second feature image; inputting the second feature image into the third ENAF module after down-sampling operation to obtain a third feature image; inputting the third feature image into the fourth ENAF module after down-sampling operation to obtain a fourth feature image;

[0041] S23: inputting the fourth feature image into the second double-branch module to perform feature extraction in the spatial domain and the frequency domain, obtaining a second fusion feature image; inputting the second fusion feature image into the fourth DNAF module to obtain a fourth output image;

[0042] S24: inputting the third feature image into the first double-branch module to perform feature extraction in the spatial domain and the frequency domain, obtaining a first fusion feature image; inputting the first fusion feature image and the fourth output image after up-sampling operation into the third DNAF module to obtain a third output image;

[0043] S25: inputting the second feature image and the third output image after up-sampling operation into the second DNAF module to obtain a second output image; inputting the first feature image and the second output image after up-sampling operation into the first DNAF module to obtain a first output image;

[0044] S26: inputting the first output image into the second FEM module to obtain an initial deblurring image; superimposing the training image and the initial deblurring image to obtain a final deblurring image;

[0045] S27: calculating a loss value through the final deblurring image, and adjusting parameters of the deblurring network;

[0046] S28: Repeat steps S21-S27 until the loss value is less than the preset value to obtain the trained deblurred network.

[0047] Preferred:

[0048] The processes for obtaining the first fused feature image and the second fused feature image are the same;

[0049] The process of obtaining the first fused feature image is as follows:

[0050] The spatial domain feature image of the third feature image is extracted by the NAF module, the frequency domain feature image of the third feature image is extracted by the LFA module, and the spatial domain feature image and the frequency domain feature image are fused by the D-FM module to obtain the first fused feature image.

[0051] Preferred:

[0052] The process of obtaining the frequency domain feature image of the third feature image is as follows:

[0053] The third feature image is subjected to two-dimensional discrete wavelet transform by the wavelet decomposition module to obtain the low-frequency subband F. LL High-frequency subband F LH High-frequency subband F HL and high-frequency subband F HH ;

[0054] Low-frequency subband F LL Inputting into a convolutional network yields enhanced low-frequency subbands. High-frequency subband F LH High-frequency subband F HL and high-frequency subband F HH Input the high-frequency feature enhancement module to obtain the enhanced high-frequency sub-bands respectively. Enhanced high-frequency subband and enhanced high-frequency subband

[0055] Enhanced low-frequency subband Enhanced high-frequency subband Enhanced high-frequency subband and enhanced high-frequency subband The input wavelet fusion module performs a two-dimensional discrete wavelet inverse transform to obtain the inverse transform feature image F′. out ;

[0056] Inverse transform feature image F′ out Input the feature distillation module to obtain the distilled feature image. Then, superimpose the distilled feature image with the third feature image to obtain the frequency domain feature image F of the third feature image. out .

[0057] A storage medium stores instructions and data for implementing the dual-branch image deblurring method based on frequency domain perception.

[0058] A dual-branch image deblurring device based on frequency domain perception comprises a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the dual-branch image deblurring method based on frequency domain perception.

[0059] The present application has the following beneficial effects:

[0060] A dual-branch structure network is arranged in the deblurring network, the dual-branch structure network comprises two parallel dual-branch modules, the dual-branch modules extract spatial domain features through NAF modules and extract frequency domain features through LFA modules, effectively combining the spatial domain and the frequency domain, and using a D-FM module to efficiently integrate the information of the spatial domain and the frequency domain; the dual-branch structure network makes the deblurring network not only maintain the characteristics of simplicity and efficiency when processing the frequency domain and the spatial domain information, but also take into account the details recovery and the calculation efficiency, and the dual branches do not interfere with each other, effectively improving the deblurring effect of the image. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The present application is an embodiment method flowchart;

[0062] Figure 2 The present application is a deblurring network structure diagram;

[0063] Figure 3 The present application is a dual-branch structure network structure diagram;

[0064] Figure 4 The present application is an LFA module structure diagram;

[0065] Figure 5 The present application is a D-FM module structure diagram;

[0066] Figure 6 The present application is a multi-scale channel attention mechanism module structure diagram;

[0067] Figure 7 The present application is an encoder data flow diagram;

[0068] Figure 8 The present application is an embodiment device structure diagram;

[0069] The implementation of the present application, functional characteristics and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0070] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0071] Referring to Figure 1 The application provides a dual-branch image deblurring method based on frequency domain perception, comprising the steps of:

[0072] S1: obtaining a sample image set, performing image preprocessing on the sample image set, and obtaining a training image set;

[0073] As an embodiment,

[0074] Step S1 is specifically:

[0075] S11: loading a sample image, sequentially performing cropping, flipping, rotating and format conversion operations on the sample image, and obtaining a training image;

[0076] Specifically,

[0077] First, for the sample image, a cropping strategy is adopted. The resolution of the original image is large, in order to reduce the amount of calculation and improve the training efficiency, all the images used for training are cropped into multiple 512x512 pixel sub-images. This strategy not only effectively reduces the size of the image and reduces the processing time, but also increases the diversity of the data, because each image will be divided into multiple independent image blocks, introducing more image variants in the training process, further improving the generalization ability of the model.

[0078] Second, in order to avoid overfitting of the model during training and improve the adaptability of the model to different image deformations, the application adopts an enhancement method based on geometric transformation. Each image is randomly flipped and rotated with a probability of 0.5. Random flipping enables the model to learn the symmetry of the image in the horizontal and vertical directions, while random rotation helps the model adapt to changes in the image at different angles. This data augmentation strategy not only increases the diversity of the training data, but also enhances the robustness of the model, enabling it to maintain good performance when facing images at different angles and directions.

[0079] Finally, in order to improve the data reading speed during training and reduce the memory occupation, all the image sub-blocks used for training are converted to lmdb format. The lmdb (Lightning Memory-Mapped Database) format is a high-efficiency data storage method that can quickly load and access data. By storing data in lmdb format, the data loading time can be reduced during large-scale data training, avoiding the memory overflow problem that may be caused by large data volume. Compared with traditional image storage methods, lmdb format can significantly improve data loading efficiency and provide better support for large-scale training through memory mapping and compression storage.

[0080] S12: repeat step S11 until all sample images are traversed, and a training image set is obtained.

[0081] S2: constructing a deblurring network, training the deblurring network through the training image set, and obtaining a trained deblurring network;

[0082] As an embodiment,

[0083] The deblurring network comprises a first FEM module, an encoder, a dual-branch structure network, a decoder and a second FEM module.

[0084] The first FEM module is connected with the encoder, the encoder is connected with the decoder, and the decoder is connected with the second FEM module.

[0085] The encoder, the dual-branch structure network and the decoder are connected in sequence.

[0086] Specifically, the structure of the deblurring network (DFNet) is as shown in Figure 2 The deblurring network takes a classic U-network framework as a baseline, and the encoder and the decoder of the DFNet mainly comprise nonlinear activation free (NAF) feature extraction modules. The dual-branch spatial domain feature processing is a NAF module, the frequency domain feature processing is a learnable frequency-domain awareness (LFA) module designed by the application, and the module for information fusion of the dual branches is a dual-branch-driven fusion module (D-FM).

[0087] As an embodiment,

[0088] The dual-branch structure network comprises a first dual-branch module and a second dual-branch module in parallel;

[0089] The first dual-branch module and the second dual-branch module have the same structure;

[0090] The first dual-branch module comprises a NAF module, an LFA module and a D-FM module;

[0091] The encoder is connected with the NAF module and the LFA module;

[0092] The NAF module and the LFA module are connected with the D-FM module;

[0093] The D-FM module is connected with the decoder;

[0094] The LFA module comprises a wavelet decomposition module, a convolution network, a high-frequency feature enhancement module, a wavelet fusion module and a feature distillation module.

[0095] The encoder is connected with the wavelet decomposition module;

[0096] The wavelet decomposition module is connected with the convolution network and the high-frequency feature enhancement module;

[0097] The convolution network and the high-frequency feature enhancement module are connected with the wavelet fusion module;

[0098] The wavelet fusion module, the feature distillation module and the D-FM module are connected in sequence.

[0099] Specifically, the structure of the double-branch network is as shown in Figure 3 The double-branch network is designed at the bottleneck layer and the next-high layer of the deblurring network, which is the design considering the deblurring performance of the model and the complexity of the model. The double-branch can fully utilize the spatial domain and frequency domain information, and the two branches do not interfere with each other. The spatial domain branch adopts a nonlinear feature extraction module without activation. The module extracts features through layer-by-layer convolution to capture local details and texture information of the image. This branch effectively reduces feature redundancy, maintains the lightweight of the model, and ensures that the local structure is maximally preserved without introducing nonlinear activation. The frequency domain branch extracts feature information at different frequencies through two-dimensional discrete wavelet transform (2D-DWT) for multi-scale frequency domain decomposition of the image.

[0100] The structure of the learnable frequency domain perception module (LFA) is as shown in Figure 4 Most current methods combine two-dimensional discrete wavelet transform with convolution network. The design steps are as follows: first, the features output by the encoder are decomposed by wavelet to obtain four sub-bands; then, the four frequency bands are subjected to equal feature enhancement and receptive field expansion operations through the convolution network; finally, the output of the convolution network is subjected to inverse wavelet transform to obtain the output of the frequency domain branch. Generally, low-frequency components can express the smooth appearance of global regions, while high-frequency components can capture rich details of local regions, such as edges, textures and other complex features. However, equal operations on the four sub-bands lack flexibility when processing different types of information. Therefore, the present application optimizes the initial frequency domain branch to obtain a learnable frequency domain perception module (LFA), which combines the low-frequency sub-band with the convolution network to increase the receptive field and extract global features, and combines the high-frequency sub-band with the feature enhancement module to extract local features, so that the frequency domain information can be more fully utilized and the complex blur types can be better handled.

[0101] The structure of the branch-driven fusion module (D-FM) is as shown in Figure 5 The fusion module performs weighted average on the features from the spatial domain branch and the frequency domain branch. The core of the module is a multi-scale channel attention mechanism module, as shown in Figure 6As shown, the attention mechanism has two paths, one is a local channel attention mechanism, and the other is a global channel attention mechanism. The local path is composed of two 1*1 convolution layers, a normalization layer and a Relu activation function. In the local path, the feature map first needs to pass through a 1*1 convolution to reduce the number of channels and reduce the calculation cost. Then, batch normalization processing is performed to ensure the stability of the model, nonlinearity is introduced through the Relu activation function, and finally the initial channel number is restored through 1*1 convolution, and the output is L(x). L(x) has the same shape as the input feature map, so it can better retain and highlight the subtle details in low-level features. The output of the local channel attention L(X) can be represented as:

[0102]

[0103] where X represents the input feature map, PWConv(·) represents the point-wise convolution operation, and δ represents the Relu activation function.

[0104] In the global path, the feature map is first reduced to C*1*1 through adaptive average pooling operation, and then a series of convolution layers, batch normalization layers and Relu activation functions are used to extract global features. The output G(X) of the global channel attention can be represented as:

[0105]

[0106] As an embodiment,

[0107] The encoder comprises a first ENAF module, a second ENAF module, a third ENAF module and a fourth ENAF module connected in sequence;

[0108] The decoder comprises a first DNAF module, a second DNAF module, a third DNAF module and a fourth DNAF module connected in sequence;

[0109] The first ENAF module is connected with the first DNAF module;

[0110] The second ENAF module is connected with the second DNAF module;

[0111] The third ENAF module, the first double-branch module and the third DNAF module are connected in sequence;

[0112] The fourth ENAF module, the second double-branch module and the fourth DNAF module are connected in sequence;

[0113] The fourth ENAF module comprises 28 NAF modules, and the first ENAF module, the second ENAF module, the third ENAF module, the first DNAF module, the second DNAF module, the third DNAF module and the fourth DNAF module each comprise one NAF module.

[0114] As an embodiment, the GoPro dataset is selected as the training set, and the HIDE dataset and the GoPro dataset are selected as the validation set. The GoPro dataset contains 3214 pairs of blurred and clear image pairs, of which 2103 pairs are used for training and 1111 pairs are used for testing. The model trained on the GoPro dataset will also be tested on the HIDE dataset.

[0115] Step S2 is specifically:

[0116] S21: input the training image into the first FEM module to obtain an initial feature image;

[0117] Specifically, the image obtained after image preprocessing in the training stage of the GoPro dataset is converted into a feature embedding image through a shallow feature extraction module (FEM). The core of the shallow feature extraction module is a convolution kernel of 3x3. The input image is preliminarily locally feature-extracted through convolution operation, and the output channel number of the feature map is adjusted. The adjusted feature map is used as the input of the encoder. This step ensures that the input image can be effectively processed by the subsequent modules.

[0118] S22: input the initial feature image into the first ENAF module to obtain a first feature image; input the first feature image into the second ENAF module after down-sampling operation to obtain a second feature image; input the second feature image into the third ENAF module after down-sampling operation to obtain a third feature image; and input the third feature image into the fourth ENAF module after down-sampling operation to obtain a fourth feature image;

[0119] Specifically, the feature extraction of the encoder is divided into four stages, each of which contains a different number of NAF modules and a down-sampling operation. The NAF module has the ability to capture local details, and the down-sampling operation is realized through a convolution layer, which is used to reduce the spatial size of the feature map and increase the number of channels. The module first loads the feature embedding image processed in step 1, initializes the channel number to 64, and then traverses the number of NAF modules in each stage. Except for the bottleneck layer which has 28 NAF modules, the other stages only have one NAF module. Then, a down-sampling layer is added to reduce the spatial size of the feature map and increase the number of channels, so that higher-level features are gradually extracted. Finally, the channel number is updated to prepare for the encoding layer of the next stage. The first three stages of the encoder only contain one nonlinear feature extraction module, which is used to quickly perform preliminary feature extraction and down-sampling on a larger feature map, while reducing the amount of calculation and memory consumption. In the last stage (bottleneck layer), a larger number of NAF modules are used to perform deeper feature extraction on a smaller spatial size, so as to capture more complex and abstract features. The data flow of the entire encoder is as followsFigure 7 as shown;

[0120] S23: input the fourth feature image into the second double-branch module to extract features in the spatial domain and the frequency domain, obtain a second fusion feature image, and input the second fusion feature image into the fourth DNAF module to obtain a fourth output image;

[0121] S24: input the third feature image into the first double-branch module to extract features in the spatial domain and the frequency domain, obtain a first fusion feature image, and input the first fusion feature image and the fourth output image after the up-sampling operation into the third DNAF module to obtain a third output image;

[0122] Specifically, the double branches extract features in the spatial domain and the frequency domain respectively, and the parallel double-branch structure is located at the bottleneck layer and the sub-high layer of the network. The double-branch structure has a spatial domain branch and a frequency domain branch, which process feature maps from the encoder respectively. The wavelet transform of the frequency domain branch can decompose and reconstruct these frequency components, helping the network to better capture and restore details, and the optimized frequency domain perception module can better adapt to the characteristics of motion blur. The sub-high layer is a layer close to the bottleneck layer in the network, and the spatial size of the feature map is slightly larger, but the number of channels is smaller. The double-branch module at this layer can extract and fuse features in a slightly larger spatial size, supplementing the feature representation of the bottleneck layer. The bottleneck layer is the deepest layer in the network, and the spatial size of the feature map is the smallest, but the number of channels is the largest. The double-branch module at this layer can extract and fuse features in a deep level on the smallest spatial size. By introducing the double-branch module at the bottleneck layer and the sub-high layer, the network can extract and fuse features at different scales, enhancing the richness and diversity of feature representation, and helping to improve the expressiveness and generalization ability of the model.

[0123] The obtaining process of the first fusion feature image and the second fusion feature image is the same;

[0124] The obtaining process of the first fusion feature image is specifically as follows:

[0125] The spatial domain feature image of the third feature image is extracted through the NAF module, the frequency domain feature image of the third feature image is extracted through the LFA module, and the spatial domain feature image and the frequency domain feature image are fused through the D-FM module to obtain the first fusion feature image.

[0126] Further,

[0127] The obtaining process of the frequency domain feature image of the third feature image is specifically as follows:

[0128] The third feature image is subjected to two-dimensional discrete wavelet transform through the wavelet decomposition module to obtain a low-frequency sub-band F LL , a high-frequency sub-band F LH , and a high-frequency sub-band F HLand high frequency subband F HH ;

[0129] Specifically, these frequency subbands can be represented as:

[0130] {F LL ,F LH ,F HL ,F HH}=DWT(F 输入 ) (3)

[0131] where DWT(·) denotes a two-dimensional discrete wavelet transform, F LL ,F LH ,F HL ,F HH represent the features of the four different frequency subbands respectively. In this way, the network can process high frequency and low frequency information separately.

[0132] The low frequency subband F LL is input into the convolutional network to obtain an enhanced low frequency subband F The high frequency subband F LH , the high frequency subband F HL , and the high frequency subband F HH are input into the high frequency feature enhancement module to obtain an enhanced high frequency subband F an enhanced high frequency subband F , and an enhanced high frequency subband F

[0133] Specifically, in order to effectively fuse and enhance these frequency features, the low frequency subband F LL is embedded into convolutional networks with 1x1 and 7x7 convolutional kernels to extract global features, and at the same time, the remaining three high frequency bands are enhanced by the high frequency feature enhancement module (FMB) to extract local details.

[0134]

[0135] where H conv×1×7 (·) denotes a convolutional network with 1x1 and 7x7 convolutional kernels, H FMB denotes a feature enhancement module, and denote the enhanced high frequency components.

[0136] The enhanced low frequency subband F , the enhanced high frequency subband F , the enhanced high frequency subband F , and the enhanced high frequency subband F are input into the wavelet fusion module for two-dimensional discrete wavelet inverse transform to obtain an inverse transform feature image F′ out ;

[0137] inverse transform the feature image F′ out The input feature distillation module obtains a distilled feature image, superimposes the distilled feature image and the third feature image, and obtains a frequency domain feature image F of the third feature image. out .

[0138] Specifically, the output four frequency bands are subjected to wavelet inverse transform to obtain a preliminarily processed feature map. Finally, the feature map is further enhanced by an efficient separable distillation module (ESDB) and added to the branch input, thereby obtaining a final frequency domain branch output, which can be expressed as follows:

[0139]

[0140] F out = H ESDB (F′ out )+ F 输入 (9)

[0141] where IDWT(·) represents a two-dimensional discrete wavelet inverse transform, H ESDB represents a feature distillation module, F′ out represents the result of inverse transform of four components, F out represents the output of the final frequency domain branch.

[0142] Further:

[0143] The branch-driven fusion mechanism (D-FM) performs weighted fusion on different branch information. In order to effectively fuse the processed double-branch information, the present application avoids directly adding the double-branch information, but instead uses a mechanism that can compare and weightedly fuse two paths of information.

[0144] S25: input the second feature image and the third output image subjected to the upsampling operation into the second DNAF module to obtain a second output image; input the first feature image and the second output image subjected to the upsampling operation into the first DNAF module to obtain a first output image;

[0145] Specifically, the decoder mainly acts on the multi-scale feature map, and performs upsampling and deblurring processing. Similar to the encoder, the decoder also has four stages, namely the lowest layer feature map processing, the intermediate layer feature map processing, the second highest layer feature map processing and the highest layer feature map processing, each stage containing a NAF module and an upsampling operation. Among them, the highest layer (bottleneck layer) and the second highest layer process the low-resolution feature map output by the double-branch. The intermediate layer stage directly fuses the output corresponding to the encoding layer, and then further extracts features. The lowest layer stage performs upsampling and resolution improvement operation on the output feature map of the second highest layer stage. The NAF module in the decoder gradually recovers the detail information of the feature map. The upsampling layer (convolution layer and pixel rearrangement) is to gradually recover the spatial size of the feature map while reducing the number of channels. The entire decoder gradually recovers the spatial size and detail information of the feature map through multiple decoding blocks and upsampling layers, and finally generates an output with the same size as the input image.

[0146] S26: input the first output image into the second FEM module to obtain an initial deblurred image; superimpose the training image and the initial deblurred image to obtain a final deblurred image;

[0147] Specifically, the original feature embedding image is fused with the deblurred inference image by using the jump connection to obtain the final deblurred effect image of the model. Here, the jump connection is used to retain the highest resolution information and alleviate gradient disappearance.

[0148] S27: calculate the loss value through the final deblurred image, and adjust the parameters of the deblurring network;

[0149] S28: repeat steps S21-S27 until the loss value is less than a preset value, and obtain a trained deblurring network.

[0150] S3: perform image deblurring operation through the trained deblurring network.

[0151] As an embodiment:

[0152] The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as evaluation indexes. In the embodiment, 16 methods are selected for comparison with the proposed method, and the selected methods are DeblurGAN-v2 (see the paper: Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better), MRDNet (see the paper: Image deblurring method based on self-attention and residual wavelet transform), DBGAN (see the paper: Deblurring by realistic blurring), MIMO-UNet (see the paper: Rethinking coarse-to-fine approach in single image deblurring), BANet (see the paper: Banet: a blur-aware attention network for dynamic scene deblurring), SPAIR (see the paper: Spatially-adaptive image restoration using distortion-guided networks), SRN (see the paper: Scale-recurrent network for deep image deblurring), SDWNet (see the paper: Sdwnet: A straight dilated network with wavelet transformation for image deblurring), SimpleNet (see the paper: Perceptual variousness motion deblurring with light global context refinement), S2SVR (see the paper: Unsupervised flow-aligned sequence-to-sequence learning for video restoration), the method proposed by Suin et al. (see the paper: Spatially-attentive patch-hierarchical network for adaptive motion deblurring), MSSNet-small (see the paper: Mssnet: Multi-scale-stage network for image restoration), and so on.MPRNet (see the paper: Multi-stage progressive image restoration), DMPHN (see the paper: Deep stacked hierarchical multi-patch network for image deblurring), the method proposed by Jiang et al. (see the paper: Image blind motion deblurring method with longitudinal channel and wavelet dynamic convolution), and MLWNet-width32 (see the paper: Efficient multi-scale network with learnable discrete wavelet transform for blind motion deblurring), and the test results on the GoPro and HIDE datasets are shown in Table 1, and Ours represents the method of the application.

[0153] Table 1 Comparison of indicators on GoPro and HIDE datasets

[0154]

[0155]

[0156] As can be seen from Table 1, on the GoPro and HIDE datasets, in terms of the comprehensive comparison of the two indicators of PSNR and SSIM, the method of the application can significantly improve the PSNR while effectively maintaining a high SSIM value. Compared with the advanced deblurring method MIMO-Unet, the method of the application improves the PSNR by 1.11 and the SSIM by 0.035 on the GoPro dataset, and improves the PSNR by 1.08 and the SSIM by 0.011 on the HIDE dataset; compared with the latest method MLWNet-width32, the method of the application improves the PSNR by 0.13 and the SSIM by 0.026 on the GoPro dataset. The experimental results show that the method of the application can effectively improve the image deblurring effect.

[0157] See Figure 8 , Figure 8 is a hardware device working schematic diagram of an embodiment of the application, and the hardware device specifically comprises: a dual-branch image deblurring device 401 based on frequency domain perception, a processor 402, and a storage medium 403.

[0158] An image deblurring device 401 based on frequency domain perception of a double branch: the image deblurring device 401 based on frequency domain perception of a double branch realizes the image deblurring method based on frequency domain perception of a double branch.

[0159] A processor 402: the processor 402 loads and executes instructions and data in the storage medium 403 for realizing the image deblurring method based on frequency domain perception of a double branch.

[0160] A storage medium 403: the storage medium 403 stores instructions and data; the storage medium 403 is used for realizing the image deblurring method based on frequency domain perception of a double branch.

[0161] It should be noted that in this paper, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0162] The above-mentioned serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments. In the unit claims of several devices, several of these devices can be embodied by the same hardware item. The use of the words first, second, and third does not represent any order, and these words can be interpreted as identification.

[0163] The above is only the preferred embodiment of the application, and does not limit the patent scope of the application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the application.

Claims

1. A dual-branch image deblurring method based on frequency domain perception, characterized in that, The method comprises the steps of: S1: obtaining a sample image set, performing image preprocessing on the sample image set, and obtaining a training image set; S2: constructing a deblurring network, training the deblurring network through the training image set, and obtaining a trained deblurring network; S3: performing deblurring operation on the image through the trained deblurring network; The deblurring network comprises a first FEM module, an encoder, a double-branch structure network, a decoder and a second FEM module; The first FEM module is connected with the encoder, the encoder is connected with the decoder, and the decoder is connected with the second FEM module; The encoder, the double-branch structure network and the decoder are connected in sequence; The double-branch structure network comprises a first double-branch module and a second double-branch module in parallel; The first double-branch module and the second double-branch module are of the same structure; The first double-branch module comprises a NAF module, an LFA module and a D-FM module; The encoder is connected with the NAF module and the LFA module; The NAF module and the LFA module are connected with the D-FM module; The D-FM module is connected with the decoder; The LFA module comprises a wavelet decomposition module, a convolution network, a high-frequency feature enhancement module, a wavelet fusion module and a feature distillation module; The encoder is connected with the wavelet decomposition module; The wavelet decomposition module is connected with the convolution network and the high-frequency feature enhancement module; The convolution network and the high-frequency feature enhancement module are connected with the wavelet fusion module; The wavelet fusion module, the feature distillation module and the D-FM module are connected in sequence.

2. The dual-branch image deblurring method based on frequency domain perception according to claim 1, characterized in that, Step S1 is specifically: S11: loading a sample image, performing cutting operation, flipping operation, rotating operation and format conversion operation on the sample image in sequence, and obtaining a training image; S12: repeating step S11 until all sample images are traversed, and obtaining a training image set.

3. The double-branch image deblurring method based on frequency domain perception according to claim 1, wherein: The encoder comprises a first ENAF module, a second ENAF module, a third ENAF module and a fourth ENAF module connected in sequence; The decoder comprises a first DNAF module, a second DNAF module, a third DNAF module and a fourth DNAF module connected in sequence; The first ENAF module is connected with the first DNAF module; The second ENAF module is connected with the second DNAF module; The third ENAF module, the first double-branch module and the third DNAF module are connected in sequence; The fourth ENAF module, the second double-branch module and the fourth DNAF module are connected in sequence; The fourth ENAF module comprises 28 NAF modules, the first ENAF module, the second ENAF module, the third ENAF module, the first DNAF module, the second DNAF module, the third DNAF module and the fourth DNAF module each comprise one NAF module.

4. The dual-branch image deblurring method based on frequency domain perception according to claim 3, characterized in that, Step S2 is specifically: S21: inputting the training image into the first FEM module to obtain an initial feature image; S22: input the initial feature image into the first ENAF module to obtain a first feature image; input the first feature image into the second ENAF module after down-sampling operation to obtain a second feature image; input the second feature image into the third ENAF module after down-sampling operation to obtain a third feature image; and input the third feature image into the fourth ENAF module after down-sampling operation to obtain a fourth feature image; S23: input the fourth feature image into the second double-branch module to extract features in the spatial domain and the frequency domain, and obtain a second fusion feature image; and input the second fusion feature image into the fourth DNAF module to obtain a fourth output image; S24: input the third feature image into the first double-branch module to extract features in the spatial domain and the frequency domain, and obtain a first fusion feature image; and input the first fusion feature image and the fourth output image after up-sampling operation into the third DNAF module to obtain a third output image; S25: input the second feature image and the third output image after up-sampling operation into the second DNAF module to obtain a second output image; and input the first feature image and the second output image after up-sampling operation into the first DNAF module to obtain a first output image; S26: input the first output image into the second FEM module to obtain an initial deblurring image; S27: calculate a loss value through the final deblurring image, and adjust parameters of the deblurring network; S28: repeat steps S21-S27 until the loss value is less than a preset value, and obtain a trained deblurring network.

5. The dual-branch image deblurring method based on frequency domain perception according to claim 4, wherein: the first fusion feature image and the second fusion feature image are obtained in the same way; the first fusion feature image is obtained in the following way: extract a spatial domain feature image of the third feature image through the NAF module, extract a frequency domain feature image of the third feature image through the LFA module, and fuse the spatial domain feature image and the frequency domain feature image through the D-FM module to obtain the first fusion feature image.

6. The dual-branch image deblurring method based on frequency domain perception according to claim 5, wherein: the frequency domain feature image of the third feature image is obtained in the following way: The storage medium stores instructions and data for implementing the dual-branch image deblurring method based on frequency domain perception according to any one of claims 1-6. The third feature image is subjected to two-dimensional discrete wavelet transform by a wavelet decomposition module to obtain low-frequency subbands , high-frequency subbands , high-frequency subbands , and high-frequency subbands ; inputting the low-frequency sub-band , obtaining an enhanced low-frequency sub-band ; inputting the high-frequency sub-band , obtaining an enhanced high-frequency sub-band , and obtaining an enhanced high-frequency sub-band ; inputting the high-frequency sub-band , obtaining an enhanced high-frequency sub-band , and obtaining an enhanced high-frequency sub-band ; the enhanced low-frequency subband , the enhanced high-frequency subband , the enhanced high-frequency subband , and the enhanced high-frequency subband input into a wavelet fusion module for two-dimensional discrete wavelet inverse transform to obtain an inverse transform feature image ; inverse transform the feature image input a feature distillation module to obtain a distilled feature image, superimpose the distilled feature image with the third feature image to obtain a frequency domain feature image of the third feature image .

7. A storage medium characterized by: including:

8. A dual-branch image deblurring device based on frequency domain perception, characterized in that: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the dual-branch image deblurring method based on frequency domain perception according to any one of claims 1-6. ​

Images