Motion deblurring method and device based on Fourier frequency domain attention mechanism
By introducing the Fourier frequency domain attention mechanism and the strip Fourier transform module in the image defuzzing method, focusing on the high-frequency and low-frequency components of the image, the problem of poor image defuzzing effect in the prior art is solved, and the realization of the realization of the realization of the image is achieved by focusing on the high-frequency and low-frequency components of the image. Clearer image recovery and more efficient computing performance.
Patent Information
- Application Number
- CN202411171672.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Existing image defuzzing methods such as CNN prefer to process low-frequency information, and cannot fully utilize the information in the blurred image, resulting in poor recovery effect.
Using the motion defuzzing method based on the Fourier frequency domain attention mechanism, the high-frequency and low-frequency components of the image are focused on the spatial domain and the temporal domain by introducing the semi-instified frequency domain channel attention module and the strip Fourier transform module. Information, realize the full use of image information.
The image debuffering effect is improved, the recovered image is clearer, and the high-frequency and low-frequency information in the image is effectively utilized, which improves the debuffering performance and reduces the computational complexity.
Smart Images

Figure CN118691503B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a motion deblurring method and device based on Fourier frequency domain attention mechanism. Background Art
[0002] Images are an important way for people to obtain information. However, in the imaging process, the images obtained are often blurred due to unfavorable factors such as camera shake, movement of objects in the scene, and camera defocus, which greatly hinders the normal use and subsequent processing of the images.
[0003] Image motion deblurring is a basic image restoration problem in image processing and computer vision. Its goal is to restore a blurred image caused by motion factors such as camera shake and object motion to a clear image. Most existing image deblurring methods usually use convolutional neural networks (CNNs) to restore the image by inputting the blurred image into CNN and outputting a clear image from CNN.
[0004] However, since CNN prefers to process low-frequency information in blurred images, it will not be able to fully utilize the information contained in the blurred images, resulting in poor restored images. Summary of the invention
[0005] The present invention provides a motion deblurring method and device based on Fourier frequency domain attention mechanism, so as to solve the defects existing in the prior art.
[0006] The present invention provides a motion deblurring method based on Fourier frequency domain attention mechanism, comprising:
[0007] Get the image to be processed;
[0008] Inputting the image to be processed into a motion deblurring model to obtain a deblurred image output by the motion deblurring model;
[0009] The motion deblurring model includes a plurality of scale encoding modules and a plurality of scale decoding modules connected in sequence; each encoding module and each decoding module includes a semi-instantiated frequency domain channel attention module;
[0010] Each semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, wherein the semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain a processing result; and the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain a calculation result.
[0011] According to a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention, each encoding module and each decoding module further includes a strip Fourier transform module;
[0012] For any module except the last decoding module in each encoding module and each decoding module, an input end of the strip Fourier transform module in any module is connected to an output end of the semi-instantiated frequency domain channel attention module in any module;
[0013] An output end of the strip Fourier transform module in any of the modules is connected to an input end of the semi-instantiated frequency domain channel attention module in the next module of any of the modules;
[0014] The strip Fourier transform module in any of the modules is used to perform strip-shaped segmentation operations, Fourier transform operations, and inverse Fourier transform operations on the input tensor in the horizontal direction and the vertical direction respectively.
[0015] According to a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention, the strip Fourier transform modules all include a first layer normalization module, a first convolution layer, a segmentation module, a Fourier transform module, a splicing module, an activation module and a first addition module;
[0016] The input tensor of the strip Fourier transform module passes through the first normalization module and the first convolution layer in sequence, and then passes through the segmentation module to obtain a first tensor and a second tensor;
[0017] The first tensor and the second tensor are respectively subjected to the Fourier transform module, and are sequentially subjected to Fourier transform operation, attention weighting operation and inverse Fourier transform to obtain a third tensor and a fourth tensor;
[0018] The third tensor and the fourth tensor are sequentially passed through the splicing module and the activation module to obtain an activation result;
[0019] The activation result and the input tensor of the strip Fourier transform module are added by the first addition module to obtain the output tensor of the strip Fourier transform module.
[0020] According to a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention, the multiple scales include at least three;
[0021] The encoding modules of the multiple scales are arranged in an order of gradually decreasing scales, and the decoding modules of the multiple scales are arranged in an order of gradually increasing scales;
[0022] The output end of the strip Fourier transform module in the first encoding module is connected to the input end of the semi-instantiated frequency domain channel attention module in the last decoding module;
[0023] The output end of the strip Fourier transform module in the next-to-last encoding module is connected to the input end of the semi-instantiated frequency-domain channel attention module in the next-to-last decoding module;
[0024] The output of the strip Fourier transform module in the third encoding module is connected to the input of the semi-instantiated frequency-domain channel attention module in the third-to-last decoding module.
[0025] According to a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention, the semi-instantiated module includes a second normalization module, a second convolutional layer, a third convolutional layer, an instantiation layer and a multiplication layer connected in sequence;
[0026] The input tensor of the semi-instantiated module is sequentially passed through the second normalization module, the second convolutional layer, and the third convolutional layer to obtain a fifth tensor;
[0027] The sixth tensor obtained by dividing the fifth tensor according to the channels is passed through the instantiation layer to obtain an eighth tensor;
[0028] The seventh tensor obtained by dividing the fifth tensor by channels and the eighth tensor are processed through the multiplication layer to obtain the processing result.
[0029] According to a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention, the frequency domain channel attention module includes a channel division module, a frequency domain attention module, a fourth convolution layer, a fifth convolution layer and a second addition module;
[0030] The processing result is passed through the channel division module to obtain multiple channel components;
[0031] The multiple channel components are cascaded after being respectively calculated by the frequency domain attention module to obtain a compressed vector;
[0032] The compressed vector is multiplied by the plurality of channel components after passing through the fourth convolutional layer to obtain a calculation result;
[0033] After the operation result passes through the fifth convolutional layer, the operation result is added to the processing result through the second addition module to obtain the calculation result.
[0034] According to a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention, the motion deblurring model further includes an input convolution layer, an output convolution layer and a fusion module;
[0035] The input end of the input convolution layer is used to receive the image to be processed, and the output end of the input convolution layer is connected to the first coding module;
[0036] The input end of the output convolution layer is connected to the output end of the last decoding module, and the output end of the output convolution layer and the input end of the input convolution layer are respectively connected to the fusion module;
[0037] The fusion module is used to fuse the image to be processed and the output result of the output convolution layer to obtain the deblurred image.
[0038] The present invention also provides a motion deblurring device based on Fourier frequency domain attention mechanism, comprising:
[0039] An image acquisition module, used for acquiring an image to be processed;
[0040] A deblurring module, used for inputting the image to be processed into a motion deblurring model to obtain a deblurred image output by the motion deblurring model;
[0041] The motion deblurring model includes a plurality of scale encoding modules and a plurality of scale decoding modules connected in sequence; each encoding module and each decoding module includes a semi-instantiated frequency domain channel attention module;
[0042] Each semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, wherein the semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain a processing result; and the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain a calculation result.
[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the motion deblurring method based on the Fourier frequency domain attention mechanism as described above is implemented.
[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the motion deblurring method based on the Fourier frequency domain attention mechanism as described above.
[0045] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described motion deblurring methods based on the Fourier frequency domain attention mechanism.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] The present invention provides a motion deblurring method and device based on Fourier frequency domain attention mechanism. The motion deblurring model adopted by the method introduces a semi-instantiated frequency domain channel attention module, which can simultaneously pay attention to the information of the high-frequency and low-frequency components of the image to be processed, and can make full use of the information contained in the image to be processed, making the restored image effect clearer. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without paying any creative work.
[0049] Figure 1 It is a flowchart of a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention;
[0050] Figure 2 It is a structural schematic diagram of the SFTM module in the motion deblurring method based on the Fourier frequency domain attention mechanism provided by the present invention;
[0051] Figure 3 It is a structural schematic diagram of a motion deblurring model in a motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention;
[0052] Figure 4 It is a structural schematic diagram of a semi-instantiated frequency domain channel attention module in the motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention;
[0053] Figure 5 It is a structural schematic diagram of the core module in the motion deblurring method based on Fourier frequency domain attention mechanism provided by the present invention;
[0054] Figure 6 It is one of the schematic diagrams of an image to be processed in the motion deblurring method based on the Fourier frequency domain attention mechanism provided by the present invention;
[0055] Figure 7 The motion deblurring model provided by the present invention is Figure 6 A schematic diagram of a deblurred image obtained by processing;
[0056] Figure 8 It is a structural schematic diagram of a motion deblurring device based on Fourier frequency domain attention mechanism provided by the present invention;
[0057] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] Since the existing CNN is more inclined to process low-frequency information in blurred images, it is unable to fully utilize the information contained in the blurred images, which results in poor restored image effects. To this end, an embodiment of the present invention provides a motion deblurring method based on a Fourier frequency domain attention mechanism.
[0060] Figure 1 Schematic diagram of a flow chart of a motion deblurring method based on Fourier frequency domain attention mechanism provided in an embodiment of the present invention. Figure 1 As shown, the method includes:
[0061] S1, obtaining the image to be processed;
[0062] S2, inputting the image to be processed into a motion deblurring model to obtain a deblurred image output by the motion deblurring model;
[0063] The motion deblurring model includes a plurality of scale encoding modules and a plurality of scale decoding modules connected in sequence; each encoding module and each decoding module includes a semi-instantiated frequency domain channel attention module;
[0064] Each semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, wherein the semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain a processing result; and the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain a calculation result.
[0065] Specifically, the motion deblurring method based on the Fourier frequency domain attention mechanism provided in the embodiment of the present invention is executed by a processor, the processor can be a computer, the computer can be a local computer or a cloud computer, the local computer can be a computer, a tablet, etc., which is not specifically limited here.
[0066] First, execute step S1 to obtain the image to be processed; the image to be processed refers to a blurred image with motion blur that needs to be deblurred, which can be a large-size substrate image or an image containing a target object such as a face or a vehicle in the monitoring field, which is not specifically limited here.
[0067] Then, step S2 is performed to input the image to be processed into the motion deblurring model to obtain a deblurred image output by the motion deblurring model.
[0068] The motion deblur model may include encoding modules of multiple scales and decoding modules of multiple scales connected in sequence, and each encoding module and each decoding module include a semi-instantiated frequency domain channel attention module. For example, the motion deblur model may include encoding modules of four scales and decoding modules of four scales connected in sequence, and the encoding modules of each scale may be arranged in a decreasing order of scale, and the decoding modules of each scale may be arranged in a increasing order of scale. The encoding modules of the same scale may include one or more, and the decoding modules of the same scale may include one or more.
[0069] Each encoding module and each decoding module includes a Half Instancenormalization Frequency Channel Attention (HIFCA) module. Each HIFCA module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence. The semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain the processing result; the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain the calculation result.
[0070] It can be understood that the input tensor here is the input of the HIFCA module, and semi-instantiation processing refers to performing an instance normalization (IN) operation on half of the input tensor, while maintaining consistency (ID) on the other side, that is, no operation is performed.
[0071] The traditional channel attention mechanism (CAM) can be expressed as formula (1):
[0072] (1)
[0073] in, is the input tensor of the channel attention mechanism, sigmoid is the activation function, fc is the fully connected layer, and pool is the pooling layer. It is the output tensor of the channel attention mechanism.
[0074] The pooling layer in formula (1) is used to perform a global average pooling (GAP) operation. The GAP operation is a special case of the two-dimensional discrete cosine transform (DCT). The result of the GAP operation is proportional to the lowest frequency component of the two-dimensional DCT. Based on this theoretical analysis, using the channel attention mechanism means that only the lowest frequency information is retained, while all components of other frequencies are discarded. The loss of high-frequency components represents blurring. Therefore, in the embodiment of the present invention, frequency domain attention calculation is performed on each channel.
[0075] The introduction of the HIFCA module can simultaneously focus on the information of the high-frequency and low-frequency components of the image to be processed. Moreover, each encoding module and each decoding module contains a HIFCA module, which can realize the stacking of multiple HIFCA modules, and can more effectively realize the extraction and application of the high-frequency and low-frequency components of the image to be processed.
[0076] It is understandable that the motion deblurring model used in the embodiment of the present invention can be obtained by training the initial model using image samples with motion blur and clear images corresponding to the image samples. The image samples can be acquired by an image acquisition device with a first shutter rate, and the clear images corresponding to the image samples can be acquired by an image acquisition device with a second shutter rate, and the first shutter rate is less than the second shutter rate.
[0077] When training the initial model, the image samples can be first input into the initial model to obtain the output results of the initial model, and then the output results and the clear image are used to calculate the loss function. According to the loss function, the structural parameters of the initial model are iteratively updated until the loss function converges to obtain the motion deblurring model.
[0078] The motion deblurring method based on Fourier frequency domain attention mechanism provided in the embodiment of the present invention first obtains the image to be processed; then the image to be processed is input into the motion deblurring model to obtain the deblurred image output by the motion deblurring model; wherein the motion deblurring model includes encoding modules of multiple scales connected in sequence and decoding modules of multiple scales; each encoding module and each decoding module includes a semi-instantiated frequency domain channel attention module; each semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, the semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain the processing result; the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain the calculation result. The semi-instantiated frequency domain channel attention module is introduced into the motion deblurring model adopted by the method, which can simultaneously pay attention to the information of the high-frequency and low-frequency components of the image to be processed, and can make full use of the information contained in the image to be processed, so that the restored image effect is clearer.
[0079] Since the traditional Fourier frequency domain transform can transform the image from the time domain to the frequency domain, it does not make full use of the horizontal and vertical information of the spatiotemporal domain. Although it can adapt to most image restoration networks, it is not ideal for restoring motion blur caused by horizontal or vertical jitter. For this reason, the Strip Fourier transform (SFTM) module is introduced in the embodiment of the present invention.
[0080] Based on the above embodiment, each encoding module and each decoding module further includes a strip Fourier transform module;
[0081] For any module except the last decoding module in each encoding module and each decoding module, an input end of the strip Fourier transform module in any module is connected to an output end of the semi-instantiated frequency domain channel attention module in any module;
[0082] An output end of the strip Fourier transform module in any of the modules is connected to an input end of the semi-instantiated frequency domain channel attention module in the next module of any of the modules;
[0083] The strip Fourier transform module in any of the modules is used to divide the input tensor into strips in the horizontal direction and the vertical direction respectively.
[0084] Specifically, each encoding module and each decoding module also includes a SFTM module.
[0085] For any module except the last decoding module in each encoding module and each decoding module, the input end of the SFTM module in each module is connected to the output end of the HIFCA module in the module, and the output end of the SFTM module in the module is connected to the input end of the HIFCA module in the next module of the module, that is, the HIFCA module in each module is connected to the SFTM module in sequence, and the output end of the SFTM module of the previous module is connected to the input end of the HIFCA module in the next module, so as to realize the cascading of adjacent modules.
[0086] The SFTM module in each module can perform strip-shaped splitting operations, Fourier transform operations, and inverse Fourier transform operations on the input tensor, that is, the calculation result output by the HIFCA module in the encoding module or decoding module, in the horizontal direction and the vertical direction respectively.
[0087] In the embodiment of the present invention, by introducing the SFTM module, blurs with different directions and amplitudes in the image to be processed can be captured, the problem of traditional Fourier transform ignoring spatial information can be solved, and the effect of deblurring the image can be improved. Moreover, stacking the HIFCA module with the SFTM module can effectively combine the information in the spatial domain and the temporal domain in the image to be processed, thereby improving the deblurring performance and reducing the computational complexity, providing a basis for more efficient and accurate image processing.
[0088] It can be understood that the motion deblurring model including the HIFCA module and the SFTM module can be regarded as a Fourier Transformer Frequency-domain Attention Network (FTFANet), which can effectively combine the information in the spatial domain and the frequency domain, thereby improving the deblurring performance and reducing the computational complexity.
[0089] Based on the above embodiments, Figure 2 As shown, the SFTM modules include a first normalization (Norm) module, a first convolution (Conv) layer, a segmentation module, a Fourier transform module, a concatenation (concat) module, an activation (Gelu) module and a first addition module.
[0090] The input tensor of the SFTM module passes through the first normalization module and the first convolution layer in sequence, and then passes through the segmentation module to obtain the first tensor and the second tensor.
[0091] The first tensor and the second tensor are respectively subjected to the Fourier transform module, and then subjected to the Fourier transform operation, the attention weighted operation and the inverse Fourier transform in sequence to obtain the third tensor and the fourth tensor.
[0092] The third tensor and the fourth tensor are sequentially passed through the concatenation module and the activation module to obtain the activation result.
[0093] The activation result and the input tensor of the SFTM module are added by the first addition module to obtain the output tensor of the SFTM module.
[0094] Specifically, the first convolutional layer can implement a 1×1 convolution operation to expand the number of channels of the input tensor of the SFTM module, thereby obtaining pixel-by-pixel cross-channel contextual information.
[0095] After the splitting module, we can get the first tensor with the same height, width and number of channels. and the second tensor The output of the first convolutional layer is split vertically into strips to obtain the first tensor , the output of the first convolutional layer is split into strips horizontally to obtain the second tensor , to generate multi-range bar features. As shown in formula (2):
[0096] (2)
[0097] in, is the strip segmentation operation implemented by the segmentation module, Conv1 is the convolution operation implemented by the first convolutional layer, is the layer normalization operation implemented by the first layer normalization module, and Y is the input tensor of the strip Fourier transform module.
[0098] After that, the first tensor and the second tensor are input into the Fourier transform module respectively. and the second tensor Perform Fourier transform row by row and column by column, and we have formula (3):
[0099] (3)
[0100] in, The first tensor The i-th component in the equation, w is the first tensor The width of For the second tensor The jth component in the matrix, h is the second tensor n is the height difference between adjacent first components or the width difference between adjacent second components in the strip-shaped slicing operation, Fourier transform implemented for the Fourier transform module.
[0101] After that, the learnable attention weight matrix is used to perform attention weighting operation to determine the required frequency information, that is, formula (4):
[0102] (4)
[0103] in, for The corresponding attention weight matrix, for The result of the attention weighted operation is for The corresponding attention weight matrix, for The result of the attention weighted operation.
[0104] Finally, the processed and Perform an inverse Fourier transform and reshape it back to its original shape to get the third tensor and the fourth tensor .
[0105] The third tensor and the fourth tensor Input to the splicing module, perform splicing operation to get the splicing result, and use the activation function The activation result is obtained by transformation, and the activation result and the input tensor of the SFTM module are added by the first addition module to obtain the output tensor of the SFTM module, that is,
[0106] (5)
[0107] in, For the splicing result, For splicing operation, is the activation function, To activate the result, It is the output tensor of the SFTM module.
[0108] Based on the above embodiments, Figure 3 As shown, the multiple scales include at least three, Figure 3 , it is shown that the encoding module and the decoding module each include 4 scales.
[0109] The encoding modules of each scale are arranged in the order of decreasing scale, and the decoding modules of each scale are arranged in the order of increasing scale;
[0110] The output end of the SFTM module in the first encoding module is connected to the input end of the HIFCA module in the last decoding module;
[0111] The output end of the SFTM module in the next-to-last bit encoding module is connected to the input end of the HIFCA module in the next-to-last bit decoding module;
[0112] The output end of the SFTM module in the third encoding module is connected to the input end of the HIFCA module in the third-to-last decoding module.
[0113] In the embodiment of the present invention, the features extracted by the motion deblurring model can be integrated through the interaction between the encoding module and the decoding module, so as to obtain a clearer deblurred image.
[0114] Based on the above embodiments, Figure 3 As shown, the motion deblurring model also includes an input convolution layer, an output convolution layer, and a fusion module;
[0115] The input end of the input convolution layer is used to receive the image to be processed, and the output end of the input convolution layer is connected to the first encoding module;
[0116] The input end of the output convolution layer is connected to the output end of the last decoding module, and the output end of the output convolution layer and the input end of the input convolution layer are connected to the fusion module respectively;
[0117] The fusion module is used to fuse the image to be processed and the output result of the output convolution layer to obtain a deblurred image.
[0118] Based on the above embodiments, Figure 4 As shown, the semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, and the semi-instantiated module includes a second normalization module, a second convolutional layer, a third convolutional layer, an instantiation layer, and a multiplication layer connected in sequence;
[0119] The input tensor of the semi-instantiated module passes through the second normalization module, the second convolutional layer, and the third convolutional layer in sequence to obtain the fifth tensor Fmid.
[0120] The fifth tensor Fmid can be divided into the sixth and seventh tensors by channel, and the sixth tensor is instantiated by the instantiation (IN) layer to obtain the eighth tensor. The seventh tensor can be passed through the consistency (ID) layer without processing, and multiplied with the eighth tensor by the multiplication layer to obtain the processing result.
[0121] Here, the instantiation operation can achieve normalization and recalibrate the features of the mean and variance of the sixth tensor, so that it is not affected by the batch size. At the same time, compared with batch normalization (Batch Normalization, BN), it can maintain more scale information and increase the stability of subsequent frequency domain attention.
[0122] Based on the above embodiments, Figure 4 As shown in , the frequency domain channel attention module includes a core module, a fifth convolutional layer, and a second addition module. Figure 5As shown, the core modules include a channel division module, a frequency domain attention module and a fourth convolutional layer.
[0123] The processing result of the semi-instantiated module is a tensor with C channels, H height and W width. , the processing result of the semi-instantiated module is obtained through the channel division module to obtain multiple channel components. Figure 5 As shown, it is used as the input tensor of the frequency domain channel attention module. After the channel division module, the multiple channel components obtained are The n channel components of For example, n channel components can be expressed as , ,and To be able to Divisible.
[0124] Each channel component is calculated by the frequency domain attention module, and the frequency domain attention is:
[0125] (6)
[0126]
[0127] in, express The component with height h and width w, They correspond to The two-dimensional (2D) index of the frequency components of To correspond to The two-dimensional DCT of yes The frequency components obtained by the frequency domain attention calculation of the frequency domain attention module, and After compression dimensional vector. for The corresponding DCT weights.
[0128] The frequency components corresponding to each channel component can be cascaded, that is, concatenated, to obtain a compressed vector:
[0129] (7)
[0130] in, is a compressed vector.
[0131] like Figure 5As shown, the n channel components are channel component 0, channel component 1, ..., channel component n-1, and the DCT bases corresponding to each channel component are DCT0, DCT1, ..., DCTn-1, respectively. Then, by performing dot multiplication between each channel component and the corresponding DCT base, the frequency components corresponding to each channel component are obtained, that is, , ,……, .
[0132] The frequency components corresponding to each channel component are cascaded to obtain the compressed vector , and then after passing through the fourth convolutional layer and Perform multiplication operation and get the result:
[0133] (8)
[0134] in, is the operation result, Convolution operation implemented for the fourth convolutional layer.
[0135] It is understandable that the calculation results is a tensor with a channel number of C, a height of H, and a width of W. In the embodiment of the present invention, the convolution kernel size of the fourth convolution layer may be 1×1, which can effectively reduce the amount of calculation.
[0136] After the operation result passes through the fifth convolution layer, it is combined with the input tensor of the semi-instantiated module , the second addition module performs the addition operation to obtain the calculation result, namely:
[0137] (9)
[0138] (10)
[0139] (11)
[0140] in, The layer normalization operation implemented by the second layer normalization module, It is the combined convolution operation implemented by the second and third convolutional layers. HIN is the combined operation of the instantiation layer, consistency layer, and multiplication layer. The convolution operation implemented for the fifth convolutional layer, is the calculation result.
[0141] In summary, the key parameters of the FTFANet provided in the embodiment of the present invention are the number of HIFCA modules and SFTM modules. Since both the HIFCA module and the SFTM module have only a few convolutional layers, blur perception can be improved by stacking multiple modules to better deblur.
[0142] In Table 1, four versions of FTFANet are shown: HIFCA-16, HIFCA-24, HIFCA-30, and HIFCA-34. The numbers represent the total number of HIFCA modules and SFTM modules stacked in FTFANet, with each module occupying half of the layers.
[0143] In the embodiment of the present invention, the peak signal-to-noise ratio (PSNR) is used as an indicator to measure image quality. As can be seen from Table 1, PSNR increases with the total number of HIFCA modules and SFTM modules stacked in FTFANet, but saturates after 30 layers. Therefore, we choose 30 layers to achieve an excellent balance between performance and efficiency.
[0144] Table 1 HIFCA layer comparison table
[0145]
[0146] Figure 6 is one of the images to be processed used in the embodiment of the present invention, Figure 7 The motion deblurring model provided in the embodiment of the present invention is used to Figure 6 The deblurred image is obtained by processing the image to be processed in .
[0147] pass Figure 6 and Figure 7 By comparison, it can be seen that the motion deblurring model adopted in the embodiment of the present invention has an obvious motion deblurring effect.
[0148] like Figure 8 As shown, based on the above embodiment, a motion deblurring device based on Fourier frequency domain attention mechanism is provided in an embodiment of the present invention, comprising:
[0149] An image acquisition module 11 is used to acquire an image to be processed;
[0150] A deblurring module 12, configured to input the image to be processed into a motion deblurring model to obtain a deblurred image output by the motion deblurring model;
[0151] The motion deblurring model includes a plurality of scale encoding modules and a plurality of scale decoding modules connected in sequence; each encoding module and each decoding module includes a semi-instantiated frequency domain channel attention module;
[0152] Each semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, wherein the semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain a processing result; and the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain a calculation result.
[0153] On the basis of the above-mentioned embodiment, in the motion deblurring device based on the Fourier frequency domain attention mechanism provided in the embodiment of the present invention, each encoding module and each decoding module further includes a strip Fourier transform module;
[0154] For any module except the last decoding module in each encoding module and each decoding module, an input end of the strip Fourier transform module in any module is connected to an output end of the semi-instantiated frequency domain channel attention module in any module;
[0155] An output end of the strip Fourier transform module in any of the modules is connected to an input end of the semi-instantiated frequency domain channel attention module in the next module of any of the modules;
[0156] The strip Fourier transform module in any of the modules is used to perform strip-shaped segmentation operations, Fourier transform operations, and inverse Fourier transform operations on the input tensor in the horizontal direction and the vertical direction respectively.
[0157] On the basis of the above embodiments, in the motion deblurring device based on the Fourier frequency domain attention mechanism provided in the embodiments of the present invention, the strip Fourier transform modules all include a first layer normalization module, a first convolution layer, a segmentation module, a Fourier transform module, a splicing module, an activation module and a first addition module;
[0158] The input tensor of the strip Fourier transform module passes through the first normalization module and the first convolution layer in sequence, and then passes through the segmentation module to obtain a first tensor and a second tensor;
[0159] The first tensor and the second tensor are respectively subjected to the Fourier transform module, and are sequentially subjected to Fourier transform operation, attention weighting operation and inverse Fourier transform to obtain a third tensor and a fourth tensor;
[0160] The third tensor and the fourth tensor are sequentially passed through the splicing module and the activation module to obtain an activation result;
[0161] The activation result and the input tensor of the strip Fourier transform module are added by the first addition module to obtain the output tensor of the strip Fourier transform module.
[0162] On the basis of the above-mentioned embodiment, in the motion deblurring device based on Fourier frequency domain attention mechanism provided in the embodiment of the present invention, the multiple scales include at least three;
[0163] The encoding modules of the multiple scales are arranged in an order of gradually decreasing scales, and the decoding modules of the multiple scales are arranged in an order of gradually increasing scales;
[0164] The output end of the strip Fourier transform module in the first encoding module is connected to the input end of the semi-instantiated frequency domain channel attention module in the last decoding module;
[0165] The output end of the strip Fourier transform module in the next-to-last encoding module is connected to the input end of the semi-instantiated frequency-domain channel attention module in the next-to-last decoding module;
[0166] The output of the strip Fourier transform module in the third encoding module is connected to the input of the semi-instantiated frequency-domain channel attention module in the third-to-last decoding module.
[0167] On the basis of the above embodiment, in the motion deblurring device based on the Fourier frequency domain attention mechanism provided in the embodiment of the present invention, the semi-instantiated module includes a second normalization module, a second convolutional layer, a third convolutional layer, an instantiation layer and a multiplication layer connected in sequence;
[0168] The input tensor of the semi-instantiated module is sequentially passed through the second normalization module, the second convolutional layer, and the third convolutional layer to obtain a fifth tensor;
[0169] The sixth tensor obtained by dividing the fifth tensor according to the channels is passed through the instantiation layer to obtain an eighth tensor;
[0170] The seventh tensor obtained by dividing the fifth tensor by channels and the eighth tensor are processed through the multiplication layer to obtain the processing result.
[0171] On the basis of the above embodiments, in the motion deblurring device based on the Fourier frequency domain attention mechanism provided in the embodiments of the present invention, the frequency domain channel attention module includes a channel division module, a frequency domain attention module, a fourth convolution layer, a fifth convolution layer and a second addition module;
[0172] The processing result is passed through the channel division module to obtain multiple channel components;
[0173] The multiple channel components are cascaded after being respectively calculated by the frequency domain attention module to obtain a compressed vector;
[0174] The compressed vector is multiplied by the plurality of channel components after passing through the fourth convolutional layer to obtain a calculation result;
[0175] After the operation result passes through the fifth convolutional layer, the operation result is added to the processing result through the second addition module to obtain the calculation result.
[0176] On the basis of the above-mentioned embodiment, in the motion deblurring device based on the Fourier frequency domain attention mechanism provided in the embodiment of the present invention, the motion deblurring model further includes an input convolution layer, an output convolution layer and a fusion module;
[0177] The input end of the input convolution layer is used to receive the image to be processed, and the output end of the input convolution layer is connected to the first coding module;
[0178] The input end of the output convolution layer is connected to the output end of the last decoding module, and the output end of the output convolution layer and the input end of the input convolution layer are respectively connected to the fusion module;
[0179] The fusion module is used to fuse the image to be processed and the output result of the output convolution layer to obtain the deblurred image.
[0180] Specifically, the functions of each module in the motion deblurring device based on Fourier frequency domain attention mechanism provided in the embodiment of the present invention correspond one-to-one to the operation flow of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further description will be given in the embodiment of the present invention.
[0181] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor (Processor) 110, a communication interface (Communications Interface) 120, a memory (Memory) 130 and a communication bus 140, wherein the processor 110, the communication interface 120, and the memory 130 communicate with each other through the communication bus 140. The processor 110 may call the logic instructions in the memory 130 to execute the motion deblurring method based on the Fourier frequency domain attention mechanism provided in the above embodiments.
[0182] In addition, the logic instructions in the above-mentioned memory 130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the motion deblurring method based on the Fourier frequency domain attention mechanism provided in the above-mentioned embodiments.
[0184] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the motion deblurring method based on the Fourier frequency domain attention mechanism provided in the above-mentioned embodiments.
[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A motion deblurring method based on Fourier frequency domain attention mechanism, It is characterized in that include: Get the image to be processed; Inputting the image to be processed into a motion deblurring model to obtain a deblurred image output by the motion deblurring model; The motion deblurring model includes a plurality of scale encoding modules and a plurality of scale decoding modules connected in sequence; each encoding module and each decoding module includes a semi-instantiated frequency domain channel attention module; Each semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, wherein the semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain a processing result; and the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain a calculation result; Each encoding module and each decoding module also includes a strip Fourier transform module; the motion deblurring model is a Fourier transform frequency domain attention network; For any module except the last decoding module in each encoding module and each decoding module, an input end of the strip Fourier transform module in any module is connected to an output end of the semi-instantiated frequency domain channel attention module in any module; An output end of the strip Fourier transform module in any of the modules is connected to an input end of the semi-instantiated frequency domain channel attention module in the next module of any of the modules; The strip Fourier transform module in any of the modules is used to perform strip-shaped segmentation operations, Fourier transform operations, and inverse Fourier transform operations on the input tensor in the horizontal direction and the vertical direction respectively.
2. The motion deblurring method based on Fourier frequency domain attention mechanism according to claim 1, It is characterized in that The strip Fourier transform modules all include a first normalization module, a first convolutional layer, a segmentation module, a Fourier transform module, a splicing module, an activation module and a first addition module; The input tensor of the strip Fourier transform module passes through the first normalization module and the first convolution layer in sequence, and then passes through the segmentation module to obtain a first tensor and a second tensor; The first tensor and the second tensor are respectively subjected to the Fourier transform module, and are sequentially subjected to Fourier transform operation, attention weighting operation and inverse Fourier transform to obtain a third tensor and a fourth tensor; The third tensor and the fourth tensor are sequentially passed through the splicing module and the activation module to obtain an activation result; The activation result and the input tensor of the strip Fourier transform module are added by the first addition module to obtain the output tensor of the strip Fourier transform module.
3. The motion deblurring method based on Fourier frequency domain attention mechanism according to claim 1, It is characterized in that The plurality of scales comprises at least three; The encoding modules of the multiple scales are arranged in an order of gradually decreasing scales, and the decoding modules of the multiple scales are arranged in an order of gradually increasing scales; The output end of the strip Fourier transform module in the first encoding module is connected to the input end of the semi-instantiated frequency domain channel attention module in the last decoding module; The output end of the strip Fourier transform module in the next-to-last encoding module is connected to the input end of the semi-instantiated frequency-domain channel attention module in the next-to-last decoding module; The output of the strip Fourier transform module in the third encoding module is connected to the input of the semi-instantiated frequency-domain channel attention module in the third-to-last decoding module.
4. The motion deblurring method based on Fourier frequency domain attention mechanism according to claim 1, It is characterized in that The semi-instantiated module includes a second normalization module, a second convolutional layer, a third convolutional layer, an instantiation layer, and a multiplication layer connected in sequence; The input tensor of the semi-instantiated module is sequentially passed through the second normalization module, the second convolutional layer, and the third convolutional layer to obtain a fifth tensor; The sixth tensor obtained by dividing the fifth tensor according to the channels is passed through the instantiation layer to obtain an eighth tensor; The seventh tensor obtained by dividing the fifth tensor by channels and the eighth tensor are processed through the multiplication layer to obtain the processing result.
5. The motion deblurring method based on Fourier frequency domain attention mechanism according to claim 1, It is characterized in that The frequency domain channel attention module includes a channel division module, a frequency domain attention module, a fourth convolutional layer, a fifth convolutional layer and a second addition module; The processing result is passed through the channel division module to obtain multiple channel components; The multiple channel components are cascaded after being respectively calculated by the frequency domain attention module to obtain a compressed vector; The compressed vector is multiplied by the plurality of channel components after passing through the fourth convolutional layer to obtain a calculation result; After the operation result passes through the fifth convolutional layer, the operation result is added to the processing result through the second addition module to obtain the calculation result.
6. The motion deblurring method based on Fourier frequency domain attention mechanism according to any one of claims 1 to 5, It is characterized in that The motion deblurring model also includes an input convolution layer, an output convolution layer and a fusion module; The input end of the input convolution layer is used to receive the image to be processed, and the output end of the input convolution layer is connected to the first coding module; The input end of the output convolution layer is connected to the output end of the last decoding module, and the output end of the output convolution layer and the input end of the input convolution layer are respectively connected to the fusion module; The fusion module is used to fuse the image to be processed and the output result of the output convolution layer to obtain the deblurred image.
7. A motion deblurring device based on Fourier frequency domain attention mechanism, It is characterized in that include: An image acquisition module, used for acquiring an image to be processed; A deblurring module, used for inputting the image to be processed into a motion deblurring model to obtain a deblurred image output by the motion deblurring model; The motion deblurring model includes a plurality of scale encoding modules and a plurality of scale decoding modules connected in sequence; each encoding module and each decoding module includes a semi-instantiated frequency domain channel attention module; Each semi-instantiated frequency domain channel attention module includes a semi-instantiated module and a frequency domain channel attention module connected in sequence, wherein the semi-instantiated module is used to perform semi-instantiated processing on the input tensor to obtain a processing result; and the frequency domain channel attention module is used to perform frequency domain attention calculation on the processing result by channel to obtain a calculation result; Each encoding module and each decoding module also includes a strip Fourier transform module; the motion deblurring model is a Fourier transform frequency domain attention network; For any module except the last decoding module in each encoding module and each decoding module, an input end of the strip Fourier transform module in any module is connected to an output end of the semi-instantiated frequency domain channel attention module in any module; An output end of the strip Fourier transform module in any of the modules is connected to an input end of the semi-instantiated frequency domain channel attention module in the next module of any of the modules; The strip Fourier transform module in any of the modules is used to perform strip-shaped segmentation operations, Fourier transform operations, and inverse Fourier transform operations on the input tensor in the horizontal direction and the vertical direction respectively.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the motion deblurring method based on the Fourier frequency domain attention mechanism as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the motion deblurring method based on the Fourier frequency domain attention mechanism as described in any one of claims 1 to 6 is implemented.
