Multi-directional deblurring method and device based on flipped deformable convolution

Abstract

The present invention relates to the field of computer vision technology, and provides a multi-directional deblurring method and device based on flipped deformable convolution. First, an image to be processed is obtained; then a flipped deformable network is applied to deblur the image to be processed to obtain a deblurred image of the image to be processed. The flipped deformable network used in the method includes an encoder, a first decoder, a second decoder and an output module. The encoder and the decoder are asymmetrically designed to achieve the effect of blur-clear information decoupling. By flipping the multi-directional deformable convolution block in the first decoder to obtain the second decoder, not only can the characteristics of the deformable convolution be used to obtain the blur features in multiple directions, but also flipping can be used to emphasize the blur information in the horizontal and vertical directions, effectively removing motion blur in multiple directions, and thus improving the deblurring performance of the flipped deformable network.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-directional deblurring method and device based on flipped deformable convolution. Background Art

[0002] Images are an important way for people to obtain information. However, in the imaging process, the images obtained are often blurred due to unfavorable factors such as camera shake, movement of objects in the scene, and camera defocus, which greatly hinders the normal use and subsequent processing of the images.

[0003] Image deblurring is a basic image restoration problem in image processing and computer vision. Its goal is to restore a blurred image caused by motion factors such as camera shake and object motion to a clear image. Most existing image deblurring methods usually rely on neural network models, which implement the image restoration process by inputting a blurred image into the neural network model and outputting a clear image from the neural network model.

[0004] However, traditional neural network models are usually not good at fully restoring the clear details of the image when dealing with image blur, which in turn affects the quality of the final image restoration. Summary of the invention

[0005] The present invention provides a multi-directional deblurring method and device based on flipped deformable convolution, so as to solve the defects existing in the prior art.

[0006] The present invention provides a multi-directional deblurring method based on flipped deformable convolution, comprising:

[0007] Get the image to be processed;

[0008] Applying a flip deformation network to deblur the image to be processed to obtain a deblurred image of the image to be processed;

[0009] The flip deformation network includes an encoder, a first decoder, a second decoder and an output module, the first decoder includes a plurality of cascaded multi-directional deformable convolution blocks, and the second decoder is obtained by flipping the multi-directional deformable convolution block in the first decoder by 90 degrees;

[0010] The encoder is used to receive the image to be processed, and encode the image to be processed to obtain image features of the image to be processed;

[0011] The multi-directional deformable convolution block is used to process the image features based on a simple frequency domain channel attention mechanism and a multi-directional deformable convolution block to obtain multi-directional fuzzy features;

[0012] The output module is used to restore the number of channels of the main axis blurred residual image output by the first decoder and the complementary residual image output by the second decoder to the same number of channels as the image to be processed, and superimpose the main axis blurred residual image, the complementary residual image and the image to be processed to obtain the deblurred image.

[0013] According to a multi-directional deblurring method based on flipped deformable convolution provided by the present invention, the encoder includes a first convolutional layer and a plurality of frequency domain channel attention fuzzy processing blocks of different scales connected in sequence, and the frequency domain channel attention fuzzy processing block includes a fuzzy processing block and a simple frequency domain channel attention block;

[0014] The blur processing block includes a first normalization layer, a second convolution layer, a third convolution layer, a first multiplication layer, a Fourier transform layer, a first Hadamard product layer, an inverse Fourier transform layer, an activation layer and a first superposition layer connected in sequence;

[0015] The output end of the second convolutional layer is also connected to the input end of the first multiplication layer;

[0016] The input end of the first superimposed layer is also connected to the input end of the first normalization layer, and the output end of the first superimposed layer is used to output the processing result of the blur processing block;

[0017] The simple frequency domain channel attention block includes a second normalization layer, a fourth convolution layer, a fifth convolution layer, a frequency domain channel attention module, a sixth convolution layer and a second multiplication layer connected in sequence, and the input end of the second multiplication layer is also connected to the input end of the second normalization layer.

[0018] According to a multi-directional deblurring method based on flipped deformable convolution provided by the present invention, the frequency domain channel attention module includes a channel division module, a frequency domain attention module, a first splicing layer, a seventh convolution layer and a third multiplication layer;

[0019] The input of the frequency domain channel attention module is passed through the channel division module to obtain multiple channel components;

[0020] The multiple channel components are respectively subjected to frequency domain attention calculation by the frequency domain attention module to obtain multiple frequency components;

[0021] The multiple frequency components are cascaded through the first splicing layer to obtain a compressed vector;

[0022] The compressed vector is multiplied by the seventh convolutional layer and the input of the frequency domain channel attention module through the third multiplication layer to obtain the output of the frequency domain channel attention module.

[0023] According to a multi-directional deblurring method based on flipped deformable convolution provided by the present invention, the multi-directional deformable convolution block includes a blur processing block, a simple frequency domain channel attention block and a multi-directional deblurring module connected in sequence;

[0024] The multi-directional deblurring module includes a first branch, a second branch and a second Hadamard product layer, wherein the first branch includes a first residual module, three deformable convolution blocks and a second splicing layer;

[0025] The three deformable convolution blocks are connected in parallel between the first residual module and the second splicing layer;

[0026] The second branch includes an eighth convolutional layer, a second residual module, a third residual module, a ninth convolutional layer and a softmax layer connected in sequence;

[0027] The second concatenation layer and the softmax layer are respectively connected to the second Hadamard product layer, and the second Hadamard product layer is used to output the multi-directional fuzzy feature.

[0028] According to a multi-directional deblurring method based on flipped deformable convolution provided by the present invention, the encoder includes three frequency domain channel attention blurring processing blocks with successively decreasing scales.

[0029] According to a multi-directional deblurring method based on flipped deformable convolution provided by the present invention, the output module comprises a first output convolution layer, a second stacking layer, a second output convolution layer and a third stacking layer connected in sequence;

[0030] The first output convolutional layer is connected to the first decoder;

[0031] The second output convolutional layer is connected to the second decoder;

[0032] The second superposition layer is used to superimpose the output of the first output convolution layer and the output of the second output convolution layer;

[0033] The third superimposed layer is used to receive the image to be processed, and superimpose the image to be processed with the output of the second superimposed layer to obtain the deblurred image.

[0034] According to a multi-directional deblurring method based on flipped deformable convolution provided by the present invention, parameters of the first decoder and the second decoder are shared.

[0035] The present invention also provides a multi-directional deblurring device based on flipped deformable convolution, comprising:

[0036] An image acquisition module, used for acquiring an image to be processed;

[0037] A deblurring module, used for deblurring the image to be processed by applying a flip deformation network to obtain a deblurred image of the image to be processed;

[0038] The flip deformation network includes an encoder, a first decoder, a second decoder and an output module, the first decoder includes a plurality of cascaded multi-directional deformable convolution blocks, and the second decoder is obtained by flipping the multi-directional deformable convolution block in the first decoder by 90 degrees;

[0039] The encoder is used to receive the image to be processed, and encode the image to be processed to obtain image features of the image to be processed;

[0040] The multi-directional deformable convolution block is used to process the image features based on a simple frequency domain channel attention mechanism and a multi-directional deformable convolution block to obtain multi-directional fuzzy features;

[0041] The output module is used to restore the number of channels of the main axis blurred residual image output by the first decoder and the complementary residual image output by the second decoder to the same number of channels as the image to be processed, and superimpose the main axis blurred residual image, the complementary residual image and the image to be processed to obtain the deblurred image.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-directional deblurring method based on flipped deformable convolution as described above is implemented.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multi-directional deblurring method based on flipped deformable convolution as described above is implemented.

[0044] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multi-directional deblurring method based on flipped deformable convolution as described above is implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The multi-directional deblurring method and device based on flipped deformable convolution provided by the present invention first obtains the image to be processed; then applies the flipped deformable network to deblur the image to be processed to obtain a deblurred image of the image to be processed. The flipped deformable network used in the method includes an encoder, a first decoder, a second decoder and an output module. The encoder and the decoder adopt an asymmetric design to achieve the effect of blur-clear information decoupling. By flipping the multi-directional deformable convolution block in the first decoder to obtain the second decoder, not only can the characteristics of the deformable convolution be used to obtain the blur features in multiple directions, but also the flipping can be used to emphasize the blur information in the horizontal and vertical directions, effectively removing motion blur in multiple directions, thereby improving the deblurring performance of the flipped deformable network. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without paying any creative work.

[0048] Figure 1 It is a flowchart of a multi-directional deblurring method based on flipped deformable convolution provided by the present invention;

[0049] Figure 2 It is a structural schematic diagram of a flip deformable network in a multi-directional deblurring method based on flip deformable convolution provided by the present invention;

[0050] Figure 3 It is a schematic diagram of the encoder structure of the flip deformable network in the multi-directional deblurring method based on flip deformable convolution provided by the present invention;

[0051] Figure 4 It is a schematic diagram of the BPB structure of the flip deformable network in the multi-directional deblurring method based on flip deformable convolution provided by the present invention;

[0052] Figure 5 It is a schematic diagram of the SFCA structure of the flip deformable network in the multi-directional deblurring method based on flip deformable convolution provided by the present invention;

[0053] Figure 6 It is a schematic diagram of the MDDM structure of the flip deformable network in the multi-directional deblurring method based on flip deformable convolution provided by the present invention;

[0054] Figure 7 It is a schematic diagram of an image to be processed in the multi-directional deblurring method based on flipped deformable convolution provided by the present invention;

[0055] Figure 8It is a schematic diagram of a deblurred image in a multi-directional deblurring method based on flipped deformable convolution provided by the present invention;

[0056] Fig. 9 It is a structural schematic diagram of a multi-directional deblurring device based on flipped deformable convolution provided by the present invention;

[0057] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Figure 1 FIG. 1 is a flow chart of a multi-directional deblurring method based on flipped deformable convolution provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] S1, obtaining the image to be processed;

[0061] S2, applying a flip deformation network to deblur the image to be processed to obtain a deblurred image of the image to be processed;

[0062] The flip deformation network includes an encoder, a first decoder, a second decoder and an output module, the first decoder includes a plurality of cascaded multi-directional deformable convolution blocks, and the second decoder is obtained by flipping the multi-directional deformable convolution block in the first decoder by 90 degrees;

[0063] The encoder is used to receive the image to be processed, and encode the image to be processed to obtain image features of the image to be processed;

[0064] The multi-directional deformable convolution block is used to process the image features based on a simple frequency domain channel attention mechanism and a multi-directional deformable convolution block to obtain multi-directional fuzzy features;

[0065] The output module is used to restore the number of channels of the main axis blurred residual image output by the first decoder and the complementary residual image output by the second decoder to the same number of channels as the image to be processed, and superimpose the main axis blurred residual image, the complementary residual image and the image to be processed to obtain the deblurred image.

[0066] Specifically, the multi-directional deblurring method based on flipped deformable convolution provided in an embodiment of the present invention is executed by a multi-directional deblurring device based on flipped deformable convolution. The device can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0067] First, step S1 is executed to obtain an image to be processed. The image to be processed refers to a blurred image with motion blur that needs to be deblurred. It can be a large-size substrate image or an image of a target object such as a face or a vehicle in the monitoring field. There is no specific limitation here.

[0068] Then, step S2 is performed to apply the flip deformable network to deblur the image to be processed, and obtain a deblurred image of the image to be processed. Here, the image to be processed can be input into the flip deformable network (Flipped Deformable Network, FDNet) to obtain a deblurred image output by the flip deformable network.

[0069] like Figure 2 As shown, the flip deformation network may include an encoder, a first decoder, a second decoder and an output module, wherein the encoder is used to receive the image to be processed and encode the image to be processed to obtain the image features of the image to be processed. The operation of the encoder can be expressed as:

[0070] ;

[0071] in, is the image feature, is the encoder, and x is the image to be processed.

[0072] The first decoder includes multiple cascaded multi-directional deformable convolution blocks (MDDBs), each of which is used to process image features based on a simple frequency channel attention (SFCA) mechanism and a multi-directional deformable convolution block to obtain multi-directional blurred features.

[0073] Compared with the frequency-domain channel attention (FCA) mechanism, the simple frequency-domain channel attention mechanism introduces layer normalization operations and two convolution operations to process the input, and then processes it through the FCA mechanism, which can reduce the amount of computation.

[0074] The multi-directional deformable convolution block corresponds to multiple deformable convolution blocks in different directions. The convolution kernel of each deformable convolution block adds a direction parameter at each position of the input, so that the convolution kernel at each position can be freely deformed and displaced within the receptive field, thereby allowing the convolution kernel to expand to a larger range during training. Here, three deformable convolution blocks (Deformable) with different expansion rates can be constructed to generate three offset maps with different receptive fields to modulate the direction of the deformable convolution, thereby generating outputs in three different directions.

[0075] The second decoder is obtained by flipping the multi-directional deformable convolution block in the first decoder by 90 degrees. Therefore, the multi-directional deformable convolution block contained in the second decoder can be expressed as (I-MDDB). MDDB is the same as I-MDDB but focuses on orthogonal content.

[0076] The first decoder and the second decoder are two independent decoders, and their parameters can be shared. According to the linear span theory, the linear reconstruction composed of the outputs of two independent decoders can contain a larger output space, because the independent regression network learns complementary features to cover the entire output space. Ideally, the outputs of the two decoders are independent regression outputs to maximize the solution space. In other words, in an embodiment of the present invention, the output of the first decoder is a principal axis blurred residual image, that is, the most variable degradation component, and the output of the second decoder is a complementary residual to complete the linear reconstruction.

[0077] The main axis blurred residual image D1 output by the first decoder can be expressed as , the complementary residual image D2 output by the second decoder can be expressed as ;in, and denote the first decoder and the second decoder respectively, represents the parameters of the first decoder, Represents the parameters of the second decoder.

[0078] The first decoder and the second decoder can decode the inherent information in the image features into a principal axis blurred residual image D1 along the horizontal axis and a complementary residual image D2 along the vertical axis, respectively, without applying any explicit constraints, thereby deblurring more effectively.

[0079] The output module can restore the number of channels of the main axis blur residual image D1 output by the first decoder and the complementary residual image D2 output by the second decoder to the same number of channels as the image to be processed, and superimpose the main axis blur residual image D1, the complementary residual image D2 and the image to be processed to obtain a deblurred image. Here, the number of channels of the image to be processed can be 3, so the number of channels of the main axis blur residual image D1 and the complementary residual image D2 can be restored to 3.

[0080] The multi-directional deblurring method based on flipped deformable convolution provided in an embodiment of the present invention first obtains an image to be processed; then applies a flipped deformable network to deblur the image to be processed to obtain a deblurred image of the image to be processed. The flipped deformable network used in the method includes an encoder, a first decoder, a second decoder and an output module. The encoder and the decoder are asymmetrically designed to achieve the effect of blur-clear information decoupling. By flipping the multi-directional deformable convolution block in the first decoder to obtain the second decoder, not only can the characteristics of the deformable convolution be used to obtain blur features in multiple directions, but also flipping can be used to emphasize the blur information in the horizontal and vertical directions, effectively removing motion blur in multiple directions, thereby improving the deblurring performance of the flipped deformable network.

[0081] On the basis of the above embodiment, the encoder includes a first convolutional layer and a plurality of frequency domain channel attention fuzzy processing blocks of different scales connected in sequence, and the frequency domain channel attention fuzzy processing block includes a fuzzy processing block and a simple frequency domain channel attention block;

[0082] The blur processing block includes a first normalization layer, a second convolution layer, a third convolution layer, a first multiplication layer, a Fourier transform layer, a first Hadamard product layer, an inverse Fourier transform layer, an activation layer and a first superposition layer connected in sequence;

[0083] The output end of the second convolutional layer is also connected to the input end of the first multiplication layer;

[0084] The input end of the first superimposed layer is also connected to the input end of the first normalization layer, and the output end of the first superimposed layer is used to output the processing result of the blur processing block;

[0085] The simple frequency domain channel attention block includes a second normalization layer, a fourth convolution layer, a fifth convolution layer, a frequency domain channel attention module, a sixth convolution layer and a second multiplication layer connected in sequence, and the input end of the second multiplication layer is also connected to the input end of the second normalization layer.

[0086] Specifically, Figure 3 As shown, the encoder includes a first convolutional layer and multiple frequency domain channel attention blurring blocks of different scales connected in sequence. Figure 3 In the embodiment, the encoder may include two frequency domain channel attention blur processing blocks, each of which includes a blur processing block (Blur Processing Block, BPB) and a simple frequency domain channel attention (SFCA) block.

[0087] like Figure 4As shown in the figure, BPB includes the first normalization (LayerNorm) layer, the second convolution (Conv) layer, the third convolution layer, the first multiplication layer, the Fourier transform (FFT) layer, the first Hadamard product layer, the inverse Fourier transform (IFFT) layer, the activation layer and the first superposition layer connected in sequence. The second convolution layer and the third convolution layer can both perform 1×1 convolution operations to expand the number of channels and obtain potential information. The activation layer can use the Gelu activation function to achieve nonlinear mapping. BPB can obtain frequency domain information through Fourier transform and inverse Fourier transform.

[0088] The output end of the second convolution layer is also connected to the input end of the first multiplication layer, the input end of the first superposition layer is also connected to the input end of the first normalization layer, and the output end of the first superposition layer is used to output the processing result of the blur processing block.

[0089] The operation of BPB can be expressed by the following formula:

[0090] ;

[0091] ;

[0092] ;

[0093] .

[0094] Among them, Y is the input of BPB, which is a tensor, The convolution operation implemented for the second convolutional layer, The layer normalization operation implemented for the first normalization layer, The convolution operation implemented for the third convolutional layer, is the output of the second convolutional layer, is the output of the first multiplication layer, is the weight coefficient, is the output of the first Hadamard product layer, It is the processing result of BPB, that is, the output of BPB.

[0095] like Figure 5 As shown, the SFCA block includes a second normalization layer, a fourth convolutional layer, a fifth convolutional layer, a frequency channel attention (FCA) module, a sixth convolutional layer, and a second multiplication layer connected in sequence, and the input end of the second multiplication layer is also connected to the input end of the second normalization layer.

[0096] The output of the fifth convolutional layer can be expressed by the following formula:

[0097] ;

[0098] in, is the output of the fifth convolutional layer, X is the input of the SFCA block, and is a tensor. The layer normalization operation implemented for the second normalization layer, The convolution operation implemented for the fourth convolutional layer, Convolution operation implemented for the fifth convolutional layer.

[0099] The FCA module may include a channel division module, a frequency domain attention module, a first splicing layer, a seventh convolutional layer and a third multiplication layer. The input of the FCA module is passed through the channel division module to obtain multiple channel components, and the multiple channel components are respectively subjected to frequency domain attention calculation by the frequency domain attention module to obtain multiple frequency components; the multiple frequency components are cascaded through the first splicing layer to obtain a compressed vector; the compressed vector is multiplied with the input of the frequency domain channel attention module after passing through the seventh convolutional layer through the third multiplication layer to obtain the output of the frequency domain channel attention module.

[0100] For example, the input of the FCA module is the output of the fifth convolutional layer, that is, , the multiple channel components obtained after the channel division module are The n channel components of , C is The number of channels must be For example, n channel components can be expressed as , , , H and W are The height and width of the

[0101] Each channel component is calculated by the frequency domain attention module, and the frequency domain attention is:

[0102] ;

[0103] ;

[0104] in, express The component with height h and width w, , They correspond to The two-dimensional (2D) index of the frequency components of For the corresponding The two-dimensional DCT of yes The frequency components obtained by the frequency domain attention calculation of the frequency domain attention module, and After compression dimensional vector. for The corresponding DCT weights.

[0105] The frequency components corresponding to each channel component are cascaded through the first splicing layer, that is, the splicing operation, and a compressed vector can be obtained:

[0106] ;

[0107] in, is a compressed vector.

[0108] The frequency components corresponding to each channel component are cascaded through the first splicing layer to obtain a compressed vector , and then after passing through the seventh convolutional layer and After multiplication operation is performed in the third multiplication layer, the operation result is obtained, and the operation result is used as the output of the FCA module, which is:

[0109] ;

[0110] in, is the operation result, Convolution operation implemented for the seventh convolutional layer.

[0111] It is understandable that the calculation results is a tensor with a channel number of C, a height of H, and a width of W. In the embodiment of the present invention, the convolution kernel size of the seventh convolution layer may be 1×1, which can effectively reduce the amount of calculation.

[0112] On the basis of the above embodiment, MDDB includes BPB, SFCA block and Multi Direction deblurring module (MDDM) connected in sequence.

[0113] like Figure 6 As shown, MDDM includes a first branch, a second branch and a second Hadamard product layer, and the first branch includes a first residual module (ResBlock), three deformable convolution blocks (Deformable) and a second concatenation (Concat) layer.

[0114] Three deformable convolution blocks are connected in parallel between the first residual module and the second concatenation layer.

[0115] The second branch includes an eighth convolutional layer, a second residual module, a third residual module, a ninth convolutional layer, and a softmax layer, which are connected in sequence.

[0116] The second concatenation layer and the softmax layer are respectively connected to the second Hadamard product layer, and the second Hadamard product layer is used to output multi-directional fuzzy features.

[0117] In MDDM, on the one hand, the input passes through the first residual module and then three residual modules to roughly obtain the blurred information, and the content generated by the three deformable convolution blocks is spliced ​​along the channel dimension through the second splicing layer to obtain the output of the second splicing layer; on the other hand, the input passes through the eighth convolution layer, the second residual module, the third residual module and the ninth convolution layer in sequence, and then passes through the Softmax layer to obtain the weight mapping.

[0118] After that, the output of the second concatenation layer and the weight map are element-wise multiplied through the second Hadamard product layer to obtain the output of MDDM, i.e., multi-directional fuzzy features.

[0119] The operation of the second Hadamard product layer can be expressed as:

[0120] ;

[0121] ;

[0122] in, is the output of the second Hadamard product layer, is the i-th channel in the output of the second concatenation layer, is the weight mapping of the jth pixel in the ith channel, h is the height of the MDDM input, and w is the width of the MDDM input.

[0123] Based on the above embodiment, the parameters of the first decoder and the second decoder are shared without increasing the computational complexity.

[0124] On the basis of the above-mentioned embodiment, the flip deformation network can be obtained by training the initial network using the image samples with motion blur and the clear images corresponding to the image samples. The image samples can be acquired by an image acquisition device with a first shutter rate, and the clear images corresponding to the image samples can be acquired by an image acquisition device with a second shutter rate, and the first shutter rate is less than the second shutter rate. The initial network can have the same structure as the flip deformation network, and the difference is only in the different structural parameters. The structural parameters of the initial network are obtained by initialization, while the structural parameters of the flip deformation network are obtained by training the initial network.

[0125] When training the initial network, the image samples can be first input into the initial network to obtain the output results of the initial network, and then the output results and the clear image are used to calculate the loss function. According to the loss function, the structural parameters of the initial network are iteratively updated until the loss function converges to obtain the flip deformation network.

[0126] like Figure 7 The image shown is the image to be processed. Figure 8is a deblurred image obtained by the multi-directional deblurring method based on flipped deformable convolution provided in an embodiment of the present invention. Figure 7 and Figure 8 By comparison, it can be seen that the multi-directional deblurring method based on flipped deformable convolution provided in the embodiment of the present invention has an excellent deblurring effect.

[0127] On the basis of the above embodiments, for the flip deformation network, the key parameter is the number of deformable convolution blocks. The number of deformable convolution blocks represents the number of directions. By designing different numbers of deformable convolution blocks, different effects are achieved on blur perception in multiple directions. In Table 1, four versions of the flip deformation network are shown: FDNet-1, FDNet-2, FDNet-4, and FDNet-8, where 1, 2, 4, and 8 represent the number of stacked deformable convolution blocks in MDDM. Peak signal-to-noise ratio (PSNR) is one of the indicators for measuring image quality. As shown in Table 1, PSNR increases with the number of stacked deformable convolution blocks, and saturation occurs after the number of deformable convolution blocks exceeds four directions, that is, more computing resources are consumed, but the improvement effect is very small. Therefore, in the embodiment of the present invention, the number of deformable convolution blocks is selected as three, which achieves an excellent balance between performance and efficiency.

[0128] Table 1 Correspondence between the number of deformable convolution blocks and PSNR

[0129]

[0130] In summary, in the multi-directional deblurring method based on flipped deformable convolution provided in the embodiment of the present invention, the flipped deformable network used focuses on blur information from multiple directions, thereby removing motion blur. The flipped deformable network uses a flipping design of one encoder and two decoders to flip multiple directions, thereby realizing the learning of multi-directional orthogonal directions and achieving feature attention in more directions. Moreover, the multi-directional deformable convolution block makes better use of frequency domain information and plays a role in acquiring blur information. The calculation parameters are greatly reduced through the parameter sharing design of the two decoders. Compared with other multi-decoder designs, this design reduces more computing consumption.

[0131] like Fig. 9 As shown, based on the above embodiment, a multi-directional deblurring device based on flipped deformable convolution is provided in an embodiment of the present invention, comprising:

[0132] An image acquisition module 91 is used to acquire an image to be processed;

[0133] A deblurring module 92, configured to apply a flip deformation network to deblur the image to be processed to obtain a deblurred image of the image to be processed;

[0134] The flip deformation network includes an encoder, a first decoder, a second decoder and an output module, the first decoder includes a plurality of cascaded multi-directional deformable convolution blocks, and the second decoder is obtained by flipping the multi-directional deformable convolution block in the first decoder by 90 degrees;

[0135] The encoder is used to receive the image to be processed, and encode the image to be processed to obtain image features of the image to be processed;

[0136] The multi-directional deformable convolution block is used to process the image features based on a simple frequency domain channel attention mechanism and a multi-directional deformable convolution block to obtain multi-directional fuzzy features;

[0137] The output module is used to restore the number of channels of the main axis blurred residual image output by the first decoder and the complementary residual image output by the second decoder to the same number of channels as the image to be processed, and superimpose the main axis blurred residual image, the complementary residual image and the image to be processed to obtain the deblurred image.

[0138] On the basis of the above embodiment, the encoder includes a first convolutional layer and a plurality of frequency domain channel attention fuzzy processing blocks of different scales connected in sequence, and the frequency domain channel attention fuzzy processing block includes a fuzzy processing block and a simple frequency domain channel attention block;

[0139] The blur processing block includes a first normalization layer, a second convolution layer, a third convolution layer, a first multiplication layer, a Fourier transform layer, a first Hadamard product layer, an inverse Fourier transform layer, an activation layer and a first superposition layer connected in sequence;

[0140] The output end of the second convolutional layer is also connected to the input end of the first multiplication layer;

[0141] The input end of the first superimposed layer is also connected to the input end of the first normalization layer, and the output end of the first superimposed layer is used to output the processing result of the blur processing block;

[0142] The simple frequency domain channel attention block includes a second normalization layer, a fourth convolution layer, a fifth convolution layer, a frequency domain channel attention module, a sixth convolution layer and a second multiplication layer connected in sequence, and the input end of the second multiplication layer is also connected to the input end of the second normalization layer.

[0143] On the basis of the above embodiment, the frequency domain channel attention module includes a channel division module, a frequency domain attention module, a first concatenation layer, a seventh convolution layer and a third multiplication layer;

[0144] The input of the frequency domain channel attention module is passed through the channel division module to obtain multiple channel components;

[0145] The multiple channel components are respectively subjected to frequency domain attention calculation by the frequency domain attention module to obtain multiple frequency components;

[0146] The multiple frequency components are cascaded through the first splicing layer to obtain a compressed vector;

[0147] The compressed vector is multiplied by the seventh convolutional layer and the input of the frequency domain channel attention module through the third multiplication layer to obtain the output of the frequency domain channel attention module.

[0148] Based on the above embodiment, the multi-directional deformable convolution block includes a blur processing block, a simple frequency domain channel attention block and a multi-directional deblurring module connected in sequence;

[0149] The multi-directional deblurring module includes a first branch, a second branch and a second Hadamard product layer, wherein the first branch includes a first residual module, three deformable convolution blocks and a second splicing layer;

[0150] The three deformable convolution blocks are connected in parallel between the first residual module and the second splicing layer;

[0151] The second branch includes an eighth convolutional layer, a second residual module, a third residual module, a ninth convolutional layer and a softmax layer connected in sequence;

[0152] The second concatenation layer and the softmax layer are respectively connected to the second Hadamard product layer, and the second Hadamard product layer is used to output the multi-directional fuzzy feature.

[0153] Based on the above embodiment, the encoder includes three frequency domain channel attention fuzzy processing blocks with successively decreasing scales.

[0154] On the basis of the above embodiment, the output module includes a first output convolution layer, a second stacking layer, a second output convolution layer and a third stacking layer connected in sequence;

[0155] The first output convolutional layer is connected to the first decoder;

[0156] The second output convolutional layer is connected to the second decoder;

[0157] The second superposition layer is used to superimpose the output of the first output convolution layer and the output of the second output convolution layer;

[0158] The third superimposed layer is used to receive the image to be processed, and superimpose the image to be processed with the output of the second superimposed layer to obtain the deblurred image.

[0159] Based on the above embodiment, parameters of the first decoder and the second decoder are shared.

[0160] Specifically, the functions of each module in the multi-directional deblurring device based on flipped deformable convolution provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0161] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor (Processor) 110, a communication interface (Communications Interface) 120, a memory (Memory) 130 and a communication bus 140, wherein the processor 110, the communication interface 120, and the memory 130 communicate with each other through the communication bus 140. The processor 110 may call the logic instructions in the memory 130 to execute the multi-directional deblurring method based on flipped deformable convolution provided in the above embodiments.

[0162] In addition, the logic instructions in the above-mentioned memory 130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0163] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-directional deblurring method based on flipped deformable convolution provided in the above-mentioned embodiments.

[0164] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the multi-directional deblurring method based on flipped deformable convolution provided in the above embodiments.

[0165] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-directional deblurring method based on flipped deformable convolution, characterized in that: include: Get the image to be processed; Applying a flip deformation network to deblur the image to be processed to obtain a deblurred image of the image to be processed; The flip deformation network includes an encoder, a first decoder, a second decoder and an output module, the first decoder includes a plurality of cascaded multi-directional deformable convolution blocks, and the second decoder is obtained by flipping the multi-directional deformable convolution block in the first decoder by 90 degrees; The encoder is used to receive the image to be processed, and encode the image to be processed to obtain image features of the image to be processed; The multi-directional deformable convolution block is used to process the image features based on a simple frequency domain channel attention mechanism and a multi-directional deformable convolution block to obtain a multi-directional fuzzy feature; compared with the frequency domain channel attention mechanism, the simple frequency domain channel attention mechanism introduces a layer normalization operation and two convolution operations to process the input, and then processes it through the frequency domain channel attention mechanism; the multi-directional deformable convolution block is a plurality of deformable convolution blocks corresponding to different directions, and the convolution kernel of each deformable convolution block adds a direction parameter at each position of the input, so that the convolution kernel at each position can be freely deformed and displaced within the receptive field; The output module is used to restore the number of channels of the main axis blur residual image output by the first decoder and the complementary residual image output by the second decoder to be the same as the number of channels of the image to be processed, and superimpose the main axis blur residual image, the complementary residual image and the image to be processed to obtain the deblurred image; The multi-directional deformable convolution block includes a blur processing block, a simple frequency domain channel attention block and a multi-directional deblurring module connected in sequence; The multi-directional deblurring module includes a first branch, a second branch and a second Hadamard product layer, the first branch includes a first residual module, three deformable convolution blocks and a second splicing layer; the three deformable convolution blocks are three deformable convolution blocks with different expansion rates, which are respectively used to generate three offset maps with different receptive fields to modulate the direction of the deformable convolution, thereby generating outputs in three different directions; The three deformable convolution blocks are connected in parallel between the first residual module and the second splicing layer; The second branch includes an eighth convolutional layer, a second residual module, a third residual module, a ninth convolutional layer and a softmax layer connected in sequence; The second concatenation layer and the softmax layer are respectively connected to the second Hadamard product layer, and the second Hadamard product layer is used to output the multi-directional fuzzy feature; The encoder comprises a first convolutional layer and a plurality of frequency domain channel attention fuzzy processing blocks of different scales connected in sequence, wherein the frequency domain channel attention fuzzy processing block comprises a fuzzy processing block and a simple frequency domain channel attention block; The blur processing block includes a first normalization layer, a second convolution layer, a third convolution layer, a first multiplication layer, a Fourier transform layer, a first Hadamard product layer, an inverse Fourier transform layer, an activation layer and a first superposition layer connected in sequence; The output end of the second convolutional layer is also connected to the input end of the first multiplication layer; The input end of the first superimposed layer is also connected to the input end of the first normalization layer, and the output end of the first superimposed layer is used to output the processing result of the blur processing block; The simple frequency domain channel attention block includes a second normalization layer, a fourth convolution layer, a fifth convolution layer, a frequency domain channel attention module, a sixth convolution layer and a second multiplication layer connected in sequence, and an input end of the second multiplication layer is also connected to an input end of the second normalization layer; The frequency domain channel attention module includes a channel division module, a frequency domain attention module, a first concatenation layer, a seventh convolution layer and a third multiplication layer; The input of the frequency domain channel attention module is passed through the channel division module to obtain multiple channel components; The multiple channel components are respectively subjected to frequency domain attention calculation by the frequency domain attention module to obtain multiple frequency components; The multiple frequency components are cascaded through the first splicing layer to obtain a compressed vector; The compressed vector is multiplied by the seventh convolutional layer and the input of the frequency domain channel attention module through the third multiplication layer to obtain the output of the frequency domain channel attention module.

2. The multi-directional deblurring method based on flipped deformable convolution according to claim 1, characterized in that: The encoder includes three frequency-domain channel attention blurring blocks of successively decreasing scales.

3. The multi-directional deblurring method based on flipped deformable convolution according to any one of claims 1-2, characterized in that: The output module includes a first output convolution layer, a second stacking layer, a second output convolution layer and a third stacking layer connected in sequence; The first output convolutional layer is connected to the first decoder; The second output convolutional layer is connected to the second decoder; The second superposition layer is used to superimpose the output of the first output convolution layer and the output of the second output convolution layer; The third superimposed layer is used to receive the image to be processed, and superimpose the image to be processed with the output of the second superimposed layer to obtain the deblurred image.

4. The multi-directional deblurring method based on flipped deformable convolution according to any one of claims 1-2, characterized in that: Parameters of the first decoder and the second decoder are shared.

5. A multi-directional deblurring device based on flipped deformable convolution, characterized in that: include: An image acquisition module, used for acquiring an image to be processed; A deblurring module, used for deblurring the image to be processed by applying a flip deformation network to obtain a deblurred image of the image to be processed; The flip deformation network includes an encoder, a first decoder, a second decoder and an output module, the first decoder includes a plurality of cascaded multi-directional deformable convolution blocks, and the second decoder is obtained by flipping the multi-directional deformable convolution block in the first decoder by 90 degrees; The encoder is used to receive the image to be processed, and encode the image to be processed to obtain image features of the image to be processed; The multi-directional deformable convolution block is used to process the image features based on a simple frequency domain channel attention mechanism and a multi-directional deformable convolution block to obtain a multi-directional fuzzy feature; compared with the frequency domain channel attention mechanism, the simple frequency domain channel attention mechanism introduces a layer normalization operation and two convolution operations to process the input, and then processes it through the frequency domain channel attention mechanism; the multi-directional deformable convolution block is a plurality of deformable convolution blocks corresponding to different directions, and the convolution kernel of each deformable convolution block adds a direction parameter at each position of the input, so that the convolution kernel at each position can be freely deformed and displaced within the receptive field; The output module is used to restore the number of channels of the main axis blur residual image output by the first decoder and the complementary residual image output by the second decoder to be the same as the number of channels of the image to be processed, and superimpose the main axis blur residual image, the complementary residual image and the image to be processed to obtain the deblurred image; The multi-directional deformable convolution block includes a blur processing block, a simple frequency domain channel attention block and a multi-directional deblurring module connected in sequence; The multi-directional deblurring module includes a first branch, a second branch and a second Hadamard product layer, the first branch includes a first residual module, three deformable convolution blocks and a second splicing layer; the three deformable convolution blocks are three deformable convolution blocks with different expansion rates, which are respectively used to generate three offset maps with different receptive fields to modulate the direction of the deformable convolution, thereby generating outputs in three different directions; The three deformable convolution blocks are connected in parallel between the first residual module and the second splicing layer; The second branch includes an eighth convolutional layer, a second residual module, a third residual module, a ninth convolutional layer and a softmax layer connected in sequence; The second concatenation layer and the softmax layer are respectively connected to the second Hadamard product layer, and the second Hadamard product layer is used to output the multi-directional fuzzy feature; The encoder comprises a first convolutional layer and a plurality of frequency domain channel attention fuzzy processing blocks of different scales connected in sequence, wherein the frequency domain channel attention fuzzy processing block comprises a fuzzy processing block and a simple frequency domain channel attention block; The blur processing block includes a first normalization layer, a second convolution layer, a third convolution layer, a first multiplication layer, a Fourier transform layer, a first Hadamard product layer, an inverse Fourier transform layer, an activation layer and a first superposition layer connected in sequence; The output end of the second convolutional layer is also connected to the input end of the first multiplication layer; The input end of the first superimposed layer is also connected to the input end of the first normalization layer, and the output end of the first superimposed layer is used to output the processing result of the blur processing block; The simple frequency domain channel attention block includes a second normalization layer, a fourth convolution layer, a fifth convolution layer, a frequency domain channel attention module, a sixth convolution layer and a second multiplication layer connected in sequence, and an input end of the second multiplication layer is also connected to an input end of the second normalization layer; The frequency domain channel attention module includes a channel division module, a frequency domain attention module, a first splicing layer, a seventh convolutional layer and a third multiplication layer; The input of the frequency domain channel attention module is passed through the channel division module to obtain multiple channel components; The multiple channel components are respectively subjected to frequency domain attention calculation by the frequency domain attention module to obtain multiple frequency components; The multiple frequency components are cascaded through the first splicing layer to obtain a compressed vector; The compressed vector is multiplied by the seventh convolutional layer and the input of the frequency domain channel attention module through the third multiplication layer to obtain the output of the frequency domain channel attention module.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-directional deblurring method based on flipped deformable convolution is implemented as described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-directional deblurring method based on flipped deformable convolution is implemented as described in any one of claims 1 to 4.

Images