A motion image deblurring model based on fourier transform

Abstract

The application discloses a motion image deblurring model based on Fourier transform, wherein a three-level symmetrical encoder-decoder architecture is arranged in the deblurring model; a self-attention mechanism FDSA is used in the encoder and the decoder to estimate the attention of scaled dot product in the frequency domain; a feedforward network FFDN is used in the encoder and the decoder to fuse blur kernel level information and pixel level information, and a learnable quantization matrix M is introduced to determine which frequency information is retained; a cross-connection channel attention module CCAM is used as an information exchange component between the encoder and the decoder; the encoder is connected with a Sobel filter to enhance edge features; three loss functions are combined to comprehensively consider the perceptual quality and frequency domain characteristics of an image, and the model is guided to focus on restoring high-frequency components in a blurred image. The application provides a deblurring model which focuses on details and enhances edges, so that a high-quality deblurred image can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image restoration, and particularly relates to a motion image deblurring model based on Fourier transform. BACKGROUND

[0002] With the popularity and development of many smart devices such as smart phones, the demand for taking pictures is becoming more and more convenient for people, and how to help smart device users take clear and non-blurred pictures is becoming more and more important. The image blur in the usual case refers to the phenomenon that the image loses the characteristics of clarity, sharpness and details due to the influence of various factors in the process of collection, transmission and display, and becomes blurred. Common types of image blur include motion blur, defocus blur and noise blur. Among them, motion blur is caused by the movement of the shooting device or the shooting object, for example, when shooting a fast-moving object, the image will appear trailing or stretching.

[0003] In recent years, with the emergence of various deep learning network architectures, many network architectures have been applied to image deblurring tasks. GAN (Generative adversarial network) is often applied to image deblurring tasks. GAN learns the distribution of images by introducing two networks of generator and discriminator. The generator is responsible for converting low-quality blurred images into high-quality clear images, and the discriminator is responsible for judging whether the image generated by the generator is similar to the true value image. Through continuous iteration training of the generator and the discriminator, GAN can generate high-quality clear images. However, GAN needs to train two networks of generator and discriminator, and the training process is relatively complex. Different degrees of image blur will also lead to unstable training of the GAN network, and thus the generated image quality is poor. RNN (Recurrent Neural Networks) is a neural network structure for processing sequence data, which can also be used for image deblurring tasks. However, the characteristics of RNN transmitting information from front to back make it difficult to handle bidirectional dependencies, and the effect is not good when capturing and retaining important information in the input sequence. Cascade network can gradually improve the quality of the image through multi-stage processing, so it is also applied to the deblurring task. However, the cascade network contains multiple sub-networks, and each sub-network needs to be trained and optimized, which has a relatively large amount of calculation, resulting in long training and inference time, and is prone to overfitting problems. SUMMARY

[0004] The purpose of the present application is to propose an end-to-end motion image deblurring model, which pays more attention to the edge information and detail information in the blurred image, thereby meeting the needs of subsequent high-level tasks such as target detection.

[0005] To achieve the above object, the application provides a motion image deblurring model based on Fourier transform, which comprises:

[0006] In the deblurring model, a three-level symmetric encoder-decoder architecture is arranged, the FDSA (frequency-domain self-attention mechanism) of the encoder and the decoder estimates the attention of the scaled dot product in the frequency domain to improve the perception ability of the model to high-frequency information; the blur kernel level information and the pixel level information are fused in the FFDN (feedforward network) of the encoder and the decoder, and a learnable quantization matrix M is introduced to determine which frequency information is worth preserving; the CCAM (cross-connection channel attention module) is used as an information exchange component between the encoder and the decoder; the Sobel filter is connected in the encoder stage to enhance the edge features and help the model enhance the perception ability of the image edges; the content loss function L count , the multi-scale frequency reconstruction loss function L MSFR and the auxiliary edge loss function L edge are used as the loss function, and the three loss functions are used jointly to comprehensively consider the perception quality and the frequency domain characteristics of the image and guide the model to focus on restoring the high-frequency components in the deblurred image.

[0007] The transformer can capture the dependency between different positions in the image by using the self-attention mechanism, so as to better understand and reconstruct the blurred image. The application converts the spatial domain transformer into the frequency domain transformer, so that the detail information and the structure information in the image are better preserved, and further uses the element-wise product in the frequency domain to replace the matrix multiplication in the spatial domain to estimate the correlation in the sequence elements, which greatly reduces the complexity of the calculation. Specifically, according to the convolution theorem, the time domain convolution (or correlation) operation of two signals is converted into the element-wise multiplication of the two signals in the frequency domain. Therefore, the attention map is effectively estimated by the element-wise multiplication in the frequency domain without the need to calculate the QK T matrix (QK T is an attention weight matrix used to assign weights between different positions of the input sequence) in the spatial domain, which greatly reduces the complexity of the model calculation. The self-attention mechanism in the encoder-decoder is named as FDSA (frequency-domain self-attention mechanism) in the application. The FDSA firstly obtains F q , F k and F v through 1x1 point convolution and 3x3 deep convolution, wherein Fq F is a Query feature k F is a Key feature v F is a Value feature. Then, the estimated features F q and F k are applied with a Fast Fourier Transform, so that F q and F k are obtained.

[0008] In order to pay more attention to the high frequency information in the frequency domain, the present application adds a Relu residual stream in the feedforward network of the encoder-decoder, and names the feedforward network as FFDN (feedforward network). The Relu residual stream in the FFDN pays more attention to the recovery of high frequency information in the frequency domain while helping the model to learn the blur kernel information, and in addition, the Relu operation in the frequency domain also brings the ability of global learning to the feedforward network. After the ReLU operation on the blurred image in the frequency domain, inverse Fourier transform can be used to generate a specific frequency blur pattern. Through the Relu residual stream, the network can learn the blur kernel level information to better complete the image deblurring task. The FFDN of the present application can fuse the generated blur kernel level information with the pixel level features, specifically: the intermediate stream first passes through a 1x1 convolution, and the obtained features are applied with a Fast Fourier Transform to obtain frequency domain features, further, in order to adaptively determine which low frequency information and high frequency information are helpful for potential image restoration, a learnable quantization matrix M is introduced, and through inverse learning of the JPEG compression algorithm, it is determined which frequency information should be retained, so that the frequency domain features are multiplied with the quantization matrix M element by element, and finally restored to pixel level features through inverse fast Fourier transform. The Relu residual stream implementation includes: first applying a Fast Fourier Transform on the obtained tensor; then using two 1x1 convolution layers, and adding a ReLU layer in the middle, selecting frequencies through ReLU, and then performing inverse Fourier transform to directly learn the blur kernel level features from the blurred image; finally, the blur kernel level features and the pixel level features are fused through element summation operation to obtain the final FFDN output.

[0009] In order to make up for the gap between different scale features, the deblurring model usually adopts the way of jump connection. But the simple jump connection cannot meet the needs of the deblurring task, therefore, the application adds a cross-connection channel attention module between the encoder and the decoder, the cross-connection channel attention module includes two parts of channel self-attention mechanism and cross-connection feedforward network. The cross-connection channel attention module makes up for the gap of semantics and resolution between the output of the encoder of different scales, and further adds the processed different scale features to the decoder, improves the multi-scale information perception ability of the network model, and solves the problem of spatial feature dislocation between the high-resolution encoder features and the low-resolution decoder features.

[0010] In order to enhance the edge information of the model, the application uses a learnable Sobel filter as an edge enhancement block. Specifically, the Sobel filter is applied to the input blurred image to obtain an edge feature image, the edge feature is connected with the low-order feature obtained by applying 3x3 depth convolution to the blurred image, and the connection operation is used as the input of the first-level encoder, and then the edge feature image is subjected to secondary downsampling operation, and the first downsampling result is connected with the output of the first-level encoder as the input of the second-level encoder, and the secondary downsampling result is connected with the output of the second-level encoder as the input of the third-level encoder.

[0011] The auxiliary edge loss function can help the network to be closer to the true value image of the target image in the high frequency part, and on the same network model, the auxiliary edge loss function shows better deblurring performance than the traditional MSE loss function. The auxiliary edge loss function of the application is based on the Sobel filter. The Sobel filter is composed of horizontal and vertical Sobel convolution kernels, and the convolution kernel can be decomposed into the product form of an averaging convolution kernel and a differential convolution kernel, wherein the averaging convolution kernel can smooth the image, and the differential convolution kernel can detect the edge of the image. This decomposition can improve the performance and stability of the filter. Therefore, the Sobel horizontal kernel G x , the Sobel vertical kernel G y can be written as:

[0012] G x =[1 2 1] T *[+1 0 -1]

[0013] G y =[+1 0 -1] T *[1 2 1]

[0014] Where [+1 0 -1] is a differential convolution kernel, which detects a clear edge map by obtaining the gradient size of the image, and [1 2 1] is an averaging convolution kernel, which reduces unnecessary noise in the image. Through the Sobel operation, the high frequency component f sobelThe equation can be expressed as follows:

[0015]

[0016] Generally, the Sobel edge map of the sharpened image contains sharpened edges, while the blurred image does not contain sharpened edges. In order to restore the clear image, the distance between the output result image and the ground truth image in the Sobel edge space is directly minimized. Therefore, the proposed auxiliary edge loss function L edge The equation can be expressed as follows:

[0017]

[0018] wherein, and I truth represent the output result image and the ground truth image respectively, and N is the total number of pixel elements. It can be seen that the auxiliary edge loss function L edge is conducive to comparing the loss value size of different resolution images.

[0019] The final model loss function L total includes three parts: the multi-scale content loss function L cont , the multi-scale frequency reconstruction loss function L MSFR and the auxiliary edge loss function L edge .

[0020]

[0021]

[0022] L total = L cont + λ1L MSFR + λ2L edge

[0023] wherein, K is the scale level, and S k represent the output result image and the ground truth image of the k scale respectively, and t k is the total number of elements of the k scale. represents the fast Fourier transform of the image signal converted to the frequency domain, and the multi-scale frequency reconstruction loss function weight λ1 and the auxiliary edge loss function weight λ2 are set to 0.1 and 0.05 respectively in the experiment.

[0024] Compared with the prior art, the deblurring model of the application focuses on details and enhances edges to obtain a high-quality deblurring image. In order to better improve the features extracted based on the frequency domain Transformer attention mechanism, a new feedforward network is designed. The new feedforward network not only considers the effects of low-frequency information and high-frequency information on restoring a clear image, but also integrates blur kernel level information into the feedforward network by applying ReLU operation and Fourier transform in the frequency domain of the blurred image, so as to provide more effective deblurring detail information. In addition, in order to better aggregate the information between the encoder and the decoder, a cross-connection channel attention module is designed as an information exchange part in the middle of the model. The module can adaptively fuse the features of each scale encoder with the semantic gap scale that may exist before the other two encoders, and add the fused features to the decoder at the same level. Further, the application integrates a learnable Sobel filter into the encoder part of the model to enhance the edge information in the image, and introduces an auxiliary edge loss function to make the restored image closer to the true value image in the high-frequency detail part. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0026] Figure 1 is the overall structure diagram of the deblurring model Efftformer, each Transformer block is composed of multiple FDSA and FFDN;

[0027] Figure 2 is the structure diagram of FDSA (frequency-domain self-attention mechanism);

[0028] Figure 3 is the structure diagram of FFDN (feedforward network);

[0029] Figure 4 is the structure diagram of the cross-connection channel attention module;

[0030] Figure 5 are four different Sobel filters, including: vertical filter (a), horizontal filter (b) and diagonal filter (c, d);

[0031] Figure 6 These are visual comparison images of the present invention on the GoPro dataset. The comparison results show that the present invention has a better effect on deblurring object detail information.

[0032] Figure 7 These are visual comparison images of the present invention on the HIDE dataset. The comparison results show that the present invention has a better effect on deblurring details of people.

[0033] Figure 8 These are visual comparison images of the present invention on the RealBlur dataset. The comparison results show that the present invention can also achieve good deblurring effect in low light environment.

[0034] Figure 9 This is a visual comparison image of edges enhanced using the Sobel filter and the auxiliary edge loss function. Specific implementation methods

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the application; that is, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0036] This embodiment provides a motion image deblurring model based on Fourier transform, with particular attention to edge information and detail information in blurred images;

[0037] like Figure 1 As shown, given a blurred image I, the input RGB blurred image I is first converted into a low-order feature F0 using a 3×3 convolution. Where H×W is the spatial dimension and C is the number of channels. Next, the low-order feature F0 is processed through a three-stage symmetric encoder-decoder architecture to obtain the final deep feature X0; the three-stage encoder progressively reduces the spatial size while increasing the channel capacity, transforming the features into multi-scale features F0, F1, and F2 at different resolutions. Furthermore, in the encoder section, to enhance the edge information aggregated by the encoder, the blurred image I is convolved with Sobel to obtain the edge feature image. Edge image features are obtained after activation using the GeLU function. right Perform secondary downsampling to further... One downsampling result and The secondary downsampling results are concatenated with F0, F1, and F2, respectively, and input to the corresponding encoders. Furthermore, to more effectively recover the image, F1, F2, and F3 are input to the cross-connected channel attention module (CCAM) for adaptive feature fusion, resulting in three scale output features X1, X2, and X3. X3 is directly input into the first-level decoder. X1 and X2 are summed with the output features of their respective previous-level decoders and used as inputs for their corresponding decoders. The deep features output by the third-level decoder are...

[0038] The image X0 is processed by 3×3 convolution and then summed with the blurred image I to obtain the final restored image.

[0039] like Figure 2 As shown, assuming a given input feature F, After layer normalization, the normalized tensor Y is obtained. F , Furthermore, through formula Get query feature F q ; through form Obtain the key feature F k ; through form Obtain the value feature F v . These are the weights of a 1×1 convolution. These are the weights of a 3×3 depthwise convolution, where Q in the superscript represents the operation on the query feature, K represents the operation on the key feature, and V represents the operation on the value feature. Then, F... q and F k Applying Fast Fourier Transform, through Calculate F q and F k In the frequency domain, the correlation A, where, Represents the Fast Fourier Transform. This represents the inverse fast Fourier transform. This represents the conjugate transpose operation. It is expressed by the formula... Calculate the aggregated feature Attention, where The symbol represents the layer normalization operation, and ⊙ represents the element-wise multiplication operation. Finally, through... Perform a 1×1 convolution operation on the aggregated feature Attention. 1×1 (·) The result obtained is summed with the FDSA input feature F to obtain the output feature of the frequency domain self-attention block.

[0040] like Figure 3 As shown, let For FFDN input features, where The height, width and channel number of the feature are represented. The Relu residual flow implementation process on the left side includes: first, the tensor Y Z The fast Fourier transform is applied, then two 1x1 convolution layers are used, and a ReLU layer is added in the middle. After frequency selection by ReLU, the inverse Fourier transform is performed to directly learn the blur kernel level information from the blurred image:

[0041]

[0042] wherein represents the fast Fourier transform, represents the convolution operation, and represents a 1x1 convolution matrix in complex form, wherein 1 represents the first convolution operation and 2 represents the second convolution operation. ReLU(·) represents the Relu nonlinear transformation, represents the fast inverse Fourier transform operation, is the spatial domain output feature of the Relu residual flow.

[0043] Since not all high-frequency information and low-frequency information contribute to the recovery of a clear image, a learnable quantization matrix M is introduced in the middle stream of the feedforward network, and is learned through the inverse method of the JPEG (Joint Photographic Experts Group) compression algorithm, so as to determine which frequency information to retain. The middle stream of the FFDN can be represented by the following formula:

[0044]

[0045] wherein, and are the operation of unfolding the patch into a one-dimensional vector according to a certain order and the operation of folding the one-dimensional vector into a patch in the JPEG compression method, respectively, represents the element-wise multiplication operation, GEGLU(·) is the GEGLU nonlinear transformation, is the spatial domain output feature of the middle stream.

[0046] Finally, the output feature of the feedforward network FFDN is obtained by the following formula

[0047]

[0048] As Figure 4As shown, to bridge the gap between features at different scales, this invention proposes a cross-connectivity channel attention module (CCAM), which consists of a channel self-attention mechanism and a cross-connectivity feedforward network. This module bridges the semantic and resolution gaps between the encoder's feature outputs at different scales and further adds the processed features at different scales to the decoder, thereby improving the model's ability to perceive multi-scale information and solving the problem of feature misalignment between high-resolution encoder features and low-resolution decoder features. The 3-level encoder features F... i The input i∈{1,2,3} is fed into the CCAM, and a self-attention mechanism is applied along the feature dimension to implement channel self-attention. Since the features of the 3-level encoder are three features at different scales, F2 and F3 are first upsampled and mapped to F1 resolution, and then tensors Y are obtained through layer normalization operations. i If i ∈ {1, 2, 3}, then perform a join operation to obtain the join tensor Y. Σ =Concat(Y1; Y2; Y3), where Concat(·) is the concatenation operation. The query projection Q, key projection K, and value projection V are obtained by aggregating pixel-level cross-channel context 1×1 convolutions and 3×3 depthwise convolutions to emphasize channel-level spatial contexts, as shown in the following formula:

[0049]

[0050]

[0051]

[0052] Among them, Q i Let i ∈ {1, 2, 3}, representing the query projections of the level 1, level 2, and level 3 encoders, respectively. Matrix It is a 1×1 convolution, matrix It is a 3×3 depthwise convolution. It reshapes the projections of the query Q, key K, and value projection V, where the query projection Q... i After reshaping the tensor, we get After reshaping the tensor by the key projection K, we get The value is obtained by reshaping the tensor by projecting V. Remodeled and Transposed attention map A is generated through dot product interaction. i , i∈{1,2,3}, where which features should flow forward is controlled by a ReLU(·) nonlinear transformation before Softmax normalization. Furthermore, with Dot product generates attention features Attention(Q) i (K,V), expressed by the following formula:

[0053]

[0054] where α is a learnable scaling parameter that controls the and size of the dot product. To make the Softmax(·) function more numerically stable and amplify the difference between the largest and smallest input values, we use a shifted Softmax function: i ,

[0055] The different attention channels are divided into different "heads" and learn different attention features, where the number of heads in the model of the application is j, and the default value is 2. Multi-head attention is represented by the following formula:

[0056]

[0057] MultiHead(Q i ,K,V)=Concat(head i 1,…,head i j )W h (2.9)

[0058] where head i j j represents the jth head attention of the query projection Q i , and K i ,i∈{1,2,3} and its weight matrix, the key projection K j , and j V j , respectively represent the key projection V of the jth head and its weight matrix. By Concat(·) connecting all the head attentions, we get the multi-head attention MultiHead(Q i ,K,V), where W h is a learnable weight matrix.

[0059] Finally, the output feature of the channel attention mechanism is obtained through the residual connection

[0060]

[0061] where W p is a 1x1 convolution.

[0062] The input features of the cross-connection feedforward network are The tensor is obtained by a layer normalization operation The application uses two 1x1 convolutions and a nonlinear Gelu activation function as a gating unit, Directly through the gating unit The gating unit processes the following representation,

[0063]

[0064] Wherein, GELU(·) represents a GELU gating unit. Directly as X1 output, And After downsampling, X2 and X3 are obtained respectively, forming the output of the CCAM module.

[0065] The output feature X3 of the CCAM module is input to the first-level decoder, and the output features X1 and X2 of the CCAM module are added to the corresponding decoder output features, respectively, as the input of the next-level decoder.

[0066] As Figure 5 The four different Sobel filters are shown, including: vertical filter (a), horizontal filter (b) and diagonal filter (c, d). The application adds Sobel filters in the model, first performs Sobel convolution operation on the input image to obtain image features containing edge information, and then uses connection operation (concatenation) to effectively concatenate the image features on each level of the model encoder after GeLU function activation, to enhance the model's perception of edge information.

[0067] As Figure 6 The comparison results show that the application has better deblurring effect on object detail information. It is illustrated that the visual comparison of the application on the GoPro dataset has better effect than other methods, and from the figure, it can be seen that the license plate and chair details are recovered.

[0068] The model is evaluated on the HIDE dataset mainly containing human motion blur, and the test on the HIDE dataset is performed using the model trained on the GoPro dataset, to prove that the application has good generalization ability, as Figure 7 For processing of human blurred detail information, the deblurring image generated by the model of the application has better image quality compared with other methods, the application has better effect on recovery of the zipper and clothes of the image person than other methods, and further, the application can more clearly recover the feet of the person, which proves the effectiveness of the application on detail information from the visual effect.

[0069] The present application further evaluates the present application on RealBlur dataset, including RealBlur-R test set and RealBlur-J test set. Figure 8 Visual comparison of different deblurring methods is shown, blurred license plate in weak light environment, the present application shows better recovery effect.

[0070] As shown in Figure 9 The present application compares the effect of Sobel convolution layer and edge loss function on edge enhancement. Through visual comparison, it can be observed that the present application obtains clearer car edge structure after adding Sobel convolution layer and edge loss function, and to some extent makes the recovered image closer to the target true value image.

[0071] In this application, the terms "comprising", "containing" or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or equipment including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles or equipment.

[0072] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A Fourier transform-based motion image deblurring model, characterized in that: a three-level symmetric encoder-decoder architecture is set in the deblurring model, the encoder processes the input blurred image, extracts its important features and converts them into a low-dimensional representation, and the decoder receives the low-dimensional features from the encoder and gradually restores the low-dimensional representation to a high-dimensional representation; a self-attention mechanism FDSA is used in the encoder and the decoder to estimate the attention of scaled dot product in the frequency domain; a feedforward network FFDN is used in the encoder and the decoder to fuse blurred kernel-level information and pixel-level information, and a learnable quantization matrix M is introduced to determine which frequency information to retain; a cross-connection channel attention module CCAM is used as an information exchange component between the encoder and the decoder; the encoder is connected with a Sobel filter to enhance edge features; The content loss function L count , the multi-scale frequency reconstruction loss function L MSFR and the auxiliary edge loss function L edge are used in combination to comprehensively consider the perceptual quality and frequency domain characteristics of the image, and guide the model to focus on restoring the high-frequency components in the blurred image. In the self-attention mechanism FDSA, F q , F k and F v are obtained by first passing through a 1x1 point convolution and a 3x3 deep convolution, where F q is the query feature, F k is the key feature, and F v is the value feature; then, the estimated features F q and F k are applied with fast Fourier transform, thereby obtaining the correlation of F q and F k in the frequency domain; the way of fusing blurred kernel-level information and pixel-level information is as follows: after the ReLU operation of the blurred image in the frequency domain, the blurred pattern is generated by selecting the frequency through inverse Fourier transform; the pixel-level information is learned through the intermediate stream of the feedforward network FFDN, and the blurred kernel-level information is learned through the ReLU residual stream of the feedforward network FFDN, and the generated blurred kernel information and the pixel-level information are fused; the feedforward network FFDN performs layer normalization on the input features to obtain a layer normalization tensor; the implementation process of the intermediate stream is as follows: the layer normalization tensor is first passed through a 1x1 convolution, the frequency domain features are obtained by applying a fast Fourier transform to the obtained features, then a learnable quantization matrix M is introduced, and through inverse learning of the JPEG compression algorithm, it is determined which frequency information should be retained; Specifically: the frequency domain features are multiplied with the quantization matrix M element by element, and the pixel-level features are restored through inverse fast Fourier transform; the formula of the intermediate stream of the FFDN is as follows: wherein, and are an operation of unfolding a patch into a one-dimensional vector in a certain order and an operation of folding a one-dimensional vector into a patch, respectively, in a JPEG compression method, denotes an element-wise multiplication operation, is GEGLU a non-linear transformation, is an intermediate stream spatial domain output feature; the implementation process of the ReLU residual stream is as follows: the fast Fourier transform is applied to the layer normalization tensor; then two 1x1 convolution layers are used, and a ReLU layer is added in the middle, the frequency selection is inverse Fourier transformed through ReLU, and the kernel-level features are directly learned from the blurred image; the formula of the FFDN residual stream is as follows: wherein denotes a fast Fourier transform, denotes a convolution operation, and denotes a 1x1 convolution matrix in complex form, wherein 1 denotes a first convolution operation and 2 denotes a second convolution operation; denotes a Relu non-linear transformation, denotes an inverse fast Fourier transform operation, is a spatial domain output feature of the Relu residual flow.

2. The motion image deblurring model based on Fourier transform according to claim 1, characterized in that: Given a blurred image I, first, the blurred image I is converted into low-order features by using a 3x3 convolution , , where HxW is the spatial dimension, and C is the number of channels; next, the low-order features Deep features are obtained through a three-level symmetric encoder-decoder architecture The three-level encoder scales down the spatial size while expanding the channel capacity, converting the features into multi-scale features with different resolutions , , .

3. The motion image deblurring model based on Fourier transform according to claim 2, characterized in that: The multi-scale features are fused The input is input into a cross-connection channel attention module CCAM for adaptive feature fusion to obtain three scale output features , , , The features are directly input into a first-level decoder, and the features , , are added to the output features of the previous decoder respectively as the input of the corresponding decoder, and the deep features output by the third-level decoder are added to the blurred image I after 3*3 convolution processing to obtain the final restored image .

4. The motion image deblurring model based on Fourier transform according to claim 1, characterized in that, Features of a Level 3 Encoder After inputting into the Cross-Connection Channel Attention Module (CCAM), firstly... Upsampling mapping is Resolution, and then tensors are obtained by layer normalization operations respectively. Then perform a connection to obtain the connection tensor. ; Connecting tensors The query projection Q, key projection K, and value projection V are obtained by aggregating pixel-level cross-channel context 1×1 convolutions and 3×3 depthwise convolutions for emphasizing channel-level spatial context. The query projection Q, key projection K, and value projection V are then reshaped to obtain... , and ; after reshaping and Generate a transposed attention map through dot product interactions. , The size of α is controlled by a learnable parameter α, where which features flow forward is controlled by a ReLU(•) nonlinear transformation before Softmax normalization; normalized features and dot product generates attention features , as follows: .

5. The motion image deblurring model based on Fourier transform according to claim 4, characterized in that, The cross-connection channel attention module CCAM divides different attention channels into different "heads" while learning different attention features; the multi-head attention features are subjected to 1x1 convolution and added to the three-level encoder features to obtain the output of the channel attention mechanism in the CCAM, which is expressed by the following formula: ; The input features of the cross-connected feedforward network are normalized by a layer normalization operation to obtain a tensor The cross-connected feedforward network uses two 1x1 convolutions and a nonlinear Gelu activation function as a gating unit, and adds the input features of the cross-connected feedforward network to obtain the output of the CCAM, as follows: at last, direct as Output, and After downsampling, the following results were obtained: and This forms the output of CCAM.

6. The motion image deblurring model based on Fourier transform according to claim 1, characterized in that, The Sobel filter enhances edge features by performing a Sobel convolution operation on the input blurred image I to obtain a feature image containing edge information. Edge image features are obtained after activation by the GeLU function. ,right Perform secondary downsampling, , One downsampling result and The secondary downsampling results were compared with multi-scale features at different resolutions. The input is connected to the corresponding encoder, thereby enhancing the model's perception of edge information.

7. The motion image deblurring model based on Fourier transform according to claim 1, characterized in that, the total loss function formula is as follows: wherein, is a multiscale frequency reconstruction loss function weight, is an auxiliary edge loss function weight; the auxiliary edge loss function is based on a Sobel filter; the Sobel filter includes horizontal and vertical Sobel convolution kernels, and the convolution kernel is decomposed into the product form of an averaging convolution kernel and a differential convolution kernel, wherein the averaging convolution kernel smooths the image, and the differential convolution kernel detects the edge of the image; in order to restore the clear image, the distance between the output result image and the true value image in the Sobel edge space is minimized; therefore, the auxiliary edge loss function is represented by the following formula: wherein, and denote the output result image and the ground truth image, respectively, and N is the total number of pixel elements.

Images