An image deblurring method based on U-Net network

Abstract

The application discloses an image deblurring method based on a U-Net network, and relates to the technical field of image processing; the method comprises the following steps: preparing a data set; constructing a network model; training the network model; optimizing the model and evaluating the performance; the application adds a multi-dense frequency domain self-attention network and a double-feature adaptive fusion network in the U-Net network. The multi-dense frequency domain self-attention network extracts features of different frequencies through multiple convolution layers and combines a self-attention mechanism to capture global long-range dependence information, which is helpful to generate higher-quality image recovery and reconstruction results in the decoding stage; the double-feature adaptive fusion network fuses feature maps obtained by a coarse and fine frequency domain feature evaluation module and feature maps obtained in the decoding stage, so that the decoder can better improve the deblurring accuracy and the quality of generated images, and the output image is clearer and more detailed.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image deblurring method based on a U-Net network. Background Technology

[0002] Image deblurring is a key technique in digital image processing, aiming to eliminate blur and improve sharpness and quality. Blur is usually caused by device motion, lens problems, or environmental factors, affecting information content and readability. Deblurring techniques improve image quality and are suitable for computer vision tasks such as object detection and image recognition.

[0003] This technology has wide applications in fields such as routine photography, medical imaging, satellite analysis, and security monitoring. In the medical field, clear images aid in accurate diagnosis; in security monitoring, they provide reliable identification capabilities. The development of deblurring technology is of great significance for improving image quality and analytical accuracy.

[0004] The Chinese patent publication number is "CN114549361B," and its patent title is "A Method for De-Motion Blurring of Images Based on an Improved U-Net Model." The improvements include the use of 3×3 convolutional kernels and the Leaky ReLU activation function. The encoder extracts features in four stages, combining depthwise separable convolution, residual convolution, and Haar wavelet transform. The decoder processes information in four stages, using skip connections and inverse wavelet transform. However, this method produces images with low contrast and unnatural colors, which do not conform to human visual perception. Furthermore, the multiple depthwise separable convolutions and residual convolutions may lead to parameter redundancy, increasing the risk of overfitting. Additionally, the lack of an attention mechanism fails to highlight key regions and enhances the model's ability to extract details. Summary of the Invention

[0005] The technical solution of this invention to solve the above-mentioned technical problems is to provide an image deblurring method based on a U-Net network, comprising the following steps:

[0006] Step 1, Prepare the datasets: Prepare three datasets for network training: Dataset 1, Dataset 2, and Dataset 3. Dataset 1 is the GoPro Dataset, which is expanded using methods such as random scaling, inversion, and translation to provide multi-scale information. Dataset 2 is the RealBlur Dataset used for model testing. Dataset 3... The Dataset dataset is used for model fine-tuning; each image is 512×512 pixels in size.

[0007] Step 2, Construct the network model: Construct a network model including U-Net network, no-reference evaluation model, stage fusion module, multi-dense frequency domain attention module, coarse and fine frequency domain feature evaluation module, convolutional prediction module and dual feature adaptive fusion module;

[0008] The U-Net network includes an encoder, a decoder, a multi-feature extraction network, a stage feature fusion module, a multi-dense frequency domain attention module, a coarse and fine frequency domain feature evaluation module, a dual-feature adaptive fusion module, and a convolutional prediction module. The encoder and decoder consist of multiple convolutional blocks. The multi-feature extraction network includes convolutional layers with different dilation rates to capture information at different scales. The stage feature fusion module allows for flexible interaction between features at different levels. The multi-dense frequency domain attention module enhances feature extraction capabilities for key regions. The coarse and fine frequency domain feature evaluation module is divided into low-frequency information extraction branches and high-frequency information extraction branches to extract low-frequency and high-frequency features respectively. The dual-feature adaptive fusion module fuses different features and enhances important features. The convolutional prediction module predicts the features of the deblurred image.

[0009] Step 3, train the network model: train the network model using the prepared dataset. During the training process, discrete wavelet transform and inverse wavelet transform are used for feature extraction and restoration, and the number of feature map channels is adjusted by 1×1 convolution to limit the model complexity.

[0010] Step 4: Optimize the model and evaluate its performance: The model is optimized using three loss functions: deblurring loss, encoder feature loss, and prediction loss. Peak signal-to-noise ratio and structural similarity are used as evaluation metrics to assess the model's performance.

[0011] Furthermore, in step 2, the encoder and decoder of the U-Net network are both composed of multiple convolutional blocks. Some convolutional blocks use 3×3 depth-separable residual convolution, while the remaining convolutional blocks are composed of 3×3 convolution, Leaky ReLU activation function, spatial attention, channel attention and 1×1 convolution.

[0012] Furthermore, in step 2, the multi-feature extraction network includes two 1×1 convolutional layers, two 3×3 convolutional layers, one convolutional layer with a dilation rate of 1, one convolutional layer with a dilation rate of 3, and an activation function. The network is divided into two branches. The 3×3 convolutions with different dilation rates are used to increase the receptive field. The feature map after the two branches are fused is then fused with the input image through residual connections after passing through a 1×1 convolution.

[0013] Furthermore, in step 2, the stage feature fusion module includes multiple 1×1 convolutional layers, max pooling, average pooling, BN layers, linear interpolation, and CBAM modules, which are used for feature fusion and enhancement of the feature maps obtained from the three stages of the encoder.

[0014] Furthermore, in step 2, the multi-dense frequency domain attention module includes a multi-scale frequency feature extraction module and a self-attention module, which are used to adaptively select important frequency information and improve feature representation capabilities.

[0015] Further, in step 2, the coarse and fine frequency domain feature evaluation module is divided into a low-frequency information extraction branch and a high-frequency information extraction branch. The low-frequency information extraction branch fuses the feature map after upsampling with the low-frequency-low-frequency feature map contained after frequency band decomposition. The high-frequency information extraction branch fuses the feature maps containing high-frequency-low-frequency, low-frequency-high-frequency, and high-frequency-high-frequency information after frequency band decomposition. Finally, the feature maps containing high-frequency and low-frequency information are fused and subjected to inverse wavelet transform, and then fused with the input feature map through a channel attention network to obtain the final output.

[0016] Furthermore, in step 2, the dual feature adaptive fusion module includes a convolutional layer, a 1×1 convolution, an activation function layer, a soft maximization layer, a linear transformation layer, and a normalization layer, which are used to fuse the feature map obtained by the coarse and fine frequency domain feature evaluation module and the feature map obtained by the decoder, and to enhance important features.

[0017] Further, in step 2, the convolutional prediction module includes a resizing layer, a 1×1 convolutional layer, an activation function, and a 3×3 convolutional layer, which are used to predict the deblurred image features and perform a loss with the quality score feature map of the no-reference evaluation model, so that the feature map obtained by the encoder approximates the quality score feature map of the no-reference evaluation model.

[0018] Furthermore, the image deblurring method based on the U-Net network also includes the following steps:

[0019] Step 5: Fine-tune the model: Input the dataset into the network model for training, compare the results with image evaluation metrics, and then fine-tune the model parameters;

[0020] Step 6, Save the model: Solidify the parameters of the final optimized U-Net network model, save the network model, and deploy it to the hardware system.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) This invention embeds image quality knowledge into the encoder and decoder, and dynamically adjusts the feature layer weights according to the image quality to adapt to different degrees of blur. Experiments show that the deblurred images produced by the network proposed in this paper are more consistent with the characteristics of human vision.

[0023] (2) This invention incorporates a multi-dense frequency domain attention module and a dual-feature adaptive fusion module into the U-Net network. The multi-dense frequency domain attention module extracts features of different frequencies through multiple convolutional layers and combines them with a self-attention mechanism to capture global long-range dependency information, which helps generate higher-quality image restoration and reconstruction results during the decoding stage. The dual-feature adaptive fusion module fuses the feature maps obtained from the coarse and fine frequency domain feature evaluation module with the feature maps obtained during the decoding stage, enabling the decoder to better improve the accuracy of deblurring and the quality of the generated image, resulting in a clearer and more detailed output image.

[0024] (3) This invention designs a coarse and fine frequency domain feature evaluation module, which extracts features from the feature map of the first stage of the decoder according to high-frequency information and low-frequency information, respectively obtaining the local details and overall structure of the image. The two are fused by convolution, and then restored by inverse wavelet transform (IDWT), finally outputting enhanced image features, which are then sent to the decoder and convolutional prediction network to improve the deblurring effect of the decoder and better predict the quality information of the feature map. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the steps of the image deblurring method based on the U-Net network of the present invention.

[0027] Figure 2 This is a schematic diagram of the network model of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of the U-net network of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of the feature fusion module of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of the multiple feature extraction network of the present invention;

[0031] Figure 6 This is a schematic diagram of the structure of the multi-dense frequency domain attention module of the present invention;

[0032] Figure 7 This is a schematic diagram of the coarse and fine frequency domain feature evaluation module of the present invention;

[0033] Figure 8This is a schematic diagram of the structure of the dual-feature adaptive fusion module of the present invention;

[0034] Figure 9 This is a schematic diagram of the structure of the convolution prediction module of the present invention;

[0035] Figure 10 This is a schematic diagram of the structure of the convolutional discrete wavelet transform of the present invention;

[0036] Figure 11 This is a schematic diagram of the inverse wavelet transform convolution structure of the present invention;

[0037] Figure 12 This is a schematic diagram of the structure of a convolutional block in the U-Net network of this invention;

[0038] Figure 13 This is a hardware schematic diagram of an image deblurring system based on an improved U-Net and quality-aware technology. Detailed Implementation

[0039] This invention proposes an image deblurring method based on U-Net network, aiming to improve the accuracy of deblurring and the quality of the generated image, making the output image clearer and more detailed.

[0040] The image deblurring method based on U-Net network proposed in this invention will be described below in specific embodiments:

[0041] In the technical solution of this embodiment, such as Figure 1 As shown, an image deblurring method based on a U-Net network includes the following steps:

[0042] Step 1, Prepare the datasets: Prepare three datasets for network training: Dataset 1, Dataset 2, and Dataset 3. Dataset 1 is the GoPro Dataset. It is expanded using methods such as random scaling, inversion, and translation to provide multi-scale information, enabling the model to extract effective features at different resolutions and improving its ability to handle blurred regions at different scales. Dataset 2 is the RealBlur Dataset, used for model fine-tuning to further optimize model performance. Dataset 3... The Dataset dataset is used to test the model and verify its generalization ability under various fuzzy conditions. Each image in each dataset is 512x512 pixels to ensure consistency in the network model's processing of different datasets and avoid instability in the training process caused by differences in image size.

[0043] Step 2, Construct the network model: Construct a network model including U-Net network, no-reference evaluation model, stage fusion module, multi-dense frequency domain attention module, coarse and fine frequency domain feature evaluation module, convolutional prediction module and dual feature adaptive fusion module;

[0044] The U-Net network includes an encoder, a decoder, a multi-feature extraction network, a stage feature fusion module, a multi-dense frequency domain attention module, a coarse and fine frequency domain feature evaluation module, a dual-feature adaptive fusion module, and a convolutional prediction module. The encoder and decoder consist of multiple convolutional blocks. The multi-feature extraction network includes convolutional layers with different dilation rates to capture information at different scales. The stage feature fusion module allows for flexible interaction between features at different levels. The multi-dense frequency domain attention module enhances feature extraction capabilities for key regions. The coarse and fine frequency domain feature evaluation module is divided into low-frequency information extraction branches and high-frequency information extraction branches to extract low-frequency and high-frequency features respectively. The dual-feature adaptive fusion module fuses different features and enhances important features. The convolutional prediction module predicts the features of the deblurred image.

[0045] Step 3, train the network model: use the dataset prepared in step (1) to train the network model. During the training process, discrete wavelet transform and inverse wavelet transform are used for feature extraction and restoration, and the number of feature map channels is adjusted by 1×1 convolution to limit the model complexity.

[0046] Step 4: Optimize the model and evaluate its performance: The model is optimized using three loss functions: deblurring loss, encoder feature loss, and prediction loss. Peak signal-to-noise ratio and structural similarity are used as evaluation metrics to assess the model's performance.

[0047] Step 5: Fine-tune the model: Input the dataset into the network model for training, compare the results with image evaluation metrics, and then fine-tune the model parameters;

[0048] Step 6, Save the model: Solidify the parameters of the final optimized U-Net network model, save the network model, and deploy it to the hardware system.

[0049] Furthermore, in step 2, the encoder and decoder of the U-Net network are both composed of multiple convolutional blocks. Some convolutional blocks use 3×3 depth-separable residual convolution, while the remaining convolutional blocks are composed of 3×3 convolution, Leaky ReLU activation function, spatial attention, channel attention and 1×1 convolution.

[0050] Furthermore, in step 2, the multi-feature extraction network includes two 1×1 convolutional layers, two 3×3 convolutional layers, one convolutional layer with a dilation rate of 1, one convolutional layer with a dilation rate of 3, and an activation function. The network is divided into two branches. The 3×3 convolutions with different dilation rates are used to increase the receptive field. The feature map after the two branches are fused is then fused with the input image through residual connections after passing through a 1×1 convolution.

[0051] Furthermore, in step 2, the stage feature fusion module includes multiple 1×1 convolutional layers, max pooling, average pooling, BN layers, linear interpolation, and CBAM modules, which are used for feature fusion and enhancement of the feature maps obtained from the three stages of the encoder.

[0052] Furthermore, in step 2, the multi-dense frequency domain attention module includes a multi-scale frequency feature extraction module and a self-attention module, which are used to adaptively select important frequency information and improve feature representation capabilities.

[0053] Further, in step 2, the coarse and fine frequency domain feature evaluation module is divided into a low-frequency information extraction branch and a high-frequency information extraction branch. The low-frequency information extraction branch fuses the feature map after upsampling with the low-frequency-low-frequency feature map contained after frequency band decomposition. The high-frequency information extraction branch fuses the feature maps containing high-frequency-low-frequency, low-frequency-high-frequency, and high-frequency-high-frequency information after frequency band decomposition. Finally, the feature maps containing high-frequency and low-frequency information are fused and subjected to inverse wavelet transform, and then fused with the input feature map through a channel attention network to obtain the final output.

[0054] Furthermore, in step 2, the dual feature adaptive fusion module includes a convolutional layer, a 1×1 convolution, an activation function layer, a soft maximization layer, a linear transformation layer, and a normalization layer, which are used to fuse the feature map obtained by the coarse and fine frequency domain feature evaluation module and the feature map obtained by the decoder, and to enhance important features.

[0055] Further, in step 2, the convolutional prediction module includes a resizing layer, a 1×1 convolutional layer, an activation function, and a 3×3 convolutional layer, which are used to predict the deblurred image features and perform a loss with the quality score feature map of the no-reference evaluation model, so that the feature map obtained by the encoder approximates the quality score feature map of the no-reference evaluation model.

[0056] Furthermore, in step 2, a quality-aware feature map is extracted using a no-reference image quality assessment model. Quality features are obtained through feature map alignment, and the quality features are compared with the features predicted by the predictor to constrain the encoder's learning. Image quality knowledge is integrated into the encoder. By reusing the encoder, the generated deblurred image is input back into the encoder, its features are extracted, and compared with the quality features of the sharp image. Image quality knowledge is embedded into the decoder, helping the model to pay more attention to quality when generating sharp images, ensuring that the output image is both sharp and has good visual effects.

[0057] Example 2:

[0058] An image deblurring method based on U-Net network includes the following steps:

[0059] Step 1, Prepare the dataset:

[0060] Dataset 1, the GoProDataset, is used for training the entire network model. It is augmented using methods such as random scaling, inversion, and translation to provide multi-scale information. Dataset 2, the RealBlur Dataset, is used for model fine-tuning. Dataset 3... The dataset is used for model testing. Each image is 512×512 pixels.

[0061] Step 2, Construct the network model:

[0062] The structure of the network model is as follows Figure 2 As shown, it includes a U-Net network, a no-reference evaluation model, a stage fusion module, a multi-dense frequency domain attention module, a coarse and fine frequency domain feature evaluation module, a convolutional prediction module, and a dual-feature adaptive fusion module.

[0063] Among them, the U-Net network, such as Figure 3 As shown, the U-Net network encoder and decoder consist of 19 convolutional blocks. Convolutional blocks 2, 4, 6, 8, 9, 12, 15, and 18 are 3×3 depthwise separable residual convolutions. The remaining convolutional blocks consist of one 3×3 convolution, a Leaky ReLU activation function, spatial attention, channel attention, and three 1×1 convolutions, as shown in the diagram. Figure 12As shown. In the encoding stage, features are extracted through convolutional blocks 1, 2, 3, 4, 5, 6, 7, and 8, and downsampled using wavelet transform. In the decoding stage, features are extracted through convolutional blocks 10, 11, 12, 13, 14, 15, 16, 17, 18, and 19, and upsampled using inverse wavelet transform to reduce information loss during reconstruction and improve deblurring. A multi-dense frequency domain attention module is used in the encoder-decoder connection process. Three stage feature fusion modules are added to the skip connection between the encoding and decoding stages. The stage feature fusion module is used for flexible interaction of features at different levels, capturing more details. The multi-dense frequency domain attention module is added in the fourth layer of U-Net to improve the feature extraction capability of key regions and reduce redundant information.

[0064] The U-Net network comprises a multi-feature extraction network, a decoder, a stage feature fusion network, a multi-dense frequency domain attention module, and a decoder. For example... Figure 5 As shown, the multi-feature extraction network comprises two 1×1 convolutional layers (convolutional layer 1 and convolutional layer 2), two 3×3 convolutional layers (convolutional layer 3 and convolutional layer 4), one convolutional layer with a dilation rate of 1 (convolutional layer 5), one convolutional layer with a dilation rate of 3 (convolutional layer 6), and an activation function. The function of these convolutional layers is to extract features from the input blurred image at different scales. To better capture information at different scales and enhance feature representation capabilities, the network is divided into two branches. The 3×3 convolutions with different dilation rates are used to increase the receptive field. The feature map fused from the two branches is then fused with the input image through a residual connection after passing through a 1×1 convolution. The feature extraction formula of the multi-feature extraction network is as follows:

[0065] F1 = Conv 3×3 (Conv 1×1 (Input))

[0066] F2 = Conv 3×3 (Conv 1×1 (Input));

[0067] F = RrLU(Conv) 1×1 (Cat(Conv 3×3,rate＝1 (F1),Conv 3×3,rate＝3 (F2)))+Input);

[0068] In the formula, F1 and F2 represent the feature maps after 1×1 and 3×3 convolutions, and F represents the final feature map.

[0069] like Figure 4As shown, the stage feature fusion module consists of seven 1×1 convolutional layers, four max pooling layers, three average pooling layers, three batch normalization (BN) layers, three linear interpolation (resize) layers, and three CBAM modules. Feature maps obtained from the three stages of the encoder are fed into the network through convolutional layers one, two, and three. Linear interpolation is used to unify the size of the feature maps. Each input feature map undergoes max pooling and average pooling to extract features at different scales. The pooled feature maps are then processed by the CRAM module to further enhance important features. Feature maps from different branches are fused through an addition operation to form a comprehensive feature map. The fused feature map is processed through multiple convolutional layers (convolutional layers four, five, six, and seven), with each convolutional layer followed by a BN layer to normalize the features. Finally, the feature maps from different paths are fused and output.

[0070] P max,i =MaxPool(F i );

[0071] P avg,i =AvgPool(F i );

[0072] A i =CBAM(P max,i ,P avg,i );

[0073] F fused =A1 + A2 + A3;

[0074] F4 = BN(Conv) 1×1 (F fused ));

[0075] G i =BN(Conv 1×1 (F4))×F i ;

[0076] G = G1 + G2 + G3;

[0077] In the formula P max,i With P avg,i This indicates that each branch undergoes max pooling and average pooling, and G is the final output.

[0078] like Figure 6As shown, the multi-dense frequency domain attention module consists of a multi-scale frequency feature extraction module and a self-attention module. The multi-scale frequency feature extraction module comprises three 1×1 convolutional layers (convolutional layer 1, convolutional layer 5, and convolutional layer 6), three 3×3 convolutional layers (convolutional layer 2, convolutional layer 3, and convolutional layer 4 with dilation rates of 1, 2, and 3 respectively), a Fourier transform, and a quantization matrix. In the multi-dense frequency domain attention module, the quantization matrix adaptively selects important frequency information, distinguishing which low-frequency and high-frequency information should be retained, thus improving feature representation capabilities. The obtained feature map is fused with the feature map obtained through the self-attention network to obtain a feature map containing frequencies at different scales. The self-attention module consists of one normalization layer, three linear transformation layers, four reshape layers, one softmax layer, and three 1×1 convolutional layers (convolutional layer 7 and convolutional layer 8). The attention module dynamically adjusts the region of interest by capturing long-range dependencies through a self-attention mechanism. It then fuses these features with a multi-scale frequency feature extraction module to obtain feature maps of frequency features at different scales. Finally, it adjusts the output channels and size using a single 1×1 convolutional layer (nine convolutional layers). The formula for the multi-dense frequency domain attention module is as follows:

[0079] Multi-scale frequency feature extraction module:

[0080] (1) Multi-scale extraction of convolutional layers:

[0081] F1 = FFT(F multi )×Quantization matrix;

[0082] F2 = IFFT(F6);

[0083] F3 = Conv 1×1 (F2);

[0084] (2) Self-attention part:

[0085]

[0086] Final output:

[0087] F = Conv 1×1 (F3+Conv 1×1 (F attention ));

[0088] like Figure 7As shown, the coarse and fine frequency domain feature evaluation module consists of seven 1×1 convolutional layers (convolutional layers 1, 2, 3, 4, 7, 10, and 12), five 3×3 convolutional layers (convolutional layers 5, 6, 8, 9, and 11), one upsampling layer, one pooling layer, one wavelet transform, and one frequency band decomposition layer. It is divided into a low-frequency information extraction branch and a high-frequency information extraction branch. The low-frequency information extraction branch fuses the upsampled feature map with the low-frequency-low-frequency feature map contained after frequency band decomposition, and further extracts low-frequency features through a residual network. The high-frequency information extraction branch fuses the feature maps containing high-frequency-low-frequency, low-frequency-high-frequency, and high-frequency-high-frequency information after frequency band decomposition, and further extracts high-frequency features through a residual network. After fusing the feature maps containing high-frequency and low-frequency information, an inverse wavelet transform is performed. It then passes through a channel attention network again, making the feature map emphasize both frequency and channel information, and finally fuses it with the input feature map to obtain the final output. The network expression formula is as follows:

[0089] Low-frequency information extraction branch:

[0090] F low =Upsample(Enhance(F t ));

[0091] F low_processed =GELU(Conv 3×3 (GELU(Conv 3×3 (Conv 1×1 (F low )))));

[0092]

[0093] In the formula F t The input feature map is the feature map after fusing the two input feature maps and passing it through a third convolutional layer. The final output feature map is the low-frequency information obtained after residual connection.

[0094] High-frequency information extraction branch:

[0095] F high =F h-h +F l-h +F h-l ;

[0096] F high_processed =

[0097] Conv 1×1 (GELU(Conv 3×3 (GELU(Conv 3×3 (Conv 1×1 (F high ))))));

[0098] F high_out =F high_processed +F high ;

[0099] In the formula F h-h F l-h and F h-l These represent the feature maps containing high-frequency-high-frequency, low-frequency-high-frequency, and high-frequency-low-frequency information after wavelet transform, respectively. F high_out The final output feature map is the high-frequency information obtained after residual connection.

[0100] Final output:

[0101] G output =Conv 1×1 (IDWT(F high_out +F low_out ))+F t ;

[0102] like Figure 8 As shown, the dual-feature adaptive fusion module consists of one 3×3 convolutional layer (convolutional layer three), four 1×1 convolutional layers (convolutional layer one, convolutional layer two, convolutional layer four kernel, and convolutional layer five), two activation function layers, two soft maximization layers, two linear transformation layers, and one normalization layer. The feature map obtained from the coarse and fine frequency domain feature evaluation module is upsampled and then fed into convolutional layer two. After segmentation, it is multiplied element-wise with the feature map obtained from the decoder to apply weights to different features, thereby enhancing important features and improving the decoder's expressive power and performance. The segmented feature map is then fed into another parallel branch consisting of convolutional layer four, a normalization layer, an activation function, and convolutional layer five, and added to another branch to extract features from different perspectives or levels and enrich feature information. Through linear transformation, the combination of activation functions and soft maximization achieves efficient integration of cross-modal information, ultimately yielding the fused feature map.

[0103] feature Figure 1 Feature extraction:

[0104] F1 = Softmax(Conv) 3×3 (Conv 1×1 (Input1)));

[0105] feature Figure 2 Feature extraction:

[0106] F2 = Split(Conv) 1×1 (Input2));

[0107] Feature map fusion and multiplication:

[0108] F3 = F1 × F2;

[0109] Residual branch:

[0110] F4 = Conv 1×1 (ReLU(BN(Conv 1×1 (F2))));

[0111] Feature map fusion and addition:

[0112] F5 = F3 × F4;

[0113] Final output:

[0114] F output =Softmax(Linear(ReLU(Linear(F5))));

[0115] like Figure 9 As shown, the convolutional prediction module consists of three reshape layers, four 1×1 convolutional layers (convolutional layer 1, convolutional layer 2, convolutional layer 3, and convolutional layer 4), one activation function, and one 3×3 convolutional layer (convolutional layer 5). It uses three feature maps obtained through coarse and fine frequency domain feature evaluation modules and two upsampling operations as input to the prediction network. After reshape and convolutional layers, these feature maps are fused. The fused feature map is then subjected to a 1×1 convolution, an activation function, and a 3×3 convolution to obtain the predicted feature map. The feature map obtained by the prediction network is compared with the quality score feature map of the no-reference evaluation model to make the feature map obtained by the encoder approximate the quality score feature map of the no-reference evaluation model, thus embedding image quality knowledge into the encoder. The formula for the convolutional prediction network is as follows:

[0116] F1 = Conv 1×1 (Reshape(Input1));

[0117] F2 = Conv 1×1 (Reshape(Input2));

[0118] F3 = Conv 1×1 (Reshape(Input3));

[0119] F4 = F1 + F2 + F3;

[0120] F output =Conv 3×3 (ReLU(Conv 1×1 (F4)));

[0121] In the formula, F1, F2, and F3 are the outputs of the three inputs after passing through the resizing layer and the convolutional layer, respectively. output To obtain the predicted feature map for the prediction network.

[0122] like Figure 12 As shown, convolutional blocks 1, 3, 5, 7, 11, 12, 14, 15, and 16 in the U-Net network consist of one 3×3 convolutional layer (convolutional layer 1), three 1×1 convolutional layers (convolutional layers 1, 2, and 3), one spatial attention module, one channel attention module, and one activation function layer. The input feature map is fused through the spatial and channel attention modules, then fused with the feature map after the 3×3 convolution, and finally passed through the Leaky ReLU activation function to obtain the output. By adding channel and spatial attention to the residual connections of the convolutional blocks, the encoder and decoder can focus on different channels and spatial locations of the feature map respectively, improving the model's ability to capture important features, enhancing the feature representation ability of the U-Net network, and improving model performance. The formula for expressing the convolutional block is as follows:

[0123] CA output =CA(Conv) 1×1 (Input));

[0124] SA output =SA(Conv 1×1 (Input));

[0125] F CA =CA output ×Input;

[0126] F SA =SA output ×Input;

[0127] F1 = Conv 1×1 (Cat(F CA ,F SA ));

[0128] F output =LeakyReLU(F1+Conv 3×3 (Input));

[0129] In the formula CA output and SA output F represents the output after spatial attention and channel attention. CA and F SA The F1 score represents the output after being fused with the input image following attention, and the F1 score represents the fusion of spatial attention and channel attention feature maps. output This represents the final output of the convolution block.

[0130] Step 3, train the network model:

[0131] The network model was trained using a dataset. During the training process, discrete wavelet transform and inverse wavelet transform were used for feature extraction and restoration. The number of feature map channels was adjusted by 1×1 convolution to limit the model complexity.

[0132] Discrete wavelet transform, such as Figure 10 As shown, the inverse wavelet transform is as follows: Figure 11 As shown, since DWT makes the number of channels in the output feature map four times that of the input feature map, the increase in the number of feature channels will lead to a significant increase in the number of parameters in subsequent network layers, which may increase the model complexity, resulting in overfitting and lower generalization ability, and also significantly increase resource consumption. Therefore, in order to be consistent with mainstream image restoration, this invention first uses a 1×1 convolution to halve the number of channels in the input feature map, and then applies DWT to limit the increase in the number of channels in the output feature map. There are two reasons for adding a 1×1 convolution after IWT: first, to match with the downsampling CDWT, matching the number of channels with the downsampling process, which helps the information correspondence process between CDWT and IWTC; second, to match the feature map size of each layer in the feature restoration stage with the layers in the feature extraction stage. Since this algorithm adds skip connections between the layers in the feature extraction stage and the feature restoration stage, by adding the feature map obtained in each layer of the feature extraction stage to the upsampling result of the previous layer in the corresponding feature restoration stage, and then inputting it into the network block of that layer for restoration, it is necessary for the feature map size of each layer in the feature restoration stage to match that of the feature extraction stage.

[0133] CDWT formula:

[0134] F CDWT =DWT(Conv 1×1 (F input ));

[0135] IWTC formula:

[0136] F IWTC =Conv 1×1 (IWT(F input ));

[0137] Step 4: Select a suitable loss function and the optimal evaluation metric;

[0138] To ensure network robustness, retain more image information, and fully extract image features, this invention selects three losses to optimize the model: the deblurring loss between the deblurred output image and the clear target image, the encoder loss between the encoder output features and the quality features extracted by the no-reference evaluation model, and the prediction loss between the deblurred image and the clear target image. All three losses are calculated using mean squared error.

[0139] Deblurring loss:

[0140] l = MSE(I out ,I gt );

[0141] Encoder feature loss:

[0142] l e =MSE(f p ,f k );

[0143] Predicted loss:

[0144] l d =MSE(f out ,f gt );

[0145] Total loss:

[0146] l t =λ1l+λ2l e +λ3l d ;

[0147] In the formula, λ1, λ2 and λ3 are the weight hyperparameters of each loss function, which are used to adjust the influence of different loss terms.

[0148] Peak signal-to-noise ratio (PSNR) and structural similarity were chosen as appropriate evaluation metrics. PSNR is an evaluation metric based on the error between pixels, used to sensitively assess image quality. Structural similarity, on the other hand, measures the similarity between images from three aspects: brightness, contrast, and structure, and is used to evaluate the degree of similarity between digital images. PSNR and structural similarity are defined as follows.

[0149]

[0150] MAX is the maximum possible pixel value of the image. I and K are the pixel values ​​of the original image and the distorted image, respectively, and M and N are the dimensions of the image.

[0151]

[0152] u x and u y It is the average of the x and y values ​​of the image. and σ is the variance of the image x and y. xy C1 and C2 are the covariances of the images x and y, and are constants used for stabilization to avoid zero denominators.

[0153] The training iterations were set to 200. The learning rate for the first 100 training iterations was set to 0.001, and the learning rate for the next 100 training iterations was gradually reduced from 0.001 to 0. The Adam optimizer was selected as the network parameter optimizer.

[0154] Step 5: Fine-tune the model: Input the dataset into the network model for training, compare the results with image evaluation metrics, and then fine-tune the model parameters;

[0155] Step 6, Save the model: Solidify the parameters of the final optimized U-Net network model, save the network model, and deploy it to the hardware system.

[0156] An image deblurring system based on an improved U-Net and quality-aware technology, such as Figure 13 As shown, the system is based on the Xilinx Zynq UltraScale+ MPSoC ZCU104 development board and consists of an FPGA hardware system, an HDMI display, and a computer. The FPGA system is divided into two parts: PS (Processing System) and PL (Programmable Logic). The PS loads the stored image data, U-Net model weights, and related parameters via an SD card and performs preprocessing operations such as normalization, resizing, and format conversion on the image data. The PL contains CNN IP cores and is responsible for performing efficient image deblurring inference tasks.

[0157] The PS (Power Sequencer) transfers preprocessed image data, model weights, and parameters from DDR memory to the PL (Programmer) via AXI-DMA. The CNN IP core in the PL consists of multiple functional modules, including a multi-dense frequency domain self-attention module and a dual-feature adaptive fusion module, used to improve detail recovery in blurred images. The CNN IP core is hardware-optimized, employing pipelined design and fixed-point quantization, enabling efficient inference tasks.

[0158] After inference, the PL (Programmer) transmits the results back to the PS (Power Controller) via an AXI-FIFO. The PS performs post-processing on the output image (denoising and format conversion) and displays the deblurring effect in real time via an HDMI monitor. The computer, as an auxiliary device, is responsible for writing the trained model and data to an SD card and supports remote monitoring and debugging.

[0159] This invention constructs an image deblurring method based on an improved U-Net and quality-aware approach. It achieves image deblurring in an end-to-end manner, eliminating the need for complex degradation models and avoiding research on different types of blur. Furthermore, hardware acceleration and data transmission technologies ensure real-time performance and processing efficiency in the deblurring process. In this invention, the PS (Power Position) end is responsible for control and data management, while the PL (Power Processing End) handles the high-load CNN inference task, forming a highly efficient image processing system. Under the same conditions, the feasibility and superiority of this method are further verified by calculating relevant image metrics compared to existing methods. Table 1 shows a comparison of relevant metrics between existing technologies (CN114549361B in the background) and the technical solution of Embodiment 2 of this invention. The table lists two commonly used metrics in deblurring networks: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM).

[0160] Table 1 Comparison of relevant indicators between existing technologies and invention proposal methods.

[0161]

[0162] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image deblurring method based on U-Net network, characterized in that, Includes the following steps: Step 1, Prepare the datasets: Prepare dataset 1, dataset 2, and dataset 3, where dataset 1 is the GoProDataset dataset; dataset 2 is the RealBlur Dataset dataset; and dataset 3 is the Köhler Dataset dataset. Step 2, Construct the network model: Construct a network model including U-Net network, no-reference evaluation model, stage fusion module, multi-dense frequency domain attention module, coarse and fine frequency domain feature evaluation module, convolutional prediction module and dual feature adaptive fusion module; The encoder and decoder consist of multiple convolutional blocks; the multi-feature extraction network contains convolutional layers with different dilation rates to capture information at different scales. The stage feature fusion module is used for flexible interaction of features at different levels; the multi-dense frequency domain attention module is used to improve the feature extraction capability of key regions; the coarse and fine frequency domain feature evaluation module is divided into low-frequency information extraction branch and high-frequency information extraction branch to extract low-frequency and high-frequency features respectively. The dual-feature adaptive fusion module is used to fuse different features and enhance important features; the convolutional prediction module is used to predict the features of the deblurred image. Step 3, train the network model: train the network model using the prepared dataset. During the training process, discrete wavelet transform and inverse wavelet transform are used for feature extraction and restoration, and the number of feature map channels is adjusted by 1×1 convolution to limit the model complexity. Step 4, optimize the model and evaluate its performance: the model is optimized using three loss functions: deblurring loss, encoder feature loss, and prediction loss, and the peak signal-to-noise ratio and structural similarity are used as evaluation metrics to evaluate the model performance. The multi-dense frequency domain attention module includes a multi-scale frequency feature extraction module and a self-attention module, which are used to adaptively select important frequency information and improve feature representation capabilities. The dual-feature adaptive fusion module includes a convolutional layer, a 1×1 convolution, an activation function layer, a soft maximization layer, a linear transformation layer, and a normalization layer. It is used to fuse the feature maps obtained by the coarse and fine frequency domain feature evaluation module and the feature maps obtained by the decoder, and to enhance important features.

2. The image deblurring method based on U-Net network according to claim 1, characterized in that, In step 2, the encoder and decoder of the U-Net network are both composed of multiple convolutional blocks. Some convolutional blocks use 3×3 depth-separable residual convolution, while the remaining convolutional blocks are composed of 3×3 convolution, Leaky ReLU activation function, spatial attention, channel attention and 1×1 convolution.

3. The image deblurring method based on U-Net network according to claim 1, characterized in that, In step 2, the multi-feature extraction network includes two 1×1 convolutional layers, two 3×3 convolutional layers, one convolutional layer with a dilation rate of 1, one convolutional layer with a dilation rate of 3, and an activation function. The network is divided into two branches. The 3×3 convolutions with different dilation rates are used to increase the receptive field. The feature map after the two branches are fused is then fused with the input image through residual connections after passing through a 1×1 convolution.

4. The image deblurring method based on U-Net network according to claim 1, characterized in that, In step 2, the stage feature fusion module includes multiple 1×1 convolutional layers, max pooling, average pooling, BN layers, linear interpolation, and CBAM modules, which are used for feature fusion and enhancement of the feature maps obtained from the three stages of the encoder.

5. The image deblurring method based on U-Net network according to claim 1, characterized in that, In step 2, the coarse and fine frequency domain feature evaluation module is divided into a low-frequency information extraction branch and a high-frequency information extraction branch. The low-frequency information extraction branch fuses the upsampled feature map with the low-frequency-low-frequency feature map contained after frequency band decomposition. The high-frequency information extraction branch fuses the feature maps containing high-frequency-low-frequency, low-frequency-high-frequency, and high-frequency-high-frequency information after frequency band decomposition. Finally, the feature maps containing high-frequency and low-frequency information are fused and subjected to inverse wavelet transform, and then fused with the input feature map through a channel attention network to obtain the final output.

6. The image deblurring method based on U-Net network according to claim 1, characterized in that, In step 2, the convolutional prediction module includes a resizing layer, a 1×1 convolutional layer, an activation function, and a 3×3 convolutional layer, which are used to predict the features of the deblurred image and perform a loss with the quality score feature map of the no-reference evaluation model, so that the feature map obtained by the encoder approximates the quality score feature map of the no-reference evaluation model.

7. The image deblurring method based on U-Net network according to claim 1, characterized in that, It also includes the following steps: Step 5: Fine-tune the model: Input the dataset into the network model for training, compare the results with image evaluation metrics, and then fine-tune the model parameters; Step 6, Save the model: Solidify the parameters of the final optimized U-Net network model, save the network model, and deploy it to the hardware system.

Images