An image deblurring method based on channel attention and cross-scale feature fusion

Abstract

The present invention discloses an image deblurring method based on channel attention and cross-scale feature fusion, comprising: constructing a training set, a validation set and a test set according to a publicly available blurred image data set; constructing an image deblurring network model based on channel attention and cross-scale feature fusion, and using the constructed training set to supervise the model to obtain a converged deblurring network model; and evaluating the deblurring performance of the network model proposed in the present invention on the constructed blurred image test set. The method proposed in the present invention achieves better deblurring performance than algorithms such as DMPHN. The present invention achieves a higher peak signal-to-noise ratio and structural similarity while reducing the overall network parameter amount and greatly reducing the forward reasoning time, thereby improving the image deblurring effect.

Description

Technical Field

[0001] The present invention relates to an image deblurring method, and in particular to an image deblurring method based on channel attention and cross-scale feature fusion. Background Art

[0002] Digital images are an important information carrier in the information age. However, they are easily affected by a variety of destructive factors during the process of acquisition, dissemination and storage, which in turn causes the quality of digital images to deteriorate. Blurring is one of the main characteristics. Image deblurring technology has therefore become a research hotspot in computer vision and image processing, and is widely used in various professional fields such as criminal investigation, traffic management, security monitoring and medical imaging. In addition, image deblurring technology can also be used as a preprocessing process for other computer vision tasks to improve the actual performance of these tasks in specific scenarios. Therefore, the study of digital image deblurring is of great significance and value.

[0003] Traditional image deblurring algorithms mainly model the image blurring process as a linear convolution model, and their solution model is mainly based on the Maximization of Posterior Probability Model (MAP) and Maximum Likelihood Estimation Model (MLE) to solve a minimized energy function. However, traditional deblurring algorithms are limited to limited scene types under the condition of spatial uniform blur, the image prior used has poor robustness, and the optimization model introduces non-convex and nonlinear constraint functions, which leads to large optimization process calculation and slow restoration speed, making it difficult to meet the real-time requirements in actual use.

[0004] With the great success of deep neural networks in the field of computer vision, many studies have proposed applying deep neural networks to image deblurring. Considering that most real scenes are complex spatial non-uniform blur scenes, it is difficult to establish a suitable degradation model for them. Therefore, researchers try to use the powerful fitting ability of deep neural networks to directly learn a nonlinear mapping function from blurred images to clear images. This type of method is often used to solve complex spatial non-uniform deblurring problems (such as dynamic scene blur) and has achieved good restoration effects. The current image deblurring algorithms based on deep neural networks mainly include multi-scale convolutional neural networks, multi-image block hierarchical networks, deep neural networks based on generative adversarial networks, and deep neural networks based on image priors. However, the existing image deblurring algorithms based on deep neural networks have a large number of overall network parameters, and the forward reasoning time is still long with the help of GPU high-performance computing. There is a lack of effective transmission and fusion of intermediate feature information of different scales in the network model, which ultimately leads to limited improvement in deblurring performance. When faced with complex and changeable spatial non-uniform blur, the restoration effect of this type of algorithm is still limited. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an image deblurring method based on channel attention and cross-scale feature fusion, reduce the overall network parameter amount, shorten the network forward reasoning time, and solve the technical problems that the existing image deblurring algorithm does not have higher practical application value and the existing image deblurring algorithm based on deep neural network has limited deblurring performance improvement.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An image deblurring method based on channel attention and cross-scale feature fusion, the method comprising:

[0008] Step S1, obtaining a blurred image data set, and then dividing it into a training set, a validation set, and a test set;

[0009] Step S2, constructing an image deblurring network model based on channel attention and cross-scale feature fusion, the image deblurring network model includes: an encoder module, a decoder module, a supervision module and a cross-scale feature fusion module, and the four modules all include a channel attention residual module; wherein the encoder module takes the blurred image as input, and takes the extracted multiple intermediate feature maps and the output feature map as output; the decoder module takes the output of the decoder module as input, and in the encoder module and the decoder module, a jump connection is set between the feature layers of the same resolution to reduce the learning difficulty and training difficulty of the encoding and decoding network; the supervision module is set in the feature layers of different scales of the encoder module and the decoder module to supervise the reconstructed images at different scales; the cross-scale feature fusion module performs step-by-step cross-scale feature fusion on the shallow features and the decoding features of different scales, and finally outputs the reconstructed deblurred image;

[0010] Step S3, supervise the image deblurring network model constructed in step S2 through the training set obtained in step S1, and then obtain a converged network model. The converged network model takes the blurred image as input and outputs a reconstructed deblurred image through forward operation.

[0011] Furthermore, the step S1 specifically includes:

[0012] Get the public blurred image dataset GOPRO;

[0013] The GOPRO dataset is then divided into a training set, a validation set, and a test set, each of which includes 2000 pairs of training images, 103 pairs of validation images, and 1111 pairs of test images.

[0014] At the same time, the public blurred image dataset HIDE is obtained, and the divided 2025 pairs of test images are used as the test set.

[0015] Furthermore, the encoder module includes: a channel attention residual module and a feature downsampling layer; wherein, when the input image is input into the encoder module, first, it passes through two simple convolutional layers, 12 channel attention residual modules with 64 feature channels in sequence to obtain an intermediate feature map encoded_feats1; then, the encoded_feats1 passes through a feature downsampling layer with a sampling factor of 0.5 and 12 channel attention residual modules with 96 feature channels to obtain an intermediate feature map encoded_feats2; finally, the encoded_feats2 passes through a feature downsampling layer with a sampling factor of 0.5 and 12 channel attention residual modules with 128 feature channels to obtain the output feature map encoded_feats3 of the encoder module.

[0016] Furthermore, the decoder module also includes multiple attention residual modules and feature upsampling layers at different scales; wherein the output feature map encoded_feats3 is used as the input of the decoder module. First, the input passes through 12 channel attention residual modules with a feature channel number of 128 in sequence to obtain the intermediate decoded feature feats3; then, the feats3 passes through a feature upsampling layer with a sampling factor of 2, wherein the feature upsampling layer reduces the number of feature channels to 96 in a 1×1 convolution manner to obtain an intermediate feature map decoded_feats3, which is compared with the encoder module The encoded_feats2 in the block is added pixel by pixel, and then input into 12 channel attention residual modules with 96 feature channels to obtain the intermediate decoded feature feats2; finally, the feats2 passes through a feature upsampling layer with a sampling factor of 2, wherein the feature upsampling layer reduces the number of feature channels to 64 by a 1×1 convolution to obtain the intermediate feature map decoded_feats2, and the decoded_feats2 is added pixel by pixel with the encoded_feats1 in the encoder module, and then input into 12 channel attention residual modules with 64 feature channels to obtain the decoded output feature map F3.

[0017] Furthermore, the channel attention residual module includes a convolutional layer, a ReLU function layer, a convolutional layer and a channel attention layer connected in sequence; wherein the channel attention module includes a global average pooling layer, a fully connected layer, an activation function layer, a fully connected layer and a Sigmoid function layer connected in sequence.

[0018] Furthermore, in the supervision module, the following operations are performed:

[0019] First, the intermediate decoding feature feats3 is subjected to convolutional layer operation to obtain a residual image with 3 channels, which is different from the original blurred image I at the corresponding resolution. blur Perform residual connection to obtain the clear image I restored at this resolution deblur ;

[0020] Then, by using the real clear image I at this resolution sharp The restored clear image I deblur Performing supervised training, wherein the generation of the decoding feature map can be constrained by the local supervised training;

[0021] Finally, in order to strengthen the counter-effect of the restored image on the decoded feature map, the restored clear image I deblur After the convolution layer operation and the sigmoid function operation, a weight tensor for weighting the intermediate decoding feature feats3 is further generated; the intermediate decoding feature feats3 is multiplied by the weight tensor pixel by pixel, and the corrected decoding feature F1 is obtained through the residual connection output; similarly, the intermediate decoding feature feats2 is passed through the supervision module to obtain the corrected decoding feature F2;

[0022] Using this weighted correction method to correct the decoded features instead of directly inputting the restored image into a higher-level network can improve the efficiency of feature information transmission, strengthen and correct important decoded features, and thus improve the quality of the restored clear image.

[0023] Furthermore, the cross-scale feature fusion module includes: an upsampling layer, a convolution layer and a deep residual attention module; and the deep residual attention module includes: 4 channel attention residual modules, a convolution layer and a residual;

[0024] Among them, in the cross-scale feature fusion module, in order to fully utilize and fuse the input shallow feature information and the decoding feature information of different scales, the cross-scale feature fusion module adopts a step-by-step fusion method, which includes:

[0025] First, the lowest resolution corrected decoding features After the upsampling layer and convolution layer operations, the shallow features The corresponding pixels are added one by one, and then input into the RIRCAB module for full feature fusion and reconstruction.

[0026] Then, correct the decoding features After the upsampling layer and convolution layer operation, the intermediate feature F M1Add the corresponding pixels one by one and then input them into the RIRCAB module to get

[0027] Next, calibrate the decoding features After the upsampling layer and convolution layer operation, the intermediate feature F M2 Add the corresponding pixels one by one and then input them into the RIRCAB module to get

[0028] at last, After the convolution layer, a clear image is finally reconstructed.

[0029] Furthermore, the step S3 includes:

[0030] Step S301, firstly perform the same preprocessing operation on the blurred image and the real clear image for training, and then input the preprocessed blurred image blocks into the network model for training, wherein the preprocessing operation includes: firstly randomly cropping an image block with a resolution of 256×256 from the original image, then performing data enhancement processing of random horizontal flipping, random vertical flipping and random 90-degree rotation on the image block, and finally normalizing the image block pixels;

[0031] Step S302: The input blurred image is subjected to encoding and decoding operations and cross-scale feature fusion by the network model constructed in step S2 to reconstruct the blurred image;

[0032] Step S303, using a multi-scale content reconstruction loss function based on L1 loss to supervise the network model;

[0033] Step S304: During the training process, the Adam optimizer is used to perform gradient updates on the network parameters. The Adam optimizer uses the default parameters, beta1 = 0.9, beta2 = 0.999; the initial learning rate is set to 2×10 -4 , use Warmup to warm up the learning rate, and continuously decay the learning rate in a cosine annealing manner. When the peak signal-to-noise ratio and structural similarity on the validation set no longer increase, the network training stops, and the optimized network model is obtained;

[0034] Step S305: After the training is completed, a converged network model is obtained, and the network model performs forward operation on the normalized input blurred image to finally reconstruct a high-quality deblurred image.

[0035] Furthermore, the multi-scale content reconstruction loss function based on L1 loss is specifically:

[0036]

[0037] In the formula, k represents different scales, and S k Respectively represent the restored image and the real clear image output by the network, N k Indicates the total number of pixels in the image.

[0038] Furthermore, the converged network model obtained in step S3 is tested and evaluated by the following method, including:

[0039] Step S401, after pixel normalization of the test set image constructed in step S1, the image is input into the converged network model trained in step S3 to obtain a reconstructed deblurred image B;

[0040] Step S402: Calculate the peak signal-to-noise ratio according to the deblurred image B and the true clear image S. The formula is:

[0041]

[0042] In this formula, MAX s represents the maximum pixel value in the real clear image S, MSE represents the mean square error between the reconstructed deblurred image B and the real clear image S, and MSE is defined as follows:

[0043] Step S403: Calculate the structural similarity based on the deblurred image B and the real clear image S. The formula is expressed as:

[0044]

[0045] In this formula, μ x , μ y , σ xy Respectively represent the mean of x, the mean of y, the variance of x, the variance of y, and the covariance of x and y; c1=(k1L) 2 , c2=(k2L) 2 are two constants, L is the range of pixel values, which is 255, k1=0.01, k2=0.03 are the default values;

[0046] Step S404: The image deblurring effect of the algorithm is measured by the PSNR value and SSIM value of the deblurred image. The larger the PSNR and SSIM values ​​are, the better the quality of the restored image is. At the same time, the deblurring performance of the network model is comprehensively evaluated in combination with the size of the network model parameters and the length of the inference time.

[0047] The beneficial effects of the present invention are:

[0048] The present invention introduces a channel attention mechanism in the basic residual module, which can adaptively adjust the importance of different channel features, strengthen important features, and thus help the network extract more effective features; the cross-scale feature fusion module gradually fuses shallow features with decoding features of different scales, fully integrating global contour information and local detail information, thereby improving the quality of restored images; the multi-layer supervision module outputs restored images at different scales and supervises training, adds additional constraints to intermediate decoding features of different scales, and promotes the effective transfer of information between multi-scale features by correcting decoding features. Compared with classic image deblurring algorithms such as DMPHN. The present invention reduces the amount of network parameters, greatly shortens the forward reasoning time, and obtains a higher peak signal-to-noise ratio and structural similarity. It is a high-performance image deblurring algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of an image deblurring method based on channel attention and cross-scale feature fusion provided in Example 1;

[0050] Figure 2 A structural diagram of the image deblurring network model based on channel attention and cross-scale feature fusion provided in Example 1;

[0051] Figure 3 It is a structural diagram of the channel attention residual module ResCAB provided in Example 1;

[0052] Figure 4 This is a structural diagram of the supervision module provided in Example 1;

[0053] Figure 5 This is a structural diagram of the cross-scale feature fusion module provided in Example 1;

[0054] Figure 6 A flowchart of network model training and prediction provided in Example 1;

[0055] Figure 7 The deblurred image effect comparison between the method of Example 1 and the current main image deblurring algorithms. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Example 1

[0058] See also Figure 1-Figure 7 This embodiment provides an image deblurring method based on channel attention and cross-scale feature fusion. The specific process of the method is as follows: Figure 1 As shown, the method includes:

[0059] Step S1, obtaining a blurred image data set, and then dividing it into a training set, a validation set, and a test set;

[0060] Specifically, in step S1, the acquired blurred image dataset is a public blurred image dataset GOPRO;

[0061] The dataset GOPRO is then divided into a training set, a validation set, and a test set, each of which includes 2000 pairs of training images, 103 pairs of validation images, and 1111 pairs of test images. At the same time, the public blurred image dataset HIDE is obtained, and the 2025 pairs of test images that have been divided are used as the test set.

[0062] Step S2, construct an image deblurring network model based on channel attention and cross-scale feature fusion, the image deblurring network model includes: an encoder module, a decoder module, a supervision module and a cross-scale feature fusion module, and the four modules all include a channel attention residual module; wherein the encoder module takes the blurred image as input and takes the extracted multiple intermediate feature maps and the output feature map as output; the decoder module takes the output of the decoder module as input, and in the encoder module and the decoder module, a jump connection is set between the feature layers of the same resolution to reduce the learning difficulty and training difficulty of the encoding and decoding network; the supervision module is set in the feature layers of different scales of the encoder module and the decoder module to supervise the reconstructed images at different scales; the cross-scale feature fusion module gradually performs cross-scale feature fusion on the shallow features and the decoding features of different scales, and finally outputs the reconstructed deblurred image.

[0063] Specifically, in this embodiment, step S2 includes:

[0064] The above image deblurring network model has the following network structure: Figure 2 The image deblurring network model is a single-stage image deblurring network, which includes: an encoder module, a decoder module, a supervision module and a cross-scale feature fusion module, among which the Residual Channel Attention Block (ResCAB) is the core module and basic module of each part.

[0065] The above-mentioned encoder module includes: a channel attention residual module and a feature downsampling layer; wherein, when the input image is input into the encoder module, first, it passes through two simple convolutional layers, 12 channel attention residual modules with 64 feature channels in sequence to obtain an intermediate feature map encoded_feats1; then, the encoded_feats1 passes through a feature downsampling layer with a sampling factor of 0.5 and 12 channel attention residual modules with 96 feature channels to obtain an intermediate feature map encoded_feats2; finally, the encoded_feats2 passes through a feature downsampling layer with a sampling factor of 0.5 and 12 channel attention residual modules with 128 feature channels to obtain the output feature map encoded_feats3 of the encoder module.

[0066] The structure of the above decoder module is relatively symmetrical with that of the above encoder module, and it also includes multiple attention residual modules and feature upsampling layers at different scales; wherein, the above output feature map encoded_feats3 is used as the input of the decoder module. First, the input passes through 12 channel attention residual modules with a feature channel number of 128 in sequence to obtain the intermediate decoded feature feats3; then, the feats3 passes through the feature upsampling layer with a sampling factor of 2, wherein the feature upsampling layer reduces the number of feature channels to 96 in a 1×1 convolution manner to obtain the intermediate feature map decoded_feats3, which is decoded_feats3. 3 is added pixel by pixel with the encoded_feats2 in the encoder module, and then input into 12 channel attention residual modules with 96 feature channels to obtain the intermediate decoded feature feats2; finally, the feats2 passes through a feature upsampling layer with a sampling factor of 2, wherein the feature upsampling layer reduces the number of feature channels to 64 by a 1×1 convolution to obtain the intermediate feature map decoded_feats2, and the decoded_feats2 is added pixel by pixel with the encoded_feats1 in the encoder module, and then input into 12 channel attention residual modules with 64 feature channels to obtain the decoded output feature map F3.

[0067] Specifically, in this embodiment, encoded_feats2 and encoded_feats1 are used as inputs of the decoder module. This jump connection between the codec network enables the codec network to only learn the residual between the blurred image and the clear image, effectively reducing the training and learning difficulty of the network.

[0068] The above-mentioned supervision module takes the intermediate decoding features feats3 and feats2 in the decoder module as input. The supervision module supervises the reconstructed images at different scales, thereby strengthening the constraints on the intermediate decoding features of different scales, and at the same time promotes the effective transmission of information between multi-scale features by correcting the decoding features.

[0069] The above-mentioned cross-scale feature fusion module gradually performs cross-scale feature fusion on shallow features and decoded features of different scales, fully integrating global contour information and local detail information, thereby improving the quality of the restored image and finally outputting a reconstructed deblurred image.

[0070] Specifically, in this embodiment, the above-mentioned channel attention residual block (Residual Channel Attention Block, ResCAB) has a specific structure as follows Figure 3 As shown. The channel attention residual module abandons the batch normalization layer and introduces the channel attention module based on the residual design. More specifically, the channel attention residual module includes a convolution layer, a ReLU function layer, a convolution layer and a channel attention layer connected in sequence; wherein, the channel attention module includes a global average pooling layer, a fully connected layer, an activation function layer, a fully connected layer and a Sigmoid function layer connected in sequence. The embedding of the channel attention mechanism can weight the feature map after the convolution operation in the channel dimension, adaptively adjust the importance of different channel features, strengthen important feature information, and enhance the feature extraction capability of the network.

[0071] Specifically, in this embodiment, the above-mentioned supervision module has a specific structure as follows: Figure 4 As shown, in this module, the following operations are performed:

[0072] First, the intermediate decoding feature feats3 is processed through a convolutional layer to obtain a residual image with 3 channels. The residual image is different from the original blurred image I at the corresponding resolution. blur Perform residual connection to obtain the clear image I restored at this resolution deblur ;

[0073] Then, by using the real clear image I at this resolution sharp The restored clear image I deblur Performing supervised training, wherein the generation of the decoding feature map can be constrained by the local supervised training;

[0074] Finally, in order to strengthen the counter-effect of the restored image on the decoded feature map, the restored clear image I deblurAfter convolution layer operation and sigmoid function operation, a weight tensor for weighting the intermediate decoding feature feats3 is further generated; the intermediate decoding feature feats3 is multiplied by the weight tensor pixel by pixel, and the corrected decoding feature F1 is obtained through the residual connection output; similarly, the intermediate decoding feature feats2 is passed through the supervision module to obtain the corrected decoding feature F2.

[0075] Using this weighted correction method to correct the decoded features instead of directly inputting the restored image into a higher-level network can improve the efficiency of feature information transmission, strengthen and correct important decoded features, and thus improve the quality of the restored clear image.

[0076] Specifically, in this embodiment, the specific structure of the cross-scale feature fusion module is as follows: Figure 5 As shown in the figure, it is mainly composed of upsampling layer, convolution layer and deep residual attention module (Residual In Residual Channel Attention Block, RIRCAB).

[0077] The above-mentioned deep residual attention module RIRCAB consists of a 4-channel attention residual module ResCAB, a convolutional layer and a residual connection.

[0078] More specifically, in this module, in order to fully utilize and fuse the input shallow feature information and the decoded feature information of different scales, the cross-scale feature fusion module adopts a step-by-step fusion method, which includes:

[0079] First, the lowest resolution corrected decoding features After the upsampling layer and convolution layer operations, the shallow features The corresponding pixels are added one by one, and then input into the RIRCAB module for full feature fusion and reconstruction.

[0080] Then, correct the decoding features After the upsampling layer and convolution layer operation, the intermediate feature F M1 Add the corresponding pixels one by one and then input them into the RIRCAB module to get

[0081] Next, calibrate the decoding features After the upsampling layer and convolution layer operation, the intermediate feature F M2 Add the corresponding pixels one by one and then input them into the RIRCAB module to get

[0082] at last, After the convolution layer, a clear image is finally reconstructed.

[0083] The process is expressed in the formula:

[0084]

[0085]

[0086]

[0087] I deblur =conv4(F M3 )

[0088] In this formula group, RIRCAB i Represents the deep residual attention module operation, conv i represents the convolutional layer operation, represents pixel-by-pixel addition, ↑ represents upsampling layer operation, F i Represents the decoding features of different scales, F Mi represents the intermediate fusion feature, I deblur This gradual fusion method can effectively transmit and fuse feature information at different levels, achieve full fusion of global contour information and high-frequency detail information, and effectively improve the quality of the restored image.

[0089] Step S3, supervise the image deblurring network model constructed in step S2 by using the training set obtained in step S1, and then obtain a converged network model, which takes the blurred image as input and outputs a reconstructed deblurred image through forward operation;

[0090] Specifically, in this embodiment, the training and prediction process of the network model is as follows: Figure 6 As shown, it specifically includes:

[0091] Step S301, first perform the same preprocessing operation on the blurred image and the real clear image for training, and then input the preprocessed blurred image blocks into the network model for training, wherein the above preprocessing operation includes: first randomly cropping image blocks with a resolution of 256×256 from the original image, and then performing data enhancement processing of random horizontal flipping, random vertical flipping and random 90-degree rotation on the image blocks, and finally normalizing the image block pixels.

[0092] Step S302: The input blurred image is subjected to encoding and decoding operations and cross-scale feature fusion by the network model constructed in step S2 to reconstruct the blurred image;

[0093] Step S303, using a multi-scale content reconstruction loss function based on L1 loss to supervise the network model;

[0094] Step S304: During the training process, the Adam optimizer is used to perform gradient updates on the network parameters. The Adam optimizer uses the default parameters, beta1 = 0.9, beta2 = 0.999; the initial learning rate is set to 2×10 -4 , use Warmup to warm up the learning rate, and continuously decay the learning rate in a cosine annealing manner. When the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) on the validation set no longer increase, the network training stops, and the optimized network model is obtained.

[0095] Step S305: After the training is completed, a converged network model is obtained, and the network model performs forward operation on the normalized input blurred image to finally reconstruct a high-quality deblurred image.

[0096] More specifically, in this embodiment, step S303 includes the following steps:

[0097] The multi-scale content reconstruction loss function is the sum of the L1 losses between the restored image and the real clear image at each scale, expressed as:

[0098]

[0099] In the formula, k represents different scales, and S k Respectively represent the restored image and the real clear image output by the network, N k Indicates the total number of pixels in the image.

[0100] Step S4: Test the converged network model obtained in step S3 using the test set obtained in step S1, and evaluate the defuzzification performance of the proposed network model based on the test results.

[0101] Specifically, step S4 includes the following steps:

[0102] Step S401: After normalizing the pixels of the test set image constructed in step S1, the image is input into the converged network model trained in step S3 to obtain a reconstructed deblurred image B;

[0103] Step S402: Calculate the Peak Signal-to-Noise Ratio (PSNR) according to the deblurred image B and the true clear image S. The formula is:

[0104]

[0105] In this formula, MAX srepresents the maximum pixel value in the real clear image S (generally each pixel is represented by 8 bits of binary, so the maximum pixel value is 255), MSE represents the mean square error (MSE) between the reconstructed deblurred image B and the real clear image S, and MSE is defined as follows:

[0106] Step S403: Calculate the structural similarity (SSIM) based on the deblurred image B and the true clear image S. The formula is:

[0107]

[0108] In this formula, μ x , μ y , σ x 2 , σ xy They represent the mean of x, the mean of y, the variance of x, the variance of y, and the covariance of x and y respectively. Where c1 = (k1L) 2 , c2=(k2L) 2 are two constants, L is the range of pixel values, generally 255, k1=0.01, k2=0.03 are the default values.

[0109] Step S404: The image deblurring effect of the algorithm is measured by the PSNR value and SSIM value of the deblurred image. The larger the PSNR and SSIM values ​​are, the better the quality of the restored image is. At the same time, the deblurring performance of the network model is comprehensively evaluated by combining the size of the network model parameters and the length of the inference time.

[0110] It should be noted that the network model provided in this embodiment is not limited to the Pytorch framework, as long as the GORPO dataset can be trained and convergence can be achieved after several iterations during the training process and the deblurred image can be reconstructed in the end.

[0111] Table 1 is a comparison of the deblurring performance of the method of the present invention and the current main image deblurring algorithms on the GOPRO test set. From the test results in the table below, the method of the present invention achieved the highest PSNR value and SSIM value on the GOPRO test set, indicating that the deblurring effect ultimately achieved by the method of the present invention is the best. The PSNR value of the method of the present invention is still 0.19dB higher than the PNSR value of the SAPHN algorithm with the best performance among other algorithms in the table, indicating that the deblurring performance of the algorithm of the present invention has been greatly improved. In terms of the amount of network parameters, compared with the Stack(2)-VMPHN and SAPHN algorithms with better deblurring effects, the method of the present invention has smaller parameters and a lighter network. In terms of the time of the network forward operation, the running time of the method of the present invention has been greatly shortened, indicating that the method of the present invention is an efficient image deblurring algorithm.

[0112] Table 1 Comparison of deblurring performance of the method in Example 1 and different image deblurring algorithms on the GOPRO test set

[0113] Model PSNR SSIM Parameter quantity (M) Run time(s) DeepDeblur 29.23 0.916 11.7 4.33 Zhang et al. 29.19 0.931 37.1 1.4 SRN 30.26 0.934 6.8 1.87 DeblurGAN-v2 29.55 0.934 60.9 0.35 DMPHN 31.2 0.945 21.7 0.424 Stack(2)-VMPHN 31.5 0.948 21.7 0.552 SAPHN 31.85 0.948 23 0.34 Method of the present invention 32.04 0.958 15.7 0.059

[0114] Table 2 is a comparison of the deblurring performance of the method in Example 1 and different image deblurring algorithms on the HIDE test set. From the test results in the table, the algorithm proposed in the present invention achieves the highest PSNR and SSIM values, where the PSNR value is 1.16 dB higher than that of DMPHN, and the SSIM value is 0.022 higher than that of DMPHN, indicating that the algorithm proposed in the present invention also achieves better results in human perception deblurring and shows excellent generalization performance.

[0115] Table 2 Comparison of deblurring performance between the method in Example 1 and different image deblurring algorithms on the HIDE test set

[0116] Model PSNR SSIM DeblurGAN 24.51 0.871 SRN 28.36 0.915 DeblurGAN-v2 26.61 0.875 DMPHN 29.09 0.924 Method of the present invention 30.25 0.946

[0117] like Figure 7As shown, it is an example diagram comparing the deblurred image effect of the method of this embodiment 1 and the current main image deblurring algorithm. The example shows the deblurring effect of two images, where the large image on the left represents the original blurred image, and in order to more intuitively highlight the deblurring effect of the algorithm, the small image on the right shows the details of the original blurred image after deblurring by different algorithms. From left to right, the four columns of images represent the deblurred image details and the real clear image details of DeepDeblur, DMPHN and the method proposed in this embodiment 1. From the first image, the restoration result of the blurred wires in the image by this method is obviously better than the other two algorithms. The restoration result can distinguish different wires, while the other two algorithms cannot handle the blur artifacts of the wires, so the restoration effect is not good. From the deblurring results of the car body advertisement, the license plate and the car logo in the second image, the method can restore clearer contours and structures. The restored car body advertisement, license plate and car logo are all clearer, while DeepDeblur and DMPHN still have artifacts in the restoration of local details such as fonts and car logos. Therefore, the method proposed in the present invention can remove a large degree of blur and restore an image with clearer contours and more accurate details.

[0118] The matters not described in detail in the present invention are all known technologies to those skilled in the art.

[0119] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An image deblurring method based on channel attention and cross-scale feature fusion, characterized in that: The method comprises: Step S1, obtaining a blurred image data set, and then dividing it into a training set, a validation set, and a test set; Step S2, construct an image deblurring network model based on channel attention and cross-scale feature fusion, the image deblurring network model includes: an encoder module, a decoder module, a supervision module and a cross-scale feature fusion module, and the four modules all include a channel attention residual module; wherein the encoder module takes the blurred image as input and takes the extracted multiple intermediate feature maps and output feature maps as output; the decoder module takes the output of the encoder module as input, and a jump connection is set between the feature layers of the same resolution of the encoder module and the decoder module to reduce the learning difficulty and training difficulty of the encoding and decoding network; the supervision module reconstructs the output features of the decoder module at different scales to supervise and constrain the reconstructed images at different scales, and the features output by the decoder module are output after the supervision module. Corrected decoding features at different scales are used as input of the cross-scale feature fusion module; the cross-scale feature fusion module performs step-by-step cross-scale feature fusion on the intermediate feature information of different resolutions and the corrected decoding features, and finally outputs a reconstructed deblurred image; Step S3, supervise the image deblurring network model constructed in step S2 through the training set obtained in step S1, and then obtain a converged network model. The converged network model takes the blurred image as input and outputs a reconstructed deblurred image through forward operation.

2. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 1, characterized in that: The step S1 specifically includes: Get the public blurred image dataset GOPRO; The GOPRO dataset is then divided into a training set, a validation set, and a test set, each of which includes 2000 pairs of training images, 103 pairs of validation images, and 1111 pairs of test images. At the same time, the public blurred image dataset HIDE is obtained, and the divided 2025 pairs of test images are used as the test set.

3. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 1, characterized in that: The encoder module comprises: a channel attention residual module and a feature downsampling layer; wherein, when an input image is input into the encoder module, first, it passes through two simple convolutional layers, 12 channel attention residual modules with 64 feature channels in sequence to obtain an intermediate feature map encoded_feats1; then, the encoded_feats1 passes through a feature downsampling layer with a sampling factor of 0.5 and 12 channel attention residual modules with 96 feature channels to obtain an intermediate feature map encoded_feats2; finally, the encoded_feats2 passes through a feature downsampling layer with a sampling factor of 0.5 and 12 channel attention residual modules with 128 feature channels to obtain an output feature map encoded_feats3 of the encoder module.

4. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 3, characterized in that: The decoder module also includes multiple attention residual modules and feature upsampling layers at different scales; wherein the output feature map encoded_feats3 is used as the input of the decoder module. First, the input passes through 12 channel attention residual modules with a feature channel number of 128 in sequence to obtain the intermediate decoded feature feats3; then, the feats3 passes through a feature upsampling layer with a sampling factor of 2, wherein the feature upsampling layer reduces the number of feature channels to 96 in a 1×1 convolution manner to obtain the intermediate feature map decoded_feats3, which is compared with the decoded_feats3 in the encoder module. The encoded_feats2 is added pixel by pixel and then input into 12 channel attention residual modules with 96 feature channels to obtain the intermediate decoded feature feats2. Finally, the feats2 passes through a feature upsampling layer with a sampling factor of 2, where the feature upsampling layer reduces the number of feature channels to 64 by a 1×1 convolution to obtain the intermediate feature map decoded_feats2. The decoded_feats2 is added pixel by pixel with the encoded_feats1 in the encoder module and then input into 12 channel attention residual modules with 64 feature channels to obtain the decoded output feature map F3.

5. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 4, characterized in that: The channel attention residual module comprises a convolutional layer, a ReLU function layer, a convolutional layer and a channel attention layer connected in sequence; wherein the channel attention module comprises a global average pooling layer, a fully connected layer, an activation function layer, a fully connected layer and a Sigmoid function layer connected in sequence.

6. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 5, characterized in that: In the supervision module, the following operations are performed: First, the intermediate decoding feature feats3 is subjected to convolutional layer operation to obtain a residual image with 3 channels, which is different from the original blurred image I at the corresponding resolution. blur Perform residual connection to obtain the clear image I restored at this resolution deblur ; Then, by using the real clear image I at this resolution sharp The restored clear image I deblur Perform supervised training, where supervised training with images at different resolutions can constrain the generation of decoded feature maps; Finally, in order to strengthen the counter-effect of the restored image on the decoded feature map, the restored clear image I deblur After the convolution layer operation and the sigmoid function operation, a weight tensor for weighting the intermediate decoding feature feats3 is further generated; the intermediate decoding feature feats3 is multiplied by the weight tensor pixel by pixel, and the corrected decoding feature F1 is obtained through the residual connection output; similarly, the intermediate decoding feature feats2 is passed through the supervision module to obtain the corrected decoding feature F2; Using this weighted correction method to correct the decoded features instead of directly inputting the restored image into a higher-level network can improve the efficiency of feature information transmission, strengthen and correct important decoded features, and thus improve the quality of the restored clear image.

7. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 6, characterized in that: The cross-scale feature fusion module includes: an upsampling layer, a convolution layer and a deep residual attention module; and the deep residual attention module includes: 4 channel attention residual modules, a convolution layer and a residual; Among them, in the cross-scale feature fusion module, in order to fully utilize and fuse the input shallow feature information and the decoding feature information of different scales, the cross-scale feature fusion module adopts a step-by-step fusion method, which includes: First, the lowest resolution corrected decoding feature After the upsampling layer and convolution layer operations, the shallow features The corresponding pixels are added one by one, and then input into the RIRCAB module for full feature fusion and reconstruction. Then, correct the decoding features After the upsampling layer and convolution layer operation, the intermediate feature F M1 Add the corresponding pixels one by one and then input them into the RIRCAB module to get Next, calibrate the decoding features After the upsampling layer and convolution layer operation, the intermediate feature F M2 Add the corresponding pixels one by one and then input them into the RIRCAB module to get at last, After the convolution layer, a clear image is finally reconstructed.

8. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 7, characterized in that: The step S3 comprises: Step S301, firstly perform the same preprocessing operation on the blurred image and the real clear image for training, and then input the preprocessed blurred image blocks into the network model for training, wherein the preprocessing operation includes: firstly randomly cropping an image block with a resolution of 256×256 from the original image, and then performing data enhancement processing of random horizontal flipping, random vertical flipping and random 90-degree rotation on the image block, and finally normalizing the image block pixels; Step S302: The input blurred image is subjected to encoding and decoding operations and cross-scale feature fusion by the network model constructed in step S2 to reconstruct the blurred image; Step S303, using a multi-scale content reconstruction loss function based on L1 loss to supervise the network model; Step S304: During the training process, the Adam optimizer is used to perform gradient updates on the network parameters. The Adam optimizer uses the default parameters, beta1 = 0.9, beta2 = 0.999; the initial learning rate is set to 2×10 -4 , use Warmup to warm up the learning rate, and continuously decay the learning rate in a cosine annealing manner. When the peak signal-to-noise ratio and structural similarity on the validation set no longer increase, the network training stops, and the optimized network model is obtained; Step S305: After the training is completed, a converged network model is obtained, and the network model performs forward operation on the normalized input blurred image to finally reconstruct a high-quality deblurred image.

9. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 8, characterized in that: The multi-scale content reconstruction loss function based on L1 loss is specifically: In the formula, k represents different scales, and S k Respectively represent the restored image and the real clear image output by the network, N k Indicates the total number of pixels in the image.

10. The image deblurring method based on channel attention and cross-scale feature fusion according to claim 9, characterized in that: The converged network model obtained in step S3 is tested and evaluated by the following method, including: Step S401, after pixel normalization of the test set image constructed in step S1, the image is input into the converged network model trained in step S3 to obtain a reconstructed deblurred image B; Step S402: Calculate the peak signal-to-noise ratio according to the deblurred image B and the true clear image S. The formula is: In this formula, MAX s represents the maximum pixel value in the real clear image S, MSE represents the mean square error between the reconstructed deblurred image B and the real clear image S, and MSE is defined as follows: Step S403: Calculate the structural similarity based on the deblurred image B and the real clear image S. The formula is expressed as: In this formula, μ x , μ y , σ xy Respectively represent the mean of x, the mean of y, the variance of x, the variance of y, and the covariance of x and y; c1=(k1L) 2 , c2=(k2L) 2 are two constants, L is the range of pixel values, which is 255, k1=0.01, k2=0.03 are the default values; Step S404: The image deblurring effect of the algorithm is measured by the PSNR value and SSIM value of the deblurred image. The larger the PSNR and SSIM values ​​are, the better the quality of the restored image is. At the same time, the deblurring performance of the network model is comprehensively evaluated in combination with the size of the network model parameters and the length of the inference time.

Images