Image deblurring method based on enhanced feature fusion mechanism

Abstract

The application discloses an image deblurring method based on an enhanced feature fusion mechanism, and comprises the following steps: step 1, establishing a data set, including a training set and a test set; step 2, establishing an image deblurring model based on the enhanced feature fusion mechanism, training according to the training set to obtain a trained image deblurring model; and step 3, inputting a blurred image into the trained image deblurring model to obtain a deblurred image; the method can solve the feature fusion effectiveness problem, the key feature loss problem and the feature redundancy problem in the image deblurring process, and realizes the image deblurring effect with details closer to reality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image deblurring method based on an enhanced feature fusion mechanism. Background Art

[0002] Image deblurring is a classic task in computer vision. Its goal is to restore a sharp image from a blurred input image. Blur can be caused by various factors, such as loss of focus, camera shake, or rapid object motion. Traditional deblurring algorithms include variational and differential methods, sparse image prior verification methods, gradient prior methods, and methods based on known blur kernels. These methods require prior conditions to reconstruct a sharp image, have limited application scenarios, and the restored image is not clear enough, making them insufficient in practical applications.

[0003] With the tremendous progress in deep learning in recent years, many networks only need to take blurred images as input and the corresponding clear images in reality as the desired output during training. By constructing a training set using image pairs, the blur kernel can be automatically learned during training, thus resolving the blur problem. Nah proposed an end-to-end multi-scale network that gradually restores a clear image in a coarse-to-fine manner. Zhang introduced a residual structure on an end-to-end basis and deepened the network, while also improving deblurring quality through spatial pyramid matching. Inspired by multi-stage networks, Zamir used a multi-scale patch model to utilize deblurring cues at different scales, passing residual feature cues to the next stage. He also used cross-stage feature fusion and supervised attention modules to strengthen feature processing, achieving excellent image deblurring results. MEGVII technology uses the HIN module based on Instance Normalization for image restoration. Based on the HIN module, a multi-stage network HINet was designed, achieving SOTA (state-of-the-art) results with reduced computational effort and inference time. The current multi-stage deblurring method based on deep learning also has certain limitations. First, there is a lack of effective feature fusion strategies, feature redundancy exists in the cross-stage feature fusion process, and feature information processing lacks specificity; second, there is a lack of effective information integration mechanism when image feature information is transmitted to the next stage, and the processing of information from the previous stage in the attention module lacks globality, resulting in the loss of many key features; in addition, the backbone network of the multi-stage deblurring method has a large number of channels, and there is a large amount of channel redundancy, resulting in performance loss. Summary of the Invention

[0004] The present invention aims to solve the problems existing in the prior art and provides an image deblurring method based on an enhanced feature fusion mechanism.

[0005] The technical solution adopted in the present invention is:

[0006] An image deblurring method based on an enhanced feature fusion mechanism comprises the following steps:

[0007] Step 1: Create a data set, including a training set and a test set;

[0008] Step 2: Establish an image deblurring model based on the enhanced feature fusion mechanism, train it according to the training set, and obtain the trained image deblurring model;

[0009] Step 3: Input the blurred image into the trained image deblurring model to obtain the deblurred image;

[0010] The image deblurring model based on the enhanced feature fusion mechanism is based on the HINet network structure, which includes an encoder, an enhanced feature fusion module, a supervised hybrid attention module and a decoder;

[0011] The encoder is used to extract image features and obtain feature maps;

[0012] The enhanced feature fusion module is used to perform feature fusion of stage 1 and stage 2 and skip connections between networks;

[0013] The supervised hybrid attention module is used to learn and integrate the features of stage 1 and pass them to stage 2;

[0014] The decoder is used to add the feature map to the original blurred image to generate the deblurred image.

[0015] Furthermore, the encoder module extracts the image as follows:

[0016] Convolution is performed on the input feature map, and the feature map channel is divided into two parts, one part is processed by IN, and the other part remains unchanged;

[0017] Combine the two parts to obtain a spliced ​​feature map;

[0018] The concatenated feature map is sequentially processed by PReLU activation function, convolution, and PReLU activation function;

[0019] Then it goes through the channel attention module;

[0020] The retained feature map is added to the initial input feature map after convolution to obtain the encoder module output;

[0021] The channel attention module performs global average pooling, convolution dimensionality reduction, PReLU activation function, convolution dimensionality increase, and sigmoid function processing on the feature map in sequence;

[0022] The decoding module has the same processing as the encoding module, excluding the process of dividing the feature map channels into equal parts.

[0023] Furthermore, the enhanced feature fusion module includes an EFF module for feature fusion and skip connections between networks;

[0024] The feature fusion calculation process is as follows:

[0025]

[0026]

[0027] Where: X is the input feature of the encoder in stage 1, Y is the input feature of the decoder in stage 1, E represents the weight operation extracted by global average pooling, one-dimensional convolution and Sigmoid function, is the element-wise product operation, is the initial feature fusion, and Z is the final feature fusion;

[0028] The weight extraction operation is as follows: after global average pooling, a dynamic convolution kernel is used to perform one-dimensional convolution, and then the weight of each channel is obtained through the Sigmoid activation function; the size of the dynamic convolution kernel is determined by the following formula:

[0029]

[0030] Where: k is the size of the convolution kernel, C is the number of channels, ||odd means that k can only be an odd number, γ and b are parameters.

[0031] Furthermore, the supervised hybrid attention module includes a SHAM module, and the processing process is as follows:

[0032] The output of stage 1 is used as input, and the dimensionality is reduced by convolution. The deblurred image is then added to the original blurred image to obtain the deblurred image.

[0033] The feature map is increased in dimension by convolution, and the feature map is processed as follows:

[0034] The first method is to perform maximum pooling and average pooling on the feature map in the channel dimension, compress it into two single-channel feature maps, splice the two single-channel feature maps, and convolute them into one to obtain the spatial attention weight map;

[0035] The second method is to perform global average pooling on the feature map to summarize the value of each feature map as an average value, and then generate a channel attention weight map through convolution, activation function, convolution, and Sigmoid function in sequence;

[0036] The convolved input image is multiplied by the spatial attention weight map and the channel attention weight map respectively, concatenated and convolved, and then added to the input feature map to enter stage 2.

[0037] Furthermore, the processing process of the stage 2 is as follows:

[0038] The output of the SHAM module is used as input and processed by the encoder, while accepting the fused multi-scale features output by the EFF module; then it is processed by the decoder.

[0039] Furthermore, the loss function Loss used in the image deblurring model training process based on the enhanced feature fusion mechanism is as follows:

[0040]

[0041] Where: PSNR is the peak signal-to-noise ratio, R i is the output of stage i, X i is the input of stage i, and Y is the real background image of each stage.

[0042] The beneficial effects of the present invention are:

[0043] (1) The enhanced feature fusion module in the present invention is responsible for the cross-stage feature fusion and the feature fusion of the jump connection part in the network. The module fully extracts the importance of different features through channel attention, uses one-dimensional convolution to improve the learning effect of channel interaction and reduce the growth of parameters, gives more weight to important input features through the soft selection weighted average structure, and uses an iterative quadratic weighted structure to break the performance bottleneck of feature fusion, thereby significantly enhancing the effectiveness of feature fusion;

[0044] (2) The supervised hybrid attention module in this invention uses different attentions to globally integrate the high-level features of the image, generating more effective weights, thereby more significantly suppressing unimportant feature information and further promoting key features to enter the next stage;

[0045] (3) In this invention, the encoder and decoder are constructed by enhanced semi-instance normalization and enhanced residual modules. SE attention is constructed in both modules to reduce channel redundancy. At the same time, PReLU is used to realize automatic learning and updating of activation function parameters, which is more adaptable to current large-scale networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the process of the present invention.

[0047] Figure 2 Schematic diagram of the image deblurring model structure constructed by the present invention.

[0048] Figure 3 Schematic diagram of the encoder and decoder structure in the present invention, a is the encoder module, b is the decoder module.

[0049] Figure 4 Schematic diagram of the enhanced feature fusion module structure in the present invention.

[0050] Figure 5 Schematic diagram of the supervised hybrid attention module structure in the present invention.

[0051] Figure 6 Schematic diagram of the deblurring effect in an embodiment of the present invention.

[0052] Figure 7 Schematic diagram comparing the test results of the method of the present invention and HINet in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 As shown, an image deblurring method based on an enhanced feature fusion mechanism is characterized by comprising the following steps:

[0055] Step 1: Create a dataset, including training and test sets. Prepare the GoPro-large dataset, which consists of 3214 blurred images (1280×720 pixels), 2103 of which are training images and 1111 are test images. This dataset consists of a one-to-one correspondence between real blurred images and ground truth clear images, all captured with a high-speed camera. Crop the training images to 512×512 pixels as the final training set, and then begin training.

[0056] Step 2: Establish an image deblurring model based on the enhanced feature fusion mechanism, train it according to the training set, and obtain the trained image deblurring model;

[0057] The image deblurring model based on the enhanced feature fusion mechanism is based on the HINet network structure, including an encoder, an enhanced feature fusion module, a supervised hybrid attention module and a decoder; the model structure is as follows Figure 2 shown.

[0058] The encoder is used to extract image features and obtain feature maps. The encoder has a U-Net structure and downsamples the feature maps multiple times, reducing the resolution of the feature maps while increasing the number of channels to extract image features.

[0059] The encoder adopts an enhanced semi-instance normalization model such as Figure 3 As shown, EHIN Block.

[0060] A 3×3 convolution is performed on the input feature map, dividing the feature map channel into two parts, one part is processed by IN, and the other part remains unchanged; the IN layer is used to correct the mean and variance of the features. Compared with the BN layer, the advantage of the IN layer is that the same normalization processing can be maintained during the training and inference stages.

[0061] Combine the two parts to obtain a spliced ​​feature map;

[0062] The concatenated feature map is then passed through a PReLU activation function, a 3×3 convolution, and a PReLU activation function for further feature extraction. PReLU iterates the parameters of the LeakyReLU as a learnable parameter, enabling the training of deeper and larger networks.

[0063] Then it passes through the channel attention module, which is used to eliminate channel redundancy caused by the high number of channels in the backbone network.

[0064] The retained feature map is added to the initial input feature map after 1×1 convolution to obtain the encoder module output;

[0065] The channel attention module performs global average pooling, convolution dimensionality reduction, PReLU activation function, convolution dimensionality increase, and sigmoid function processing on the feature map in sequence;

[0066] The decoder upsamples the feature map multiple times to increase the resolution of the feature map while reducing the number of channels, generating a 512×512×3 residual image at the end of a single stage. This image is added to the original blurred image to generate the deblurred image.

[0067] The decoding module structure is as follows Figure 3 As shown in (b), the process is the same as that of the encoding module, excluding the process of bisecting the feature map channels. The ERES module is used to build the decoder, which is responsible for generating more advanced detailed features.

[0068] The enhanced feature fusion module includes the EFF module for feature fusion and jump connections between networks; the network structure is as follows Figure 4 As shown; the location is as Figure 2As shown in the figure, this module has been added in six locations. The feature maps of inputs 1 and 2 are convolved for feature extraction, then added together for feature fusion. The resulting mixed features are then globally average pooled to produce a 1×1×C feature map. Next, to address the high number of channels in the input feature maps, EFF uses channel attention to obtain weights. This module applies a one-dimensional convolution and a sigmoid function to the feature maps to obtain weights for each channel, with output values ​​ranging from 0 to 1. EFF uses one-dimensional convolution with weight sharing to obtain channel attention, as this avoids dimensionality reduction and allows for better extraction of inter-channel dependencies. Next, to perform a weighted average of the features of inputs 1 and 2, the output value (α) of the previous attention function is used as the weight for input 1, and the output value (1-α) is subtracted from 1 as the weight for input 2. This achieves the effect of emphasizing input 1 and deemphasizing input 2, creating a softly selected weighted average. These two weights are multiplied by the corresponding input features and then added together to produce the fusion result, completing the initial feature fusion. To further overcome performance bottlenecks, EFF performs another one-dimensional convolution and normalization on the initial feature fusion results. It then performs a weighted average of the feature maps of inputs 1 and 2 to obtain the final output, completing the enhanced feature fusion. This iterative feature fusion approach provides a more complete understanding of the input feature maps and a secondary summary of the fusion weights, which positively impacts the final fusion result.

[0069] The feature fusion calculation process is as follows:

[0070]

[0071]

[0072] Where: X is the input feature of the encoder in stage 1, Y is the input feature of the decoder in stage 1, E represents the weight operation extracted by global average pooling, one-dimensional convolution and Sigmoid function, is the element-wise product operation, is the initial feature fusion, and Z is the final feature fusion.

[0073] The weight extraction operation is as follows: after global average pooling, a dynamic convolution kernel is used to perform one-dimensional convolution, and then the weight of each channel is obtained through the Sigmoid activation function; the size of the dynamic convolution kernel is determined by the following formula:

[0074]

[0075] Where: k is the size of the convolution kernel, C is the number of channels, ||odd means that k can only be an odd number, γ and b are parameters used to change the ratio between the number of channels C and the convolution kernel size, which are set to 2 and 1 respectively.

[0076] The Sigmoid function formula is:

[0077]

[0078] Where x represents the value of the matrix.

[0079] The main function of the EFF module is to eliminate feature redundancy, increase the weight of Input 1, and achieve better feature fusion performance. In cross-stage feature fusion, the EFF module connects Stage 1 and Stage 2 at the encoder. Because the features transmitted by the Stage 1 encoder are more important to the Stage 2 encoder than the decoder, the EFF module uses the encoder input as Input 1 and the decoder input as Input 2. The two EFF modules in this section preserve network information, making network optimization more stable, and using the multi-scale features of the previous stage to enrich the features of the next stage.

[0080] The present invention also inserts an EFF module into the jump connection part of the backbone network. The jump connection is responsible for transferring the features of the shallow network to the deep network to save low-level position information. However, for the decoder part, the effectiveness of extracting features in the shallow network is not as good as that in the deep network. Transferring too much low-level position information will affect the detailed information in the deep network. At the same time, there is a large amount of feature redundancy in the features transferred by the jump connection. Therefore, the present invention uses the EFF module, takes the feature map of the decoder part as input 1, and takes the features transferred by the encoder through the jump connection as input 2. The features of these two parts are fused and spliced ​​into the feature map of the backbone network. The purpose is to reduce the weight of the shallow features, increase the weight of the deep features, and eliminate feature redundancy in the process of feature fusion, thereby improving network performance.

[0081] The supervised hybrid attention module is used to learn and integrate the features of stage 1 and pass them to stage 2; structure diagram Figure 5 As shown, SHAM. Set the location in the network as Figure 2 shown.

[0082] The supervised hybrid attention module includes the SHAM module, and the processing process is as follows:

[0083] The output of stage 1 is used as input, and the dimensionality is reduced by convolution. The deblurred image is added to the original blurred image. The loss function is also calculated here.

[0084] The loss function used is based on the PSNR loss function. Before calculating PSNR, you need to know how to calculate MSE. Given a clean image I and a noisy image K of size m×n, the mean square error (MSE) is defined as follows:

[0085]

[0086] Where: (i, j) represents the pixel coordinates of the image.

[0087] The definition of PSNR is as follows:

[0088]

[0089] Among them MAX1 2 The maximum possible pixel value of the image. If each pixel is represented by 8 bits of binary, then it is 255.

[0090] The loss function is as follows:

[0091]

[0092] Where: PSNR is the peak signal-to-noise ratio, R i is the output of stage i, X i is the input of stage i, and Y is the real background image of each stage.

[0093] To address the problem that the Sigmoid function can only learn and weight pixel information and lacks global feature extraction capabilities, SHAM first increases the dimension of the feature map through convolution after obtaining the deblurred image, increases the number of channels, and then performs two different processes on the feature map with high channel numbers at the same time.

[0094] The first method is to perform maximum pooling and average pooling on the feature map in the channel dimension, compress it into two single-channel feature maps, splice the two single-channel feature maps, and convolute and compress them into a spatial attention weight map. Average pooling is used to learn the degree information of the target, maximum pooling is used to obtain the discriminative features of the object, and compression in the channel dimension is performed to summarize the global spatial features.

[0095] The second method is to perform global average pooling on the feature map to summarize the value of each feature map as an average value, and generate a channel attention weight map through convolution, activation function, convolution, and Sigmoid function in sequence; the purpose is to capture the channel dependency between any two channel maps and weight and update each channel.

[0096] The convolved input image is multiplied by the spatial attention weight map and the channel attention weight map, respectively, and then concatenated to further enhance feature representation. Convolution halves the number of channels, and the image is added to the input feature map to proceed to stage 2, completing the module. This design globally learns the spatial global features and channel importance of all feature maps, effectively enhancing feature information.

[0097] Phase 2 uses the output of the SHAM module as input and begins processing the image using the encoder. This process receives the fused multi-scale features output by the EFF module. Finally, the decoder produces the final result. Once the model is built, training can begin.

[0098] In the following examples, the experimental equipment uses a single NVIDIA V100 32G and the deep learning framework is PyTorch. During the training process, the Adam optimizer is used for training, and the initial learning rate is set to 2×10 -4 , and then gradually reduced to 1×10 -7 .

[0099] The loss function used in training is shown above. R1 and R2 are trained together in training, and R2+X2 in stage 2 is used as the final output during inference.

[0100] Step 3: Input the blurred image into the trained image deblurring model to obtain the deblurred image;

[0101] This invention is an improvement based on the existing HINet network, a paper proposed by Chen in 2021: HalfInstance Normalization Network (abbreviated as: HINet). HINet uses Instance Normalization as a feature enhancement method, constructs a HIN module for application in the field of image restoration, and achieves industry-leading performance for the first time through InstanceNormalization. It also designs a multi-stage network HINet based on the HIN module, which achieves SOTA while significantly reducing the amount of calculation and inference time. HINet is based on a multi-stage encoding and decoding structure network with a total of two stages. Each stage network consists of an encoder and a decoder. The encoder consists of four HIN Blocks (each stage consists of two HIN Blocks), which are responsible for extracting features. The decoder consists of four Res Blocks (each stage consists of two Res Blocks), which are mainly used to generate features.

[0102] In the encoder, HINet uses the IN layer to construct a HIN Block. The idea behind this is that IN can maintain the same normalization during training and inference, while also recalibrating the mean and variance of features to preserve more scale information. Compared to the Res Block, the HIN Block primarily splits the feature map channels after the 3×3 convolution into two parts: one half undergoes IN processing, while the other half retains contextual information. These two parts are then merged together through splicing. The remaining operations are consistent with the Res Block.

[0103] In the two stages of HINet, the first stage generates a preliminary deblurred image, and the second stage further generates a clearer image. The codecs of different stages are connected by a cross-stage feature fusion module (CSFF), which enriches the features of the next stage by fusing the multi-scale features of the previous stage. The two stages are transitioned through a supervised attention module (SAM), which is located at the end of the first stage and the beginning of the second stage. It is mainly used to generate an attention map to suppress features with less information in the current stage and promote the propagation of useful features to the next stage.

[0104] HINet consists of two sub-networks, both of which are UNet structures. In the UNet structure of each stage, a 3×3 convolution is first used to extract the initial features, and then the features are fed into the subsequent encoder-decoder structure. The encoder performs downsampling by 4×4 convolution and extracts features at each scale through the HIN Block, while the decoder uses 2×2 convolution for upsampling and uses the Res Block to extract high-level features. These features are then fused with the encoded partial features to compensate for the information loss caused by repeated sampling. Downsampling and upsampling are each performed four times. Finally, the network uses a 3×3 convolution to obtain the final residual output. In addition, there is a CSFF module and a SAM module between the two stages, which are connected at different locations to achieve supervision and guidance functions.

[0105] At different encoder and decoder positions, there will be a cross-stage feature fusion (CSFF) module. The CSFF module fuses the feature maps of the encoder and decoder of the first stage through a 3×3 convolution, and then performs an element-wise product operation, which is equivalent to matrix addition, and transmits it to the corresponding encoder position of the second stage. Its main functions are three points: (1) It makes the network more stable and less likely to cause information loss due to repeated up and down sampling. (2) The multi-scale features of a single stage can help enrich the features of the next stage. (3) The introduction of CSFF makes the structural optimization more stable because it promotes the flow of information.

[0106] The end of the first stage and the beginning of the second stage are connected in series by a supervised attention module (SAM). This module takes the results of the first stage as input, convolves the residual image generated in the first stage with the blurred image, and then generates a restored image. The restored image is then processed by convolution and a sigmoid function to generate an attention map. The convolved input image is then dot-multiplied with the attention map to enhance the features. Finally, the input feature map is added to the enhanced feature map and fed into the second-stage network. In general, the main role of SAM is to provide a real-world supervision signal that is useful for progressive image restoration at each stage. With the help of local supervision predictions, it generates an attention map to suppress features with less information in the current stage and promote the propagation of useful features to the next stage.

[0107] After being processed by the CSFF and SAM modules, the features of the first stage are transmitted to the two encoder positions and the entrance position of the second stage network respectively. Then, according to the same process as the first stage, the encoder extracts features and the decoder generates features, and finally the enhanced residual image is obtained. Combined with the original image, a clear image after deblurring can be generated.

[0108] However, this structural model has the following problems

[0109] (1) There is a problem of cross-stage feature fusion, which is specifically due to three points. The first point is that feature fusion is performed in a simple linear manner by adding corresponding elements of the matrix. The effect is relatively general and is not good enough for processing complex images. The second point is that the features are not enhanced before addition, and the semantic features cannot be well fused. The third point is that for the encoder of stage 2, the features transmitted by the encoder of stage 1 are more important than those of the decoder. Equally adding the features of the encoder and decoder is not conducive to the transmission of important features. The present invention proposes a new feature fusion mechanism, which reasonably enhances and weights the features of stage 1, increases the effectiveness of feature fusion, and thus improves the deblurring effect.

[0110] (2) During the transition from stage 1 to stage 2, the supervisory attention module responsible for this process has an attention generation problem. The reason for this problem is that the attention mechanism of this module can only learn the pixel information of each single feature map, without highlighting the deeper feature information, and there is no global learning of the overall features of all feature maps. The present invention can learn global information from the channel dimension and spatial dimension during the transition between the two stages, fully integrate high-level features, and better retain key features during deblurring.

[0111] (3) The backbone network has a large number of channels. The channel information obtained based solely on the features transmitted by the encoder-decoder network is insufficient, and there is a large amount of channel redundancy. In addition, the current deblurring method uniformly uses the Leaky ReLU activation function, whose slope is a fixed value. The uniform fixed value cannot adapt well to various positions in the network. In view of the fact that the backbone network has a large number of channels, the present invention improves the basic modules HIN Block and RES Block of the network, adds a channel attention mechanism, and at the same time, in order to address the shortcomings of Leaky ReLU, adds PReLU to the basic module. PReLU can iterate the parameters of LeakyReLU as a learnable parameter, which can be used to train deeper and larger networks.

[0112] The present invention is aimed at improving the above-mentioned shortcomings and solves the above-mentioned problems in a targeted manner.

[0113] In order to verify the effectiveness of the algorithm, the experiment uses the test set of the GoPro-large dataset for testing. Figure 6 The left image is used as the input of the model, and after model inference, we get Figure 6 As shown in the right image, the model can clearly sharpen blurry images, restoring object and environmental details to a high degree, while also achieving a relatively good visual effect on text deblurring. We demonstrate the effectiveness of this method using evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM).

[0114] The proposed method was experimentally compared with mainstream multi-stage deblurring methods DMPHN, SRN, MPRNet, HINet, and the classic method DeBlurGAN-v2. The results were tested on the GoPro-large dataset under the same experimental conditions (training on a single Tesla V100 GPU). The results are shown in Table 1.

[0115] Table 1. Comparison of objective evaluation indicators of experimental results of GoPro-large dataset

[0116] method PSNR SSIM Params(M) SRN 29.63 0.932 6.8 DeBlurGAN-v2 28.89 0.911 60.9 DMPHN 30.54 0.935 21.7 MPRNet 31.63 0.949 20.1 HINet 31.90 0.951 88.7 Ours 32.49 0.957 89.1

[0117] As can be seen from the table, the PSNR and SSIM indicators are both better than the existing mainstream methods. Compared with HINet, the method of the present invention significantly improves PSNR and SSIM with a very small increase in the number of parameters, proving the practicality and efficiency of the method. Figure 7 shown.

[0118] The method of the present invention replaces the cross-stage feature fusion module in HINet with an enhanced feature fusion module (EFF) to complete cross-stage feature fusion with better performance. At the same time, EFF is used in the jump connection part of the backbone network to increase the effectiveness of jump connection feature fusion. Compared with the original feature fusion method, EFF uses one-dimensional convolution with weight sharing to extract channel attention based on the multi-channel characteristics of the feature map, avoiding dimensionality reduction operations, thereby better extracting the dependency between channels, and solving the feature redundancy problem of fused features while suppressing the growth of the parameter amount; a soft selection weighting strategy is used to give more weight to the key feature map to promote the transmission of important features; and the weight is adjusted twice through an iterative structure to further enhance the feature fusion effect.

[0119] This paper introduces a supervised hybrid attention module (SHAM). Building on the supervised attention module, this module expands the number of channels in the restored image through convolution. This module uses hybrid attention to globally learn important information in both spatial and channel dimensions, achieving more targeted data enhancement. This module significantly suppresses unimportant feature information, further promoting key features to the next stage.

[0120] The present invention uses an enhanced half-instance normalization module (EHIN) to construct an encoder, responsible for feature extraction. An enhanced residual module (ERES) is used to construct a decoder, responsible for feature generation. Channel attention is added to the module to reduce channel redundancy caused by the high number of channels in the backbone network. The LeakyReLU activation function is also replaced with a PReLU. PReLU can iterate the LeakyReLU parameters as a learnable parameter, enabling the training of deeper and larger networks.

Claims

1. An image deblurring method based on an enhanced feature fusion mechanism, characterized in that: The following steps are involved: Step 1: Create a data set, including a training set and a test set; Step 2: Establish an image deblurring model based on the enhanced feature fusion mechanism, train it according to the training set, and obtain the trained image deblurring model; Step 3: Input the blurred image into the trained image deblurring model to obtain the deblurred image; The image deblurring model based on the enhanced feature fusion mechanism is based on the HINet network structure, which includes an encoder, an enhanced feature fusion module, a supervised hybrid attention module and a decoder; The encoder is used to extract image features and obtain feature maps; The enhanced feature fusion module is used to perform feature fusion of stage 1 and stage 2 and skip connections between networks; The supervised hybrid attention module is used to learn and integrate the features of stage 1 and pass them to stage 2; The decoder is used to add the feature map to the original blurred image to generate a deblurred image; The supervised hybrid attention module includes the SHAM module, and the processing process is as follows: The output of stage 1 is used as input, and the dimensionality is reduced by convolution. The deblurred image is then added to the original blurred image to obtain the deblurred image. The feature map is increased in dimension by convolution, and the feature map is processed as follows: The first method is to perform maximum pooling and average pooling on the feature map in the channel dimension, compress it into two single-channel feature maps, splice the two single-channel feature maps, and convolute them into one to obtain the spatial attention weight map; The second method is to perform global average pooling on the feature map to summarize the value of each feature map as an average value, and then generate a channel attention weight map through convolution, activation function, convolution, and Sigmoid function in sequence; The convolved input image is multiplied by the spatial attention weight map and the channel attention weight map respectively, concatenated and convolved, and then added to the input feature map to enter stage 2.

2. The image deblurring method based on enhanced feature fusion mechanism according to claim 1, characterized in that: The encoder module extracts the image process as follows: Convolution is performed on the input feature map, and the feature map channel is divided into two parts, one part is processed by IN, and the other part remains unchanged; Combine the two parts to obtain a spliced ​​feature map; The concatenated feature map is sequentially processed by PReLU activation function, convolution, and PReLU activation function; Then it goes through the channel attention module; The retained feature map is added to the initial input feature map after convolution to obtain the encoder module output; The channel attention module performs global average pooling, convolution dimensionality reduction, PReLU activation function, convolution dimensionality increase, and sigmoid function processing on the feature map in sequence; The decoding module has the same processing as the encoding module, excluding the process of dividing the feature map channels into equal parts.

3. The image deblurring method based on enhanced feature fusion mechanism according to claim 2, characterized in that: The enhanced feature fusion module includes an EFF module for feature fusion and skip connections between networks; The feature fusion calculation process is as follows: Where: X is the input feature of the encoder in stage 1, Y is the input feature of the decoder in stage 1, E represents the weight operation extracted by global average pooling, one-dimensional convolution and Sigmoid function, is the element-wise product operation, is the initial feature fusion, and Z is the final feature fusion; The weight extraction operation is as follows: after global average pooling, a dynamic convolution kernel is used to perform one-dimensional convolution, and then the weight of each channel is obtained through the Sigmoid activation function; the size of the dynamic convolution kernel is determined by the following formula: Where: k is the size of the convolution kernel, C is the number of channels, ||odd means that k can only be an odd number, γ and b are parameters.

4. The image deblurring method based on enhanced feature fusion mechanism according to claim 1, characterized in that: The process of stage 2 is as follows: The output of the SHAM module is used as input and processed by the encoder, while accepting the fused multi-scale features output by the EFF module; then it is processed by the decoder.

5. The image deblurring method based on enhanced feature fusion mechanism according to claim 4, characterized in that: The loss function Loss used in the image deblurring model training process based on the enhanced feature fusion mechanism is as follows: Where: PSNR is the peak signal-to-noise ratio, R i is the output of stage i, X i is the input of stage i, and Y is the real background image of each stage.

Images