Image deblurring method and device based on multiple-input-multiple-output codec

Abstract

The invention discloses an image deblurring method and device based on a multiple-input-multiple-output codec, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a GoPro blurred image data set, and dividing the GoPro blurred image data set into a training set and a test set; inputting the training set of the GoPro fuzzy data set into an improved MIMO-UNet model for training to obtain a deblurring training weight, and applying the weight to a test set of the GoPro fuzzy data set to obtain a deblurring image and a deblurring effect index result; the blurred image is deblurred through the improved MIMO-UNet model to obtain the deblurred image, so that the network performance is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image deblurring method and apparatus based on a multiple-input multiple-output codec. Background Technology

[0002] Multi-scale networks have been widely applied in image processing, such as in text detection and image denoising. Given the good results achieved by multi-scale neural networks in other image processing fields, applying them to image deblurring is a natural choice. For example, a non-uniform blurring method was studied by combining deep learning with classical optimization methods. This method decomposes complex problems into several simple sub-tasks and establishes a multi-scale network structure, thereby expanding the perceptual range of the image and improving the deblurring effect. Considering that networks at different scales are essentially consistent, a scale recurrent network is constructed by sharing model parameters, which reduces network parameters while also improving the deblurring effect.

[0003] However, while multi-scale networks have achieved good results in image deblurring, they still suffer from texture blurring and loss of detail. For example, multi-scale networks involve a large number of training parameters and coarse-to-fine upsampling, resulting in very high computational costs. Furthermore, increasing the model's depth and refinement level has not improved the deblurring effect.

[0004] Therefore, how to provide an image deblurring method and device based on a multiple-input multiple-output codec is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an image deblurring method and device based on a multiple-input multiple-output (MIMO) codec. First, a multi-scale kernel attention module is designed to integrate blur kernel information into the multi-scale feature map, thereby achieving differentiated processing of targets with different degrees of blur and improving the deblurring effect. Second, by constructing a multi-scale feature fusion module, the importance of features in space and channels is fully explored at both the overall and local levels, thereby improving network performance.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An image deblurring method based on a multiple-input multiple-output codec includes:

[0008] Obtain the GoPro blurred image dataset and divide it into training and test sets;

[0009] The training set of the GoPro blurred dataset is input into the improved MIMO-UNet model for training to obtain deblurring training weights. These weights are then applied to the test set of the GoPro blurred dataset to obtain deblurred images and deblurring effect metrics.

[0010] The blurred image is deblurred by using an improved MIMO-UNet model to obtain a deblurred image.

[0011] Furthermore, the improved MIMO-UNet model includes:

[0012] The multi-scale kernel attention module is used to estimate the fuzzy kernel and calculate the attention weights, enabling the network to differentiate the processing of regions with different degrees of fuzziness.

[0013] The multi-scale feature fusion module is used to integrate feature information at different scales and optimize feature extraction through Fourier transform, inverse Fourier transform and average pooling.

[0014] Furthermore, the multi-scale kernel attention module includes:

[0015] A kernel estimation network is used to predict the blur kernel and standard deviation of each pixel, and an uncertainty learning mechanism is introduced to generate the uncertain components of the latent encoding through Gaussian resampling;

[0016] The kernel attention module is used to perform attention calculations on the fuzzy kernel information and multi-scale feature maps, and integrate them into the defuzzification network.

[0017] Furthermore, the kernel attention module includes:

[0018] The first coding module EB1 consists of a 3×3 convolutional layer and eight residual blocks. Each residual block contains two 3×3 convolutional layers and a ReLU activation function.

[0019] The second encoding module EB2 and the third encoding module EB3 are based on the first encoding module EB1, with the addition of a multi-scale feature fusion module MFFM.

[0020] The first decoding module DB1 corresponds to the first encoding module EB1;

[0021] The second decoding module DB2 and the third decoding module DB3 introduce transpose convolution operation based on the first decoding module DB1.

[0022] Furthermore, the multi-scale feature fusion module (MFFM) includes:

[0023] Channel connection and splitting strategies are used to fuse features at different scales;

[0024] The Fast Fourier Transform (FFT) branch is used to extract frequency domain features and then recover them to the spatial domain using the Inverse Fast Fourier Transform (IFFT).

[0025] A 1×1 convolutional layer is used to select the modulation frequency threshold and optimize the efficiency of feature utilization.

[0026] Average pooling layers are used to reduce noise interference and extract global features.

[0027] Furthermore, the kernel attention module also includes multiple asymmetric feature fusion modules (AFFs), wherein each AFF receives the outputs of all encoding modules as input, integrates multi-scale features through convolutional layers, and passes them to the decoding module.

[0028] Furthermore, the multi-scale feature fusion network includes:

[0029] Features at different scales are fused through channel connection and splitting strategies;

[0030] Frequency domain features are extracted using Fast Fourier Transform and restored to the spatial domain using Inverse Fast Fourier Transform, thus enhancing feature richness.

[0031] Average pooling is introduced to reduce noise interference and extract overall image features.

[0032] On the other hand, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an image deblurring method based on a multiple-input multiple-output codec.

[0033] As described above, this invention discloses an image deblurring method and apparatus based on a multiple-input multiple-output (MIMO) codec. Unlike traditional multi-scale deblurring methods that use simple sub-network stacking strategies, this invention employs a U-Net network to simulate the cascaded structure of sub-networks. A multi-scale kernel attention module is provided to fully integrate information obtained from estimating the blur kernel, effectively incorporating this information into the multi-scale feature map. This allows for differentiated attention to regions with varying degrees of blur, achieving superior results in non-uniform blur removal. Furthermore, a multi-scale feature fusion module is constructed to further enhance the detail representation of the restored image, particularly focusing on structural details and effectively extracting global and local features. Simultaneously, a series of experiments evaluated the deblurring performance of the proposed invention, demonstrating its robustness. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0036] Figure 2 This is a schematic diagram of the overall network framework structure for image deblurring provided by the present invention;

[0037] Figure 3 This is a schematic diagram of the kernel estimation network structure provided by the present invention;

[0038] Figure 4 A schematic diagram of the core attention module structure provided by the present invention;

[0039] Figure 5 The multi-scale feature fusion module provided by this invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1:

[0042] See Figure 1 This invention discloses an image deblurring method based on a multiple-input multiple-output codec, comprising:

[0043] Obtain the GoPro blurred image dataset and divide it into training and test sets;

[0044] The training set of the GoPro fuzzy dataset is input into the improved MIMO-UNet model for training to obtain deblurring training weights. These weights are then applied to the test set of the GoPro fuzzy dataset to obtain deblurred images and deblurring effect metrics.

[0045] The blurred image is deblurred by using an improved MIMO-UNet model to obtain a deblurred image.

[0046] See the overall network structure. Figure 2Before the encoding stage, a Multi-scale Kernel Attention Module (MKAM) is used to input the blurred image into the kernel estimation network, obtaining an estimated blur kernel for each pixel. Subsequently, the blur and information are transmitted to the encoder, where attention is calculated with features at different scales, thus integrating the blur kernels into the deblurring network. This allows image features to provide differentiated attention to regions with varying degrees of blur. Encoding module EB1 consists of a single 3×3 convolutional layer with a stride of 1 and a residual group containing eight residual blocks. Each residual block includes two 3×3 convolutional layers with a stride of 1 and a ReLU activation function. Skip connections are used to propagate feature information to deeper layers, avoiding the vanishing gradient problem. Encoding blocks EB2 and EB3 add a Multi-scale Feature Fusion Module (MFFM) to EB1. This module optimizes the image deblurring network by combining kernel-level information with Fourier transform, inverse Fourier transform, and average pooling techniques. Furthermore, by introducing 1×1 convolutions, the network can flexibly adjust the threshold to achieve frequency selection, thereby more effectively utilizing multi-scale features. The design of the decoding block DB1 is similar to that of the encoding block EB1, but a transposed convolution operation is introduced between decoding blocks DB2 and DB3. In the encoding process of the blurred image, firstly, the blurred image B1 is downsampled twice to reduce the image size, generating images of size [missing information]. and Blurred images B2 and B3 of different sizes are used. Simultaneously, blurred image B1 is input to coding block EB1 to extract feature information. The processing result is then passed to the next coding block, and the output result is denoted as... To improve computational efficiency, a Shallow Convolutional Module (SCM) is used. This module applies two consecutive 3×3 and 1×1 convolution operations to the input image. The resulting features are then concatenated with the original input image along the channel dimension, followed by an additional 1×1 convolution to obtain the final SCM output. This process is not only efficient, but also fully extracts the features of downsampled blurred images.

[0047] In the feature decoding stage, asymmetric feature fusion (AFF) enables the transmission of cross-scale information within a single U-Net architecture. Specifically, each AFF receives the outputs of all encoded blocks (EB) as input and integrates these multi-scale features through convolutional layers. Subsequently, the output of the AFF is forwarded to its corresponding decoder (DB), a process described by Equation 1.

[0048]

[0049] In the formula, the output of the nth level AFF module is: and This represents the output of the nth level coded block. To achieve the fusion of features at different scales, upsampling (↑) and downsampling (↓) techniques are employed. Furthermore, each decoded block has the ability to utilize multi-scale features, which helps improve the model's performance in deblurring tasks.

[0050] The final output of the image during the decoding stage is shown in Equation 2.

[0051]

[0052] In the formula, B n , Let represent the blurred image at level n, the output of the AFF module, the output of the coded block, and the output of the decoded block, respectively. The output of the decoded block is actually a feature map rather than a direct image, therefore a mapping function o is needed to generate the final output image. In this algorithm, a single-layer convolution is used as this mapping function o to achieve the transformation from feature map to image.

[0053] Specifically, given the lack of precise benchmark data for spatially variable blur kernels in real-world images, numerous deep learning-based blind image deblurring methods focus on estimating blur representations and reconstructing sharp images from observed blurry images. However, these methods often neglect crucial prior knowledge about motion blur, resulting in significant performance degradation in real-world scenarios. To address these challenges, this paper employs the MKAM module, utilizing normalized flow techniques to characterize blur kernels in the latent space and incorporating an uncertainty learning mechanism. By employing varying convolution strides at different depth levels to adapt to various feature scales, the kernel attention module more effectively integrates blur kernel information into multi-scale feature maps, further improving the accuracy and robustness of kernel estimation. This module mainly consists of a kernel estimation network and a kernel attention module.

[0054] Specifically, in practical applications, accurate estimation of fuzzy kernels faces challenges due to the complexity and spatial variability of kernel features. To enhance the effectiveness and robustness of prediction results, an uncertainty learning mechanism is introduced into the kernel estimation network. For example... Figure 3 As shown, the nuclear network first receives the blurred image. As input, predict the normalized latent code for each pixel i. Where L = k 2 This represents the size of the fuzzy kernel, and z i It follows a standard normal distribution. The latent encoding of the entire image is represented as: Meanwhile, the standard deviation of the latent encoding of each pixel It is also predicted synchronously, and the standard deviation of the entire image is denoted as . Each latent code z is obtained by performing Gaussian resampling. i Uncertain component n i As shown in Equation 3.

[0055]

[0056] in, With z i Same size. (Passed) To transform z i The standard deviation of makes Assumption and n i They are independent of each other, and the final potential code can be seen as the sum of them, and the result still satisfies the standard normal distribution, as shown in Equation 4.

[0057]

[0058] in, Represents the identity components of the latent encoding, n i It is an uncertain component. Due to the training process, Random noise n i destroy, Instead of deterministic point embeddings, the model is more robust to estimation errors, thus improving accuracy.

[0059] With the help of kernel attention modules, defuzzification networks can more effectively utilize the information provided by the estimated fuzzy kernel, as follows: Figure 4 As shown. Let the input feature map be... First, a convolutional layer is used to process it. Then, a non-uniform blur kernel is applied to each pixel location. The kernel attention map is introduced into the convolutional layer and obtained in the following way, as shown in Equation 5.

[0060] F att =Conv3(concat[Conv1(X),Conv2(K)]) (5)

[0061] The output feature map is denoted as f(X) = F. att⊙Conv1(X)+X, where ⊙ represents element-wise multiplication. Given that the image feature size processed by the encoder within the network gradually decreases, correspondingly varying convolution strides are adopted in kernel attention modules of different depths to adapt to the feature size requirements of each layer. This design allows the information from the blur kernel to be effectively integrated across multi-scale feature maps. Thanks to the proposed kernel attention module, image features can be given differentiated attention to regions with different degrees of blur, thus achieving superior performance in non-uniform deblurring tasks.

[0062] Existing multi-scale analysis methods often employ channel concatenation or element-wise addition for feature fusion. However, existing algorithms do not effectively leverage the complementarity between features at different scales, thus reducing the deblurring effect. To address this issue, this invention introduces MFFM based on a channel concatenation and splitting strategy, with the following structure: Figure 5 As shown. The MFFM module will input feature E k-1 and F k The features are combined by channel concatenation. Subsequently, features X are thinned by 3×3 convolution and split into {X1, X2}. In the processing of X1, a branch based on Fast Fourier Transform (FFT) is introduced to effectively model frequency information, thereby eliminating blur in the image. The advantage of this branch is that it can model the differences between blurred and clear images in the frequency domain, while capturing the interaction of long-range and short-range information. To achieve this goal, a frequency domain branch based on channel convolution is created to extract global information in the feature frequency domain. Today, the Discrete Fourier Transform (DFT) is widely used in modern signal processing algorithms, and its one-dimensional version can be obtained through Equation 6.

[0063]

[0064] Wherein, the nth item in the sequence is a complex number x[n] of length N, with frequency... The spectrum at point X is represented as X[k], where j represents the imaginary unit.

[0065] Specifically, the frequency contains global characteristics of all frequencies. If the signal x[n] is a real number, its DFT will exhibit a conjugate symmetric form, as shown in Equation 7.

[0066]

[0067] Among them, X * It is considered as the conjugate symmetric value of X. Furthermore, Equation 3.7 can also be applied to the two-dimensional discrete Fourier transform, i.e., X[Mu,Nu]=X * [u,v]. Let the real number input characteristics be... Where H, W, and C represent the height, width, and number of channels of the feature, respectively. Therefore, inspired by the principle of discrete Fourier transform, Fourier transform is introduced, and the calculation process is shown in Equation 8.

[0068] X(k)=∑(x(n)×e -2πikn / N (8)

[0069] Here, X(k) represents the amplitude and phase information of the k-th frequency component in the frequency domain, while x(n) corresponds to the value of the n-th sampling point in the time domain, and N represents the total number of sampling points. The main idea of ​​the Fast Fourier Transform (FFT) algorithm is to decompose the discrete Fourier transformation (DFT) operation into multiple smaller-scale DFT calculations and achieve efficient computation through recursion. This computational method fully utilizes the symmetry and periodicity of the signal, integrating a pair of conjugate frequency components into a complex number expression, thereby significantly reducing computational complexity.

[0070] Therefore, the processing flow of the X1 branch is as follows: First, the data is transformed using real-number fast Fourier transform, and its real and imaginary parts are separated. Features are then further extracted using a 3×3 convolutional layer. Afterward, the data is restored to its original domain using inverse fast Fourier transform, and then multiplied element-wise with the original feature X1 to enhance feature richness. The specific steps are as follows:

[0071] (1) Perform a two-dimensional real fast Fourier transform on X1 to obtain its frequency domain representation.

[0072] (2) Extraction The real and imaginary parts are concatenated and joined together along the channel dimension, denoted as . Among them ⊙ c This indicates a concatenation operation at the channel dimension.

[0073] (3) After splicing The feature map is fed into a 3×3 convolutional layer, and the output feature map f is generated.

[0074] (4) For f = f real ⊙ c f imag Applying the inverse fast Fourier transform, we convert back to the spatial domain to obtain the final output feature map.

[0075] In one specific embodiment, given that blurred images are often subject to various noise interferences, average pooling is introduced in MFFM. This technique effectively smooths the image and reduces noise interference by calculating the average pixel values ​​within a local region. Simultaneously, average pooling can also extract overall image features and summarize core image information, such as approximate shape, color distribution, and other global attributes, capturing these features through averaging of local regions.

[0076] In the evaluation system of image deblurring tasks, it is usually necessary to rely on the quantitative results of the network's deblurring effect to assess its performance. This invention will use two major indicators, PSNR and SSIM, to measure the network's deblurring effectiveness.

[0077] PSNR, measured in dB, is a commonly used metric for evaluating the similarity between two images. Its specific calculation formula is shown in Equation 9.

[0078]

[0079] In the formula, the mean squared error (MSE) between the two images is calculated, and this error is measured based on the number of bits n for each sample value. SSIM is another widely used metric for evaluating the similarity between two images; its specific calculation formula is given in Equation 10.

[0080]

[0081] In the formula, μ x μ is the average value of x. y σ is the average value of y. xy Let x and y be the covariances. Let x be the variance. Let be the variance of y. c1 and c2 are constants that maintain stability, respectively (k1L) 2 and (k2L) 2 L represents the dynamic range of pixel values, k1 = 0.01, k2 = 0.03. The SSIM index ranges from -1 to +1. When two images are identical, the SSIM value reaches its upper limit, which is equal to 1.

[0082] On the other hand, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an image deblurring method based on a multiple-input multiple-output codec.

[0083] Example 2:

[0084] This embodiment discloses an image deblurring method based on a multiple-input multiple-output codec, including:

[0085] Obtain the GoPro blurred image dataset;

[0086] A modified MIMO-UNet model is constructed. Based on the MIMO-UNet model, a multi-scale kernel attention module is built. Using the kernel estimation network and kernel attention module within it, image features will give different attention to regions with varying degrees of blur. Simultaneously, a multi-scale feature fusion network is proposed, allowing the network to flexibly modulate thresholds for frequency selection, thereby better utilizing useful features at different scales.

[0087] The improved MIMO-UNet model is trained by inputting the test set of the GoPro blurred dataset to obtain deblurring training weights. These weights are then applied to the test set of the GoPro blurred dataset to obtain deblurred images and deblurring effect metrics.

[0088] Specifically, this embodiment covers three public datasets: GoPro, RealBlur, and the PascalVOC2007+2012 dataset for object detection. For the PascalVOC dataset, the images are divided into three groups, and three different blurring processes are applied to each group: motion blur, Gaussian blur, and dot-matrix blur.

[0089] Specifically, this embodiment will demonstrate the following two sets of experimental results:

[0090] (1) The model was trained on 2103 pairs of images in the GoPro training set and validated on 1111 pairs of images in the test set. At the same time, the trained model was directly tested on 7167 pairs of motion-blurred images, 7167 pairs of Gaussian-blurred images, and 7169 pairs of dot-matrix blurred images in the Pascal VOC2007+2012 dataset.

[0091] (2) Training was performed using 3758 pairs of images in RealBlur-R and the same number of image pairs in RealBlur-J, and performance was tested on their respective 980-pair test sets.

[0092] This embodiment is built on the PyTorch framework, runs on the Windows 10 operating system, and is equipped with an Intel(R) Core(TM) i5-10400F CPU (2.90GHz) and an Nvidia GeForce RTX 2080Ti GPU (22GB VRAM). The development language is Python 3.8, the deep learning framework is PyTorch-GPU version 2.0.1, the CUDA version is 12.1, and the cuDNN version is 7.6.5.

[0093] During the training phase, the batch size was set to 4, the input image size was 256×256 pixels, and the Adam optimization algorithm was used for model optimization. The learning rate was initialized and halved every 500 iterations, for a total of 3000 training rounds. In addition, to enhance data diversity, each image was horizontally flipped with a probability of 0.5.

[0094] In one specific embodiment, the experimental results and analysis are as follows:

[0095] Specifically, this embodiment compares the performance of the proposed improved deblurring method based on a multiple-input multiple-output codec with that of MPRNet, HINet, and MIMO-UNet networks on the Pascal VOC2007+2012 fuzzy dataset. As shown in Table 1, the experimental results demonstrate that the network designed in this invention achieves better deblurring performance. Although MPRNet's PSNR is slightly higher than that of the improved network in this invention when handling motion blur, and HINet's PSNR is slightly higher when handling Gaussian blur and granular blur, the network in this invention significantly outperforms the other three networks in terms of SSIM. Furthermore, compared to the baseline model MIMO-UNet, the improved network in this invention significantly improves image sharpness in handling Gaussian blur, motion blur, and granular blur.

[0096] Table 1. Test results of the Pascal VOC2007+2012 fuzzy dataset.

[0097]

[0098] Specifically, this embodiment further verifies the performance of the proposed network structure on a public dataset and conducts experimental comparisons with other networks.

[0099] A deblurring network integrating a multi-scale kernel attention module and a multi-scale feature fusion module was trained using the GoPro training set and evaluated using the GoPro test set. These results were also compared with those of other commonly used networks on the GoPro test set. Quantitative analysis results are detailed in Table 2. Compared to the baseline model MIMO-UNet, the improved MIMO-UNet model of this invention achieves a 0.38 dB improvement in PSNR and an increase in SSIM. Although the improved MIMO-UNet model has a slightly lower PSNR than the SAPHN deblurring network, its SSIM is slightly better, indicating that this network also exhibits good performance in deblurring. Furthermore, compared to other methods, the improved MIMO-UNet model of this invention restores images with richer detail, better deblurring effect, and clearer images that are closer to real-world images, especially in the details of letters, numbers, and lines.

[0100] Table 2 Test results of the GoPro test suite

[0101] Model PSNR SSIM DeepDeblur 29.23 0.916 SRN 30.26 0.934 DeblurGAN 28.70 0.858 DeblurGAN-v2 29.55 0.934 PSS-NSC 30.92 0.942 DMPHN 31.20 0.945 SAPHN 31.85 0.948 MTRNN 31.15 0.945 MIMO-UNet 31.06 0.957 This invention 31.44 0.958

[0102] In this embodiment, the proposed network was trained and tested on the RealBlur dataset, and the results are shown in Table 3. Compared with the baseline model MIMO-UNet, the improved MIMO-UNet model of this invention achieved PSNR improvements of 0.61dB and 0.88dB, and SSIM gains of 0.019 and 0.017, respectively, on the RealBlur-R and RealBlur-J datasets. Although the PSNR of this network is slightly lower than that of the MTRNN network on the RealBlur-J dataset, it significantly outperforms the MTRNN network in terms of SSIM. This result demonstrates that the proposed deblurring improvement method based on a multiple-input multiple-output codec exhibits good performance for real-world blurring and defocusing phenomena.

[0103] Specifically, by comparing the clear, blurred, and deblurred versions of the entire image, and also by comparing clear, blurred, and deblurred images of large and small blurred areas, it can be concluded that in areas with a high degree of blur, deblurring significantly improves sharpness. Similarly, this conclusion also applies to other locally blurred images. In summary, the improved MIMO-UNet model of this invention not only effectively handles global blur but also demonstrates good processing capabilities for locally blurred or non-uniformly blurred situations.

[0104] Table 3 Test results of the RealBlur test set

[0105]

[0106]

[0107] To verify the performance of each module in the deblurring task, a series of combined experiments were conducted, and the results are shown in Table 4. By using the multi-scale kernel attention (MKAM) module, the PSNR was improved by 0.1 dB, effectively integrating the blur kernel information into the multi-scale feature map and enhancing the deblurring capability. After adopting the multi-scale feature fusion (MFFM) module, the PSNR was improved by 0.21 dB, and the SSIM also increased slightly, demonstrating the positive effect of the MFFM module in extracting global and local features of blurred images. When the MKAM and MFFM modules were combined, the obtained PSNR and SSIM values ​​reached 31.44 dB and 0.958, respectively. This result indicates that the proposed algorithm has good deblurring performance.

[0108] Table 4 Ablation experiments with different module combinations

[0109] MKAM MFFM PSNR SSIM 31.06 0.957 √ 31.16 0.957 √ 31.27 0.958 √ √ 31.44 0.958

[0110] Specifically, this invention proposes an image deblurring method and device based on a multiple-input multiple-output (MIMO) codec. Unlike traditional multi-scale deblurring methods that rely on simple sub-network stacking, this method employs a U-Net network to simulate the cascaded structure of sub-networks. A multi-scale kernel attention module is provided to fully integrate information obtained from estimating the blur kernel, effectively incorporating this information into the multi-scale feature map. This allows for differentiated attention to regions with varying degrees of blur, achieving superior results in removing non-uniform blur. Furthermore, a multi-scale feature fusion module is constructed to further enhance the detail representation of the restored image, particularly focusing on structural details and effectively extracting global and local features. A series of experiments evaluated the deblurring performance of the proposed method, and the results verify its robustness.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0112] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image deblurring method based on a multiple-input multiple-output codec, characterized in that, include: Obtain the GoPro blurred image dataset and divide it into training and test sets; The training set of the GoPro fuzzy dataset is input into the improved MIMO-UNet model for training to obtain deblurring training weights. These weights are then applied to the test set of the GoPro fuzzy dataset to obtain deblurred images and deblurring effect metrics. The blurred image is deblurred by using an improved MIMO-UNet model to obtain a deblurred image.

2. The image deblurring method based on a multiple-input multiple-output codec according to claim 1, characterized in that, The improved MIMO-UNet model includes: The multi-scale kernel attention module is used to estimate the fuzzy kernel and calculate the attention weights, enabling the network to differentiate the processing of regions with different degrees of fuzziness. The multi-scale feature fusion module is used to integrate feature information at different scales and optimize feature extraction through Fourier transform, inverse Fourier transform and average pooling.

3. The image deblurring method based on a multiple-input multiple-output codec according to claim 2, characterized in that, The multi-scale kernel attention module includes: A kernel estimation network is used to predict the blur kernel and standard deviation of each pixel, and an uncertainty learning mechanism is introduced to generate the uncertain components of the latent encoding through Gaussian resampling; The kernel attention module is used to perform attention calculations on the fuzzy kernel information and multi-scale feature maps, and integrate them into the defuzzification network.

4. The image deblurring method based on a multiple-input multiple-output codec according to claim 3, characterized in that, The kernel attention module includes: The first coding module EB1 consists of a 3×3 convolutional layer and eight residual blocks. Each residual block contains two 3×3 convolutional layers and a ReLU activation function. The second encoding module EB2 and the third encoding module EB3 are based on the first encoding module EB1, with the addition of a multi-scale feature fusion module MFFM. The first decoding module DB1 corresponds to the first encoding module EB1; The second decoding module DB2 and the third decoding module DB3 introduce transpose convolution operation based on the first decoding module DB1.

5. The image deblurring method based on a multiple-input multiple-output codec according to claim 4, characterized in that, The multi-scale feature fusion module (MFFM) includes: Channel connection and splitting strategies are used to fuse features at different scales; The Fast Fourier Transform (FFT) branch is used to extract frequency domain features and then recover them to the spatial domain using the Inverse Fast Fourier Transform (IFFT). A 1×1 convolutional layer is used to select the modulation frequency threshold and optimize the efficiency of feature utilization. Average pooling layers are used to reduce noise interference and extract global features.

6. The image deblurring method based on a multiple-input multiple-output codec according to claim 4, characterized in that, The kernel attention module further includes multiple asymmetric feature fusion modules (AFFs), wherein each AFF receives the outputs of all encoding modules as input, integrates multi-scale features through convolutional layers, and passes them to the decoding module.

7. The image deblurring method based on a multiple-input multiple-output codec according to claim 2, characterized in that, The multi-scale feature fusion network includes: Features at different scales are fused through channel connection and splitting strategies; Frequency domain features are extracted using Fast Fourier Transform and restored to the spatial domain using Inverse Fast Fourier Transform, thus enhancing feature richness. Average pooling is introduced to reduce noise interference and extract overall image features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements an image deblurring method based on a multiple-input multiple-output codec as described in any one of claims 1 to 7.

Images