Lightweight multi-path image deblurring method for edge deployment

Abstract

The invention belongs to the technical field of image deblurring, and particularly relates to an edge deployment-oriented lightweight multi-path image deblurring method, which comprises the following steps of: 1, preparing a data set; 2, constructing a network model; step 3, training a network model; 4, optimizing the model and evaluating the performance; 5, finely adjusting the model; 6, storing the model; and step 7, hardware deployment and debugging. According to the method, the lightweight image deblurring network is designed, the performance of the model is improved while the parameters of the model are reduced, the lightweight deblurring network is used in the field of image deblurring, and the problems that an existing network model is large and actual deployment is difficult are solved; channel-level adaptive feature fusion is carried out on the feature map by designing a dual-response enhanced fusion network, and the contrast and significance of edge information are enhanced to realize efficient extraction of multi-level features; complementary optimization of global and local features is realized by designing a lightweight multi-path feature sensing network.

Description

Technical Field

[0001] This invention relates to the field of image deblurring technology, specifically a lightweight multipath image deblurring method for edge deployment. Background Technology

[0002] Image deblurring is a core technology in digital image processing, primarily used to eliminate visual quality degradation caused by blurring effects. Blurring is usually caused by factors such as device motion, lens defects, environmental interference, or sensor noise. This type of distortion significantly reduces the information density and visual readability of an image.

[0003] In computer vision applications, low-resolution images can negatively impact the accuracy of tasks such as target localization and feature recognition. Deblurring can effectively optimize image quality, providing a more reliable data foundation for subsequent intelligent analysis. For example, in medical imaging diagnosis, high-resolution images support more accurate pathological assessments; in remote sensing monitoring, clear satellite imagery helps improve the accuracy of geographic information analysis; and for security systems, deblurred surveillance footage can enhance key functions such as facial recognition and behavior analysis.

[0004] This technology is not only widely needed in civilian photography and industrial inspection, but also plays an irreplaceable role in professional scenarios such as biomedicine, aerospace, and public safety. Its core value lies in laying a high-quality data foundation for cross-domain image analysis and decision support systems by improving the ability to restore image details.

[0005] The Chinese patent publication number is "CN119809980A", and its patent title is "A Lightweight Deblurring Method Based on Frequency Domain and Spatial Domain Feature Interaction". This method improves the U-Net model and achieves efficient restoration through a frequency domain and spatial domain feature interaction mechanism. The specific process includes: First, low-level features of the blurred image are extracted using 1×1 convolution, and then input into a U-shaped network consisting of a 3-level symmetric encoder-decoder. Each level performs multi-scale feature fusion through multiple FSTBlock modules (frequency-spatial domain interaction modules). Inside FSTBlock, frequency domain attention (dividing features into 8×8 windows, transforming them to the frequency domain using FFT, filtering high-frequency components using a learnable quantization matrix, and generating weights to strengthen key regions after inverse transformation) and multi-head pyramid channel attention (splitting features into four paths, extracting local details through 1×1, 3×3, and 5×5 convolutions respectively, and dynamically filtering important information using global pooling and probability weighting) are used to extract global frequency dependence and local spatial features. The advantages of both are then fused through pixel-level maximum value interaction. The deep features are further optimized by a pyramid feedforward network (using multi-scale depth-separable convolution and gating mechanisms, combined with random deactivation to suppress overfitting) to generate residual images. This method mainly relies on frequency-spatial domain interaction, which is less adaptable to complex blur patterns. Furthermore, the system needs to process all input images, lacks preprocessing and filtering, and has low computational resource utilization. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a lightweight multipath image deblurring method for edge deployment, which solves the problems mentioned in the background section.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0010] A lightweight multipath image deblurring method for edge deployment includes the following steps:

[0011] Step 1: Prepare the datasets; Prepare three datasets: Dataset 1 (GoProDataset), Dataset 2 (RealBlur Dataset), and Dataset 3 (Köhler Dataset).

[0012] Step 2: Construct a lightweight dynamic image deblurring network model: The lightweight dynamic image deblurring network model includes a U-Net network and a residual dynamic gated blur discrimination network. The U-Net network includes a Manling gated network, a dual-response enhancement fusion network, and a lightweight multi-path feature perception network.

[0013] Step 3: Train the network model; train the network model using the prepared dataset;

[0014] Step 4: Optimize the model and evaluate its performance; the model is optimized using three loss functions: deblurring loss, encoder feature loss, and prediction loss, and the peak signal-to-noise ratio and structural similarity are used as evaluation metrics to evaluate the model performance;

[0015] Step 5: Fine-tune the model; Input the dataset into the network model for training, compare the results with image evaluation metrics, and then fine-tune the model parameters;

[0016] Step 6: Save the model; solidify the parameters of the final optimized network model and save the network model;

[0017] Step 7: Hardware deployment and debugging; Convert the saved network model into C language using STM32cube.AI and deploy it to the embedded system, and debug the embedded system.

[0018] Furthermore, in step 1, random scaling, inversion, and translation are used to invert and translate dataset 1 to expand the dataset, while providing multi-scale information; dataset 2 is used for model fine-tuning; dataset 3 is used to test the model and verify its generalization ability under various fuzzy conditions; the size of each image in each dataset is 512x512.

[0019] Furthermore, in step 2, the acquired image data is processed by a residual dynamic gated fuzzy discrimination network to determine whether the image needs to be deblurred. If no deblurring operation is required, the image is directly output. This operation can reduce unnecessary calculations in the system and improve the system's running speed. If deblurring operation is required, the image is processed by the U-Net network.

[0020] Furthermore, in step 2, the Manning gated network consists of two branches. The first branch consists of Linear, 3×3 depthwise separable convolution, SuLU activation function and state space network (SSM), and the second branch consists of Linear and SuLU activation. The result of the cross product of the two branches is then subjected to 1×1 convolution. This is used to enhance the network's ability to model temporal and global dependency information during the deblurring process.

[0021] The dual-response enhancement fusion network consists of two branches. The inputs to the two branches are the feature maps after 3×3 convolution of the output of the upper-level decoder / encoder and the feature maps without 3×3 convolution, respectively. One branch performs channel-level adaptive feature fusion on the feature map without 3×3 convolution. The other branch preprocesses the feature map after 3×3 convolution using the SimAM attention mechanism, and then extracts edge features through pooling, subtraction, convolution, and multiple residual connections to enhance the contrast and saliency of edge information. The results of the two branches are fused to obtain richer contextual information and achieve efficient multi-level feature extraction.

[0022] The lightweight multi-path feature perception network consists of lightweight convolution, multi-attention collaboration, and multi-path fusion mechanisms, which achieve complementary optimization of global and local features, enhance the deep network's ability to perceive key information, and obtain more scale feature information while ensuring lightweight computation.

[0023] Furthermore, in step 2, the residual dynamic gated fuzzy discrimination network enhances the contrast between fuzzy and non-fuzzy features by extracting residual deep features and combining them with a dual-gated feedforward output layer, thereby reducing the misclassification rate and achieving high-precision fuzzy classification.

[0024] (III) Beneficial Effects

[0025] Compared with existing technologies, this invention provides a lightweight multipath image deblurring method for edge deployment, which has the following advantages:

[0026] This invention designs a lightweight image deblurring network, which improves model performance while reducing model parameters. In the field of image deblurring, the use of a lightweight deblurring network addresses the problem of large existing network models and difficulties in practical deployment. It designs a dual-response enhancement fusion network to perform channel-level adaptive feature fusion of feature maps and enhance the contrast and saliency of edge information, achieving efficient extraction of multi-level features. Furthermore, it designs a lightweight multi-path feature perception network to achieve complementary optimization of global and local features, improving the deep network's ability to perceive key information and obtaining more scale-based feature information while maintaining lightweight computation. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method steps of the present invention;

[0028] Figure 2 This is a schematic diagram of the overall image deblurring network structure in the method of the present invention;

[0029] Figure 3 This is a schematic diagram of the residual dynamic gated fuzzy discrimination network structure in the method of the present invention;

[0030] Figure 4This is a schematic diagram of the Manning gated network structure in the method of the present invention;

[0031] Figure 5 This is a schematic diagram of the dual-response enhanced fusion network structure in the method of the present invention;

[0032] Figure 6 This is a schematic diagram of the lightweight multipath feature perception network structure in the method of the present invention;

[0033] Figure 7 This is a schematic diagram of the hardware deployment structure in the method of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example

[0036] like Figure 1-7 As shown, an embodiment of the present invention proposes a lightweight multipath image deblurring method for edge deployment, comprising the following steps:

[0037] Step 1: Prepare the datasets; Prepare three datasets for network training: Dataset 1, Dataset 2, and Dataset 3. Dataset 1 is the GoPro Dataset, which is expanded using random scaling, inversion, and translation to provide multi-scale information, enabling the model to extract effective features at different resolutions and improve its ability to handle blurred regions at different scales. Dataset 2 is the RealBlur Dataset, used for model fine-tuning to further optimize model performance. Dataset 3 is the Köhler Dataset; the model is tested on this dataset to verify its generalization ability under various blur conditions. Each image in each dataset is 512x512 pixels to ensure consistency in processing different datasets in the network model and avoid instability during training due to differences in image size.

[0038] Step 2: Construct a lightweight dynamic image deblurring network model; the lightweight dynamic image deblurring network model includes a U-Net network and a residual dynamic gated blur discrimination network. The U-Net network includes a Manling gated network, a dual-response enhancement fusion network, and a lightweight multi-path feature perception network.

[0039] Lightweight dynamic image deblurring networks, such as Figure 2As shown, the image data captured by the camera is processed by a residual dynamic gated blur discrimination network to determine whether the image needs deblurring. If no deblurring is needed, the image is directly output. This reduces unnecessary computation and improves system speed. If deblurring is required, the image is processed through the U-Net network.

[0040] Residual dynamic gated fuzzy discrimination network, such as Figure 3 As shown, the residual dynamic gated fuzzy discriminant network enhances the contrast between fuzzy and unfuzzy features and reduces the false positive rate by combining a residual deep feature extraction branch with a dual-gated feedforward output layer. The input image is divided into two branches after passing through three 1×1 convolutions and three ReLU activation functions. One branch consists of convolutions, normalization layers, and ReLU activation functions for the deep feature extraction path of the input image. In this path, convolutional layers four and seven are 5×5 pointwise convolutions, convolutional layers six and nine are 1×1 pointwise convolutions, convolutional layers five and eight are 3×3 depthwise separable convolutions, and convolutional layer ten is a 1×1 ordinary convolution. Another branch is a dual-gate feedforward network consisting of convolutional layers, ReLU, and GeLU activation functions. Convolutional layers 11, 14, and 15 are 1×1 ordinary convolutions, convolutional layers 12 and 13 are 3×3 depthwise separable convolutions, and convolutional layer 14 is a 5×5 depthwise separable convolution. This path is used to aggregate contextual information from the input image to achieve high-accuracy classification. The network formula is as follows:

[0041]

[0042] The output of the residual dynamic gated fuzzy discrimination network, The output of the residual depth feature extraction branch, This is the output of the dual-gate feedforward network.

[0043] Manling gated networks, such as Figure 4 As shown, the Manning gated network consists of two branches. The first branch comprises a Linear network, a 3×3 depthwise separable convolution, a SuLU activation function, and a State-Space Network (SSM). The second branch comprises a Linear network and a SuLU activation function. The cross product of the two branches is then subjected to a 1×1 convolution. This enhances the network's ability to model temporal and global dependencies during the deblurring process. The network formula is as follows:

[0044]

[0045] For the output of the Manning gated network, This is the input to the Manning gated network.

[0046] Dual-response enhanced converged networks, such as Figure 5 As shown, the dual-response enhancement fusion network consists of two branches. The inputs to the two branches are the feature map after 3×3 convolution of the output of the upper-level decoder / encoder and the feature map without 3×3 convolution, respectively. One branch performs channel-level adaptive feature fusion on the feature map without 3×3 convolution. The other branch preprocesses the feature map after 3×3 convolution using the SimAM attention mechanism, and then extracts edge features through pooling, subtraction, convolution, and multiple residual connections to enhance the contrast and saliency of edge information. The input to the first branch is the feature map after 3×3 convolution of the decoder / encoder output. The result is obtained by subtracting the result of SimAM from the result of average pooling. , After passing through 3×3 convolutions with Sigmoid activation functions, we obtain... Obtained through residual connection and , After another 1×1 convolution, we get The first branch contains multiple residual connections. The specific formula is as follows:

[0047]

[0048] The input to the second branch is the feature map output by the decoder / encoder. The spatial features are captured sequentially through spatial attention, average pooling, 1×1 convolution, and a sigmoid activation function. , Again with We get the cross product. Finally, after a 1×1 convolution, we get... .

[0049]

[0050]

[0051] The final output of the dual-response enhanced fusion network is:

[0052] Lightweight multipath feature-aware networks, such as Figure 6 As shown, the lightweight multi-path feature perception network consists of lightweight convolution, multi-attention collaboration, and a multi-path fusion mechanism. This achieves complementary optimization of global and local features, enhancing the deep network's ability to perceive key information and obtaining more scale-based feature information while maintaining lightweight computation. (Input feature map) After channel attention and 1×1 convolution, we obtain , After passing through the Norm layer and undergoing a 1×1 pointwise convolution, we obtain... , After passing through a Norm layer, a 3×3 depthwise separable convolution, and a 1×1 convolution, we obtain... ,at the same time After spatial attention and 1×1 convolution, we obtain . and Dot product to get . After 3×3 depthwise separable convolution and 1×1 convolution, we obtain... . and Dot product to get . After 1×1 convolution and Adding them together gives . and Adding them together gives , and After addition, a 1×1 convolution is performed to obtain The formula is as follows:

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] Hardware deployment architecture, such as Figure 7As shown, the core hardware system includes a camera driver circuit, a Wi-Fi driver circuit, and a main control system. The main control chip internally deploys a residual dynamic gated fuzzy discrimination network, a U-shaped deblurring network, a camera driver, and a Wi-Fi driver. Image information captured by the external camera is transmitted to the residual dynamic gated fuzzy discrimination network inside the main control chip to determine if the image information is blurry. If blurry, deblurring is performed. If not blurry, the image information is uploaded to the cloud via the Wi-Fi circuit so that users can view the captured image information in real time. For the deployment of the network model on edge devices, the Cube.AI tool of STM32CubeMX software is used to convert the Python code of the network model into C code. The converted code is then quantized and trimmed before being deployed on the edge devices.

[0064] Step 3: Train the network model; train the network model using the prepared dataset.

[0065] Step 4: Optimize the model and evaluate its performance; the model is optimized using three loss functions: mean squared error, mean absolute error, and frequency domain loss, and the peak signal-to-noise ratio and structural similarity are used as evaluation metrics to evaluate the model performance.

[0066] Mean Squared Error Loss (MSE):

[0067]

[0068] Mean Absolute Error (MAE):

[0069]

[0070] Frequency domain loss:

[0071]

[0072] Total loss:

[0073]

[0074] In the formula , and These are the weight hyperparameters of the various loss functions, used to adjust the influence of different loss terms. In this invention... , and The values ​​are 0.3, 0.5 and 0.2 respectively.

[0075] Peak signal-to-noise ratio (PSNR) and structural similarity were chosen as appropriate evaluation metrics. PSNR is an evaluation metric based on the error between pixels, used to sensitively assess image quality. Structural similarity, on the other hand, measures the similarity between images from three aspects: brightness, contrast, and structure, and is used to evaluate the degree of similarity between digital images. PSNR and structural similarity are defined as follows.

[0076] ;

[0077] MAX is the maximum possible pixel value of the image. I and K are the pixel values ​​of the original image and the distorted image, respectively, and M and N are the dimensions of the image.

[0078] ;

[0079] and It is the average of the x and y values ​​of the image. and It is the variance of the x and y values ​​of the image. It is the covariance of the x and y axes of the image. and It is a constant used for stability, to avoid the denominator being zero.

[0080] The training iterations were set to 2000. The learning rate for the first 1000 training iterations was set to 0.001, and the learning rate for the next 100 training iterations was gradually reduced from 0.001 to 0. The Adam optimizer was selected as the network parameter optimizer.

[0081] Step 5: Fine-tune the model; Input the dataset into the network model for training, compare the results with image evaluation metrics, and then fine-tune the model parameters;

[0082] Step 6: Save the model; solidify the parameters of the final optimized network model and save the network model;

[0083] Step 7: Hardware deployment and debugging; Convert the saved network model into C language using STM32cube.AI and deploy it to the embedded system, and debug the embedded system.

[0084] This invention presents a lightweight multipath image deblurring method and system for edge deployment. It achieves image deblurring in an end-to-end manner, eliminating the need for complex degradation models and avoiding the need to study different types of image blur. The system is then deployed to edge devices for real-time processing.

[0085] Table 1 compares the relevant indicators of existing technologies and the method proposed in this invention.

[0086] method Peak signal-to-noise ratio Structural similarity Parameters CN119809980A 32.42 0.956 5.9 DeepDeblur 29.23 0.914 11.7 HINet 32.71 0.959 170.71 Example 32.69 0.955 5.4

[0087] Table 1 shows the average PSNR and average SSIM of some methods on the GoPro test set, as well as the number of parameters and computational complexity of these methods. According to the results in Table 1, the method proposed in this invention outperforms most existing methods on the GoPro dataset, and its performance is most outstanding in terms of fewer parameters and less computational complexity.

[0088] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight multipath image deblurring method for edge deployment, characterized in that: Includes the following steps: Step 1: Prepare the datasets; Prepare three datasets: Dataset 1 (GoProDataset), Dataset 2 (RealBlur Dataset), and Dataset 3 (Köhler Dataset). Step 2: Construct a lightweight dynamic image deblurring network model: The lightweight dynamic image deblurring network model includes a U-Net network and a residual dynamic gated blur discrimination network. The U-Net network includes a Manling gated network, a dual-response enhancement fusion network, and a lightweight multi-path feature perception network. Step 3: Train the network model; train the network model using the prepared dataset; Step 4: Optimize the model and evaluate its performance; the model is optimized using three loss functions: mean squared error, mean absolute error, and frequency domain loss, and the peak signal-to-noise ratio and structural similarity are used as evaluation metrics to evaluate the model performance; Step 5: Fine-tune the model; Input the dataset into the network model for training, compare the results with image evaluation metrics, and then fine-tune the model parameters; Step 6: Save the model; solidify the parameters of the final optimized network model and save the network model; Step 7: Hardware deployment and debugging; Convert the saved network model into C language using STM32cube.AI and deploy it to the embedded system, and debug the embedded system.

2. The lightweight multipath image deblurring method for edge deployment according to claim 1, characterized in that: In step 1, random scaling, inversion, and translation are used to invert and translate dataset 1 to expand the dataset, while providing multi-scale information; dataset 2 is used for model fine-tuning; dataset 3 is used to test the model and verify its generalization ability under various fuzzy conditions; the size of each image in each dataset is 512x512.

3. The lightweight multipath image deblurring method for edge deployment according to claim 1, characterized in that: In step 2, the acquired image data is processed by a residual dynamic gated fuzzy discrimination network to determine whether the image needs to be deblurred. If no deblurring is required, the image is output directly. This operation can reduce unnecessary calculations and improve the system's running speed. If deblurring is required, the image is processed by the U-Net network.

4. The lightweight multipath image deblurring method for edge deployment according to claim 1, characterized in that: In step 2, the Manning gated network consists of two branches. The first branch consists of Linear, 3×3 depthwise separable convolution, SuLU activation function and state space network (SSM). The second branch consists of Linear and SuLU activation. The cross product of the two branches is then subjected to 1×1 convolution. This is used to enhance the network's ability to model temporal and global dependency information during the deblurring process. The dual-response enhancement fusion network consists of two branches. The inputs of the two branches are the feature map after the output of the upper-level decoder / encoder has undergone 3×3 convolution and the feature map without 3×3 convolution, respectively. One of the branches performs channel-level adaptive feature fusion on the feature map without 3×3 convolution. Another branch preprocesses the feature map after 3×3 convolution using the SimAM attention mechanism, and then extracts edge features through pooling, subtraction, convolution and multiple residual connections to enhance the contrast and saliency of edge information; the results of the two branches are fused to obtain richer contextual information and achieve efficient extraction of multi-level features; The lightweight multi-path feature perception network consists of lightweight convolution, multi-attention collaboration, and multi-path fusion mechanisms, which achieve complementary optimization of global and local features, enhance the deep network's ability to perceive key information, and obtain more scale feature information while ensuring lightweight computation.

5. A lightweight multipath image deblurring method for edge deployment according to claim 1, characterized in that: In step 2, the residual dynamic gated fuzzy discriminant network enhances the contrast between fuzzy and non-fuzzy features by extracting residual deep features and combining them with a dual-gated feedforward output layer, thereby reducing the misclassification rate and achieving high-precision fuzzy classification.

Images