A multi-scale image deblurring method based on latent space conditional diffusion model

Abstract

The present application relates to the field of image deblurring, and discloses a multi-scale image deblurring method based on a latent space conditional diffusion model, comprising the following steps: constructing a multi-scale image deblurring network, the network comprising a conditional diffusion model and a sliding window attention module, the conditional diffusion model being used to generate multi-scale prior features from a blur condition vector in a latent space; the sliding window attention module being a U-shaped network based on an encoder-decoder, and being used to perform image deblurring feature regression guided by the multi-scale prior features; the network being trained using a two-stage strategy comprising pre-training and post-training; and a blur image to be processed being input into the trained multi-scale image deblurring network, and a final deblurred image being output. The method disclosed in the present application can effectively suppress the common problems of over-smoothing and artifacts in the deblurring process while ensuring computational efficiency, and improve the recovery accuracy of texture details and edge structures.

Description

Technical Field

[0001] This invention relates to the field of image deblurring, and in particular to a multi-scale image deblurring method based on a latent spatial conditional diffusion model. Background Technology

[0002] With the continuous improvement of industrial intelligence and automation, image processing technology has become a key component of the manufacturing quality inspection system. In numerous application scenarios such as machining, traffic monitoring, security monitoring, and consumer electronics, image quality directly affects the accuracy of subsequent visual tasks, including target detection, surface defect identification, dimensional measurement, and status monitoring. However, during actual data acquisition, factors such as equipment vibration, motion blur, lighting changes, insufficient depth of field, and atmospheric scattering often lead to varying degrees of blurring and degradation in images, resulting in loss of image details and unclear edges, severely impacting the reliability of visual algorithms.

[0003] Traditional image deblurring methods mainly include end-to-end learning methods based on convolutional neural networks (CNNs) and reconstruction methods based on generative models. CNN-like algorithms typically rely on fixed receptive fields and local convolution operators, making it difficult to effectively handle multi-scale blur characteristics in scenes, and their generalization ability is limited. When the blur type is complex and the blur kernel is non-uniformly distributed, their reconstruction performance deteriorates significantly. With the development of deep learning, diffusion models have gradually gained attention in image reconstruction tasks due to their powerful generation capabilities and stable training characteristics. However, most existing diffusion models perform noise addition and inversion processes in pixel space, resulting in high computational complexity and slow inference speed. At the same time, their performance in real-world complex blurry scenes is still limited by insufficient conditional modeling and limited prior representation capabilities. In addition, existing image deblurring methods generally suffer from degradation when processing high-frequency texture restoration, and the generated results may contain artifacts, missing details, or structural deformations, making it difficult to meet the requirements of high fidelity and high robustness for industrial inspection. In real environments such as machining workshops, image degradation factors are diverse and highly random, making it difficult for traditional methods to achieve a balance between generalization and speed.

[0004] To address the aforementioned issues, the academic community has gradually explored solutions based on the Latent Diffusion Model (LDM). LDM maps the diffusion process to a low-dimensional latent space, learning more compact image representations and significantly improving computational efficiency while maintaining generation quality. However, existing LDMs still have shortcomings in handling multi-scale fuzzy information fusion, cross-layer feature interactions, and structural prior representations, leading to unstable reconstruction quality in realistic and complex scenes and an inability to fully capture the correlation between global structure and local details.

[0005] Therefore, there is an urgent need for a novel image deblurring method that can efficiently model multi-scale fuzzy information, integrate diffusion priors with strong expressive power, and take into account both high-quality reconstruction and fast inference, so as to meet the comprehensive requirements of industrial applications for stability, speed and accuracy. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a multi-scale image deblurring method based on a latent spatial conditional diffusion model. While ensuring computational efficiency, it effectively suppresses common problems such as oversmoothing and artifacts during the deblurring process, and improves the accuracy of texture detail and edge structure restoration.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A multi-scale image deblurring method based on a latent spatial conditional diffusion model includes the following steps:

[0009] Step 1: Construct a multi-scale image deblurring network, the network comprising:

[0010] The conditional diffusion model includes a sharp encoder, a fuzzy encoder, a denoising U-shaped network, and a multi-scale prior generation module, which is used to generate multi-scale prior features from fuzzy conditional vectors in the latent space; the conditional diffusion model performs noise modeling and feature reconstruction in the low-dimensional feature space by forward noise addition and reverse denoising.

[0011] The sliding window attention module, a U-shaped encoder-decoder network, is used to perform image deblurring feature regression guided by multi-scale prior features. This module includes an encoder path and a decoder path. The encoder path performs multi-level downsampling of the input features to extract multi-scale contextual information; the decoder path performs multi-level upsampling of the features to progressively restore spatial resolution. At least one sliding window attention deblurring module is integrated after the dual-channel integration module in each level of the encoder and decoder paths. The levels corresponding to the encoder and decoder paths also include cross-level skip connections to concatenate features from each level of the encoder path with features from an intermediate level of the decoder path to preserve high-frequency details.

[0012] Step 2, Model Training:

[0013] The multi-scale image deblurring network is trained using a two-stage strategy including pre-training and post-training to learn the mapping relationship from blurred images to sharp images.

[0014] Step 3, Model Application:

[0015] The blurred image to be processed is input into the trained multi-scale image deblurring network, and then processed sequentially by the conditional diffusion model and the sliding window attention module to output the final deblurred image.

[0016] In the above scheme, in the conditional diffusion model, the sharp encoder is used to stitch the sharp image and the blurry image together and map them to the latent space to obtain the initial feature vector; the blurry encoder is used to compress the blurry image separately into a blurry conditional vector.

[0017] The denoising U-shaped network is composed of multiple residual blocks and attention blocks stacked together. It receives a fuzzy condition vector and a noise vector from the forward diffusion process, predicts the noise estimate and variance interpolation at each step of the diffusion process, and obtains the denoised feature vector based on the noise estimate and variance interpolation through reparameterized sampling.

[0018] The multi-scale prior generation module generates at least three prior features at different scales by performing one-dimensional convolution downsampling on the denoised feature vector.

[0019] In the above scheme, the residual blocks in the denoising U-shaped network include:

[0020] The feature transformation branch includes, in sequence, a group normalization layer, an activation function layer, and a convolutional layer;

[0021] The conditional embedding branch includes a temporal embedding sub-branch and a conditional embedding sub-branch, which map time step information and conditional information into embedding vectors, respectively, and fuse them in the channel dimension. In the temporal embedding sub-branch, the time step information is mapped to a temporal embedding vector with the same number of channels as the feature transformation branch by passing through a temporal embedding layer, a linear layer, an activation function layer, and another linear layer. In the conditional embedding sub-branch, the conditional information is fused with the temporal embedding vector in the channel dimension through a label embedding layer.

[0022] The residual connection branch adds the output after feature transformation and conditional fusion to the original input element by element.

[0023] In the above scheme, the attention block is a cross-location attention module, which is used to establish global dependencies on low-resolution feature maps and enhance the ability to recover large-scale blurred structures.

[0024] In the above scheme, the multi-scale prior generation module achieves multi-scale downsampling through the following formula:

[0025] ;

[0026] in, The denoised feature vectors are Let represent the prior features at the s-th scale.

[0027] In the above scheme, the dual-channel integrated module performs the following operations:

[0028] After reshaping the intermediate features of the blurred image, the query vector Q is obtained through layer normalization and linear mapping.

[0029] The prior features are split into two branches along the channel dimension, and after layer normalization and linear mapping, they are concatenated into key vector K and value vector V.

[0030] Perform cross-self-attention computation on Q, K, and V;

[0031] The attention output is added to the residual of the original intermediate features, and the local texture recovery capability is enhanced by two convolutional layers.

[0032] In the above scheme, the sliding window attention deblurring module includes, in sequence:

[0033] In the first attention stage, a window self-attention mechanism is adopted to divide the input features into multiple non-overlapping windows. Attention is calculated independently within each window to reduce computational complexity and focus on local detail recovery.

[0034] In the second attention stage, a sliding window self-attention mechanism is adopted. By performing regular translation of the window, the fixed window boundary is broken, and feature interaction between adjacent windows is realized to model long-distance dependencies.

[0035] Feedforward networks are used to enhance the nonlinear expressive power of features;

[0036] The first residual connection adds the features processed by the first and second attention stages to the original input features;

[0037] The second residual connection adds the output of the feedforward network to the output of the first residual connection.

[0038] In the above scheme, the pre-training phase during network training includes:

[0039] Prior features are extracted from sharp-blurred image pairs using a sharp encoder and downsampled into multi-scale prior features;

[0040] The blurred image and the multi-scale prior features are input into the sliding window attention module, and the sliding window attention module is trained end-to-end using the L1 loss function.

[0041] In the above scheme, the post-training phase during network training includes:

[0042] Freeze the pre-trained sharp encoder parameters;

[0043] The conditional diffusion model is trained to learn to generate multi-scale prior features consistent with the semantics of a clear image from fuzzy conditional vectors and random noise.

[0044] The overall network is optimized using a joint loss function, which includes the mean squared error loss of the conditional diffusion model, the KL divergence loss, and the defuzzing L1 loss.

[0045] The multi-scale image deblurring method based on a latent spatial conditional diffusion model provided by the present invention, through the above technical solution, has the following beneficial effects:

[0046] 1. This invention proposes a multi-scale image deblurring network (MSdiff). This network uses a conditional diffusion model to generate multi-scale prior features in the latent space, reducing computational overhead while improving the model's generalization ability. Then, the priors are integrated into the sliding window attention module to improve deblurring performance.

[0047] 2. This invention proposes a dual-channel integration module (DCIM) based on the fusion of diffusion prior features and blurred image features. This module can decompose cross-attention information into a dual-channel architecture according to the channel dimension, so as to achieve better information fusion between prior features and blurred intermediate features. DCIM can improve the ability to extract multi-level information from prior features, improve the processing of high-frequency features of blurred images, and help to recover image texture, details and other information;

[0048] 3. Based on the sliding window self-attention characteristics of the multi-scale diffusion model combined with the regression method, this invention proposes a sliding window attention deblurring module (ID-SwinT). This module mainly includes window attention and feedforward network channels. ID-SwinT can reduce the overall complexity of the deblurring architecture while capturing the details of blurred feature images over long distances, thereby optimizing the impact of the diffusion model easily generating additional artifacts and improving the performance of the deblurred image in terms of distortion accuracy. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0050] Figure 1 This is a schematic diagram of a multi-scale image deblurring network (MSdiff) disclosed in an embodiment of the present invention;

[0051] Figure 2 Here are structural diagrams of a clear encoder and a fuzzy encoder;

[0052] Figure 3 Here are the structure diagrams for residual blocks and attention blocks; (a) shows the residual block; (b) shows the attention block.

[0053] Figure 4 This is a structural diagram of a dual-channel integrated module (DCIM).

[0054] Figure 5 Structure diagram of the sliding window attention deblurring module (ID-SwinT);

[0055] Figure 6 This is a schematic diagram of the multi-scale image deblurring process. Detailed Implementation

[0056] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0057] This invention provides a multi-scale image deblurring method based on a latent spatial conditional diffusion model, comprising the following steps:

[0058] Step 1: Construct a multi-scale image deblurring network, named MSdiff. Through joint modeling and learning of blurred images and their corresponding clear images, high-quality deblurring and reconstruction of complex, non-uniformly blurred images is achieved. This is particularly suitable for industrial and real-world scene images with strong motion blur, spatial non-uniform blur, and high-frequency detail degradation. The method proposed in this invention can effectively suppress common problems such as over-smoothing and artifacts in the deblurring process while ensuring computational efficiency, and improve the accuracy of texture detail and edge structure recovery.

[0059] like Figure 1 As shown, the MSdiff network consists of two parts: a conditional diffusion model and a sliding window attention module.

[0060] I. Conditional Diffusion Model

[0061] The conditional diffusion model includes a sharp encoder, a fuzzy encoder, a denoising U-shaped network, and a multi-scale prior generation module, which is used to generate multi-scale prior features from fuzzy conditional vectors in the latent space. The conditional diffusion model performs noise modeling and feature reconstruction in the low-dimensional feature space by adding noise in the forward direction and denoising in the reverse direction.

[0062] 1. Clear encoder

[0063] A sharp encoder is used to concatenate sharp and blurry images and map them into a latent space to obtain an initial feature vector. The structure is as follows: Figure 2As shown, the stitched image is first spatially rearranged by pixel ungrouping, reducing resolution and increasing the number of channels. Then, a convolutional layer is used for initial mapping and alignment of the rearranged features. These features are then input into L residual modules to improve the ability to represent high-frequency details while ensuring the stability of deep network training. Following this, convolutional and pooling layers compress local to global features, and finally, linear layers reshape and project the final latent feature representation.

[0064] 2. Fuzzy encoder

[0065] A fuzz encoder is used to compress a fuzzy image into a fuzzy conditional vector; the fuzz encoder has the same structure as the sharp encoder.

[0066] 3. Noise-reducing U-shaped network

[0067] The denoising U-shaped network consists of multiple stacked residual blocks and attention blocks, following an overall upsampling-downsampling structure. For example... Figure 1 As shown, this U-shaped network receives a fuzzy conditional vector and a noise vector from the forward diffusion process. The two vectors are fused along the channel dimension and then input into a convolutional layer, mapping them to a unified channel dimension. After a downsampling step, the network passes through multiple residual blocks, with attention blocks inserted at key layers to model long-range dependencies and global contextual information. Intermediate features from the downsampling stage are directly passed to the corresponding upsampling layer via skip connections, ensuring that high-resolution details are not lost during the diffusion denoising process. Finally, through upsampling, normalization layers, and convolutional operations, the noise estimate and variance interpolation at each step of the diffusion process are predicted. Based on the noise estimate and variance interpolation, the denoised feature vector is obtained through reparameterized sampling.

[0068] (1) Residual block

[0069] like Figure 3 As shown in (a), the residual block includes a feature transformation branch, a conditional embedding branch, and a residual connection branch.

[0070] The feature transformation branch consists of a group normalization layer, an activation function layer, and a convolutional layer. Input features first pass through group normalization and activation functions to stabilize the feature distribution and enhance non-linear expressive power. Then, the output of the convolutional layer is added element-wise with the features from the conditional embedding branch, injecting conditional information into the feature transformation process.

[0071] The conditional embedding branch includes a temporal embedding sub-branch and a conditional embedding sub-branch, which map time step information and conditional information into embedding vectors, respectively, and fuse them in the channel dimension. In the temporal embedding sub-branch, the time step information is mapped to a temporal embedding vector with the same number of channels as the feature transformation branch by passing through a temporal embedding layer, a linear layer, an activation function layer, and another linear layer. In the conditional embedding sub-branch, the conditional information is fused with the temporal embedding vector in the channel dimension through a label embedding layer and used as a feature input to the conditional node.

[0072] The residual connection branch adds the output after feature transformation and conditional fusion to the original input element by element through residual connection to obtain the final output.

[0073] Through the above structure, the residual block, while maintaining the continuous transmission of input features, introduces time step information and conditional information to jointly modulate the feature transformation process, thereby enhancing the adaptive feature modeling capability of the denoising U-shaped network under different diffusion time steps.

[0074] (2) Attention block

[0075] like Figure 3 As shown in (b), the attention block in this embodiment is a cross-location attention module, which is used to establish global dependencies on low-resolution feature maps. It is good at capturing long-distance dependencies and enhances the ability to recover large-scale fuzzy structures through dynamic calculation of global feature dependencies.

[0076] like Figure 1 As shown in the conditional diffusion model, the sharp encoder maps the features of the concatenated sharp and blurred images to the latent space to obtain the initial feature vector. Immediately following Add T-step Gaussian noise to The blurred image is then compressed using a fuzz encoder to... As a fuzzy conditional vector and Gaussian noise The inputs are fed into a denoising U-shaped network, which predicts the noise vector at each step of the forward denoising process. Finally, after T iterations of sampling, the predicted feature vector is output. .

[0077] Forward-spreading Gaussian noise vector With fuzzy condition vector After concatenation, the data is transmitted to multiple residual blocks and attention blocks, establishing global dependencies on temporal and spatial information at low resolution. Then, downsampling extracts global information within a larger receptive field. The intermediate layer further fuses temporal and spatial features, and in upsampling, skip connections preserve high-resolution details, improving denoising accuracy. Finally, the estimated noise and variance interpolation are output. The following details the entire diffusion model process in the latent space:

[0078] This invention generates prior features using a conditional diffusion model in the latent space. The diffusion model includes forward and backward processes. In the forward process, the blurred image and the sharp labeled image are stitched together and then compressed into an initial feature vector via a sharp encoder. ,Towards Gradually add T-order Gaussian noise until... The values ​​are independent standard Gaussian distributions, as shown in the following formula:

[0079] (1);

[0080] In the formula Indicates forward diffusion distribution, Indicates a Gaussian distribution. Indicates the form of noise addition. This represents the joint distribution of the forward diffusion process.

[0081] This invention employs a cosine noise-adding scheme, as shown in equation (2):

[0082] (2);

[0083] In the formula, From equation (1), we can derive:

[0084] (3);

[0085] Equation (3) shows that given an initial eigenvector Then the diffusion distribution at any time t can be obtained. I represents an identity matrix with all diagonals equal to 1.

[0086] The goal of the reverse process is to gradually denoise from independent Gaussian noise to generate prior features. This invention assumes that the posterior distribution of learnable mean and variance is as follows:

[0087] (4);

[0088] Among them, the mean ,variance Represented as:

[0089] (5);

[0090] (6);

[0091] In equations (5) and (6), noise Interpolation v It is an uncertain variable. This invention utilizes a denoising U-shaped network to estimate the noise. Interpolate v, and then use The mean and variance of the sample are calculated with v, and finally, reparameter sampling is performed. The distribution is shown in equation (7):

[0092] (7);

[0093] In the above formula, It is Gaussian noise. These represent the noise and variance interpolation estimated by the denoising U-shaped network, respectively. This is the fuzzy conditional vector of the fuzzy image after compression by the fuzzy encoder. The feature vector can be generated by repeatedly iterating the sampling formula in equation (7) T times. As a priori feature, it can also be denoted as ( Figure 1 (Prior 1 in the middle).

[0094] 4. Multi-scale prior generation module

[0095] To construct multi-scale prior features, the multi-scale prior generation module generates prior features at three different scales by performing one-dimensional convolutional downsampling on the denoised feature vectors. ,correspond Figure 1 Prior 1, Prior 2, and Prior 3.

[0096] The multi-scale prior generation module achieves multi-scale downsampling through the following formula:

[0097] (8);

[0098] in, The denoised feature vectors are Let represent the prior features at the s-th scale.

[0099] The aforementioned multi-scale prior features are aligned and fused with feature maps of different resolutions in the sliding window attention module.

[0100] Will Figure 1 Predicted feature vector The prior features, multi-scale, and the intermediate vector of the blurred image are fused in the dual-channel integration module (DCIM), and the fused features are then input into the regression module ID-SwinT (structure as follows). Figure 5 As shown, features of the blurred image are extracted, and then the blurred image information is further refined through downsampling / upsampling stages. In order to retain more image details, the resolution features from the encoder are stitched together layer by layer in the decoder part. Finally, the original image size is restored by convolution to obtain the deblurred image.

[0101] II. Sliding Window Attention Module

[0102] The sliding window attention module is an encoder-decoder-based U-shaped network used to perform image deblurring feature regression guided by multi-scale prior features. This module employs a sampling-downsampling multi-level U-shaped network architecture to extract and reconstruct deblurring features layer by layer within the regression framework. The module uses the intermediate features of the blurred image as the backbone and introduces multi-scale prior features generated by a conditional diffusion model at different resolution levels to achieve prior-guided hierarchical image deblurring.

[0103] The sliding window attention module includes an encoder path and a decoder path. The encoder path is used to perform multi-level downsampling of the input features to extract multi-scale contextual information; the decoder path is used to perform multi-level upsampling of the features to progressively restore spatial resolution. In each level of the encoder and decoder paths, at least one sliding window attention deblurring module is integrated after the dual-channel integration module. The levels corresponding to the encoder and decoder paths also include cross-level skip connections, which are used to concatenate the features of each level in the encoder path with the features of the intermediate level in the decoder path to preserve high-frequency details.

[0104] First, the blurred image is mapped to the feature space through convolutional layers to obtain an initial feature representation. This feature then undergoes a multi-level downsampling stage, each stage containing several sliding window attention deblurring modules (ID-SwinT). The downsampling operation progressively reduces the feature resolution and expands the receptive field. Specifically, in the encoder path, the first layer stacks three ID-SwinTs and fuses the prior 1 generated by the conditional diffusion model with the current blurred feature through a dual-channel integration module (DCIM). The second layer stacks five ID-SwinTs and fuses the prior 2 generated by the conditional diffusion model with the first-level output through DCIM. The third layer stacks six ID-SwinTs and fuses the prior 2 with the second-level output through DCIM. In the intermediate layer stage, six ID-SwinTs are stacked and the prior 3 is fused with the downsampled third-level output through DCIM. In each layer, DCIM is embedded before the ID-SwinTs to perform cross-attention computation, enabling the diffused prior features to effectively guide the representation learning of the blurred features at different scales. In the decoder path, the upsampling and downsampling sections are structurally symmetrical. They employ a layer-by-layer approach to restore spatial resolution, and use skip connections to stitch features from the corresponding downsampling levels into the upsampling to preserve high-frequency details. Specifically, the first upsampling level first stitches features from the downsampling stage according to their dimensions, then stacks six ID-SwinTs, and fuses the prior 2 generated by the conditional diffusion model with the ID-SwinT output using DCIM. Similarly, the second level stitches features from the corresponding downsampling stage according to their resolution, stacks five ID-SwinTs, and fuses the prior 2 generated by the conditional diffusion model with the ID-SwinT output of the second level using DCIM. Finally, the third level stitches features through skip connections, stacks three ID-SwinTs, and again fuses the prior 1 generated by the conditional diffusion model with the ID-SwinT output of the third level using DCIM. This design ensures that during feature reconstruction, both low-level spatial details are integrated, and high-level semantics and diffusion prior information are fully utilized. Finally, at the end of the upsampling stage, the features are mapped back to the image space through a convolutional layer to obtain the final deblurred output image.

[0105] 1. Dual-channel integrated module (DCIM)

[0106] To more fully integrate prior features and intermediate features of blurred images, this invention proposes a dual-channel integrated module (DCIM), the detailed architecture of which is as follows: Figure 4 As shown, the dual-channel integration module first performs cross-attention calculations on the prior features and the intermediate features of the blurred image, and then aggregates the output information into the sliding window attention deblurring module (ID-SwinT), thereby effectively guiding the sliding window attention module architecture for image deblurring training.

[0107] Specifically, the blurred image is processed through a convolutional module to obtain intermediate feature input. The input is then reshaped, normalized, and processed through linear layers, before being mapped to... Q This refers to the "query" in the attention mechanism. Prior features are reshaped into two branches, which are then split into dual-channel architectures along the channel dimension. These dual-channel architectures undergo layer normalization and linearization respectively, and the two channels are then concatenated and mapped to... K and V This refers to the "keys" and "values" in the attention mechanism. Q, K, V Perform cross-self-attention computation ( Figure 4 (Only one attention point is displayed in the text), the formula is as follows:

[0108] (9);

[0109] in, express Q and K The feature dimension, namely the number of channels.

[0110] Finally, the cross-attention result is reshaped. To better reconstruct the details of the deblurred image, this invention adds the result to the intermediate feature input residual to obtain the output feature. Furthermore, to compensate for the shortcomings of the attention mechanism in local detail modeling, this invention introduces two convolutional operations before the residual addition to enhance the recovery capability of fine-grained textures.

[0111] Compared to single-channel integrated modules, the core innovation of dual-channel integrated modules lies in:

[0112] (1) Dual-channel feature interaction: Each attention head uses twice the number of channels for "keys" and "values" during computation, enabling the network to learn complementary feature subspaces under the same prior input, thereby enhancing its ability to extract multi-layer information from the prior.

[0113] (2) Combination of global and local modeling: Cross-self-attention is good at capturing long-distance dependencies, while additional convolutional layers enhance local detail recovery. The combination of the two makes the deblurred image more refined in edge and texture reconstruction.

[0114] (3) Multi-scale prior design: Blurring in real-world scenarios often manifests as complex nonlinearity and spatial inhomogeneity, while single-scale prior features are insufficient to handle complex ambiguity situations. Therefore, this invention downsamples the original prior twice via one-dimensional convolution to obtain multi-scale priors. For example... Figure 1 As shown, multi-scale prior features can be used to adapt to intermediate features of blurred images of different sizes, which can adapt to different degrees of blur and receptive field sizes, thereby improving the model's generalization ability and information fusion effect in diverse blurred scenes.

[0115] The dual-channel integrated module not only achieves dual-channel feature fusion and cross-scale information guidance in terms of structure, but also improves the model's ability to recover detailed features in terms of performance, providing stronger support for deblurring complex and blurred images in industrial scenarios.

[0116] Therefore, it can be seen that the dual-channel integrated module performs the following operations:

[0117] After reshaping the intermediate features of the blurred image, the query vector Q is obtained through layer normalization and linear mapping.

[0118] The prior features are split into two branches along the channel dimension, and after layer normalization and linear mapping, they are concatenated into key vector K and value vector V.

[0119] Perform cross-self-attention computation on Q, K, and V;

[0120] The attention output is added to the residual of the original intermediate features, and the local texture recovery capability is enhanced by two convolutional layers.

[0121] 2. Sliding window attention deblurring module (ID-SwinT)

[0122] This invention combines the characteristics of sliding window attention with image deblurring tasks, and proposes a sliding window attention deblurring module (ID-SwinT), the structure of which is as follows: Figure 5 After the intermediate features of the blurred image are fused with prior information in the dual-channel integration module, the window self-attention mechanism (W-MHSA) and the sliding window self-attention mechanism (SW-MHSA) are calculated sequentially.

[0123] First-stage window self-attention mechanism:

[0124] The features are divided into multiple non-overlapping windows, and attention is computed independently within each window. This design reduces the computational complexity of attention from the traditional... Reduce to ,in, L The length of the feature vector. C For the feature dimension, the formula is as follows:

[0125] (10);

[0126] (11);

[0127] In the formula, 𝜴 represents the computational complexity. W This indicates the number of windows to be divided.

[0128] This complexity optimization is particularly important for deblurring high-resolution images, significantly improving the model's scalability and efficiency.

[0129] Second-stage sliding window self-attention mechanism:

[0130] Building upon the window self-attention mechanism, the limitation of fixed window boundaries is broken through window translation, enabling connections to be established between adjacent windows. This design effectively eliminates boundary effects and enhances the model's ability to model long-distance dependencies and cross-regional fuzzy patterns. After two-stage attention modeling, the features and original input are summed using residuals to form the first residual branch. The result then enters the feedforward network to further enhance nonlinear representation capabilities. Finally, residuals are summed again to form the second residual branch, ensuring the model maintains stable training while possessing stronger feature representation capabilities.

[0131] The innovation of the sliding window attention deblurring module in the image deblurring process is mainly reflected in the following aspects:

[0132] (1) The window self-attention mechanism limits the computational attention range and is better at capturing local high-frequency details (such as edges and textures) of blurred images, making the deblurring results more focused and sharper in detail reconstruction and less prone to blur artifacts; while the sliding window self-attention mechanism can integrate a large range of contextual information layer by layer, which is particularly effective for repairing large-scale degradation such as motion blur and out-of-focus blur.

[0133] (2) Through window partitioning and translation mechanism, sliding window attention can significantly reduce computational complexity while still maintaining the ability to comprehensively model global and local information, thus taking into account both efficiency and image deblurring quality.

[0134] Therefore, it can be seen that the sliding window attention deblurring module includes, in sequence:

[0135] In the first attention stage, a window self-attention mechanism is adopted to divide the input features into multiple non-overlapping windows. Attention is calculated independently within each window to reduce computational complexity and focus on local detail recovery.

[0136] In the second attention stage, a sliding window self-attention mechanism is adopted. By performing regular translation of the window, the fixed window boundary is broken, and feature interaction between adjacent windows is realized to model long-distance dependencies.

[0137] Feedforward networks are used to enhance the nonlinear expressive power of features;

[0138] The first residual connection adds the features processed by the first and second attention stages to the original input features;

[0139] The second residual connection adds the output of the feedforward network to the output of the first residual connection.

[0140] Step 2, Model Training:

[0141] 1. Dataset preparation stage:

[0142] This invention evaluates a multi-scale diffusion model on synthetic and real-world datasets. The synthetic datasets include GoPro and MSB. The GoPro dataset contains 2103 pairs of blurred and sharp images for training and 1111 pairs for testing. The MSB dataset contains 1934 pairs of blurred and sharp images, all with a resolution of 1920×1080, covering various typical scenes in a machining workshop, such as the working states of lathes, milling machines, and drilling machines, as well as complex lighting and material reflection conditions. The real-world dataset primarily uses the RealBlur dataset, which has two subsets: RealBlur-J and RealBlur-R. Each subset consists of 3758 training pairs and 980 test pairs. This invention trains the multi-scale diffusion model on the GoPro training dataset and evaluates it on four test datasets.

[0143] 2. Training Phase:

[0144] The multi-scale diffusion model provided by this invention undergoes two training processes: pre-training and post-training. In the pre-training stage, the purpose of this invention is to pre-train the sharp encoder and the sliding window attention module. After the sharp image and the blurred image are stitched together, the sharp encoder compresses them from the pixel space to the latent space to obtain prior features. Then, the prior features are downsampled through one-dimensional convolution to generate multi-scale prior features (prior 1, prior 2, and prior 3). These prior features are then integrated into the sliding window attention module architecture at multiple scales via a dual-channel integration module (DCIM). During image deblurring training, the prior features provide clear guidance for the sliding window attention module, thereby better reconstructing the detailed features of the deblurred image. In the pre-training stage, this invention uses the L1 loss function for training optimization, as shown in equation (12). , These represent the clear image with labels corresponding to the deblurred image and the blurred image, respectively.

[0145] (12).

[0146] This invention loads the model parameters of the well-trained clear encoder and sliding window attention module from the pre-training stage into the post-training stage. Simultaneously, to alleviate computational bottlenecks, this invention employs a training strategy of freezing the clear encoder parameters in the post-training stage, effectively reducing the computational burden on the conditional diffusion model during this training phase. Figure 1As shown. Unlike the pre-training stage, which directly obtains prior features through a clear encoder, the post-training stage learns to generate prior features by training a conditional diffusion model of the latent space. ( (Estimated prior after diffusion process). Wherein, the fuzzy conditional vector... It is obtained by compressing a blurred image through a blur encoder. After training, the generated prior features are downsampled through convolution to generate multi-scale priors. Finally, the multi-scale prior features are used by a dual-channel ensemble module to guide the deblurring process in the sliding window attention module.

[0147] In the post-training phase, the training loss consists of two parts: the conditional diffusion model and the sliding window attention module. The MSE loss is used in the training of the conditional diffusion model. Furthermore, to achieve better log-likelihood on image datasets, this invention adds a forward diffusion distribution to the original conditional diffusion model training loss. With backward denoising distribution KL divergence loss between distributions As shown in equation (13):

[0148] (13);

[0149] in, This represents the training loss function during the conditional diffusion phase. It represents the expectation of a variable with parameters. Indicates noise. The network represents the noise estimation network, where is a hyperparameter and is a local minimum value. During training, this invention sets it to 0.002. Indicates the KL divergence loss;

[0150] Defined as:

[0151] (14)

[0152] in, This is the reverse denoising process, where q is the forward distribution. This indicates the final denoising step, which involves retrieving samples during the reverse denoising process. Recover clean sample , Indicates the forward process from the sample , reasoning The posterior distribution, This indicates the reverse denoising process from the sample Predicted recovery , To calculate the KL divergence, it is defined as follows: , is used to represent the degree of difference between the true distribution q and the predicted distribution p; Indicates the forward process from the sample , reasoning The posterior distribution, This represents the first step of the reverse denoising process, i.e., from pure Gaussian noise. Restore to the previous moment The sample, This represents the joint distribution of the forward diffusion process, i.e., given the initial image. The sample distribution at time T can be sampled based on the noise-adding scheme. This represents the distribution of pure Gaussian noise at time T.

[0153] Finally, the joint training loss designed in the post-training stage of this invention is:

[0154] (15);

[0155] in, , These represent the clear image with labels corresponding to the deblurred image and the blurred image, respectively.

[0156] The specific training process for image deblurring based on the multi-scale diffusion model of this invention is as follows:

[0157] S1: In the pre-training stage, the blurred image is stitched together with the sharp image, and then mapped to the latent space by the sharp encoder to obtain the prior features.

[0158] S2: Convolutionally downsample the prior features to generate multi-scale priors (prior 1, prior 2, and prior 3). Then, input the blurred image and the multi-scale priors into the sliding window attention module to output the deblurred image. Train the deblurred image and the clear image according to the loss formula (12).

[0159] S3: In the post-training phase, the weights of the sharp encoder and the sliding window attention module, which were trained in the pre-training phase, are first loaded, and the sharp encoder is "frozen" (its parameters are no longer updated). The blurred image is then stitched together with the sharp image and mapped to the latent space through the "frozen" sharp encoder to obtain the prior features.

[0160] S4: The blurred image is processed by a fuzzy encoder to obtain a fuzzy conditional vector. The prior features are then forward-diffused to add noise, resulting in a noise vector. The fuzzy conditional vector and the noise vector are input into a denoising U-shaped network to predict the noise and variance interpolation for each state. The initial prior features are then calculated using the reparameter recalculation technique, and finally, downsampling is performed to obtain multi-scale prior features.

[0161] S5: Input the multi-scale prior features and the blurred image into the sliding window attention module, and output the deblurred image. Train according to the loss of Equation (15) and save the optimal model during the training process.

[0162] 3. Test sampling phase

[0163] After completing the pre-training and post-training phases, the test set needs to be evaluated and tested. For example... Figure 1 As shown, the blurred image is first compressed into a feature vector using a fuzzy encoder. , This will be used as a fuzzy condition vector, along with direct random sampling from a Gaussian distribution. The images are fed into a denoising U-shaped network to generate prior features, which are then downsampled to obtain multi-scale priors: prior 1, prior 2, and prior 3. Finally, the blurred image and the multi-scale priors are distributed together into a sliding window attention module architecture to output the final deblurred image.

[0164] Step 3, Model Application:

[0165] like Figure 6 As shown, the blurred image to be processed and the Gaussian noise image are input into the trained multi-scale image deblurring network. The training weights of the blur encoder, the denoising U-shaped network, and the sliding window attention model are loaded. The blurred image is mapped to the latent space by the blur encoder to obtain the blur condition vector. The blur condition vector and the Gaussian noise image are interpolated by the denoising U-shaped network to predict the noise and variance at each time step and update the time step. The denoising feature vector for the next step is sampled by the reparameter method. The final denoising feature vector is iteratively output and then downsampled to obtain the multi-scale prior. The multi-scale prior and the blurred image are input into the sliding window attention model. After being processed by the encoder-decoder based U-shaped network, it is mapped back to the pixel space and the final deblurred image is output.

[0166] To verify the effectiveness of the proposed MSdiff image deblurring method, seven state-of-the-art image deblurring methods were compared with MSdiff on a synthetic dataset: DeblurGAN, DeepDeblur, DeblurGAN-v2, DBGAN, Restormer, Stripformer, and HI-Diff. This invention uses common evaluation metrics in the deblurring field: PSNR and SSIM. The quantitative comparison results are shown in Table 1.

[0167] Table 1. Comparison of the performance of different deblurring methods on synthetic datasets.

[0168]

[0169] According to Table 1, for the GoPro test set, MSdiff achieved PSNR and SSIM scores of 33.23 dB and 0.965, respectively. Its PSNR performance is 1.13 dB higher than the best deblurring model DBGAN based on GAN networks, and 0.15 dB higher than the best deblurring model Stripformer based on Transformer networks. It is only 0.1 dB lower than the best model HI-Diff. However, MSdiff slightly outperforms HI-Diff in the SSIM score. When testing the synthetic dataset MSB, MSdiff's deblurring performance comprehensively surpasses the compared state-of-the-art methods, achieving PSNR and SSIM scores of 33.01 dB and 0.942, respectively. Furthermore, it leads the HI-Diff model by 0.08 dB in PSNR, verifying MSdiff's powerful deblurring generalization performance.

[0170] This invention further compares MSdiff with three state-of-the-art deblurring methods on real datasets. The quantitative results of each method are shown in Table 2. Each method was trained on the GoPro dataset and tested on the RealBlur-J and RealBlur-R datasets.

[0171] Table 2 Comparison of the performance of different deblurring methods on real datasets

[0172]

[0173] As shown in Table 2, MSdiff outperforms all compared image deblurring methods. In terms of PSNR, MSdiff achieves 29.22 dB and 36.62 dB in RealBlur-J and RealBlur-R, respectively, exceeding the second-best method HI-Diff by 0.07 dB and 0.34 dB. The results in Table 2 further demonstrate the superiority of the MSdiff deblurring method.

[0174] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-scale image deblurring method based on a latent spatial conditional diffusion model, characterized in that, Includes the following steps: Step 1: Construct a multi-scale image deblurring network, the network comprising: The conditional diffusion model includes a sharp encoder, a fuzzy encoder, a denoising U-shaped network, and a multi-scale prior generation module, which is used to generate multi-scale prior features from fuzzy conditional vectors in the latent space; the conditional diffusion model performs noise modeling and feature reconstruction in the low-dimensional feature space by forward noise addition and reverse denoising. The sliding window attention module, a U-shaped encoder-decoder network, is used to perform image deblurring feature regression guided by multi-scale prior features. This module includes an encoder path and a decoder path. The encoder path performs multi-level downsampling of the input features to extract multi-scale contextual information; the decoder path performs multi-level upsampling of the features to progressively restore spatial resolution. At least one sliding window attention deblurring module is integrated after the dual-channel integration module in each level of the encoder and decoder paths. The levels corresponding to the encoder and decoder paths also include cross-level skip connections to concatenate features from each level of the encoder path with features from an intermediate level of the decoder path to preserve high-frequency details. Step 2, Model Training: The multi-scale image deblurring network is trained using a two-stage strategy including pre-training and post-training to learn the mapping relationship from blurred images to sharp images. Step 3, Model Application: The blurred image to be processed is input into the trained multi-scale image deblurring network, and then processed by the conditional diffusion model and the sliding window attention module in sequence to output the final deblurred image. In the conditional diffusion model, the sharp encoder is used to stitch the sharp image and the blurry image together and map them to the latent space to obtain an initial feature vector; the blurry encoder is used to compress the blurry image separately into a blurry conditional vector. The denoising U-shaped network is composed of multiple residual blocks and attention blocks stacked together. It receives a fuzzy condition vector and a noise vector from the forward diffusion process, predicts the noise estimate and variance interpolation at each step of the diffusion process, and obtains the denoised feature vector based on the noise estimate and variance interpolation through reparameterized sampling. The multi-scale prior generation module generates at least three prior features at different scales by performing one-dimensional convolution downsampling on the denoised feature vector. The dual-channel integrated module performs the following operations: After reshaping the intermediate features of the blurred image, the query vector Q is obtained through layer normalization and linear mapping. The prior features are split into two branches along the channel dimension, and after layer normalization and linear mapping, they are concatenated into key vector K and value vector V. Perform cross-self-attention computation on Q, K, and V; The attention output is added to the residual of the original intermediate features, and the local texture recovery capability is enhanced by two convolutional layers.

2. The multi-scale image deblurring method based on the latent spatial conditional diffusion model according to claim 1, characterized in that, The residual blocks in the denoising U-shaped network include: The feature transformation branch includes, in sequence, a group normalization layer, an activation function layer, and a convolutional layer; The conditional embedding branch includes a temporal embedding sub-branch and a conditional embedding sub-branch, which map time step information and conditional information into embedding vectors, respectively, and fuse them in the channel dimension. In the temporal embedding sub-branch, the time step information is mapped to a temporal embedding vector with the same number of channels as the feature transformation branch by passing through a temporal embedding layer, a linear layer, an activation function layer, and another linear layer. In the conditional embedding sub-branch, the conditional information is fused with the temporal embedding vector in the channel dimension through a label embedding layer. The residual connection branch adds the output after feature transformation and conditional fusion to the original input element by element.

3. The multi-scale image deblurring method based on the latent spatial conditional diffusion model according to claim 1, characterized in that, The attention block is a cross-location attention module used to establish global dependencies on low-resolution feature maps, thereby enhancing the ability to recover large-scale blurred structures.

4. The multi-scale image deblurring method based on the latent spatial conditional diffusion model according to claim 1, characterized in that, The sliding window attention deblurring module includes, in sequence: In the first attention stage, a window self-attention mechanism is adopted to divide the input features into multiple non-overlapping windows. Attention is calculated independently within each window to reduce computational complexity and focus on local detail recovery. In the second attention stage, a sliding window self-attention mechanism is adopted. By performing regular translation of the window, the fixed window boundary is broken, and feature interaction between adjacent windows is realized to model long-distance dependencies. Feedforward networks are used to enhance the nonlinear expressive power of features; The first residual connection adds the features processed by the first and second attention stages to the original input features; The second residual connection adds the output of the feedforward network to the output of the first residual connection.

5. The multi-scale image deblurring method based on the latent spatial conditional diffusion model according to claim 1, characterized in that, During network training, the pre-training phase includes: Prior features are extracted from sharp-blurred image pairs using a sharp encoder and downsampled into multi-scale prior features; The blurred image and the multi-scale prior features are input into the sliding window attention module, and the sliding window attention module is trained end-to-end using the L1 loss function.

6. The multi-scale image deblurring method based on the latent spatial conditional diffusion model according to claim 1, characterized in that, During network training, the post-training phase includes: Freeze the pre-trained sharp encoder parameters; The conditional diffusion model is trained to learn to generate multi-scale prior features consistent with the semantics of a clear image from fuzzy conditional vectors and random noise. The overall network is optimized using a joint loss function, which includes the mean squared error loss of the conditional diffusion model, the KL divergence loss, and the defuzzing L1 loss.

Images