An image deblurring method and apparatus

Abstract

This invention provides an image deblurring method and apparatus. The method includes: inputting a blurred image into a state-space noise prediction model to obtain predicted noise; the state-space noise prediction model is a model constructed based on a state-space model with a selective scanning mechanism; the state-space noise prediction model simultaneously extracts local features of the input features and long-range spatial dependency features obtained by selective scanning along four directions: horizontal, vertical, reverse horizontal, and reverse vertical; denoising the blurred image based on the predicted noise to obtain a denoised image; inputting the denoised image into a space-frequency ensemble model to obtain space-frequency domain features; and obtaining the deblurred image based on the space-frequency domain features. This invention improves the texture and edge features in the deblurred image by capturing long-range spatial dependencies and through joint optimization of the frequency and spatial domains, thereby improving image quality.

Description

Technical Field

[0001] This invention relates to the field of image deblurring technology, and specifically to an image deblurring method and apparatus. Background Technology

[0002] Image deblurring technology, as a core task of underlying vision, plays a fundamental role in ensuring the accuracy of downstream recognition in key fields such as medical diagnosis, remote sensing mapping, and industrial inspection. However, existing methods still have many shortcomings.

[0003] Diffusion models, as an emerging generative AI framework, generate high-quality images through a progressive denoising process, demonstrating great potential in deblurring tasks. CN116645287A discloses an image deblurring method based on a diffusion model, which achieves image deblurring by constructing an extended model. However, it suffers from the following technical problems: 1. The backbone network relies on traditional convolution results, which is insufficient in capturing the long-range structural dependencies of the image, resulting in limited feature extraction effects for complex degradation modes such as dynamic motion blur, making it difficult to accurately restore the coherent texture and edge information of the image. 2. Feature processing is only performed in the spatial domain, without utilizing frequency domain information to specifically repair the attenuation of high-frequency details, leading to problems such as edge blurring and texture loss in the deblurred image due to insufficient high-frequency features.

[0004] Therefore, there is an urgent need to provide an image deblurring method and apparatus that can efficiently model the global contextual dependencies of an image and accurately restore high-frequency details and natural colors, thereby improving the quality of the deblurred image. Summary of the Invention

[0005] In view of this, it is necessary to provide an image deblurring method and apparatus to solve the technical problems of insufficient global information capture and insufficient detail recovery in the prior art, which leads to poor image quality after deblurring.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides an image deblurring method, comprising:

[0007] A blurred image is input into a state-space noise prediction model to obtain predicted noise. The state-space noise prediction model is a model constructed based on a state-space model with a selective scanning mechanism. The state-space noise prediction model includes an encoder and a decoder. Both the encoder and the decoder contain multiple cascaded visual state-space modules. Each visual state-space module includes a first feature extraction branch, a second feature extraction branch, and a feature fusion unit. The first feature extraction branch is used to extract local features of the input features. The second feature extraction branch is used to selectively scan the input features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, to obtain long-range spatially dependent features. The feature fusion unit is used to fuse the local features and the long-range spatially dependent features to obtain fused features. The fused features are then residually connected to the input features to obtain output features.

[0008] The blurred image is denoised based on the predicted noise to obtain a denoised image.

[0009] The denoised image is input into the space-frequency ensemble model, which includes a frequency domain processing unit and a spatial processing unit. The frequency domain processing unit performs a Fourier transform on the denoised image to convert it to the frequency domain for feature optimization, and then converts it back to the spatial domain through an inverse Fourier transform to obtain frequency domain optimized features. The frequency domain optimized features and the denoised image are then residually fused to obtain fused features. The spatial processing unit extracts spatial domain features from the fused features in the spatial domain to obtain space-frequency domain features.

[0010] The spatial frequency domain features and the frequency domain optimized features are residually fused to obtain a deblurred image.

[0011] In one possible implementation, the encoder includes:

[0012] The first encoding module includes multiple cascaded visual state space modules and a first block merging layer;

[0013] The second encoding module includes multiple cascaded visual state space modules and a second block merging layer;

[0014] The third encoding module includes multiple cascaded visual state space modules and a third block merging layer;

[0015] The fourth encoding module includes one of the aforementioned visual state space modules;

[0016] The input to the first encoding module is a feature map obtained by segmenting and linearly embedding the blurred image; the first segmentation merging layer, the second segmentation merging layer and the third segmentation merging layer are used to downsample the input feature map.

[0017] In one possible implementation, the decoder includes:

[0018] The first decoding module includes a visual state space module, and the input features of the first decoding module are the output features of the fourth encoding module;

[0019] The second decoding module includes multiple cascaded visual state space modules and a first block extension layer. The input feature of the second decoding module is the concatenation result of the output features of the first decoding module and the output features of the third encoding module.

[0020] The third decoding module includes multiple cascaded visual state space modules and a second block extension layer. The input of the third decoding module is the concatenation result of the output features of the second encoding module and the output features of the second decoding module.

[0021] The fourth decoding module includes multiple cascaded visual state space modules and a third block extension layer. The input of the fourth decoding module is the concatenation result of the output features of the third decoding module and the output features of the first encoding module.

[0022] The first block expansion layer, the second block expansion layer, and the third block expansion layer are used to upsample the input features.

[0023] In one possible implementation, the visual state space module further includes a time step feature fusion unit, which encodes the current diffusion time step into a time step embedding vector, performs linear projection and activation processing on the time step embedding vector to obtain time step features, and adds the time step features to the input features of the visual state space module element by element.

[0024] In one possible implementation, the frequency domain processing unit includes a two-dimensional fast Fourier transform layer, a first convolutional layer, and a two-dimensional fast Fourier inverse transform layer.

[0025] The two-dimensional fast Fourier transform layer is used to transform the denoised image from the spatial domain to the frequency domain to obtain complex frequency domain features, and to decompose the complex frequency domain features into real features and imaginary features;

[0026] The first convolutional layer is used to concatenate the real part features and the imaginary part features along the channel dimension to obtain concatenated frequency domain features, and to perform convolution processing on the concatenated frequency domain features to obtain second convolutional features;

[0027] The two-dimensional fast Fourier inverse transform layer is used to transform the second convolutional feature from the frequency domain to the spatial domain to obtain the frequency domain optimized feature.

[0028] In one possible implementation, the spatial processing unit includes a fusion layer, a normalization layer, a second convolutional layer, a simple gating layer, and a third convolutional layer;

[0029] The fusion layer is used to fuse the denoised image and the frequency domain optimization features to obtain fused features;

[0030] The normalization layer is used to normalize the fused features to obtain normalized features;

[0031] The second convolutional layer is used to perform convolution processing on the normalized features to obtain the second convolutional features;

[0032] The simple gating layer is used to perform gating activation processing on the second convolutional features to obtain gating features;

[0033] The third convolutional layer is used to perform convolution processing on the gated features to obtain the spatial domain features.

[0034] In one possible implementation, the method further includes:

[0035] Obtain a training dataset, which includes multiple data pairs, each data pair including a blurred image sample and a clear image sample corresponding to the blurred image sample;

[0036] Gaussian noise is gradually added to clear image samples to generate noisy images at different time steps;

[0037] The blurred image sample and the noisy noise are input into the initial noise prediction model to obtain the model prediction noise;

[0038] Based on the noise predicted by the model, the blurred image samples are denoised to obtain a preliminary denoised image.

[0039] The initial denoised image is input into the initial spatial frequency ensemble model to obtain a predicted clear image;

[0040] The image restoration loss is determined based on the predicted sharp image and the sharp image sample, and the noise prediction loss is determined based on the model predicted noise and the added Gaussian noise;

[0041] Based on the loss of the image restoration loss and the loss of the noise prediction loss, the parameters of the initial noise prediction model and the initial space-frequency ensemble model are jointly optimized until the model converges, thereby obtaining the state-space noise prediction model and the space-frequency ensemble model.

[0042] In one possible implementation, the image restoration loss includes frequency loss, image quality loss, and total variation loss, wherein the image quality loss includes L1 loss and structural similarity loss.

[0043] In one possible implementation, before inputting the blurred image into the state-space noise prediction model, the following is also included:

[0044] The blurred image is then subjected to pixel normalization and downsampling enhancement.

[0045] Secondly, the present invention also provides an image deblurring apparatus, comprising:

[0046] A noise prediction unit is used to input a blurred image into a state-space noise prediction model to obtain predicted noise. The state-space noise prediction model is a model constructed based on a state-space model with a selective scanning mechanism. The state-space noise prediction model includes an encoder and a decoder. Both the encoder and the decoder contain multiple cascaded visual state-space modules. Each visual state-space module includes a first feature extraction branch, a second feature extraction branch, and a feature fusion unit. The first feature extraction branch is used to extract local features of the input features. The second feature extraction branch is used to selectively scan the input features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, to obtain long-range spatially dependent features. The feature fusion unit is used to fuse the local features and the long-range spatially dependent features to obtain fused features, and then perform a residual connection between the fused features and the input features to obtain output features.

[0047] A denoising unit is used to perform denoising processing on the blurred image based on the predicted noise to obtain a denoised image;

[0048] The feature enhancement unit is used to input the denoised image into the space-frequency ensemble model. The space-frequency ensemble model includes a frequency domain processing unit and a spatial processing unit. The frequency domain processing unit performs a Fourier transform on the denoised image to convert it to the frequency domain for feature optimization, and then converts it back to the spatial domain through an inverse Fourier transform to obtain frequency domain optimized features. The frequency domain optimized features and the denoised image are then residually fused to obtain fused features. The spatial processing unit performs spatial domain feature extraction on the fused features in the spatial domain to obtain space-frequency domain features.

[0049] The deblurring processing unit is used to perform residual fusion of the spatial frequency domain features and the frequency domain optimized features to obtain a deblurred image.

[0050] The beneficial effects of this invention are as follows: The image deblurring provided by this invention constructs a state-space noise prediction model based on a state-space model with a selective scanning mechanism. This model utilizes the second feature extraction branch of the visual state-space module to scan features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, efficiently capturing long-range spatial dependencies between any pixels in a blurred image. This fundamentally solves the problem of poor image deblurring results caused by insufficient ability to capture long-range dependencies due to reliance solely on convolution results in existing technologies. It accurately restores the coherent texture and edge features of the image, improving the quality of the deblurred image. Simultaneously, compared to the Transformer model used in traditional methods, the state-space noise prediction model significantly reduces computational complexity, substantially shortens deblurring time, and effectively improves deblurring efficiency.

[0051] Furthermore, after obtaining the denoised image, the present invention performs joint optimization of the frequency domain and spatial domain based on the space-frequency integrated model. Compared with the pure spatial domain method, the present invention introduces the frequency domain optimization method, which can accurately restore edge features and texture features, making the edges of the obtained deblurred image clearer and the texture richer, thereby further improving the image quality of the deblurred image. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic flowchart of an embodiment of the image deblurring method provided by the present invention;

[0054] Figure 2 A schematic diagram of an embodiment of the visual state space module provided by the present invention;

[0055] Figure 3 This is a schematic diagram of an embodiment of the space-frequency integration model provided by the present invention;

[0056] Figure 4 A schematic diagram of an embodiment of the state-space noise prediction model provided by the present invention;

[0057] Figure 5 A schematic diagram of a specific embodiment of the space-frequency integration model provided by the present invention;

[0058] Figure 6 A schematic diagram illustrating an embodiment of the complete process of the image deblurring method provided by the present invention;

[0059] Figure 7 This is a schematic diagram of an embodiment of the image deblurring device provided by the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0061] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may be implemented out of order, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0062] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0063] This invention provides an image deblurring method and apparatus, which will be described below.

[0064] Figure 1 This is a schematic flowchart of an embodiment of the image deblurring method provided by the present invention. Figure 2 This is a schematic diagram of an embodiment of the visual state space module provided by the present invention. Figure 3 This is a schematic diagram of an embodiment of the space-frequency integration model provided by the present invention, as shown below. Figures 1-3 As shown, image deblurring methods include:

[0065] S101. Input the blurred image into the state space noise prediction model to obtain the predicted noise. The state space noise prediction model is a model constructed based on the state space model with selective scanning mechanism (VSSM). The state space noise prediction model includes an encoder and a decoder. Both the encoder and the decoder contain multiple cascaded visual state space modules. The visual state space module includes a first feature extraction branch, a second feature extraction branch, and a feature fusion unit. The first feature extraction branch is used to extract local features of the input features. The second feature extraction branch is used to selectively scan the input features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, to obtain long-range spatially dependent features. The feature fusion unit is used to fuse local features and long-range spatially dependent features to obtain fused features. The fused features are then residually connected with the input features to obtain the output features.

[0066] S102. Denoise the blurred image based on the predicted noise to obtain a denoised image;

[0067] S103. Input the denoised image into the space-frequency fusion model. The space-frequency fusion model includes a frequency domain processing unit and a spatial processing unit. The frequency domain processing unit performs a Fourier transform on the denoised image, converts it to the frequency domain for feature optimization, and then converts it back to the spatial domain through an inverse Fourier transform to obtain frequency domain optimized features. The frequency domain optimized features and the denoised image are fused using residuals to obtain fused features. The spatial processing unit extracts spatial domain features from the fused features in the spatial domain to obtain space-frequency domain features.

[0068] S104. Perform residual fusion of spatial frequency domain features and frequency domain optimization features to obtain a deblurred image.

[0069] Specifically, the state update method of each visual state space module can be represented as follows:

[0070]

[0071] In the formula, Let be the state vector at time t. This is the state transition matrix. , The projection matrix that depends on the input. Given the input sequence, The decay factor controls the degree of forgetting of historical states, and is ensured to be non-negative through the Softplus activation function; This is the Hadamard product operator.

[0072] The output equation of the visual state space module is expressed as follows:

[0073]

[0074] In the formula, Let D be the output feature at time t, and D be the residual weights used to fuse the input and state output. It is a linear projection of the state vector, mapping the N-dimensional state back to the D-dimensional features.

[0075] Because the input features were selectively scanned along four directions—horizontal, vertical, reverse horizontal, and reverse vertical—the final output features should be:

[0076]

[0077] In the formula, , , , Selective scanning outputs are performed in four directions: horizontal, vertical, reverse horizontal, and reverse vertical. The integrity of image features is improved through stitching and fusion.

[0078] From the expression of the state update equation, we can see that: the decay factor By multiplying element by element, the state vector of the previous time step is processed. By selectively modulating each dimension, dynamic and input-dependent control of historical information can be achieved, thus capturing long-range spatial dependence features.

[0079] It should be understood that the image deblurring method in the embodiments of the present invention can be implemented in any device based on the image deblurring method, such as an image processing device or an image restoration device. Specifically, the image deblurring method is stored in the aforementioned device as a pre-programmed program, and when the device is started, the program is invoked and the image deblurring method is implemented.

[0080] Compared with existing technologies, the image deblurring method provided in this invention constructs a state-space noise prediction model based on a state-space model with a selective scanning mechanism. It utilizes the second feature extraction branch of the visual state-space module to scan features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, efficiently capturing long-range spatial dependencies between any pixels in a blurred image. This fundamentally solves the problem of poor image deblurring results caused by insufficient ability to capture long-range dependencies due to reliance solely on convolution results in existing technologies. It accurately restores the coherent texture and edge features of the image, improving the quality of the deblurred image. Simultaneously, compared to the Transformer model used in traditional methods, the state-space noise prediction model significantly reduces computational complexity, substantially shortens deblurring time, and effectively improves deblurring efficiency.

[0081] Furthermore, in this embodiment of the invention, after obtaining the denoised image, the frequency domain and spatial domain are jointly optimized based on the space-frequency integrated model. Compared with the pure spatial domain method, the frequency domain optimization method introduced in this invention can accurately restore edge features and texture features, making the edges of the obtained deblurred image clearer and the texture richer, thereby further improving the image quality of the deblurred image.

[0082] In some embodiments of the present invention, such as Figure 4 As shown, the encoder includes:

[0083] The first encoding module includes multiple cascaded visual state space modules and a first block merging layer;

[0084] The second encoding module includes multiple cascaded visual state space modules and a second block merging layer;

[0085] The third encoding module includes multiple cascaded visual state space modules and a third block merging layer;

[0086] The fourth encoding module includes a visual state space module;

[0087] The first encoding module takes as input a feature map obtained by segmenting and linearly embedding the blurred image; the first segmentation merging layer, the second segmentation merging layer, and the third segmentation merging layer are used to downsample the input feature map.

[0088] Specifically, the downsampled channels are doubled in size, while the resolution is reduced by half.

[0089] In a specific embodiment of the present invention, the first encoding module, the second encoding module and the third encoding module each include two cascaded visual state space modules.

[0090] In this embodiment of the invention, the blurred image is segmented and linearly embedded before being input into the first encoding module, so that the two-dimensional blurred image is adapted to the encoder. At the same time, by setting multiple cascaded encoding modules, the extraction of features at different scales can be achieved.

[0091] Specifically, the 3-channel blurred image is segmented into 4×4 image blocks and linearly projected into a 96-dimensional embedding space.

[0092] In some embodiments of the present invention, such as Figure 4 As shown, the decoder includes:

[0093] The first decoding module includes a visual state space module, and the input features of the first decoding module are the output features of the fourth encoding module.

[0094] The second decoding module includes multiple cascaded visual state space modules and a first block extension layer. The input features of the second decoding module are the concatenation results of the output features of the first decoding module and the output features of the third encoding module.

[0095] The third decoding module includes multiple cascaded visual state space modules and a second block extension layer. The input of the third decoding module is the concatenation result of the output features of the second encoding module and the output features of the second decoding module.

[0096] The fourth decoding module includes multiple cascaded visual state space modules and a third block extension layer. The input of the fourth decoding module is the concatenation result of the output features of the third decoding module and the output features of the first encoding module.

[0097] The first block expansion layer, the second block expansion layer, and the third block expansion layer are used to upsample the input features.

[0098] It should be noted that the features in the input decoding module are the output features of the visual state space module in the decoder.

[0099] In a specific embodiment of the present invention, the second decoding module, the third decoding module and the fourth decoding module each include two visual state space modules.

[0100] In particular, the resolution was doubled by reducing the size by a factor of two after upsampling.

[0101] It should be noted that in order to restore the image to the same size as the input image, the output of the fourth decoding module needs to be divided into blocks and linearly projected.

[0102] This invention achieves skip connections between encoding and decoding by setting the output of the encoding module as the input of the decoding module at the same level, thereby reducing iteration steps, avoiding gradient explosion, and further improving the deblurring efficiency of images.

[0103] To enable the state-space noise prediction model to perceive the current diffusion stage and thus perform accurate denoising operations adapted to the noise level, in some embodiments of the present invention, such as... Figure 2 As shown, the visual state space module also includes a time step feature fusion unit. The time step feature fusion unit is used to encode the current diffusion time step into a time step embedding vector, and to perform linear projection and activation processing on the time step embedding vector to obtain time step features. The time step features are then added element-wise to the input features of the visual state space module.

[0104] In a specific embodiment of the present invention, the time step embedding vector adopts sine and cosine coding, which can be expressed as:

[0105]

[0106] In the formula, Embedded vectors for time steps; For the embedded dimension; The dimension index of the embedding vector for each time step.

[0107] Linear projection and activation processing can be represented as:

[0108]

[0109] In the formula, For time step features; For linear projection processing, , This represents two linear projections; For activation processing.

[0110] Furthermore, the element-wise summation of the time-step features and the input features of the visual state space module can be expressed as:

[0111]

[0112] in, This is a fusion feature that integrates time-step features; The input features for the visual state space module are image features; To incorporate time step features Features of the spatial dimension of the image after broadcasting.

[0113] This invention, through the establishment of a time-step feature fusion unit, uses the fused features as input to the visual state space module, participating in subsequent local feature extraction and four-directional selective scanning. This mechanism ensures that the model's state update and output generation are modulated by the current diffusion time step, thereby enabling it to dynamically adjust feature selection and information transmission strategies to accurately predict noise components at corresponding noise levels.

[0114] In specific embodiments of the present invention, such as Figure 5 As shown, the frequency domain processing unit includes a two-dimensional fast Fourier transform layer, a first convolutional layer, and a two-dimensional fast Fourier inverse transform layer.

[0115] A two-dimensional fast Fourier transform layer is used to transform the denoised image from the spatial domain to the frequency domain, obtain complex frequency domain features, and decompose the complex frequency domain features into real features and imaginary features.

[0116] The complex frequency domain characteristics can be expressed as:

[0117]

[0118] In the formula, It is a negative frequency domain characteristic. It is a two-dimensional fast Fourier transform; For denoising the image; For the normalization option, it indicates that the transformation is orthogonal.

[0119] The first convolutional layer is used to concatenate real and imaginary features along the channel dimension to obtain concatenated frequency domain features, and then convolutional processing is performed on the concatenated frequency domain features to obtain the second convolutional features.

[0120] Specifically, the second convolutional feature can be represented as:

[0121]

[0122]

[0123] In the formula, This is the second convolution feature; This is a two-dimensional convolution process; To splice frequency domain features; For splicing operations; Features of the real part; This is a feature of the imaginary part.

[0124] A two-dimensional inverse fast Fourier transform layer is used to transform the second convolutional features from the frequency domain to the spatial domain to obtain frequency domain optimized features.

[0125] This invention transforms the denoised image from the spatial domain to the frequency domain, enabling the model to directly manipulate and optimize the image's frequency components. Blur is essentially the attenuation of high-frequency information; by transforming to the frequency domain, the high-frequency components lost due to blurring can be directly enhanced in the frequency space, thereby restoring details. Compared to methods that only perform convolution operations in the spatial domain, the restored image has sharper edges and clearer textures.

[0126] Furthermore, by setting up channel concatenation of the real and imaginary parts of complex features and performing cross-channel optimization through a one-dimensional convolutional layer, the coupling relationship between RGB color channels can be naturally learned and maintained in the frequency domain. Through this cross-channel frequency domain convolution, the frequency response of the three channels can be adjusted collaboratively, ensuring that the restored color proportions are coordinated and the saturation is natural. This fundamentally avoids the color distortion and color cast problems caused by the independent processing of each channel in traditional methods, further improving the quality of the deblurred image.

[0127] To ensure the stability of the training of the space-frequency integrated optimization module and the effectiveness of feature fusion, the frequency domain processing unit also includes a batch normalization layer and a third convolutional layer before the two-dimensional fast Fourier transform layer, an activation function layer after the first convolutional layer, and a fourth convolutional layer after the two-dimensional inverse fast Fourier transform, wherein the activation function is GELU.

[0128] The process flow of the frequency domain processing unit can be described as follows: The two-dimensional fast Fourier transform layer captures the global structural features of the image through channel expansion, frequency domain optimization and channel compression. First, the channels are expanded to 2C through 1×1 convolution. After GELU activation, the channel is input into the two-dimensional fast Fourier inverse transform layer. The output dimension is compressed back to C. The dependencies between channels are enhanced through 1×1 convolution.

[0129] In some embodiments of the present invention, such as Figure 5 As shown, the spatial processing unit includes a fusion layer, a normalization layer, a second convolutional layer, a simple gating layer, and a third convolutional layer;

[0130] The fusion layer is used to fuse the denoised image and frequency domain optimization features to obtain fused features;

[0131] The normalization layer is used to normalize the fused features to obtain normalized features.

[0132] Specifically, the normalization layer is a layer normalization, which can be expressed as:

[0133]

[0134] In the formula, Normalized features; Features of fusion; Spatial mean for each channel; Spatial variance for each channel; , represents the learnable affine transformation parameters; represents a very small value. Ensure the denominator is not zero.

[0135] The second convolutional layer is used to perform convolution processing on the normalized features to obtain the second convolutional features;

[0136] A simple gating layer is used to perform gating activation processing on the second convolutional features to obtain gating features.

[0137] It should be noted that in the simple gated layer, the second convolutional feature is divided into two parts along the channel dimension, and then element-wise multiplication is performed to obtain the enhanced feature. The process can be represented as follows:

[0138]

[0139] In the formula, These are the two features obtained after dividing the second convolutional feature along the channel dimension.

[0140] The third convolutional layer is used to perform convolution processing on the gated features to obtain spatial domain features.

[0141] This invention, through the introduction of layer normalization, effectively alleviates the internal covariate shift problem in deep learning models, enabling subsequent convolutional and gated activation layers to operate under a stable input distribution. This significantly reduces the difficulty of model training and accelerates the convergence process. Simultaneously, the introduction of simple gated layers dynamically suppresses unimportant feature responses and enhances key features, allowing the model to focus more on details that contribute to image restoration in the spatial domain, thereby outputting more expressive spatial domain features.

[0142] It should be understood that the state-space noise prediction model and the space-frequency ensemble model need to be trained before use. Therefore, in some embodiments of the present invention, such as Figure 6 As shown, image deblurring methods also include:

[0143] S601. Obtain the training dataset. The training dataset includes multiple data pairs, each of which includes a blurred image sample and a clear image sample corresponding to the blurred image sample.

[0144] It should be noted that the training dataset needs to cover image samples from multiple indoor and outdoor scenes, with different blur intensities (such as motion blur and out-of-focus blur), and different resolutions, in order to improve the generalization ability and applicability of the trained model.

[0145] S602. Gaussian noise is gradually added to clear image samples to generate noisy images at different time steps.

[0146] Specifically, the process of adding Gaussian noise involves gradually adding a fixed variance. The Gaussian noise is converted into input data X0, and after T steps, the input X0 is transformed into data that is close to pure noise. This process can be represented as follows:

[0147]

[0148]

[0149] In the formula, and Let represent the noise data and predefined variance at time step t, respectively, and N represent a Gaussian distribution. The noisy image at each time step can be calculated using the formula:

[0150]

[0151] In the formula, This represents the cumulative noise reduction coefficient. It is random Gaussian noise.

[0152] S603. Input the blurred image sample and the noisy noise into the initial noise prediction model to obtain the model prediction noise;

[0153] S604. Based on the model prediction noise, perform denoising processing on the blurred image samples to obtain a preliminary denoised image.

[0154] The process of determining the initial denoised image essentially involves capturing the long-range dependencies of the noisy image through the visual state space module, accurately predicting the noise added by forward diffusion, and then estimating the initial clear image based on the state space noise. This process can be represented as follows:

[0155]

[0156] In the formula, Predict noise for the model; This is the original sharp image estimated at time step t based on the current noisy image and the noise predicted by the model during the reverse denoising stage of the diffusion process.

[0157] By balancing the signal and noise using Gaussian noise terms, a noisy image from the previous time step is generated, achieving iterative denoising. The process can be represented as follows:

[0158]

[0159] Repeat this process until t=0 to obtain the final, preliminary, clear image. Among these steps, and The coefficient is used to balance noise and signal.

[0160] S605. Input the initially denoised image into the initial space-frequency ensemble model to obtain the predicted clear image;

[0161] S606. Determine the image restoration loss based on the predicted sharp image and sharp image samples, and determine the noise prediction loss based on the model prediction noise and the added Gaussian noise;

[0162] S607. Based on the image restoration loss and noise prediction loss, the parameters of the initial noise prediction model and the initial space-frequency ensemble model are jointly optimized until the model converges, thus obtaining the state-space noise prediction model and the space-frequency ensemble model.

[0163] In addition to the technical problems mentioned in the background, existing methods also suffer from poor image restoration accuracy and color fidelity after deblurring. To solve this problem, in a specific embodiment of the present invention, the image restoration loss is a weighted sum of frequency loss, image quality loss, and total variation loss. The image quality loss includes L1 loss and structural similarity loss.

[0164] This invention incorporates image color restoration accuracy and color deviation into the model training process by setting image restoration loss to include frequency loss, image quality loss, and total variation loss. This avoids the saturation decay and color shift problems caused by neglecting color correlation in existing methods. Subjectively, the restored image color accuracy is closer to the true value, and the color deviation value is reduced, effectively solving the problems of large color representation differences and poor visual effects in existing deblurring algorithms.

[0165] Meanwhile, the image quality loss is set as a weighted sum of L1 loss, structural similarity loss, and brightness contrast consistency loss. Compared with the existing single loss scheme, the model convergence speed is improved and the artifact occurrence rate is reduced. Moreover, the PSNR is significantly reduced when tested across datasets, and the generalization and stability are significantly better than the existing technology.

[0166] In a specific embodiment of the present invention, the formula for calculating the noise prediction loss is as follows:

[0167]

[0168] Frequency loss includes high-frequency loss and low-frequency loss in different frequency bands. The total variation loss serves as a regularization term for the frequency loss. Specifically, the formulas for calculating frequency loss and total variation loss are:

[0169]

[0170] In the formula, Predict noise for the model; High-speed noise added to simulate reality; This is the mean square error loss; , , To predict the first high-frequency band, the second high-frequency band, and the low-frequency band of a clear image; , , The first high-frequency band, the second high-frequency band, and the low-frequency band are for clear image samples; , , This represents the total variation across the first high-frequency band, the second high-frequency band, and the low-frequency band.

[0171] In a specific embodiment of the present invention, image quality loss The calculation formula is:

[0172]

[0173] In the formula, To predict clear images; For true and clear images; This is a structural similarity index.

[0174] It should be understood that the total loss is a weighted sum of noise prediction loss, frequency loss, image quality loss, and total variation loss.

[0175] It should be noted that, in order to further improve the quality of the deblurred images, the peak signal-to-noise ratio (PSNR) of the predicted sharp images is periodically calculated using the validation set during the training process, and is used as an additional evaluation metric for the model.

[0176] In some embodiments of the present invention, before step S101, the method further includes:

[0177] Pixel normalization and downsampling enhancement are performed on blurred images.

[0178] This invention adjusts the blurred image to a fixed size (e.g., 256×256) by setting pixel normalization, ensuring that the size is an integer multiple of 32, to accommodate subsequent upsampling and downsampling by the model. Furthermore, by setting downsampling enhancement, jagged artifacts can be reduced, further improving the quality of the deblurred image.

[0179] In summary, the image deblurring method proposed in this invention first utilizes a state-space noise prediction model constructed based on a selective scanning mechanism as a noise predictor. This model captures long-distance dependencies through selective scanning, adapting to high-resolution image processing with linear complexity, thus improving the efficiency of the diffusion model while ensuring feature capture capabilities. Simultaneously, after denoising, a frequency domain processing unit is added to optimize the diffusion model output, ensuring enhanced image detail quality and edge features. Finally, to comprehensively optimize the image restoration effect, a multi-dimensional loss combination strategy is proposed, fusing noise loss, frequency loss, image quality loss, and TV loss to guarantee the denoising capabilities, structural similarity, and spatial smoothness of the generated image.

[0180] To better implement the image deblurring method in the embodiments of the present invention, based on the image deblurring method, the embodiments of the present invention also provide an image deblurring device, such as... Figure 7 As shown, the image deblurring device 700 includes:

[0181] The noise prediction unit 701 is used to input the blurred image into the state space noise prediction model to obtain the predicted noise. The state space noise prediction model is a model constructed based on a state space model with a selective scanning mechanism. The state space noise prediction model includes an encoder and a decoder. Both the encoder and the decoder contain multiple cascaded visual state space modules. The visual state space module includes a first feature extraction branch, a second feature extraction branch, and a feature fusion unit. The first feature extraction branch is used to extract local features of the input features. The second feature extraction branch is used to selectively scan the input features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, to obtain long-range spatially dependent features. The feature fusion unit is used to fuse local features and long-range spatially dependent features to obtain fused features, and then performs a residual connection between the fused features and the input features to obtain output features.

[0182] The denoising unit 702 is used to denoise the blurred image based on the predicted noise to obtain a denoised image;

[0183] The feature enhancement unit 703 is used to input the denoised image into the space-frequency fusion model. The space-frequency fusion model includes a frequency domain processing unit and a spatial processing unit. The frequency domain processing unit performs a Fourier transform on the denoised image, converts it to the frequency domain for feature optimization, and then converts it back to the spatial domain through an inverse Fourier transform to obtain frequency domain optimized features. The frequency domain optimized features and the denoised image are fused by residual to obtain fused features. The spatial processing unit extracts spatial domain features from the fused features in the spatial domain to obtain space-frequency domain features.

[0184] The deblurring processing unit 704 is used to perform residual fusion of spatial frequency domain features and frequency domain optimized features to obtain a deblurred image.

[0185] The image deblurring device 700 provided in the above embodiments can implement the technical solutions described in the above image deblurring method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content of the above image deblurring method embodiments, and will not be repeated here.

[0186] The above provides a detailed description of the image deblurring method and apparatus provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will know that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image deblurring method, characterized in that, include: The blurred image is input into the state space noise prediction model to obtain the predicted noise. The state space noise prediction model is a model constructed based on a state space model with a selective scanning mechanism. The state-space noise prediction model includes an encoder and a decoder. Both the encoder and the decoder contain multiple cascaded visual state-space modules. Each visual state-space module includes a first feature extraction branch, a second feature extraction branch, and a feature fusion unit. The first feature extraction branch is used to extract local features of the input features. The second feature extraction branch is used to selectively scan the input features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, to obtain long-range spatially dependent features. The feature fusion unit is used to fuse the local features and the long-range spatially dependent features to obtain fused features, and then perform residual connection between the fused features and the input features to obtain output features. The blurred image is denoised based on the predicted noise to obtain a denoised image. The denoised image is input into the space-frequency ensemble model, which includes a frequency domain processing unit and a spatial processing unit. The frequency domain processing unit performs a Fourier transform on the denoised image to convert it to the frequency domain for feature optimization, and then converts it back to the spatial domain through an inverse Fourier transform to obtain frequency domain optimized features. The frequency domain optimized features and the denoised image are then residually fused to obtain fused features. The spatial processing unit extracts spatial domain features from the fused features in the spatial domain to obtain space-frequency domain features. The spatial frequency domain features and the frequency domain optimized features are residually fused to obtain a deblurred image.

2. The image deblurring method according to claim 1, characterized in that, The encoder includes: The first encoding module includes multiple cascaded visual state space modules and a first block merging layer; The second encoding module includes multiple cascaded visual state space modules and a second block merging layer; The third encoding module includes multiple cascaded visual state space modules and a third block merging layer; The fourth encoding module includes one of the aforementioned visual state space modules; The input to the first encoding module is a feature map obtained by segmenting and linearly embedding the blurred image; the first segmentation merging layer, the second segmentation merging layer and the third segmentation merging layer are used to downsample the input feature map.

3. The image deblurring method according to claim 2, characterized in that, The decoder includes: The first decoding module includes a visual state space module, and the input features of the first decoding module are the output features of the fourth encoding module; The second decoding module includes multiple cascaded visual state space modules and a first block extension layer. The input feature of the second decoding module is the concatenation result of the output features of the first decoding module and the output features of the third encoding module. The third decoding module includes multiple cascaded visual state space modules and a second block extension layer. The input of the third decoding module is the concatenation result of the output features of the second encoding module and the output features of the second decoding module. The fourth decoding module includes multiple cascaded visual state space modules and a third block extension layer. The input of the fourth decoding module is the concatenation result of the output features of the third decoding module and the output features of the first encoding module. The first block expansion layer, the second block expansion layer, and the third block expansion layer are used to upsample the input features.

4. The image deblurring method according to claim 1, characterized in that, The visual state space module further includes a time step feature fusion unit, which encodes the current diffusion time step into a time step embedding vector, performs linear projection and activation processing on the time step embedding vector to obtain time step features, and adds the time step features to the input features of the visual state space module element by element.

5. The image deblurring method according to claim 1, characterized in that, The frequency domain processing unit includes a two-dimensional fast Fourier transform layer, a first convolutional layer, and a two-dimensional fast Fourier inverse transform layer. The two-dimensional fast Fourier transform layer is used to transform the denoised image from the spatial domain to the frequency domain to obtain complex frequency domain features, and to decompose the complex frequency domain features into real features and imaginary features; The first convolutional layer is used to concatenate the real part features and the imaginary part features along the channel dimension to obtain concatenated frequency domain features, and to perform convolution processing on the concatenated frequency domain features to obtain second convolutional features; The two-dimensional fast Fourier inverse transform layer is used to transform the second convolutional feature from the frequency domain to the spatial domain to obtain the frequency domain optimized feature.

6. The image deblurring method according to claim 1, characterized in that, The spatial processing unit includes a fusion layer, a normalization layer, a second convolutional layer, a simple gating layer, and a third convolutional layer; The fusion layer is used to fuse the denoised image and the frequency domain optimization features to obtain fused features; The normalization layer is used to normalize the fused features to obtain normalized features; The second convolutional layer is used to perform convolution processing on the normalized features to obtain the second convolutional features; The simple gating layer is used to perform gating activation processing on the second convolutional features to obtain gating features; The third convolutional layer is used to perform convolution processing on the gated features to obtain the spatial domain features.

7. The image deblurring method according to claim 1, characterized in that, The method further includes: Obtain a training dataset, which includes multiple data pairs, each data pair including a blurred image sample and a clear image sample corresponding to the blurred image sample; Gaussian noise is gradually added to clear image samples to generate noisy images at different time steps; The blurred image sample and the noisy image are input into the initial noise prediction model to obtain the model's predicted noise. Based on the noise predicted by the model, the blurred image samples are denoised to obtain a preliminary denoised image. The initial denoised image is input into the initial spatial frequency ensemble model to obtain a predicted clear image; The image restoration loss is determined based on the predicted sharp image and the sharp image sample, and the noise prediction loss is determined based on the model predicted noise and the added Gaussian noise; Based on the loss of the image restoration loss and the loss of the noise prediction loss, the parameters of the initial noise prediction model and the initial space-frequency ensemble model are jointly optimized until the model converges, thereby obtaining the state-space noise prediction model and the space-frequency ensemble model.

8. The image deblurring method according to claim 7, characterized in that, The image restoration loss includes frequency loss, image quality loss, and total variation loss, and the image quality loss includes L1 loss and structural similarity loss.

9. The image deblurring method according to claim 1, characterized in that, Before inputting the blurred image into the state-space noise prediction model, the method further includes: The blurred image is then subjected to pixel normalization and downsampling enhancement.

10. An image deblurring device, characterized in that, include: The noise prediction unit is used to input the blurred image into the state space noise prediction model to obtain the predicted noise. The state space noise prediction model is a model constructed based on a state space model with a selective scanning mechanism. The state-space noise prediction model includes an encoder and a decoder. Both the encoder and the decoder contain multiple cascaded visual state-space modules. Each visual state-space module includes a first feature extraction branch, a second feature extraction branch, and a feature fusion unit. The first feature extraction branch is used to extract local features of the input features. The second feature extraction branch is used to selectively scan the input features along four directions: horizontal, vertical, reverse horizontal, and reverse vertical, to obtain long-range spatially dependent features. The feature fusion unit is used to fuse the local features and the long-range spatially dependent features to obtain fused features, and then perform residual connection between the fused features and the input features to obtain output features. A denoising unit is used to perform denoising processing on the blurred image based on the predicted noise to obtain a denoised image; The feature enhancement unit is used to input the denoised image into the space-frequency ensemble model. The space-frequency ensemble model includes a frequency domain processing unit and a spatial processing unit. The frequency domain processing unit performs a Fourier transform on the denoised image to convert it to the frequency domain for feature optimization, and then converts it back to the spatial domain through an inverse Fourier transform to obtain frequency domain optimized features. The frequency domain optimized features and the denoised image are then residually fused to obtain fused features. The spatial processing unit performs spatial domain feature extraction on the fused features in the spatial domain to obtain space-frequency domain features. The deblurring processing unit is used to perform residual fusion of the spatial frequency domain features and the frequency domain optimized features to obtain a deblurred image.

Images