Single-frame image motion deblurring method and device based on direction perception and diffusion refinement

Abstract

The application provides a single-frame image motion deblurring method and device based on direction perception and diffusion refinement, wherein the method comprises the following steps: acquiring a single-frame blurred image, and estimating spatial motion blur parameters of the single-frame blurred image to obtain a blur direction, a blur length and a corresponding confidence degree of each pixel in the single-frame blurred image; performing direction regularization deconvolution on the single-frame blurred image based on the blur direction, the blur length and the confidence degree to obtain a coarse restoration image; inputting the coarse restoration image and the blur direction and the blur length into a pre-trained conditional latent space diffusion model to perform refinement processing to obtain a refined restoration image; and performing frequency domain angular consistency constraint on the refined restoration image to output a final restoration image. Through the estimation of the spatial motion blur parameters, the combination of the direction regularization deconvolution and the conditional diffusion refinement, and the introduction of the frequency domain angular constraint, the directionality artifacts are effectively inhibited, and the restoration quality is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a method and apparatus for motion deblurring of single-frame images based on direction perception and diffusion refinement. Background Technology

[0002] In intelligent transportation and highway monitoring scenarios, significant motion blur often appears in the acquired images due to the combined effects of high-speed vehicle movement and camera exposure time. This blur exhibits typical spatial variability: within the same frame, vehicles passing quickly in the foreground are more blurred, while stationary backgrounds or slow-moving vehicles in the distance are less blurred, and the direction of blurring changes with the vehicle's trajectory. Blurred images lose crucial texture details and edge information, severely impacting the performance of subsequent high-level vision tasks such as license plate recognition, vehicle type classification, and traffic violation detection.

[0003] Existing motion deblurring techniques can be broadly categorized into two types. The first type is based on traditional optimization methods, including blind deconvolution (such as the Richardson-Lucy algorithm), Wiener filtering, and regularization methods based on sparse priors. These methods typically assume that the blur kernel is spatially invariant throughout the entire image, meaning the entire image is uniformly blurred. However, in real-world highway scenes, the speed and direction of motion vary across different regions, causing the blur kernel to exhibit significant spatial variation. This makes the above assumptions difficult to uphold, leading to severe ringing artifacts or loss of detail in the restored results. The second type is based on data-driven deep learning methods, particularly end-to-end encoder-decoder networks. These methods learn the mapping from blurred to sharp images using large-scale synthetic data, generating visually high-quality restored results. However, purely data-driven methods lack explicit modeling of the blurring physical process, have limited generalization ability when there are distributional differences between the training data and the real scene, and are prone to artifacts such as oversmoothing or phantom textures, making it difficult to reliably recover the true edges and detailed structures of moving targets.

[0004] To address the aforementioned issues, how to effectively model spatially variable motion blur parameters under single-frame image input conditions, guide the restoration process to maintain consistency with real physical degradation, and suppress directional artifacts has become a pressing technical challenge in this field. Summary of the Invention

[0005] Therefore, it is necessary to provide a method and apparatus for motion deblurring of single-frame images based on direction perception and diffusion refinement to address the aforementioned technical problems.

[0006] In a first aspect, the present invention provides a single-frame image motion deblurring method based on direction awareness and diffusion refinement, comprising:

[0007] Acquire a single-frame blurred image and estimate the spatial motion blur parameters of the single-frame blurred image to obtain the blur direction, blur length and corresponding confidence level of each pixel in the single-frame blurred image;

[0008] Based on the blur direction, blur length and confidence, a direction-regularized deconvolution is performed on a single-frame blurred image to obtain a coarse restored image.

[0009] The coarse restored image is processed by inputting the blur direction and blur length into a pre-trained conditional latent space diffusion model to obtain a refined restored image.

[0010] Apply frequency domain angular consistency constraints to the refined and restored image, and output the final restored image.

[0011] Optionally, the spatial motion blur parameters of a single-frame blurred image are estimated to obtain the blur direction, blur length, and corresponding confidence level of each pixel in the single-frame blurred image, including:

[0012] The RGB channel feature map, grayscale image, horizontal gradient map, vertical gradient map, and Gabor filter response map in multiple preset directions of a single-frame blurred image are concatenated along the channels to obtain the input feature tensor.

[0013] A multi-scale convolutional neural network is used to extract features from the input feature tensor, resulting in a multi-scale feature map that includes a shallow feature map.

[0014] Calculate the gradient covariance matrix of each pixel in a single-frame blurred image, and calculate the principal orientation and anisotropy of each pixel as statistical features of the local structure tensor based on the gradient covariance matrix of each pixel.

[0015] The enhanced feature map is obtained by fusing the statistical features of the local structure tensor with the shallow feature map in the multi-scale feature map.

[0016] Based on the enhanced feature map, the direction regression head, length regression head, and confidence head are used to output the blur direction, blur length, and corresponding confidence of each pixel in a single frame of blurred image.

[0017] Optionally, based on the blur direction, blur length, and confidence level, a direction-regularized deconvolution is performed on a single-frame blurred image to obtain a coarse restored image, including:

[0018] Construct an inverse filter kernel library, wherein the inverse filter kernel library includes multiple inverse filter kernels corresponding to multiple discrete directions and multiple discrete length combinations, and a preset regularization parameter is introduced into each inverse filter kernel;

[0019] For each pixel, based on the blur direction and blur length, select multiple inverse filter kernels that are adjacent in direction and length from the inverse filter kernel library as a candidate kernel set;

[0020] A hybrid gating mechanism is adopted to weight and fuse candidate kernels in the candidate kernel set based on confidence level, thereby generating an adaptive inverse filter kernel for the corresponding pixel;

[0021] The single-frame blurred image is converted from the RGB color space to the luminance-chrominance color space, the luminance channel and the chrominance channel are separated, and the luminance channel is deconvolved by applying an adaptive inverse filter kernel to obtain the deconvolved luminance channel. The chrominance channel is weakened to obtain the weakened chrominance channel.

[0022] The luminance channel after deconvolution is merged with the chroma channel after weakening, and then converted back to the RGB color space to obtain a coarsely restored image.

[0023] Optionally, the coarse restored image is processed by inputting the blur direction and blur length into a pre-trained conditional latent space diffusion model to obtain a refined restored image, including:

[0024] The coarse restored image, along with the blur direction and blur length, is input into a pre-trained conditional latent space diffusion model to generate a temporary decoded image.

[0025] Based on the data consistency mechanism, the temporary decoded image is constrained within a preset neighborhood of the coarse restored image to obtain the refined restored image.

[0026] Optionally, the coarsely restored image and the blur direction and blur length are input into a pre-trained conditional latent space diffusion model to generate a temporary decoded image, including:

[0027] The coarse restored image is encoded using a variational autoencoder in a pre-trained conditional latent space diffusion model to obtain the first latent space features;

[0028] The fuzzy direction and fuzzy length are normalized respectively, and the normalized fuzzy direction and fuzzy length are used as geometric conditions.

[0029] The first latent space features and geometric conditions are input into the U-Net network in the conditional latent space diffusion model for iterative denoising to obtain the second latent space features.

[0030] The second latent space feature is input into the variational autoencoder in the conditional latent space diffusion model for decoding to obtain a temporary decoded image.

[0031] Optionally, the temporary decoded image is constrained within a preset neighborhood of the coarse restored image according to a data consistency mechanism to obtain a refined restored image, including:

[0032] The same low-pass filtering is applied to both the temporary decoded image and the coarse restored image to extract the low-frequency components of the temporary decoded image and the coarse restored image.

[0033] Calculate the first low-frequency difference between the low-frequency components of the temporary decoded image and the low-frequency components of the coarse restored image, and correct the temporary decoded image based on the first low-frequency difference so that the low-frequency components of the corrected temporary decoded image are consistent with the low-frequency components of the coarse restored image within a preset neighborhood, thereby obtaining a refined restored image.

[0034] Optionally, the method further includes:

[0035] The first blur direction, first blur length and corresponding first confidence of each pixel in the first single frame training blurred image are obtained, and the first single frame training blurred image is subjected to directional regularized deconvolution based on the first blur direction, first blur length and first confidence to obtain the first coarse restoration reference image.

[0036] The first coarse restored reference image, along with the first blur direction and the first blur length, are input into the conditional latent space diffusion model to obtain the predicted image;

[0037] The same low-pass filtering is applied to the predicted image and the first coarse restoration reference image respectively. The low-frequency components of the predicted image and the first coarse restoration reference image are extracted, and the second low-frequency difference between the low-frequency components of the predicted image and the first coarse restoration reference image is calculated as the low-frequency consistency loss.

[0038] The global color statistical features of the predicted image and the global color statistical features of the first coarse restoration reference image are calculated respectively, and the difference between the global color statistical features of the predicted image and the global color statistical features of the first coarse restoration reference image is calculated as the color consistency loss.

[0039] The weighted sum of low-frequency consistency loss and color consistency loss is added as the total penalty term to the training objective function of the conditional latent space diffusion model, and the model parameters of the conditional latent space diffusion model are adjusted by the optimizer.

[0040] Optionally, the refined and restored image is subjected to frequency domain angular consistency constraints to output the final restored image, including:

[0041] The thinned and restored image is subjected to Fast Fourier Transform in a sliding window manner to obtain the spectrum of each window. The spectrum of each window is then mapped from Cartesian coordinates to polar coordinates to obtain the spectrum representation in polar coordinates.

[0042] The angular energy distribution of each window is statistically analyzed to obtain the angular energy curve of each window, and the angular energy curve of each window is checked for abnormal peaks in the preset direction.

[0043] If an abnormal spike is detected, a preset amplitude scaling suppression is applied to the frequency band corresponding to that direction within the window to obtain the suppressed spectrum.

[0044] The suppressed spectrum is subjected to inverse fast Fourier transform to obtain the final restored image.

[0045] Optionally, the method further includes:

[0046] The second blur direction, second blur length and corresponding second confidence of each pixel in the second single-frame training blurred image are obtained. Based on the second blur direction, second blur length and second confidence, the second single-frame training blurred image is subjected to directional regularized deconvolution to obtain the second coarse restoration reference image.

[0047] The second coarse restored reference image, along with the second blur direction and the second blur length, are input into the conditional latent space diffusion model to obtain the predicted restored image;

[0048] Frequency domain processing was performed on the predicted restored image and the sharp reference image corresponding to the second single-frame training blurred image, respectively, and the angular energy curves of the predicted restored image and the sharp reference image were statistically analyzed.

[0049] The difference between the angular energy curve of the predicted restored image and the angular energy curve of the sharp reference image is calculated as the frequency domain angular consistency loss function;

[0050] The frequency domain angular consistency loss function is added to the overall training objective function of the conditional latent space diffusion model, and the model parameters of the conditional latent space diffusion model are adjusted by the optimizer.

[0051] Secondly, the present invention provides a single-frame image motion deblurring device based on direction perception and diffusion refinement, comprising:

[0052] The acquisition module is used to acquire a single-frame blurred image;

[0053] The parameter estimation module, connected to the acquisition module, is used to estimate the spatial motion blur parameters of a single-frame blurred image, and obtain the blur direction, blur length and corresponding confidence level of each pixel in the single-frame blurred image;

[0054] The directional regularized deconvolution module, connected to the parameter estimation module, is used to perform directional regularized deconvolution on a single-frame blurred image based on the blur direction, blur length, and confidence level to obtain a coarse restored image.

[0055] The Conditional Latent Space Diffusion Model, connected to the Direction Regularized Deconvolution Module, is used to refine the coarse restored image by inputting the blur direction and blur length into the pre-trained Conditional Latent Space Diffusion Model, resulting in a refined restored image.

[0056] The frequency domain constraint module, connected to the conditional latent space diffusion model, is used to apply frequency domain angular consistency constraints to the refined restored image and output the final restored image.

[0057] The present invention provides a single-frame image motion deblurring method and apparatus based on direction awareness and diffusion refinement. By estimating the blur direction and blur length of each pixel, a spatially variable motion blur model aligned with the actual degradation process is constructed, enabling physically reliable restoration of the structural edges of the coarsely restored image. Based on this, a conditional latent spatial diffusion model is introduced to supplement high-frequency details, and a data consistency mechanism constrains the generated result within a preset neighborhood of the coarsely restored image. This leverages the advantages of the generation model in texture restoration while effectively preventing content fabrication. Furthermore, frequency domain angular consistency constraints are used to directionally eliminate residual ringing and stripe artifacts along the main blur direction, significantly improving the visual quality of the image. Attached Figure Description

[0058] Figure 1a This is a flowchart illustrating a single-frame image motion deblurring method based on direction perception and diffusion refinement provided in an embodiment of the present invention.

[0059] Figure 1b This is another flowchart illustrating the single-frame image motion deblurring method based on direction perception and diffusion refinement provided in an embodiment of the present invention.

[0060] Figure 1c This is another flowchart illustrating the single-frame image motion deblurring method based on direction perception and diffusion thinning provided in this embodiment of the invention.

[0061] Figure 2 A schematic diagram of the circuit module structure of the single-frame image motion deblurring device based on direction perception and diffusion refinement provided in an embodiment of the present invention;

[0062] Figure 3 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0064] like Figure 1a As shown, this invention provides a single-frame image motion deblurring method based on direction awareness and diffusion thinning, comprising:

[0065] Step S10: Obtain a single-frame blurred image and estimate the spatial motion blur parameters of the single-frame blurred image to obtain the blur direction, blur length and corresponding confidence level of each pixel in the single-frame blurred image;

[0066] The blurred single-frame image can be captured by surveillance cameras or cameras deployed along highways, or it can be an image collected by urban traffic monitoring, intelligent driving vehicle recorders, security monitoring, drone aerial photography, handheld shooting devices, or other complex motion scenarios. Those skilled in the art should understand that the single-frame image motion deblurring method based on direction perception and diffusion refinement provided by this invention is applicable to the restoration of motion blur in single-frame images caused by high-speed object movement, camera shake, or excessively long exposure times in various scenarios.

[0067] In an optional embodiment of the present invention, the step S10 of estimating the spatial motion blur parameters of a single-frame blurred image to obtain the blur direction, blur length, and corresponding confidence level of each pixel in the single-frame blurred image includes:

[0068] Step S101: The RGB channel feature map, grayscale image, horizontal gradient map, vertical gradient map, and Gabor filter response maps of multiple preset directions of a single frame blurred image are concatenated along the channels to obtain the input feature tensor.

[0069] In this invention, the RGB channel feature map, grayscale map, horizontal gradient map, vertical gradient map, and Gabor filter response map of multiple preset directions of a single frame blurred image can all be obtained using existing methods. Those skilled in the art can flexibly choose according to actual needs, and no limitation is made here.

[0070] In this invention, those skilled in the art can select the number and angle of multiple preset directions according to actual needs, and no limitation is made here. Preferably, the multiple preset directions are selected from four directions: 0°, 90°, 180°, and 270°.

[0071] In one specific embodiment of the present invention, a single-frame blurred image can be preprocessed using the OpenCV image processing library (i.e., the OpenCV library) to obtain the RGB channel feature map, grayscale image, horizontal gradient map, vertical gradient map, and Gabor filter response map of multiple preset directions of the single-frame blurred image to construct the input feature tensor. The specific steps are as follows:

[0072] Assume a single-frame blurred image is in BGR color space format (the default reading format of the OpenCV library) with a resolution of 1920×1080.

[0073] First, the color is converted to RGB format using the `cv2.cvtColor` function in the OpenCV library. Then, the `cv2.split` function in the OpenCV library is used to separate the R, G, and B channels. Each channel is a 1920×1080 two-dimensional matrix, which serves as the RGB channel feature map. It should be noted that the RGB channel feature map includes the R channel feature map, G channel feature map, and B channel feature map. Figure 3 Each channel feature map is independent.

[0074] Secondly, the cv2.cvtColor function in the OpenCV library is used, specifying the conversion code cv2.COLOR_RGB2GRAY, to convert the RGB image into a single-channel grayscale image, which is also a two-dimensional matrix with the same size of 1920×1080.

[0075] Next, the cv2.Sobel operator from the OpenCV library is used to calculate the horizontal and vertical gradient maps of the grayscale image, both of which are two-dimensional matrices of 1920×1080.

[0076] Then, multiple directions are preset (e.g., 6 directions, namely 0°, 30°, 60°, 90°, 120° and 150°), and Gabor kernels are generated for each direction using the cv2.getGaborKernel function in the OpenCV library; after generating the kernels, the grayscale image is filtered using the cv2.filter2D function in the OpenCV library to obtain the Gabor filter response map for the corresponding direction. Each Gabor filter response map is a 1920×1080 two-dimensional matrix, for a total of 6 Gabor filter response maps.

[0077] Finally, the RGB channel feature maps (R, G, and B channels, a total of 3 channels), the grayscale image (1 channel), the horizontal gradient map (1 channel), the vertical gradient map (1 channel), and the 6 Gabor filter response maps (6 channels) are concatenated along the channel dimensions using the `np.stack` function in the NumPy (NumericalPython) library to obtain an input feature tensor of size 1920×1080×12. This input feature tensor includes the color, brightness, edge direction, and multi-scale directional texture information of a single-frame blurred image.

[0078] Step S102: Use a multi-scale convolutional neural network to extract features from the input feature tensor to obtain a multi-scale feature map including a shallow feature map;

[0079] Optionally, the multi-scale convolutional neural network adopts a lightweight multi-scale convolutional neural network with ResNet-18 as the backbone and FPN (Feature Pyramid Network) structure as the neck. Both ResNet-18 and FPN are existing ResNet-18 and FPN architectures.

[0080] ResNet (Residual Network) is a deep neural network architecture that addresses the vanishing gradient and representation bottleneck problems during deep neural network training by introducing residual connections. ResNet-18 is a lightweight model in the ResNet series.

[0081] Optionally, the shallow feature map is the feature map output by the first convolutional block in the multi-scale convolutional neural network. Those skilled in the art can choose according to actual needs, and there is no limitation here.

[0082] Step S103: Calculate the gradient covariance matrix of each pixel in a single-frame blurred image, and calculate the principal direction and anisotropy of each pixel as statistical features of the local structure tensor based on the gradient covariance matrix of each pixel.

[0083] In one specific embodiment of the present invention, the calculation of local structure tensor statistical features can be implemented using the OpenCV library. The specific steps are as follows:

[0084] ① If a single-frame blurred image is an RGB image, use the cv2.cvtColor function in the OpenCV library to convert it to a grayscale image.

[0085] ② Use the Sobel operator (such as a 3×3 convolution kernel) to perform convolution operations on the grayscale image to obtain the horizontal gradient map of the grayscale image. and vertical gradient plot Both are two-dimensional matrices with the same size as the blurred image in a single frame. In the OpenCV library, this operation is implemented using the cv2.Sobel function, by setting the kernel size kme=3.

[0086] ③ Use the cv2.multiply function from the OpenCV library to calculate the results element-wise. square matrix , square matrix as well as and product matrix All three are two-dimensional matrices with the same size as the single-frame blurred image.

[0087] ④ Use the cv2.boxFilter function from the OpenCV library, and use a rectangular window of size w×w (e.g., w=5) to filter the boxes. , as well as Mean filtering is performed by averaging the second moments of all pixels within the rectangular window to obtain the corresponding first gradient covariance component matrix. Second gradient covariance component matrix and the third gradient covariance component matrix .

[0088] ⑤ Based on the first gradient covariance component matrix Second gradient covariance component matrix and the third gradient covariance component matrix This allows us to obtain the gradient covariance matrix for each pixel: ,in, The coordinates of the pixels. For pixels The first gradient covariance component, For pixels The second gradient covariance component, For pixels The third gradient covariance component.

[0089] ⑥ According to the eigenvalue formula of a real symmetric matrix ,in, For pixels The feature values ​​are used to calculate the first feature value of each pixel. Second eigenvalue ,in, .Right now:

[0090] ,

[0091] .

[0092] ⑦ Based on the gradient covariance matrix of each pixel, and according to the formula Calculate the principal direction of each pixel .

[0093] ⑧ Based on the first feature value of each pixel Second eigenvalue And according to the formula Calculate the anisotropy degree of each pixel. ; where the anisotropy degree of each pixel The value range is [0,1]; To prevent zero constant, .

[0094] ⑨ The principal orientation and anisotropy of each extracted pixel are used as local structure tensor statistical features. That is, the local structure tensor statistical features include a directional feature map composed of the principal orientations of all pixels. and anisotropic feature map composed of the anisotropic degrees of all pixels. .

[0095] It should be noted that the above specific implementation method is only one way of implementation, and the present invention is not limited thereto. Those skilled in the art can choose other existing methods for calculating the statistical characteristics of local structure tensors according to actual needs.

[0096] Step S104: Fuse the local structure tensor statistical features with the shallow feature map in the multi-scale feature map to obtain the enhanced feature map;

[0097] In one specific embodiment of the present invention, firstly, directional feature maps are detected respectively. and anisotropy degree feature map Compared with the shallow feature map obtained from step S102 Alignment. (If the orientation feature map...) and anisotropy degree feature map Compared with the shallow feature map obtained from step S102 Misalignment can be addressed by upsampling or downsampling the orientation feature map. and anisotropy degree feature map Spatial dimensions and shallow feature maps The spatial dimensions remain consistent. Upsampling can employ bilinear interpolation, while downsampling can utilize max pooling or convolution with a stride of 2. Those skilled in the art can choose flexibly according to actual needs, and no limitations are imposed here.

[0098] Then, the resized orientation feature map and anisotropy degree feature map With shallow feature map The feature maps are concatenated along the channel dimension to obtain the concatenated feature maps. Its number of channels is the number of channels in the shallow feature map plus 2.

[0099] Finally, the stitched feature map Input a 1×1 convolutional layer (or a 3×3 convolutional layer, which can be padded to maintain size), perform cross-channel information fusion and dimensionality reduction, and the number of output channels can be set according to the needs of subsequent networks (such as maintaining the same number of channels as the shallow feature map) to obtain the enhanced feature map. The enhanced feature map includes both the original shallow semantic information and the local structural tensor statistical feature information, providing a more robust feature representation for subsequent direction, length, and confidence regression.

[0100] Step S105: Based on the enhanced feature map, output the blur direction, blur length and corresponding confidence of each pixel in the single frame blurred image through the direction regression head, length regression head and confidence head respectively.

[0101] Optionally, the orientation regression head is composed of a The system consists of convolutional layers with two output channels, corresponding to... and No activation function is used for subsequent recovery of the blur direction. .

[0102] Optionally, the length regression head adopts a two-stage structure of "coarse classification + fine shift," including a classification branch and a regression branch. The classification branch consists of a... The system consists of convolutional layers with N output channels (N being the preset number of length intervals, typically 10 to 50; those skilled in the art can choose flexibly according to actual needs, and this is not limited here), followed by a Softmax activation function to output the probability of each length interval; the regression branch consists of a... The system consists of convolutional layers, each outputting a single-channel offset. The final blur length is obtained by adding the corresponding offset to the center value of the interval with the highest probability.

[0103] Optionally, the confidence header consists of a The system consists of convolutional layers with a single output channel followed by a sigmoid activation function. It should be noted that the confidence score for each pixel in the single-frame blurred image output by the confidence header ranges from [0,1].

[0104] It should be understood that those skilled in the art can flexibly select existing directional regression headers, length regression headers, and confidence headers and make appropriate adjustments according to actual needs, without any limitations here.

[0105] Step S20: Based on the blur direction, blur length and confidence, perform directional regularized deconvolution on the single-frame blurred image to obtain a coarse restored image;

[0106] In one optional embodiment of the present invention, such as Figure 1b As shown, step S20 specifically includes:

[0107] Step S201: Construct an inverse filter kernel library, wherein the inverse filter kernel library includes multiple inverse filter kernels corresponding to multiple discrete directions and multiple discrete length combinations, and a preset regularization parameter is introduced into each inverse filter kernel;

[0108] Among them, 12 discrete directions can be selected, with a step size of 15, namely 0°, 15°, 30°, 45°, 60°, 75°, 90°, 105°, 120°, 135°, 150°, and 165°; 5 discrete lengths can be selected, with a value range of [1, 50]. Specific values ​​can be flexibly set by those skilled in the art according to actual needs. Furthermore, each inverse filter kernel in the inverse filter kernel library can be equipped with preset regularization parameters, such as Tikhonov regularization. Those skilled in the art can flexibly adjust the image noise level according to the actual application requirements, and the present invention does not limit this.

[0109] In an optional embodiment of the present invention, for each pixel According to its fuzzy direction and fuzzy length In a two-dimensional empty matrix of a predetermined size, along the direction Draw a line of length The line segment, with each discrete grid cell it passes through assigned a value. Thus, the positive blur kernel corresponding to that pixel is obtained. Subsequently, in the frequency domain, the Tikhonov regularization method is used, based on the positive fuzzy kernel. Calculate the corresponding frequency domain inverse filter kernel Finally, the frequency domain inverse filter kernel is... The spatial domain inverse filter kernel is obtained by transforming the image back to the spatial domain using an inverse Fourier transform. This spatial domain inverse filter kernel is then stored in an inverse filter kernel library as a separable one-dimensional kernel. In this way, we can define the fuzzy direction for each pair of discrete fuzzy directions. and fuzzy length The corresponding inverse filter kernel is pre-calculated to form a lookup table.

[0110] For the two-dimensional empty matrix, those skilled in the art can flexibly set it according to actual needs, and no limitation is made here. For example, according to the formula Calculate the size of a two-dimensional empty matrix. ;in, It represents the maximum discrete length in the inverse filter kernel library.

[0111] Optionally, in the frequency domain, to suppress noise amplification during the deconvolution process, the Tikhonov regularization method is used to construct the corresponding inverse filter kernel. Let... For the Fourier transform of the positive fuzzy kernel, Let λ be its conjugate complex number, and λ be a preset regularization parameter (e.g., λ=0.01). Then, the frequency domain inverse filter kernel is expressed as: This formula, by introducing a preset regularization parameter λ, in At frequencies close to zero, the inverse filter gain is limited to within 1 / λ, thereby preventing high-frequency noise from being excessively amplified.

[0112] Step S202: For each pixel, select multiple inverse filter kernels with adjacent directions and lengths from the inverse filter kernel library as a candidate kernel set based on the blur direction and blur length;

[0113] Step S203: A hybrid gating mechanism is adopted to weight and fuse the candidate kernels in the candidate kernel set according to the confidence level, so as to generate the adaptive inverse filter kernel of the corresponding pixel;

[0114] The hybrid gating mechanism can be any existing hybrid gating mechanism, and those skilled in the art can choose flexibly according to actual needs; no limitation is made here.

[0115] Step S204: Convert the single-frame blurred image from the RGB color space to the luminance-chrominance color space, separate the luminance channel and the chrominance channel, and apply an adaptive inverse filter kernel to perform deconvolution processing on the luminance channel to obtain the deconvolution-processed luminance channel and perform weakening processing on the chrominance channel to obtain the weakened chrominance channel.

[0116] Step S205: Merge the luminance channel after deconvolution and the chroma channel after weakening, and convert them back to the RGB color space to obtain the coarsely restored image.

[0117] To enable those skilled in the art to more clearly understand steps S201 to S205, a detailed explanation is provided below, using a 3×3 pixel single-frame blurred image as an example:

[0118] ① A pre-built inverse filter kernel library is constructed, including inverse filter kernels corresponding to two discrete directions (0° and 90°) and two discrete lengths (2 pixels and 4 pixels). In other words, there are a total of 4 inverse filter kernels in the library for each pixel. A regularization parameter is introduced into each inverse filter kernel during offline calibration. (e.g., using Tikhonov regularization), and stored as a separable one-dimensional kernel, denoted as: , , , ,in, Represents a pixel with a direction of 0° and a length of 2. The inverse filter kernel is denoted as , and others follow the same pattern, which will not be elaborated here.

[0119] ②Assuming that step S10 has estimated the blur direction, blur length, and corresponding confidence level of each pixel, the center pixel is used as the reference point. For example, its fuzzy direction fuzzy length Pixels, confidence level Select from the inverse filter kernel library... Adjacent directions 0° and 90° and with Adjacent lengths 2 and 4 yield the candidate kernel set: , , , .

[0120] ③ Calculate the fusion weight of each candidate kernel in the candidate kernel set.

[0121] Based on the angle difference, calculate the directional weights of the candidate kernels in different directions (using softmax normalization, with the temperature parameter set to 30°): , ;in, This represents the orientation weight of the candidate kernel for the pixel with an orientation of 0°. This represents the directional weight of the candidate kernel for the pixel with a 90° orientation.

[0122] Using linear interpolation or an equal-weighting method, the length weights for different lengths of the candidate kernels are calculated based on the length difference. For example, using an equal-weighting method, the length weights are calculated based on the length difference. , ;in, The length weight of the candidate kernel for this pixel is 2. This represents the length weight of the candidate kernel for this pixel, which has a length of 4.

[0123] Calculate the product of the orientation weight and length weight for each candidate kernel as the base weight, and then multiply it by the confidence level. The base weights are modulated as a whole to obtain the modulated weights for each candidate kernel of the pixel:

[0124] ,in, This represents the modulated weight of the candidate kernel for the pixel with a direction of 0° and a length of 2.

[0125] ,in, This represents the modulated weights of the candidate kernels for this pixel, which has a direction of 0° and a length of 4.

[0126] ,in, This represents the modulated weight of the candidate kernel for this pixel, which has a direction of 90° and a length of 2.

[0127] ,in, This represents the modulated weight of the candidate kernel for this pixel, which has a direction of 90° and a length of 4.

[0128] center pixel The adaptive inverse filter kernel is a weighted sum of the above candidate kernels, and the result is as follows:

[0129] ,in, For pixels The adaptive inverse filter kernel.

[0130] The calculation method for the adaptive inverse filter kernel of other pixels follows the same logic, and will not be elaborated here.

[0131] ④ A pixel-by-pixel blurred image is converted from the RGB color space to the YUV color space to obtain the luminance channel. and chroma channels , Each channel is matrix.

[0132] For each pixel, perform deconvolution based on its adaptive inverse filter kernel. Using the center pixel... For example, the new brightness value after deconvolution is used as the pixel's... The components are calculated similarly for other pixels, which will not be elaborated here, resulting in the overall brightness image. (i.e., the brightness channel after deconvolution).

[0133] right and Instead of strong deconvolution, the channels are weakened to suppress color noise, such as the chroma channel. , Perform Gaussian filtering (e.g., standard deviation) ), to obtain the smoothed color channel , (i.e., the weakened chroma channel).

[0134] ⑤ The brightness channel after deconvolution processing With weakened chroma channels , Merge and convert back to RGB color space to obtain the desired result. Pixel coarse reconstruction of the image.

[0135] Step S30: The coarse restored image is processed by inputting the blur direction and blur length into the pre-trained conditional latent space diffusion model to obtain the refined restored image;

[0136] The conditional latent space diffusion model used here is an existing one, such as the StableDiffusion model. Those skilled in the art can flexibly choose the appropriate model based on actual needs, and no limitation is imposed here. For example, conditional latent space diffusion models include variational autoencoders, U-Net networks, noise scheduling strategies, sampling algorithms, etc. All of these can be flexibly selected and adjusted by those skilled in the art according to actual requirements, and this invention does not impose any limitations on them.

[0137] It should be noted that the blur direction and blur length in step S30 are calculated based on each pixel in a single-frame blurred image, and both the blur direction and blur length are numerical matrices corresponding to the spatial size of a single-frame blurred image.

[0138] In one optional embodiment of the present invention, such as Figure 1c As shown, step S30 specifically includes:

[0139] Step S301: Input the coarse restored image, along with the blur direction and blur length, into the pre-trained conditional latent space diffusion model to generate a temporary decoded image;

[0140] Optionally, step S301 specifically includes:

[0141] Step S3011: Encode the coarse restored image using the encoder of the variational autoencoder in the pre-trained conditional latent space diffusion model to obtain the first latent space features;

[0142] Step S3012: Normalize the fuzzy direction and fuzzy length respectively, and use the normalized fuzzy direction and fuzzy length as geometric conditions.

[0143] Regarding the normalization methods for fuzzy direction and fuzzy length, those skilled in the art can choose existing normalization methods according to actual needs, and no limitation is made here. For example, for fuzzy direction... It can be calculated by its sine and cosine values. Normalization is performed to facilitate subsequent integration with polar coordinate representation; for fuzzy length... The min-max normalization method can be used: ,in, For the maximum fuzzy length, Minimum fuzzy length, This represents the normalized fuzzy length. After the above processing, the normalized fuzzy direction... The range of the value is [-1, 1], and the normalized fuzzy length is... The range of values ​​is [0,1], which makes it convenient to input geometric conditions into the U-Net network.

[0144] Step S3013: Iteratively denoise the first latent space features and the U-Net network in the geometric condition input latent space diffusion model to obtain the second latent space features;

[0145] In one optional embodiment of the present invention, firstly, the normalized fuzzy direction matrix (size: H×W×2, each pixel corresponds to...) is... (Two channels) and the normalized blur length matrix (size: H×W×1, each pixel corresponds to...) One channel is directly concatenated along the channel dimension to form a geometric condition (H×W×3). Then, the geometric condition is input into the encoder of the variational autoencoder for encoding, and the encoded geometric condition is used as a guide to iteratively denoise the U-Net network in the conditional latent space diffusion model with the first latent space feature input to obtain the second latent space feature.

[0146] It should be noted that before inputting the fuzzy direction and fuzzy length as geometric conditions into the latent space diffusion model, those skilled in the art can perform necessary preprocessing operations such as alignment, dimensional transformation, or feature mapping according to the specific input interface requirements of the selected model. Such processing is a conventional technique in this field and will not be elaborated here. Other models follow the same principle and will not be elaborated here either.

[0147] Step S3014: Input the second latent space features into the decoder of the variational autoencoder in the conditional latent space diffusion model to obtain a temporary decoded image.

[0148] Step S302: Based on the data consistency mechanism, the temporary decoded image is constrained within a preset neighborhood of the coarse restored image to obtain the refined restored image.

[0149] In an optional embodiment of the present invention, step S302 specifically includes:

[0150] Step S3021: Perform the same low-pass filtering on the temporary decoded image and the coarse restored image respectively, and extract the low-frequency components of the temporary decoded image and the coarse restored image.

[0151] In one specific embodiment of the present invention, a Gaussian filter can be used as a low-pass filter, and the kernel size is set to [value missing]. The standard deviation is 1.5. The `cv2.GaussianBlur` function from the OpenCV library is applied to both the temporary decoded image and the coarsely restored image to obtain their respective low-frequency components. This operation preserves the low-frequency structural information in both the temporary decoded image and the coarsely restored image, providing a basis for subsequent calculation of the first low-frequency difference. Those skilled in the art can adjust the filter kernel size and standard deviation according to the image resolution and noise level; this invention does not limit this adjustment.

[0152] Step S3022: Calculate the first low-frequency difference between the low-frequency components of the temporary decoded image and the low-frequency components of the coarse restored image, and correct the temporary decoded image according to the first low-frequency difference, so that the low-frequency components of the corrected temporary decoded image and the low-frequency components of the coarse restored image are consistent in a preset neighborhood, thereby obtaining a refined restored image.

[0153] It should be understood that in step S3022, the low-frequency components of the temporary decoded image and the low-frequency components of the coarse restored image are subtracted pixel by pixel to obtain the first low-frequency difference.

[0154] In an optional embodiment of the present invention, the method further includes:

[0155] Step S31: Obtain the first blur direction, first blur length and corresponding first confidence of each pixel in the first single-frame training blurred image, and perform directional regularized deconvolution on the first single-frame training blurred image based on the first blur direction, first blur length and first confidence to obtain the first coarse restoration reference image.

[0156] The implementation of step S31 can be referred to the implementation of steps S10 and S20 of the present invention, and will not be repeated here.

[0157] The first single-frame training blurred image is derived from a pre-built training image library. This library contains a large number of blurred images and their corresponding sharp reference images. The blurred images can be generated in various ways, such as by capturing images in real highway scenes with a high-speed camera, or by applying a known motion blur kernel to a sharp image and then performing simulation synthesis; no specific method is used here. To ensure the effectiveness of model training, the blurred images in the training image library cover various motion blur types, including different blur directions and different blur lengths.

[0158] Step S32: Input the first coarse restored reference image, the first blur direction, and the first blur length into the conditional latent space diffusion model to obtain the predicted image;

[0159] The implementation of step S32 can be referred to the implementation of step S301 of the present invention, and will not be repeated here.

[0160] Step S33: Perform the same low-pass filtering on the predicted image and the first coarse restoration reference image respectively, extract the low-frequency components of the predicted image and the first coarse restoration reference image, and calculate the second low-frequency difference between the low-frequency components of the predicted image and the low-frequency components of the first coarse restoration reference image as the low-frequency consistency loss.

[0161] The implementation of step S33 can be referred to the implementation of step S302 of the present invention, and will not be repeated here.

[0162] Step S34: Calculate the global color statistical features of the predicted image and the global color statistical features of the first coarse restoration reference image respectively, and calculate the difference in global color statistical features between the global color statistical features of the predicted image and the global color statistical features of the first coarse restoration reference image as the color consistency loss.

[0163] In one specific embodiment of the present invention, the predicted images are calculated respectively. and the first coarse restoration reference image The global mean and standard deviation across the three RGB channels form a 6-dimensional feature vector:

[0164] ;

[0165] ;

[0166] in, To predict the global color statistical features of an image, To predict the mean of the R channel of an image, To predict the mean of the G channel of an image, To predict the mean of the B channel of an image, To predict the standard deviation of the R channel of an image, To predict the standard deviation of the G channel of an image, To predict the standard deviation of the B channel of an image, The global color statistical features of the first coarsely restored reference image. The mean value of the R channel of the first coarsely restored reference image. The mean value of the G channel of the first coarsely restored reference image. The mean value of the B channel of the first coarsely restored reference image. The standard deviation of the R channel of the first coarsely restored reference image. The standard deviation of the G channel of the first coarsely restored reference image. The standard deviation of the B channel of the first coarse restoration reference image is given.

[0167] In implementation, the cv2.meanStdDev function in the OpenCV library can be used to quickly calculate the global mean and standard deviation of each channel, and then combine them into the corresponding feature vector. Those skilled in the art can also use other global color statistical features (such as color histograms, tone moments, etc.), and this invention does not limit them.

[0168] The difference between two feature vectors (i.e., the difference in global color statistics between the predicted image and the first coarse restored reference image) is calculated using Euclidean distance (or Manhattan distance) as the color consistency loss. For example, the difference between two feature vectors is calculated using Euclidean distance: ;in, This represents the color consistency loss; the smaller the value, the closer the overall color distribution of the predicted image is to that of the first coarsely restored reference image. This represents the Euclidean norm or L2 norm.

[0169] Step S35: The weighted sum of the low-frequency consistency loss and the color consistency loss is added as the total penalty term to the training objective function of the conditional latent space diffusion model, and the model parameters of the conditional latent space diffusion model are adjusted by the optimizer.

[0170] The optimizer can be an existing optimizer, such as the Adam optimizer. Those skilled in the art can choose flexibly according to actual needs, and there is no limitation here. Specifically, the Adam optimizer is used to adjust the parameters of the conditional latent space diffusion model through the backpropagation algorithm.

[0171] The training objective function for the conditional latent space diffusion model can be flexibly set by those skilled in the art according to actual needs, and is not limited here. Specifically, the training objective function for the conditional latent space diffusion model can be: .in, This represents the noise prediction loss (e.g., mean square error) inherent in the conditional latent space diffusion model itself. This represents the low-frequency consistency loss, constraining the low-frequency structure of the generated image to be consistent with the first coarse restoration reference image; This represents the color consistency loss, constraining the global color distribution of the generated image to be consistent with the first coarse restoration reference image; and The hyperparameters representing the balance of various losses are set experimentally (e.g., ...). , ).

[0172] Step S40: Apply frequency domain angular consistency constraints to the refined and restored image, and output the final restored image.

[0173] In an optional embodiment of the present invention, step S40 specifically includes:

[0174] Step S401: Perform a fast Fourier transform on the thinned and restored image using a sliding window method to obtain the spectrum of each window, and map the spectrum of each window from Cartesian coordinates to polar coordinates to obtain the spectrum representation of each window in polar coordinates.

[0175] The Fast Fourier Transform method and the method of mapping Cartesian coordinates to polar coordinates can both be employed using existing methods. Those skilled in the art can choose flexibly according to actual needs, and no limitation is imposed here. In addition, those skilled in the art can also flexibly choose the size of the sliding window according to actual needs, and no limitation is imposed here.

[0176] Step S402: Statistically analyze the angular energy distribution of each window to obtain the angular energy curve of each window, and detect whether there are abnormal peaks in the angular energy curve of each window in the preset direction.

[0177] For example, assuming that after step S401, the amplitude and corresponding azimuth angle (in degrees) of each frequency point in a 3×3 spectrum obtained are shown in Table 1 (the DC component (0,0) does not participate in the azimuth analysis and is not shown in the table below):

[0178] Table 1

[0179]

[0180] Divide the 0° to 180° range into four intervals, each 45° wide: interval A: [0°, 45°), interval B: [45°, 90°), interval C: [90°, 135°), and interval D: [135°, 180°].

[0181] The frequency points are assigned to the corresponding intervals and the energy (i.e., amplitude) is accumulated, as follows:

[0182] Interval A: Includes amplitudes of 20 (-1,0) and 18 (1,0), with a total energy of 38.

[0183] Interval B: Contains amplitudes of 6 at (1,1) and 4 at (-1,-1), with a total energy of 10.

[0184] Interval C: contains amplitudes of 10 at (0,1) and 8 at (0,-1), with a total energy of 18.

[0185] Interval D: contains amplitudes of 5 at (-1,1) and 3 at (1,-1), with a total energy of 8.

[0186] This yields the angular energy curve: , , , .

[0187] Step S403: If an abnormal spike is detected, apply a preset amplitude scaling suppression to the frequency band corresponding to that direction within the window to obtain the suppressed spectrum;

[0188] Continuing with the previous example, calculate the average energy across the four intervals. Standard deviation And set the detection threshold as The energy of interval A, 38, is greater than 30. Therefore, it is determined that there is an abnormal spike in the horizontal direction (near 0°) of this window.

[0189] Multiply the amplitude of the frequency points within interval A by the suppression coefficient. The amplitude at frequency (-1, 0) becomes The amplitude at frequency (1,0) becomes The total energy of the modified interval A is reduced to about 7.6, while the energy of other intervals remains unchanged.

[0190] It should be understood that the above example is for calculating the angular energy curve and detecting abnormal spikes for a single window. Other windows can be calculated similarly to obtain the suppressed spectrum, which will not be elaborated further here. Furthermore, those skilled in the art can flexibly choose the suppression coefficient according to actual needs; no limitations are imposed here.

[0191] Step S404: Perform inverse fast Fourier transform on the suppressed spectrum to obtain the final restored image.

[0192] In an optional embodiment of the present invention, the method further includes:

[0193] Step S41: Obtain the second blur direction, second blur length and corresponding second confidence of each pixel in the second single-frame training blurred image, and perform directional regularized deconvolution on the second single-frame training blurred image based on the second blur direction, second blur length and second confidence to obtain the second coarse restoration reference image.

[0194] The implementation of step S41 can be referred to the implementation of steps S10 and S20 of the present invention, and will not be repeated here.

[0195] The second single-frame training blurred image is derived from a pre-built training image library. This library contains a large number of blurred images and their corresponding sharp reference images. The blurred images can be generated in various ways, such as by capturing images in real highway scenes with a high-speed camera, or by applying a known motion blur kernel to a sharp image and then performing simulation synthesis; no specific method is used here. To ensure the effectiveness of model training, the blurred images in the training image library cover various motion blur types, including different blur directions and different blur lengths.

[0196] Step S42: Input the second coarse restored reference image, the second blur direction, and the second blur length into the conditional latent space diffusion model to obtain the predicted restored image;

[0197] The implementation of step S42 can be referred to the implementation of step S301 of the present invention, and will not be repeated here.

[0198] Step S43: Perform frequency domain processing on the predicted restored image and the clear reference image corresponding to the second single-frame training blurred image respectively, and statistically analyze the angular energy curve of the predicted restored image and the angular energy curve of the clear reference image.

[0199] The implementation of step S43 can be referred to the implementation of steps S401 and S402 of the present invention, and will not be repeated here.

[0200] Step S44: Calculate the difference between the angular energy curve of the predicted restored image and the angular energy curve of the sharp reference image as the frequency domain angular consistency loss function;

[0201] The frequency domain angular consistency loss function is calculated based on Manhattan distance or Euclidean distance. One approach is to subtract the energy value of the predicted restored image's angular energy curve from the energy value of the sharp reference image for each corresponding angular interval, take the absolute value, and then sum the absolute values ​​of all angular intervals to obtain a scalar. Alternatively, the energy difference can be obtained by subtracting the energy value of the predicted restored image's angular energy curve from the energy value of the sharp reference image for each corresponding angular interval, then squaring each energy difference, summing all the squares, and finally taking the square root.

[0202] Step S45: Add the frequency domain angular consistency loss function to the overall training objective function of the conditional latent space diffusion model, and adjust the model parameters of the conditional latent space diffusion model through the optimizer.

[0203] The optimizer can be an existing optimizer, such as the Adam optimizer. Those skilled in the art can choose flexibly according to actual needs, and there is no limitation here. Specifically, the Adam optimizer is used to adjust the parameters of the conditional latent space diffusion model through the backpropagation algorithm.

[0204] The overall training objective function for the conditional latent space diffusion model can be flexibly set by those skilled in the art according to actual needs, and is not limited here. Specifically, the overall training objective function for the conditional latent space diffusion model can be: ,in, This represents the noise prediction loss (e.g., mean square error) inherent in the conditional latent space diffusion model itself. This represents the low-frequency consistency loss, constraining the low-frequency structure of the generated image to be consistent with the first coarse restoration reference image; This represents the color consistency loss, constraining the global color distribution of the generated image to be consistent with the first coarse restoration reference image; This represents the frequency domain angular consistency loss, which constrains the angular energy distribution of the generated image to be consistent with that of the sharp reference image; , and The hyperparameters representing the balance of various losses are set experimentally (e.g., ...). , , ).

[0205] It should be noted that steps S101 to S105, S3011 to S3014, S3021 to S3022, S31 to S35, S401 to S404, and S41 to S45 are for ease of description only and are not shown in the figure.

[0206] The single-frame image motion deblurring method provided by this invention, based on direction awareness and diffusion refinement, constructs a spatially variable motion blur model aligned with the actual degradation process by estimating the blur direction and blur length of each pixel, enabling physically reliable restoration of the structural edges of the coarsely restored image. Furthermore, a conditional latent spatial diffusion model is introduced to supplement high-frequency details, and a data consistency mechanism constrains the generated result within a preset neighborhood of the coarsely restored image, leveraging the advantages of the generation model in texture restoration while effectively preventing content fabrication. Finally, frequency domain angular consistency constraints are used to directionally eliminate residual ringing and stripe artifacts along the main blur direction, significantly improving the visual quality of the image.

[0207] Based on the same inventive concept, embodiments of the present invention also provide a single-frame image motion deblurring device based on direction perception and diffusion thinning for implementing the aforementioned single-frame image motion deblurring method based on direction perception and diffusion thinning. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more embodiments of the single-frame image motion deblurring device based on direction perception and diffusion thinning provided below can be found in the limitations of the single-frame image motion deblurring method based on direction perception and diffusion thinning described above, and will not be repeated here.

[0208] like Figure 2 As shown, this invention provides a single-frame image motion deblurring device based on direction awareness and diffusion refinement, characterized by comprising: an acquisition module 21, a parameter estimation module 22, a direction regularization deconvolution module 23, a conditional latent space diffusion model 24, and a frequency domain constraint module 25; wherein,

[0209] Acquisition module 21 is used to acquire a single-frame blurred image;

[0210] The parameter estimation module 22, connected to the acquisition module 21, is used to estimate the spatial motion blur parameters of a single-frame blurred image, and obtain the blur direction, blur length and corresponding confidence level of each pixel in the single-frame blurred image.

[0211] The directional regularized deconvolution module 23, connected to the parameter estimation module 22, is used to perform directional regularized deconvolution on a single-frame blurred image based on the blur direction, blur length, and confidence level to obtain a coarse restored image.

[0212] The conditional latent space diffusion model 24 is connected to the orientation regularized deconvolution module 23. It is used to refine the coarse restored image and the blur direction and blur length input into the pre-trained conditional latent space diffusion model to obtain a refined restored image.

[0213] The frequency domain constraint module 25, connected to the conditional latent space diffusion model 24, is used to perform frequency domain angular consistency constraints on the refined restored image and output the final restored image.

[0214] Optionally, the parameter estimation module 22 is specifically used to: concatenate the RGB channel feature map, grayscale image, horizontal gradient map, vertical gradient map, and Gabor filter response maps of multiple preset directions of a single-frame blurred image along the channels to obtain an input feature tensor; use a multi-scale convolutional neural network to extract features from the input feature tensor to obtain a multi-scale feature map including a shallow feature map; calculate the gradient covariance matrix of each pixel in the single-frame blurred image, and calculate the principal direction and anisotropy of each pixel as local structure tensor statistical features based on the gradient covariance matrix of each pixel; fuse the local structure tensor statistical features with the shallow feature map in the multi-scale feature map to obtain an enhanced feature map; based on the enhanced feature map, output the blur direction, blur length, and corresponding confidence of each pixel in the single-frame blurred image through the direction regression header, length regression header, and confidence header, respectively.

[0215] Optionally, the directional regularization deconvolution module 23 is specifically used for: constructing an inverse filter kernel library, wherein the inverse filter kernel library includes multiple inverse filter kernels corresponding to multiple discrete directions and multiple discrete length combinations, and each inverse filter kernel introduces a preset regularization parameter; for each pixel, according to the blur direction and blur length, selecting multiple inverse filter kernels with adjacent directions and lengths from the inverse filter kernel library as a candidate kernel set; using a hybrid gating mechanism, weighting and fusing the candidate kernels in the candidate kernel set with confidence to generate an adaptive inverse filter kernel for the corresponding pixel; converting the single-frame blurred image from the RGB color space to the luminance and chrominance color space, separating the luminance channel and the chrominance channel, and applying the adaptive inverse filter kernel to perform deconvolution processing on the luminance channel to obtain the deconvolution-processed luminance channel and weakening the chrominance channel to obtain the weakened chrominance channel; merging the deconvolution-processed luminance channel and the weakened chrominance channel, and converting them back to the RGB color space to obtain a coarse restored image.

[0216] Optionally, the conditional latent space diffusion model 24 is specifically used to: input the coarse restored image and the blur direction and blur length into the pre-trained conditional latent space diffusion model to generate a temporary decoded image; and constrain the temporary decoded image within a preset neighborhood of the coarse restored image according to the data consistency mechanism to obtain a refined restored image.

[0217] Optionally, the coarse restored image, along with the blur direction and blur length, is input into a pre-trained conditional latent space diffusion model to generate a temporary decoded image. This includes: encoding the coarse restored image using a variational autoencoder in the pre-trained conditional latent space diffusion model to obtain first latent space features; normalizing the blur direction and blur length respectively, and using the normalized blur direction and normalized blur length as geometric conditions; inputting the first latent space features and geometric conditions into a U-Net network in the conditional latent space diffusion model for iterative denoising to obtain second latent space features; and inputting the second latent space features into the variational autoencoder in the conditional latent space diffusion model for decoding to obtain a temporary decoded image.

[0218] Optionally, the temporary decoded image is constrained within a preset neighborhood of the coarse restored image according to the data consistency mechanism to obtain a refined restored image. This includes: applying the same low-pass filtering to both the temporary decoded image and the coarse restored image to extract the low-frequency components of the temporary decoded image and the coarse restored image; calculating the first low-frequency difference between the low-frequency components of the temporary decoded image and the coarse restored image; and correcting the temporary decoded image based on the first low-frequency difference so that the low-frequency components of the corrected temporary decoded image are consistent with the low-frequency components of the coarse restored image within the preset neighborhood to obtain a refined restored image.

[0219] Optionally, the apparatus of the present invention further includes: an optimizer (not shown in the figure), which is connected to the conditional latent space diffusion model 24, for adding the weighted sum of low-frequency consistency loss and color consistency loss as a total penalty term to the training objective function of the conditional latent space diffusion model, and adjusting the model parameters of the conditional latent space diffusion model through the optimizer. During this training phase, the parameter estimation module 22 is further configured to: obtain the first blur direction, first blur length, and corresponding first confidence level of each pixel in the first single-frame training blurred image; the directional regularization deconvolution module 23 is further configured to: perform directional regularization deconvolution on the first single-frame training blurred image based on the first blur direction, first blur length, and first confidence level to obtain the first coarse restoration reference image; the conditional latent space diffusion model 24 is further configured to: input the first coarse restoration reference image, the first blur direction, and the first blur length into the conditional latent space diffusion model to obtain the prediction image; perform the same low-pass filtering on the prediction image and the first coarse restoration reference image respectively, extract the low-frequency components of the prediction image and the first coarse restoration reference image, and calculate the second low-frequency difference between the low-frequency components of the prediction image and the low-frequency components of the first coarse restoration reference image as the low-frequency consistency loss; calculate the global color statistical features of the prediction image and the global color statistical features of the first coarse restoration reference image respectively, and calculate the difference in global color statistical features between the global color statistical features of the prediction image and the global color statistical features of the first coarse restoration reference image as the color consistency loss.

[0220] Optionally, the frequency domain constraint module 25 is specifically used for: performing a fast Fourier transform on the thinned restored image in a sliding window manner to obtain the spectrum of each window, and mapping the spectrum of each window from Cartesian coordinates to polar coordinates to obtain the spectrum representation in polar coordinates; statistically analyzing the angular energy distribution of each window to obtain the angular energy curve of each window, and detecting whether there are abnormal peaks in the angular energy curve of each window in a preset direction; if abnormal peaks are detected, applying a preset amplitude scaling suppression to the frequency band corresponding to that direction within the window to obtain the suppressed spectrum; and performing an inverse fast Fourier transform on the suppressed spectrum to obtain the final restored image.

[0221] Optionally, the optimizer is also connected to the frequency domain constraint module 25 to add the frequency domain angular consistency loss function to the overall training objective function of the conditional latent space diffusion model, and to adjust the model parameters of the conditional latent space diffusion model through the optimizer. During this training phase, the parameter estimation module 22 is further used to: obtain the second blur direction, second blur length, and corresponding second confidence level of each pixel in the second single-frame training blurred image; the orientation regularization deconvolution module 23 is further used to: perform orientation regularization deconvolution on the second single-frame training blurred image based on the second blur direction, second blur length, and second confidence level to obtain the second coarse restored reference image; the conditional latent space diffusion model 24 is further used to: input the second coarse restored reference image, the second blur direction, and the second blur length into the conditional latent space diffusion model to obtain the predicted restored image; the frequency domain constraint module 25 is further used to: perform frequency domain processing on the predicted restored image and the clear reference image corresponding to the second single-frame training blurred image respectively, and statistically analyze the angular energy curve of the predicted restored image and the angular energy curve of the clear reference image; calculate the difference between the angular energy curve of the predicted restored image and the angular energy curve of the clear reference image as the frequency domain angular consistency loss function.

[0222] The single-frame image motion deblurring device provided by this invention, based on direction awareness and diffusion refinement, constructs a spatially variable motion blur model aligned with the actual degradation process by estimating the blur direction and blur length of each pixel, enabling physically reliable restoration of the structural edges of the coarsely restored image. Furthermore, a conditional latent spatial diffusion model is introduced to supplement high-frequency details, and a data consistency mechanism constrains the generated result within a preset neighborhood of the coarsely restored image, leveraging the advantages of the generation model in texture restoration while effectively preventing content fabrication. Finally, frequency domain angular consistency constraints are used to directionally eliminate residual ringing and stripe artifacts along the main blur direction, significantly improving the visual quality of the image.

[0223] It should be noted that "multiple" in this invention includes two or more.

[0224] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0225] Each module in the devices of this invention can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0226] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data required for or generated by the aforementioned direction-aware and diffusion-refinement-based single-frame image motion deblurring method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a direction-aware and diffusion-refinement-based single-frame image motion deblurring method.

[0227] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a single-frame image motion deblurring method based on direction perception and diffusion refinement. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device casing, or an external keyboard, touchpad, or mouse.

[0228] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0229] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0230] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0231] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0232] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0233] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0234] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0235] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A single-frame image motion deblurring method based on direction awareness and diffusion thinning, characterized in that, include: Acquire a single-frame blurred image and estimate the spatial motion blur parameters of the single-frame blurred image to obtain the blur direction, blur length and corresponding confidence level of each pixel in the single-frame blurred image; Based on the blur direction, blur length and confidence, a direction-regularized deconvolution is performed on a single-frame blurred image to obtain a coarse restored image. The coarse restored image is processed by inputting the blur direction and blur length into a pre-trained conditional latent space diffusion model to obtain a refined restored image. Apply frequency domain angular consistency constraints to the refined and restored image, and output the final restored image.

2. The method according to claim 1, characterized in that, The estimation of spatial motion blur parameters of a single-frame blurred image yields the blur direction, blur length, and corresponding confidence level of each pixel in the single-frame blurred image, including: The RGB channel feature map, grayscale image, horizontal gradient map, vertical gradient map, and Gabor filter response map in multiple preset directions of a single-frame blurred image are concatenated along the channels to obtain the input feature tensor. A multi-scale convolutional neural network is used to extract features from the input feature tensor, resulting in a multi-scale feature map that includes a shallow feature map. Calculate the gradient covariance matrix of each pixel in a single-frame blurred image, and calculate the principal orientation and anisotropy of each pixel as statistical features of the local structure tensor based on the gradient covariance matrix of each pixel. The enhanced feature map is obtained by fusing the statistical features of the local structure tensor with the shallow feature map in the multi-scale feature map. Based on the enhanced feature map, the direction regression head, length regression head, and confidence head are used to output the blur direction, blur length, and corresponding confidence of each pixel in a single frame of blurred image.

3. The method according to claim 1, characterized in that, The process of performing directionally regularized deconvolution on a single-frame blurred image based on blur direction, blur length, and confidence level to obtain a coarse restored image includes: Construct an inverse filter kernel library, wherein the inverse filter kernel library includes multiple inverse filter kernels corresponding to multiple discrete directions and multiple discrete length combinations, and a preset regularization parameter is introduced into each inverse filter kernel; For each pixel, based on the blur direction and blur length, select multiple inverse filter kernels that are adjacent in direction and length from the inverse filter kernel library as a candidate kernel set; A hybrid gating mechanism is adopted to weight and fuse candidate kernels in the candidate kernel set based on confidence level, thereby generating an adaptive inverse filter kernel for the corresponding pixel; The single-frame blurred image is converted from the RGB color space to the luminance-chrominance color space, the luminance channel and the chrominance channel are separated, and the luminance channel is deconvolved by applying an adaptive inverse filter kernel to obtain the deconvolved luminance channel. The chrominance channel is weakened to obtain the weakened chrominance channel. The luminance channel after deconvolution is merged with the chroma channel after weakening, and then converted back to the RGB color space to obtain a coarsely restored image.

4. The method according to claim 1, characterized in that, The process of refining the coarse restored image by inputting the blur direction and blur length into a pre-trained conditional latent space diffusion model to obtain a refined restored image includes: The coarse restored image, along with the blur direction and blur length, is input into a pre-trained conditional latent space diffusion model to generate a temporary decoded image. Based on the data consistency mechanism, the temporary decoded image is constrained within a preset neighborhood of the coarse restored image to obtain the refined restored image.

5. The method according to claim 4, characterized in that, The step of generating a temporary decoded image by inputting the coarsely restored image, blur direction, and blur length into a pre-trained conditional latent space diffusion model includes: The coarse restored image is encoded using a variational autoencoder in a pre-trained conditional latent space diffusion model to obtain the first latent space features; The fuzzy direction and fuzzy length are normalized respectively, and the normalized fuzzy direction and fuzzy length are used as geometric conditions. The first latent space features and geometric conditions are input into the U-Net network in the conditional latent space diffusion model for iterative denoising to obtain the second latent space features. The second latent space feature is input into the variational autoencoder in the conditional latent space diffusion model for decoding to obtain a temporary decoded image.

6. The method according to claim 4 or 5, characterized in that, The step of constraining the temporary decoded image within a preset neighborhood of the coarse restored image according to the data consistency mechanism to obtain the refined restored image includes: The same low-pass filtering is applied to both the temporary decoded image and the coarse restored image to extract the low-frequency components of the temporary decoded image and the coarse restored image. Calculate the first low-frequency difference between the low-frequency components of the temporary decoded image and the low-frequency components of the coarse restored image, and correct the temporary decoded image based on the first low-frequency difference so that the low-frequency components of the corrected temporary decoded image are consistent with the low-frequency components of the coarse restored image within a preset neighborhood, thereby obtaining a refined restored image.

7. The method according to claim 1, characterized in that, The method further includes: The first blur direction, first blur length and corresponding first confidence of each pixel in the first single frame training blurred image are obtained, and the first single frame training blurred image is subjected to directional regularized deconvolution based on the first blur direction, first blur length and first confidence to obtain the first coarse restoration reference image. The first coarse restored reference image, along with the first blur direction and the first blur length, are input into the conditional latent space diffusion model to obtain the predicted image; The same low-pass filtering is applied to the predicted image and the first coarse restoration reference image respectively. The low-frequency components of the predicted image and the first coarse restoration reference image are extracted, and the second low-frequency difference between the low-frequency components of the predicted image and the first coarse restoration reference image is calculated as the low-frequency consistency loss. The global color statistical features of the predicted image and the global color statistical features of the first coarse restoration reference image are calculated respectively, and the difference between the global color statistical features of the predicted image and the global color statistical features of the first coarse restoration reference image is calculated as the color consistency loss. The weighted sum of low-frequency consistency loss and color consistency loss is added as the total penalty term to the training objective function of the conditional latent space diffusion model, and the model parameters of the conditional latent space diffusion model are adjusted by the optimizer.

8. The method according to claim 1, characterized in that, The step of applying frequency domain angular consistency constraints to the refined and restored image and outputting the final restored image includes: The thinned and restored image is subjected to Fast Fourier Transform in a sliding window manner to obtain the spectrum of each window. The spectrum of each window is then mapped from Cartesian coordinates to polar coordinates to obtain the spectrum representation in polar coordinates. The angular energy distribution of each window is statistically analyzed to obtain the angular energy curve of each window, and the angular energy curve of each window is checked for abnormal peaks in the preset direction. If an abnormal spike is detected, a preset amplitude scaling suppression is applied to the frequency band corresponding to that direction within the window to obtain the suppressed spectrum. The suppressed spectrum is subjected to inverse fast Fourier transform to obtain the final restored image.

9. The method according to claim 1 or 8, characterized in that, The method further includes: The second blur direction, second blur length and corresponding second confidence of each pixel in the second single-frame training blurred image are obtained. Based on the second blur direction, second blur length and second confidence, the second single-frame training blurred image is subjected to directional regularized deconvolution to obtain the second coarse restoration reference image. The second coarse restored reference image, along with the second blur direction and the second blur length, are input into the conditional latent space diffusion model to obtain the predicted restored image; Frequency domain processing was performed on the predicted restored image and the sharp reference image corresponding to the second single-frame training blurred image, respectively, and the angular energy curves of the predicted restored image and the sharp reference image were statistically analyzed. The difference between the angular energy curve of the predicted restored image and the angular energy curve of the sharp reference image is calculated as the frequency domain angular consistency loss function; The frequency domain angular consistency loss function is added to the overall training objective function of the conditional latent space diffusion model, and the model parameters of the conditional latent space diffusion model are adjusted by the optimizer.

10. A single-frame image motion deblurring device based on direction perception and diffusion refinement, characterized in that, include: The acquisition module is used to acquire a single-frame blurred image; The parameter estimation module, connected to the acquisition module, is used to estimate the spatial motion blur parameters of a single-frame blurred image, and obtain the blur direction, blur length and corresponding confidence level of each pixel in the single-frame blurred image; The directional regularized deconvolution module, connected to the parameter estimation module, is used to perform directional regularized deconvolution on a single-frame blurred image based on the blur direction, blur length, and confidence level to obtain a coarse restored image. The Conditional Latent Space Diffusion Model, connected to the Direction Regularized Deconvolution Module, is used to refine the coarse restored image by inputting the blur direction and blur length into the pre-trained Conditional Latent Space Diffusion Model, resulting in a refined restored image. The frequency domain constraint module, connected to the conditional latent space diffusion model, is used to apply frequency domain angular consistency constraints to the refined restored image and output the final restored image.

Images