Blurred image enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides a blurred image enhancement method and device, electronic equipment and a storage medium. The method comprises the following steps: extracting a target blurring probability matrix and a blurring type matrix; extracting a high-frequency detail feature map and a low-frequency structure feature map of the original blurred image on the basis of target pixel points of which the existence blurring probability is higher than a preset blurring degree; performing feature fusion on the high-frequency detail feature map and the low-frequency structure feature map to obtain a fused feature map; and inputting the fused feature map and the blurring type matrix into a decoder, and performing up-sampling on the fused feature map by using an up-sampling core generated by the blurring type matrix to obtain an enhanced clear image corresponding to the original blurred image. Through the mode of the invention, the features of the blurred image can be extracted based on the target pixel points determined by the blurring probability matrix, the time consumption of image enhancement is shortened, the up-sampling core generated by the blurring type matrix is utilized to perform up-sampling on the fused feature map, the image noise is prevented from being amplified, and the image enhancement effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a fuzzy image enhancement method and device, electronic equipment and storage medium. BACKGROUND

[0002] The fuzzy image enhancement algorithm is to improve the visual quality and valuable details of the image, and to provide a better foundation for subsequent image processing and analysis tasks. The fuzzy image enhancement algorithm adjusts the gray scale distribution of the image to enhance the contrast between different regions, so that the objects in the image are more clear and distinguishable.

[0003] However, the traditional algorithm often amplifies image noise during image enhancement, resulting in poor enhancement effect and long time consumption. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a fuzzy image enhancement method, device, electronic equipment and storage medium, which can extract the features of the fuzzy image based on the target pixel points determined by the fuzzy probability matrix, shorten the image enhancement time, and use the up-sampling kernel generated by the fuzzy type matrix to up-sample the fusion feature map, avoid amplifying image noise, and improve the image enhancement effect.

[0005] In a first aspect, the present application provides a fuzzy image enhancement method, which comprises: inputting an original fuzzy image into a fuzzy feature extraction module to obtain a target fuzzy probability matrix and a fuzzy type matrix; the target fuzzy probability matrix includes the target probability of each pixel point in the original fuzzy image being fuzzy; the fuzzy type matrix is used to represent the fuzzy type corresponding to each pixel point in the original fuzzy image; inputting the fuzzy probability matrix, the high-resolution image and the low-resolution image corresponding to the original fuzzy image into the sparse attention module of the encoder, to extract a high-frequency detail feature map of the original fuzzy image from the high-resolution image and a low-frequency structure feature map of the original fuzzy image from the low-resolution image based on the target pixel points; the target pixel points are the pixel points higher than the preset blur degree selected from each pixel point; the high-frequency detail feature map represents the edges and textures contained in the original fuzzy image; the low-frequency structure feature map reflects the structure and semantic content of the original fuzzy image; inputting the high-frequency detail feature map and the low-frequency structure feature map into the feature fusion module of the encoder for feature fusion to obtain a fusion feature map; The fused feature map and the blur type matrix are input into the decoder to upsample the fused feature map using the upsampling kernel generated by the blur type matrix, thereby obtaining an enhanced clear image corresponding to the original blurred image.

[0006] In one possible implementation, the step of inputting the original blurred image into the blurred feature extraction module to obtain the target blurred probability matrix and the blurred type matrix includes: The original blurred image is input into a lightweight convolutional neural network to extract feature maps at multiple scales; For a feature map at any scale, predict the initial fuzzy probability matrix and fuzzy type score matrix corresponding to the feature map at that scale; the fuzzy type score matrix includes the score of each pixel corresponding to the target fuzzy type under the feature map at that scale; the target fuzzy type corresponding to the pixel refers to the fuzzy type that the pixel obtains the highest score among all preset fuzzy types under the feature map at that scale. The target fuzzy probability matrix is ​​obtained by weighting and fusing all initial fuzzy probability matrices based on the weights corresponding to each fuzzy type. The fuzzy type matrix is ​​obtained by weighting and fusing the score matrices of all fuzzy types based on the weights corresponding to each fuzzy type.

[0007] In one possible implementation, the step of weighted fusion of all fuzzy type score matrices based on the weights corresponding to each fuzzy type to obtain the fuzzy type matrix includes: Substituting the weights corresponding to each fuzzy type and the score matrix of all fuzzy types into the following formula, the fuzzy type matrix is ​​obtained; ; in, The fuzz type value is any pixel in the fuzz type matrix. The number of scales. Let be the weight of the target blur type for any pixel in the feature map at the i-th scale. Let be the score of the target blur type corresponding to any pixel point in the feature map at the i-th scale.

[0008] In one possible implementation, the step of inputting the blur probability matrix, the high-resolution image corresponding to the original blurred image, and the low-resolution image into the sparse attention module of the encoder to extract high-frequency detail feature maps of the original blurred image from the high-resolution image based on target pixels, and extracting low-frequency structural feature maps of the original blurred image from the low-resolution image, includes: The feature values ​​corresponding to the target pixel in the fuzzy probability matrix are set to 1, and the feature values ​​corresponding to other pixels are set to 0 to obtain the global sparse mask matrix. For each pixel, the feature values ​​of all pixels outside the preset window corresponding to the pixel in the global sparse mask matrix are set to 0 to obtain a local sparse mask matrix; based on the local sparse mask matrix, the feature values ​​of the pixels in the high-frequency detail feature map are extracted from the high-resolution image. Based on the global sparse mask matrix, feature values ​​of each pixel in the low-frequency structural feature map are extracted from the low-resolution image.

[0009] In one possible implementation, the upsampling kernel is generated using the fuzzy type matrix through the following steps: Based on the fuzzy type matrix, the high-frequency detail feature map and the low-frequency structural feature map are weighted and fused to obtain a weighted fused feature map; Based on the weighted fusion feature map, the target upsampling kernel is obtained.

[0010] In one possible implementation, obtaining the target upsampling kernel based on the weighted fusion feature map includes: Perform a convolution operation on the weighted fused feature map to obtain an initial upsampling kernel; The blur probability matrix and the original blurred image are input into the sparse attention module to obtain the sparse attention weight matrix; the sparse attention weight matrix is ​​used to characterize the contribution of each pixel to detail recovery and sharpness improvement in the process of enhancing the original blurred image; The initial upsampling kernel is weighted and corrected using the sparse attention weight matrix to obtain the target upsampling kernel.

[0011] In one possible implementation, the image enhancement model includes the blur feature extraction module, the encoder, and the decoder; the loss function of the image enhancement model includes pixel-level differences, edge information differences, and visual feature differences between the blurred image sample and the enhanced clear image corresponding to the blurred image sample output by the image enhancement model. The pixel-level differences reflect the degree of closeness between the blurred image sample and the corresponding enhanced sharp image in terms of detail recovery; the edge information differences reflect the differences between the outlines and shapes of objects contained in the blurred image sample and the outlines and shapes of objects contained in the corresponding enhanced sharp image; the visual feature differences reflect the differences between the semantic information and visual quality of the blurred image sample and the semantic information and visual quality of the corresponding enhanced sharp image.

[0012] Secondly, embodiments of this application also provide an image enhancement device, the device comprising: An extraction module is used to input the original blurred image into a blurred feature extraction module to obtain a target blurred probability matrix and a blurred type matrix; the target blurred probability matrix includes the target probability that each pixel in the original blurred image is blurred; the blurred type matrix is ​​used to characterize the blurred type corresponding to each pixel in the original blurred image. The extraction module is further configured to input the blur probability matrix, the high-resolution image corresponding to the original blurred image, and the low-resolution image into the sparse attention module of the encoder, so as to extract the high-frequency detail feature map of the original blurred image from the high-resolution image based on the target pixel, and extract the low-frequency structural feature map of the original blurred image from the low-resolution image; the target pixel is a pixel selected from each pixel that has a higher blur level than a preset blur level; the high-frequency detail feature map represents the edges and textures contained in the original blurred image; the low-frequency structural feature map reflects the structure and semantic content of the original blurred image; The feature fusion module is used to input the high-frequency detail feature map and the low-frequency structural feature map into the feature fusion module of the encoder for feature fusion to obtain a fused feature map; The decoding module is used to input the fused feature map and the blur type matrix into the decoder, so as to upsample the fused feature map using the upsampling kernel generated by the blur type matrix, and obtain the enhanced clear image corresponding to the original blurred image.

[0013] In one possible implementation, the extraction module is specifically used to input the original blurred image into a lightweight convolutional neural network to extract feature maps at multiple scales; for any feature map at any scale, predict the initial blur probability matrix and blur type score matrix corresponding to the feature map at that scale; the blur type score matrix includes the score of each pixel corresponding to the target blur type under the feature map at that scale; the target blur type corresponding to the pixel refers to the blur type that the pixel obtains the highest score among all preset blur types under the feature map at that scale; weightedly fuse all initial blur probability matrices based on the weights corresponding to each blur type to obtain the target blur probability matrix; weightedly fuse all blur type score matrices based on the weights corresponding to each blur type to obtain the blur type matrix.

[0014] In one possible implementation, the extraction module is further configured to: Substituting the weights corresponding to each fuzzy type and the score matrix of all fuzzy types into the following formula, the fuzzy type matrix is ​​obtained; ; in, The fuzz type value is any pixel in the fuzz type matrix. The number of scales. Let be the weight of the target blur type for any pixel in the feature map at the i-th scale. Let be the score of the target blur type corresponding to any pixel point in the feature map at the i-th scale.

[0015] In one possible implementation, the extraction module is specifically configured to set the feature value corresponding to the target pixel in the blur probability matrix to 1, and set the feature values ​​corresponding to other pixels other than the target pixel to 0, to obtain a global sparse mask matrix; for each pixel, set the feature values ​​corresponding to all pixels outside the preset window corresponding to the pixel in the global sparse mask matrix to 0, to obtain a local sparse mask matrix; extract the feature values ​​of the pixels in the high-frequency detail feature map from the high-resolution image based on the local sparse mask matrix; and extract the feature values ​​of each pixel in the low-frequency structural feature map from the low-resolution image based on the global sparse mask matrix.

[0016] In one possible implementation, the decoding module is further configured to generate an upsampling kernel using the fuzzy type matrix through the following steps: Based on the fuzzy type matrix, the high-frequency detail feature map and the low-frequency structural feature map are weighted and fused to obtain a weighted fused feature map; Based on the weighted fusion feature map, the target upsampling kernel is obtained.

[0017] In one possible implementation, the decoding module is further configured to: Perform a convolution operation on the weighted fused feature map to obtain an initial upsampling kernel; The blur probability matrix and the original blurred image are input into the sparse attention module to obtain the sparse attention weight matrix; the sparse attention weight matrix is ​​used to characterize the contribution of each pixel to detail recovery and sharpness improvement in the process of enhancing the original blurred image; The initial upsampling kernel is weighted and corrected using the sparse attention weight matrix to obtain the target upsampling kernel.

[0018] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for enhancing a blurred image as described in any of the first aspects.

[0019] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method for enhancing a blurred image as described in any of the first aspects.

[0020] This application provides a method, apparatus, electronic device, and storage medium for enhancing blurred images. The method includes: extracting a target blur probability matrix and a blur type matrix; extracting a high-frequency detail feature map of the original blurred image from a high-resolution image based on target pixels, and extracting a low-frequency structural feature map of the original blurred image from a low-resolution image; the target pixels are pixels selected from all pixels with a blur level higher than a preset blur degree; fusing the high-frequency detail feature map and the low-frequency structural feature map to obtain a fused feature map; and inputting the fused feature map and the blur type matrix into a decoder to upsample the fused feature map using an upsampling kernel generated by the blur type matrix, thereby obtaining an enhanced clear image corresponding to the original blurred image. This method allows for the extraction of features from blurred images based on target pixels determined by the blur probability matrix, shortening image enhancement time, and using an upsampling kernel generated by the blur type matrix to upsample the fused feature map, avoiding amplification of image noise and improving image enhancement effect. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 The network structure diagram of the image enhancement model provided in the embodiments of this application is shown; Figure 2 A flowchart illustrating a method for enhancing a blurred image according to an embodiment of this application is shown; Figure 3 A schematic diagram of the structure of a blurred image enhancement device provided in an embodiment of this application is shown; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0025] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0026] To enable those skilled in the art to utilize the content of this application, and in conjunction with the specific application scenario of "image processing technology," the following embodiments are provided. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is primarily described within the "image processing technology field," it should be understood that this is merely an exemplary embodiment.

[0027] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0028] Reference Figure 1 The diagram shown is a network structure diagram of the image enhancement model provided in this application embodiment. The image enhancement method for blurred images provided in this application embodiment is implemented by applying this image enhancement model.

[0029] The following will combine Figure 1 The image enhancement model shown illustrates a method for enhancing a blurred image provided in this application embodiment.

[0030] Reference Figure 2The diagram shown is a flowchart illustrating a method for enhancing a blurred image according to an embodiment of this application. The exemplary steps of this embodiment are described below: S201. Input the original blurred image into the blurred feature extraction module to obtain the target blurred probability matrix and blurred type matrix.

[0031] In this embodiment, the original blurred image refers to an image with low contrast due to poor shooting conditions, uneven lighting, or other reasons. The blur feature extraction module is used to extract blur features from the original blurred image. Blur features refer to the probability of blurring and the blur type (such as motion blur, out-of-focus blur, Gaussian blur, etc.) in the original blurred image. Blur features include a target blur probability matrix and a blur type matrix. The target blur probability matrix includes the target probability of blurring for each pixel in the original blurred image, used to characterize the degree of blur; the blur type matrix is ​​used to characterize the blur type corresponding to each pixel in the original blurred image. The blur type corresponding to each pixel is determined based on the score of the target blur type for each pixel under each scale feature map corresponding to the original blurred image. The target blur type corresponding to a pixel under any scale feature map refers to the blur type that the pixel obtains the highest score among all preset blur types under that scale feature map.

[0032] Here, the original blurred image, the target blurred probability matrix, and the blurred type matrix are all the same size, which means that they have the same number of rows and columns in the spatial dimension, thus ensuring that each pixel position corresponds one-to-one in the three matrices during processing.

[0033] For example, if the original blurred image is 640×480 pixels, then the target blur probability matrix and blur type matrix are also 640×480 pixels in size.

[0034] Specifically, the original blurred image is input into the blur feature extraction module to obtain the target blur probability matrix and blur type matrix, including: Step 1: Input the original blurred image into a lightweight convolutional neural network (such as a shallow module of MobileNetV3) to extract feature maps at multiple scales.

[0035] For example, the original blurred image is input into a lightweight convolutional neural network (such as a shallow module of MobileNetV3) to extract feature maps {S1, S2, S3} at three scales.

[0036] Step 2: For a feature map at any scale, predict the initial fuzzy probability matrix and fuzzy type score matrix corresponding to that scale.

[0037] In this embodiment, while predicting the initial blur probability matrix corresponding to the feature map at that scale based on the Sigmoid activation function, a blur type score matrix corresponding to the feature map at that scale is predicted based on the Softmax output. The initial blur probability matrix corresponding to the feature map at any scale includes the initial probability that each pixel in the original blurred image is blurred at that scale. The blur type score matrix corresponding to the feature map at any scale includes the score of each pixel corresponding to the target blur type at that scale; the target blur type corresponding to a pixel refers to the blur type that the pixel obtains the highest score among all preset blur types at that scale.

[0038] Step 3: Based on the weights corresponding to each fuzzy type, perform weighted fusion of all initial fuzzy probability matrices to obtain the target fuzzy probability matrix.

[0039] In this embodiment of the application, the weights corresponding to each fuzzy type and all initial fuzzy probability matrices are substituted into the following formula to obtain the target fuzzy probability matrix; ; in, This represents the probability that any pixel in the fuzziness type matrix contains a blurred target. For the number of scales, Let be the weight of the target blur type for any pixel in the feature map at the i-th scale. Let be the initial probability that any pixel in the feature map at the i-th scale is blurred.

[0040] Step 4: Based on the weights corresponding to each fuzzy type, perform weighted fusion of all initial fuzzy probability matrices and fuzzy type score matrices to obtain the fuzzy type matrix.

[0041] In this embodiment of the application, the weights corresponding to each fuzzy type, all initial fuzzy probability matrices, and fuzzy type score matrices are substituted into the following formula to obtain the fuzzy type matrix; ; in, The fuzz type value is any pixel in the fuzz type matrix. For the number of scales, Let be the weight of the target blur type for any pixel in the feature map at the i-th scale. Let be the score of the target blur type corresponding to any pixel point in the feature map at the i-th scale.

[0042] Here, in the field of blurred image enhancement, the method for determining the target blur probability matrix and blur type matrix proposed in this application has significant advantages: (1) Accurately identify blur types: By employing multi-scale feature extraction and classification techniques, it is possible to more accurately identify various blur types in the image, such as motion blur, defocus blur, or Gaussian blur. This step is crucial for subsequent blurred image enhancement processing because different blur types require different processing strategies. (2) Enhance blur detection capability: The target blur probability matrix provides probability information about the presence of blur for each pixel in the image. This information helps to identify the distribution and degree of blurred regions in the image, thereby guiding the enhancement algorithm to process these regions more effectively. (3) Introduce lightweight CNN for fast analysis: In the blurred image input processing stage, a lightweight convolutional neural network (CNN) is introduced to predict the blur probability map. This lightweight CNN can quickly and efficiently analyze the original blurred image, identify blurred regions, and generate the corresponding target blur probability matrix. This allows the model to intuitively perceive the differences in blur degree in different regions of the original blurred image, providing an important basis for subsequent enhancement processing. In summary, the embodiments of this application provide an efficient and accurate solution for the field of blurred image enhancement by accurately identifying blur types, enhancing blur detection capabilities, and introducing a lightweight CNN for rapid analysis.

[0043] S202. Input the blur probability matrix, the high-resolution image and the low-resolution image corresponding to the original blurred image into the sparse attention module of the encoder, so as to extract the high-frequency detail feature map of the original blurred image from the high-resolution image based on the target pixel, and extract the low-frequency structural feature map of the original blurred image from the low-resolution image.

[0044] In this embodiment, the high-frequency detail feature map represents the edges and textures contained in the original blurred image, referring to the subtle changes and local features contained in the original blurred image. High-frequency detail features can significantly improve the clarity and sharpness of the original blurred image and are key elements in image enhancement and detail restoration. The low-frequency structure feature map reflects the structure and semantic content of the original blurred image, referring to the overall shape, large-scale structure, and global layout features contained in the image, which plays an important role in maintaining the overall consistency, semantic integrity, and visual effect of the image.

[0045] Here, a higher probability of a pixel being blurred indicates a greater degree of blurriness, making it more important for image enhancement. Target pixels are selected from all pixels based on their higher blurriness level than a preset threshold. Therefore, the blur probability matrix is ​​transformed into a signal that guides the sparse attention module, enabling it to focus more precisely on blurred regions (i.e., target pixels) when processing images. The sparse attention module dynamically allocates computational resources based on the degree of blurriness in the blurred regions, significantly optimizing the efficiency of image enhancement tasks. This allows the model to efficiently enhance blurred images with limited computational resources, improving the overall quality of the original blurred image.

[0046] Specifically, the blur probability matrix, the high-resolution image corresponding to the original blurred image, and the low-resolution image are input into the sparse attention module of the encoder to extract high-frequency detail feature maps of the original blurred image from the high-resolution image based on target pixels, and to extract low-frequency structural feature maps of the original blurred image from the low-resolution image, including: Step 1: Set the eigenvalues ​​corresponding to the target pixel in the fuzzy probability matrix to 1, and set the eigenvalues ​​corresponding to other pixels to 0 to obtain the global sparse mask matrix.

[0047] In this embodiment, by setting the feature value corresponding to the target pixel to 1 and the feature value corresponding to other pixels to 0, intelligent judgment of blurred regions in the original blurred image can be achieved. This allows the sparse attention module to prioritize focusing on areas with more severe blurring during subsequent processing, allocating more computing resources to these blurred regions, thereby more efficiently restoring details and greatly improving the enhancement effect on blurred regions.

[0048] Step 2: For each pixel, set the feature values ​​of all pixels outside the preset window corresponding to the pixel in the global sparse mask matrix to 0 to obtain the local sparse mask matrix; extract the feature values ​​of pixels in the high-frequency detail feature map from the high-resolution image based on the local sparse mask matrix.

[0049] In the embodiments of this application, when extracting high-frequency detail features of any pixel in the shallow structure of the sparse attention module, only other pixels with a feature value of 1 in the local sparse mask matrix corresponding to the pixel are considered (i.e., all pixels within the preset window corresponding to the pixel), which can accurately restore high-frequency details.

[0050] Step 3: Extract the feature values ​​of each pixel in the low-frequency structural feature map from the low-resolution image based on the global sparse mask matrix.

[0051] In this embodiment, for each pixel, a key pixel is selected from the global sparse mask matrix starting from that pixel and following a preset step size (represented by the number of pixels). The values ​​of the key pixels and all other pixels in the global sparse mask matrix are set to 0, resulting in the global sparse mask matrix for that pixel. When extracting low-frequency detail features of any pixel in the deep structure of the sparse attention module, it is necessary to focus on all pixels with a feature value of 1 in the global sparse mask matrix, which can reconstruct the low-frequency structural features of the original blurred image.

[0052] Here, in traditional attention mechanisms, for a blurry image with n pixels, the computation process requires calculating the similarity between the query vector of each pixel and the key vectors of all pixels, thereby generating n attention weights. Finally, a weighted summation operation is performed on the value vector of each pixel based on these weights. This process has a computational complexity of up to [missing information]. This application limits the number of other elements that each element needs to focus on based on the probability of blurring at each pixel, allowing the sparse attention module to focus on pixels with high blurring levels and avoid indiscriminate comprehensive calculations for all pixels. The embodiments of this application limit the similarity calculation of the query vector of each pixel to only with key vectors with high blurring probabilities, thereby significantly reducing the computational load. This approach greatly reduces computational complexity and memory requirements without significantly affecting model performance, providing a more efficient and economical solution for blurred image enhancement tasks. Furthermore, the sparse attention module in the embodiments of this application alternately uses local window attention (i.e., using a local sparse mask matrix to obtain pixel attention weights) and strided sparse attention (i.e., using a global sparse mask matrix to obtain pixel attention weights) when extracting high-frequency detail features and low-frequency structural features. Local window attention focuses on local regions of the image, effectively capturing high-frequency detail features; strided sparse attention, through a specific striding strategy, filters key information over a wider range, effectively capturing low-frequency structural features. The combination of the two improves the model's ability to focus on and process image information while ensuring computational efficiency.

[0053] Specifically, this application extracts feature values ​​of pixels in high-frequency detail feature maps from high-resolution images based on a local sparse mask matrix, or extracts feature values ​​of each pixel in low-frequency structural feature maps from low-resolution images based on a global sparse mask matrix, through the following steps: i. Mapping high-resolution or low-resolution images to corresponding embedding vectors: Taking the high-resolution or low-resolution image as input, a linear transformation is used to map each pixel to a query vector Q, a key vector K, and a value vector V. These vectors are used to represent the features of each pixel in the image and prepare for capturing the correlation information between pixels.

[0054] ii. For each pixel, calculate the similarity between the query vector Q and the key vector K. This is achieved through a dot product operation and scaled by the square root of the key vector dimension d to obtain the similarity score for that pixel. Then, apply a local sparse mask matrix or a global sparse mask matrix M to control the model to focus only on blurred regions, thereby calculating sparse attention. The specific formula is as follows: :in, It can be a local sparse mask matrix or a global sparse mask matrix.

[0055] iii. Attention Weight Normalization: The adjusted similarity scores are normalized using the softmax activation function to obtain the attention weight for each pixel. These weights reflect the importance of each pixel's value vector in the final output. The specific formula is as follows: ; in, The attention weights for each pixel.

[0056] iv. For each pixel, the pixel's value vector V is weighted using the pixel's attention weight to obtain the pixel's feature value in either the high-frequency detail feature map or the low-frequency structural feature map. This step integrates high-frequency detail features and low-frequency structural features, providing accurate and effective feature representations for image enhancement tasks.

[0057] Here, in the field of blurred image enhancement, by utilizing local and global sparse mask matrices generated from target pixels with a blur level higher than a preset threshold, high-frequency detail features and low-frequency structural features of the original blurred image are extracted. This demonstrates significant advantages in multiple dimensions, including computational efficiency, feature capture, information filtering, and model generalization ability, enabling more efficient image enhancement tasks under limited memory resources. The key roles of local and global sparse mask matrices in blurred image enhancement are as follows: 1. Enhanced targeting, accurately focusing on pixels with high blur levels: Using local and global sparse mask matrices ensures that local contextual information is more effectively integrated into the attention mechanism, accurately preserving areas with high blur levels while effectively filtering out information with low blur levels. In blurred image enhancement tasks, this means the model can focus more on image information that plays a crucial role in improving image clarity. 2. Improved efficiency, reduced computational burden: Traditional attention mechanisms, due to a lack of effective pixel filtering, often extract a large amount of irrelevant image information when processing images. This not only significantly increases computational load but also prolongs processing time. By utilizing local and global sparse mask matrices, the full attention weight matrix can be dynamically pruned, transforming it into a fine-grained structured sparse pattern. This transformation significantly reduces unnecessary computation and greatly improves the speed of image enhancement. 3. Enhanced Anomalous Data Handling Capabilities: Blurry images often contain anomalous data such as noise and missing data, which severely impact the quality of image enhancement. Local and global sparse mask matrices possess powerful information filtering capabilities, effectively filtering out these anomalous data and significantly improving the quality of image enhancement, preventing poor enhancement results due to amplified image noise during the enhancement process. Therefore, in blurry image enhancement tasks, utilizing local and global sparse mask matrices not only significantly improves the targeting and efficiency of image enhancement but also better addresses the challenges posed by anomalous data, providing innovative ideas and methods for the field of blurry image enhancement.

[0058] In addition, the high-resolution and low-resolution images corresponding to the original blurred image are obtained based on the Visual Transformer (ViT). In the field of blurred image enhancement, the organic integration of the sparse attention module and the Visual Transformer (ViT) has many significant advantages: (1) From the perspective of computational efficiency and high resolution adaptation, the global self-attention mechanism adopted by the traditional Transformer has high computational complexity and extremely high computational cost. In the embodiment of this application, the sparse attention module focuses on the region with more severe blurring and allocates more computational resources to these blurred regions, which can effectively reduce computational complexity and significantly improve inference speed, thereby achieving a good match between computational efficiency and high-resolution image processing. In terms of feature fusion, multi-scale feature fusion can be achieved by fusing the two. Under different resolution images, the model can give full play to its respective advantages, capture the global low-frequency structural information of the image under low resolution, and focus on the local high-frequency detail information under high resolution. This effective capture and integration of multi-scale features greatly improves the model's processing efficiency for blurred images, enabling the model to understand and process blurred images more comprehensively and deeply.

[0059] Optionally, before the sparse attention module in the encoder, the encoder may also include a normalization module, an image segmentation module, and a positional encoding module. Specifically, 1. Input of the original blurred image and normalization processing: The original blurred image is input into the normalization module of the encoder to perform a normalization operation on the original blurred image, ensuring effective processing and consistency of the original blurred image in the model. Normalization, through a specific mathematical transformation, maps the pixel values ​​of the original blurred image to a uniform range, eliminating the adverse effects that may be caused by the scale differences of the image data itself, laying the foundation for subsequent processing. The high-resolution and low-resolution images corresponding to the normalized original blurred image are also obtained. 2. Image segmentation and embedding: The normalized original blurred image, high-resolution image, and low-resolution image are segmented into fixed-size image blocks using an overlapping segmentation method with convolutional layers. This overlapping segmentation strategy can effectively enhance local continuity, making information overlap between adjacent image blocks, helping the model to better capture the feature correlations of local image regions, and providing richer and more continuous local information for subsequent feature extraction and processing. 3. Introduction of Positional Encoding: To further enhance the sparse attention module's ability to model spatial relationships in images, learnable relative positional encoding is introduced. Relative positional encoding provides the model with relative positional information between different image blocks, enabling the model to not only focus on the features of individual pixels but also effectively utilize their spatial relationships, thus more accurately modeling and analyzing information in the image. 4. Inputting the positionally encoded image blocks into the sparse attention module.

[0060] S203. Input the high-frequency detail feature map and the low-frequency structural feature map into the feature fusion module of the encoder for feature fusion to obtain the fused feature map.

[0061] In this application's embodiments, high-frequency detail features and low-frequency structural features are two important features in image processing, used to describe local details and global structure in an image, respectively. High-frequency detail features enhance the sharpness and detail of a clear image by capturing local changes such as edges and textures; low-frequency structural features maintain global consistency and semantic integrity by capturing the overall shape and large-scale structure of the image. This application provides two methods for fusing high-frequency detail feature maps and low-frequency structural feature maps.

[0062] The first fusion method involves upsampling and downsampling to extract high-frequency detail feature maps from high-resolution images and low-frequency structural feature maps from low-resolution images, then concatenating them. Upsampling expands low-frequency structural feature maps to higher resolution, while downsampling does the opposite. This method integrates high-frequency detail features and low-frequency structural features, achieving multi-scale feature fusion. This provides comprehensive and rich feature support for blurred image enhancement, improves the image enhancement model's understanding of image content, and thus enhances the image enhancement effect. The specific formula is as follows: ; in, To fuse feature maps; This is a low-frequency structural feature map. This is the original scale feature map corresponding to the original blurred image. This is a high-frequency detail feature map. For upsampling operation, For downsampling operation, This is a channel-level splicing operation. This is a function used for feature fusion (such as convolution or MLP).

[0063] The second fusion method involves the interaction of global attention from low-frequency structural feature maps and local attention from high-frequency detail feature maps through a cross-attention module. The specific formula is as follows: ; in, This is a cross-attention mechanism.

[0064] Here, the second fusion method allows the low-frequency structural feature map to focus on the subtle textures and edges in the high-frequency detail feature map, improving the visual clarity of the image. Simultaneously considering both the global structure and local details of the image significantly enhances the enhancement effect, resulting in a more natural and realistic output image. Selectively focusing on key information in the high-frequency feature map reduces unnecessary computation and improves the model's computational efficiency, which is particularly important for processing images with varying degrees of blur and feature distributions.

[0065] In addition to the normalization processing module, image segmentation module, position encoding module, and sparse attention module in the first network, the encoder also includes a second network consisting of nine Resblock residual convolutional network blocks connected sequentially. Resblock is the residual block. The introduction of residual blocks is significant in the visual Transformer-based blurred image enhancement system. On the one hand, the addition of residual blocks optimizes the network structure, removing some unnecessary components from the visual Transformer, thus simplifying the overall structure and significantly improving computational efficiency. On the other hand, Resblock possesses powerful feature extraction capabilities, enabling the visual Transformer to more accurately capture key information in the image. Especially when dealing with complex images, this capability can greatly improve the visual Transformer's understanding and representation of images, thereby enhancing the effect and quality of blurred image enhancement.

[0066] If the encoder includes a first network and a second network, the feature fusion module in the encoder is also used to fuse the fused feature map obtained by fusing the high-frequency detail feature map and the low-frequency structural feature map with the feature map output by the second network to obtain the final fused feature map.

[0067] S204. Input the fused feature map and the blur type matrix into the decoder, and use the upsampling kernel generated by the blur type matrix to upsample the fused feature map to obtain the enhanced clear image corresponding to the original blurry image.

[0068] In this application embodiment, the decoder is used to output an enhanced clear image with the same resolution as the original blurred image. Compared with the traditional fixed interpolation or transposed convolution method, this application introduces a "blur type matrix" in the upsampling stage to dynamically generate a spatially variable upsampling kernel for the fusion feature map, which has the following two major advantages: (1) Region-adaptive kernel generation. The blur type matrix first determines the blur category of the original blurred image pixel by pixel, and then generates an upsampling kernel that matches it in real time. For example, in motion blur region: the kernel shape is anisotropic along the motion direction, which can enhance edge direction information; in defocus blur region: the kernel is isotropic and the transition is smooth, that is, adaptive smoothing or sharpening is performed according to the blur type of each region. This avoids the introduction of chessboard artifacts and effectively reduces artificial traces; in Gaussian blur region: the kernel weights are concentrated and regularly distributed, which effectively suppresses noise diffusion. Thus, each pixel can obtain a tailor-made upsampling operator, which significantly improves the restoration accuracy. (2) Adaptive recombination strategy. During resolution enhancement, since the blur type matrix is ​​determined based on the score of the target blur type for each pixel under the feature maps of each scale corresponding to the original blurred image, when upsampling the fused feature map using the upsampling kernel generated by the blur type matrix, the feature recombination method can be automatically adjusted according to the blur degree of each region. This allows for the keen capture of high-frequency details such as edges and textures in the image, with priority given to enhancing blurred regions: in blurred regions, high-frequency detail recovery is strengthened to compensate for the structural information lost due to blurring; in clear regions, the original features are kept stable to prevent over-sharpening or distortion. This strategy achieves differentiated processing of "de-blurring blurred regions and preserving clarity in clear regions" globally, ultimately outputting a high-resolution image with rich and natural details, achieving accurate recovery and optimization of high-frequency details, and significantly improving the ability to recover high-frequency details.

[0069] Specifically, the upsampling kernel is generated using the fuzzy type matrix through the following steps: Step 1: Based on the fuzzy type matrix High-frequency detail feature map and low-frequency structural feature map Perform weighted fusion to obtain a weighted fusion feature map. The calculation formula is as follows: .

[0070] Step 2: Based on the weighted fusion feature map, obtain the target upsampling kernel.

[0071] i. Perform a convolution operation on the weighted fused feature map to obtain the initial upsampling kernel.

[0072] ii. Input the blur probability matrix and the original blurred image into the sparse attention module to obtain the sparse attention weight matrix; the sparse attention weight matrix is ​​used to characterize the contribution of each pixel to detail recovery and sharpness improvement in the process of enhancing the original blurred image.

[0073] In this embodiment, the sparse attention weight matrix is ​​determined using the previously determined global sparse mask matrix.

[0074] iii. Utilizing sparse attention weight matrix For the initial upsampling kernel After weighted correction, the target upsampling kernel is obtained. The specific formula is as follows: .

[0075] Here, by combining sparse attention with multi-scale features (high-frequency detail feature maps and low-frequency structural feature maps) in the upsampling kernel calculation, a complementary "position-scale-content" closed loop can be formed in image enhancement tasks. The synergistic advantages are reflected in the following four points: **Fine Structure Restoration:** The upsampling kernel can rearrange features at the sub-pixel level, directly restoring fine edges during the upsampling process; sparse attention only triggers calculations on high-uncertainty tokens (such as edges and textures), avoiding the smoothing effect of global attention and further sharpening details; multi-scale fusion provides a source of information where low-frequency contours and high-frequency textures coexist, providing sufficient original cues for both. **Computationally Efficient and Scalable:** The generation of the upsampling kernel and the token sparsification of sparse attention jointly reduce FLOPs, enabling real-time high-resolution enhancement even on the edge; multi-scale feature fusion can reduce redundant calculations by sharing backbone features. **Region Adaptive Enhancement:** Sparse attention dynamically determines which regions need secondary refinement based on residual errors; multi-scale fusion provides corresponding receptive fields for different blur types (motion / defocus / Gaussian), making kernel generation more targeted. Artifact suppression: Multi-scale fusion first provides a consistent context, reducing the probability of structural artifacts generated by upsampling in extremely blurred regions; sparse attention then performs secondary verification and correction on potential artifact regions, achieving a closed-loop control of "enhancement-detection-re-enhancement". In summary, the synergy of these three methods can significantly improve image edge sharpness, texture realism, and overall visual naturalness while maintaining low computational cost, making it suitable for joint enhancement tasks such as super-resolution, deblurring, and denoising.

[0076] Furthermore, the decoder is composed of an upsampling module, a multi-scale skip connection fusion module, a residual feature refinement block module, and a repeated upsampling and fusion module working together. Throughout the image enhancement process, the decoder's functions also encompass the following important aspects: 1. Precise feature mapping to pixels: The decoder transforms the fused feature map extracted by the encoder into a pixel-level image. This transformation process is gradually advanced using the upsampling technology provided in this embodiment and the residual refinement mechanism of the residual feature refinement block module. The upsampling module amplifies features through flexible parameter adjustments; the residual feature refinement optimizes the details of the upsampled result. Together, they gradually restore spatial details, concretizing abstract features into pixel representations of the image. 2. Precise reconstruction of high-frequency details: Through a multi-scale fusion strategy, skip connections are used to introduce shallow detail features from the encoder. This cross-layer information transfer method effectively avoids the loss of high-frequency detail features during processing, ensuring the complete preservation of the image's fine structure. 3. Effective suppression of artifacts: The residual feature refinement block module in the decoder corrects these potential artifacts through local convolution operations. Furthermore, a normalization layer is used to standardize features, maintain feature stability, prevent overfitting during model training, and ensure the accuracy and reliability of image enhancement results. 4. Strong guarantee of multi-scale consistency: The decoder adopts a progressive upsampling strategy, gradually restoring image resolution in stages. In this process, attention is paid to maintaining the consistency of local and global structures, ensuring that each part of the image maintains structural coordination and unity while the resolution is improved. At the same time, through a cross-layer feature complementarity mechanism, encoder features of different scales are fully integrated to achieve a reasonable balance between high-frequency and low-frequency information, making the enhanced image rich in detail and with a more natural and harmonious overall structure. 5. Multi-scale adaptability and optimized computational efficiency: Upsampling can automatically adjust the size of the upsampling kernel according to the resolution of the input image. While ensuring computational efficiency close to bilinear interpolation, it can achieve the restoration effect of transposed convolution. This multi-scale adaptability allows upsampling to be performed efficiently on images of different resolutions, optimizing the utilization of computational resources. 6. Synergy with sparse attention mechanism: There is a good synergistic effect between upsampling and sparse attention mechanism. In the ViT encoder, the sparse attention mechanism achieves precise focusing on blurred regions; while in the decoder, upsampling further refines these blurred regions by applying a finer upsampling kernel. This global-local collaborative optimization strategy significantly increases the priority of blurred regions in the image restoration process, resulting in a further improvement in the enhancement effect of blurred images.

[0077] Furthermore, the image enhancement model includes a blur feature extraction module, an encoder, and a decoder. The loss function of the image enhancement model includes pixel-level differences, edge information differences, and visual feature differences between the real sharp image corresponding to the blurry image sample and the enhanced sharp image corresponding to the blurry image sample output by the image enhancement model. The pixel-level differences reflect the degree of closeness between the blurry image sample and the corresponding enhanced sharp image in terms of detail recovery. The edge information differences reflect the differences between the outline and shape of the object contained in the blurry image sample and the outline and shape of the object contained in the corresponding enhanced sharp image. The visual feature differences reflect the differences between the semantic information and visual quality of the blurry image sample and the semantic information and visual quality of the corresponding enhanced sharp image.

[0078] Here, this application employs a comprehensive loss function strategy to reduce the distribution gap between the generated reconstructed image and the original image. Specifically, this loss function consists of Charbonnier L1 loss (pixel-level difference) and edge-aware loss. (Differences in edge information) and perceptual loss (Visual feature differences) collectively constitute the problem. Charbonnier L1 loss is a commonly used loss function in image processing and computer vision. It optimizes the traditional L1 loss (i.e., absolute value loss), significantly improving its sensitivity to small errors, thus more accurately measuring the difference between the real sharp image corresponding to the blurred image sample and the enhanced sharp image corresponding to the blurred image sample output by the image enhancement model. Edge-aware loss focuses on special processing of the image edge, enhancing edge information to improve the overall quality and detail of the image, making the edges of the enhanced image clearer and more realistic. Perceptual loss, on the other hand, takes the perspective of the human perception system, comparing the perceptual differences between the real sharp image corresponding to the blurred image sample and the enhanced sharp image corresponding to the blurred image sample output by the image enhancement model, guiding the model to optimize and generate images that are more consistent with human visual perception.

[0079] Given that this application employs a supervised training scheme, during training, the input blurred image samples must be strictly paired with their corresponding real sharp images. This pairing method ensures that the model can learn the accurate mapping relationship from blurred images to real sharp images, and thus, by minimizing the aforementioned comprehensive loss function, effectively enhances the blurred images, generating sharp images that are closer to the distribution of real images.

[0080] In summary, the loss function of the model is: ; ; ; ; in, These are weighting coefficients and need to be adjusted experimentally. It is the error term between the enhanced clear image and the real clear image corresponding to the blurred image sample output by the image enhancement model. It is a small positive number used to prevent the denominator from being zero. The output features of the l-th layer of the model, For gradient.

[0081] The Charbonnier L1 loss employs a square root operation, imposing a more significant penalty when the error term approaches zero. Compared to the conventional L1 loss, this design greatly enhances the robustness of the loss function. In many tasks with stringent image quality requirements, this loss function exhibits unique advantages: it maintains high sensitivity when handling small errors, accurately capturing subtle differences in the image; simultaneously, it also has a certain degree of suppression when facing larger errors, preventing them from excessively interfering with the overall image effect. In deep learning image reconstruction tasks, Charbonnier L1, with its precise error handling capabilities, effectively preserves rich image details, making the generated image closer to the real image. This provides solid technical support for tasks such as blurred image enhancement, resulting in enhanced images with superior detail representation. In deep learning image reconstruction tasks, edge-aware functions can enhance the clarity of edges and textures by amplifying image gradient differences. Furthermore, in deep learning image reconstruction tasks, the awareness function extracts high-level features through a pre-trained network, constraining the semantic similarity between the generated image and the real image, thereby improving visual naturalness and avoiding over-smoothing.

[0082] Based on the same inventive concept, this application also provides a device for enhancing a blurred image corresponding to the method for enhancing a blurred image. Since the principle of the device in this application is similar to the method for enhancing a blurred image described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0083] Reference Figure 3 The diagram shown is a schematic representation of an image enhancement device according to an embodiment of this application. The device includes: The extraction module 301 is used to input the original blurred image into the blur feature extraction module to obtain a target blur probability matrix and a blur type matrix; the target blur probability matrix includes the target probability that each pixel in the original blurred image is blurred; the blur type matrix is ​​used to characterize the blur type corresponding to each pixel in the original blurred image. The extraction module 301 is further configured to input the blur probability matrix, the high-resolution image corresponding to the original blurred image, and the low-resolution image into the sparse attention module of the encoder, so as to extract the high-frequency detail feature map of the original blurred image from the high-resolution image based on the target pixel, and extract the low-frequency structural feature map of the original blurred image from the low-resolution image; the target pixel is a pixel selected from each pixel that has a higher blur degree than a preset blur degree; the high-frequency detail feature map represents the edges and textures contained in the original blurred image; the low-frequency structural feature map reflects the structure and semantic content of the original blurred image; Feature fusion module 302 is used to input the high-frequency detail feature map and the low-frequency structural feature map into the feature fusion module of the encoder for feature fusion to obtain a fused feature map; The decoding module 303 is used to input the fused feature map and the blur type matrix into the decoder, so as to use the upsampling kernel generated by the blur type matrix to upsample the fused feature map and obtain the enhanced clear image corresponding to the original blur image.

[0084] This device can extract features from blurred images based on target pixels determined by the fuzzy probability matrix, shortening the image enhancement time. It also uses an upsampling kernel generated by the fuzzy type matrix to upsample the fused feature map, avoiding amplification of image noise and improving the image enhancement effect.

[0085] like Figure 4 As shown in the embodiment of this application, an electronic device 400 includes a processor 401, a memory 402, and a bus. The memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device is running, the processor 401 communicates with the memory 402 via the bus, and the processor 401 executes the machine-readable instructions to perform the steps of the above-described method for enhancing blurred images.

[0086] Specifically, the memory 402 and processor 401 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 401 runs the computer program stored in the memory 402, it can execute the above-mentioned method for enhancing blurred images.

[0087] Corresponding to the above-described method for enhancing blurred images, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method for enhancing blurred images.

[0088] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0089] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0091] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the feature processing methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0092] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for enhancing a blurred image, characterized in that, The method includes: The original blurred image is input into the blur feature extraction module to obtain a target blur probability matrix and a blur type matrix; the target blur probability matrix includes the target probability that each pixel in the original blurred image is blurred; the blur type matrix is ​​used to characterize the blur type corresponding to each pixel in the original blurred image. The blur probability matrix, the high-resolution image corresponding to the original blurred image, and the low-resolution image are input into the sparse attention module of the encoder to extract high-frequency detail feature maps of the original blurred image from the high-resolution image based on target pixels, and to extract low-frequency structural feature maps of the original blurred image from the low-resolution image. The target pixels are pixels selected from all pixels that have a higher blur level than a preset blur. The high-frequency detail feature maps represent the edges and textures contained in the original blurred image. The low-frequency structural feature maps reflect the structure and semantic content of the original blurred image. The high-frequency detail feature map and the low-frequency structural feature map are input into the feature fusion module of the encoder for feature fusion to obtain a fused feature map. The fused feature map and the blur type matrix are input into the decoder to upsample the fused feature map using the upsampling kernel generated by the blur type matrix, thereby obtaining an enhanced clear image corresponding to the original blurred image.

2. The method for enhancing a blurred image according to claim 1, characterized in that, The step of inputting the original blurred image into the blurred feature extraction module to obtain the target blurred probability matrix and blurred type matrix includes: The original blurred image is input into a lightweight convolutional neural network to extract feature maps at multiple scales; For a feature map at any scale, predict the initial fuzzy probability matrix and fuzzy type score matrix corresponding to the feature map at that scale; the fuzzy type score matrix includes the score of each pixel corresponding to the target fuzzy type under the feature map at that scale; the target fuzzy type corresponding to the pixel refers to the fuzzy type that the pixel obtains the highest score among all preset fuzzy types under the feature map at that scale. The target fuzzy probability matrix is ​​obtained by weighting and fusing all initial fuzzy probability matrices based on the weights corresponding to each fuzzy type. The fuzzy type matrix is ​​obtained by weighting and fusing the score matrices of all fuzzy types based on the weights corresponding to each fuzzy type.

3. The method for enhancing a blurred image according to claim 2, characterized in that, The weighted fusion of all fuzzy type score matrices based on the weights corresponding to each fuzzy type yields the fuzzy type matrix, which includes: Substituting the weights corresponding to each fuzzy type and the score matrix of all fuzzy types into the following formula, the fuzzy type matrix is ​​obtained; ； in, The fuzz type value is any pixel in the fuzz type matrix. The number of scales. Let be the weight of the target blur type for any pixel in the feature map at the i-th scale. Let be the score of the target blur type corresponding to any pixel point in the feature map at the i-th scale.

4. The method for enhancing a blurred image according to claim 1, characterized in that, The step of inputting the blur probability matrix, the high-resolution image corresponding to the original blurred image, and the low-resolution image into the sparse attention module of the encoder to extract the high-frequency detail feature map of the original blurred image from the high-resolution image based on the target pixel, and to extract the low-frequency structural feature map of the original blurred image from the low-resolution image, includes: The feature values ​​corresponding to the target pixel in the fuzzy probability matrix are set to 1, and the feature values ​​corresponding to other pixels are set to 0 to obtain the global sparse mask matrix. For each pixel, the feature values ​​of all pixels outside the preset window corresponding to the pixel in the global sparse mask matrix are set to 0 to obtain a local sparse mask matrix; based on the local sparse mask matrix, the feature values ​​of the pixels in the high-frequency detail feature map are extracted from the high-resolution image. Based on the global sparse mask matrix, feature values ​​of each pixel in the low-frequency structural feature map are extracted from the low-resolution image.

5. The method for enhancing a blurred image according to claim 1, characterized in that, The upsampling kernel is generated using the fuzzy type matrix through the following steps: Based on the fuzzy type matrix, the high-frequency detail feature map and the low-frequency structural feature map are weighted and fused to obtain a weighted fused feature map; Based on the weighted fusion feature map, the target upsampling kernel is obtained.

6. The method for enhancing a blurred image according to claim 5, characterized in that, The process of obtaining the target upsampling kernel based on the weighted fusion feature map includes: Perform a convolution operation on the weighted fused feature map to obtain an initial upsampling kernel; The blur probability matrix and the original blurred image are input into the sparse attention module to obtain the sparse attention weight matrix; the sparse attention weight matrix is ​​used to characterize the contribution of each pixel to detail recovery and sharpness improvement in the process of enhancing the original blurred image; The initial upsampling kernel is weighted and corrected using the sparse attention weight matrix to obtain the target upsampling kernel.

7. The method for enhancing a blurred image according to claim 1, characterized in that, The image enhancement model includes the blur feature extraction module, the encoder, and the decoder; the loss function of the image enhancement model includes pixel-level differences, edge information differences, and visual feature differences between the blurred image sample and the enhanced clear image corresponding to the blurred image sample output by the image enhancement model. The pixel-level difference reflects the degree of detail recovery between the blurred image sample and the corresponding enhanced sharp image; The edge information difference reflects the difference between the outline and shape of the object contained in the blurred image sample and the outline and shape of the object contained in the corresponding enhanced sharp image; the visual feature difference reflects the difference between the semantic information and visual quality of the blurred image sample and the semantic information and visual quality of the corresponding enhanced sharp image.

8. An image enhancement device, characterized in that, The device includes: An extraction module is used to input the original blurred image into a blurred feature extraction module to obtain a target blurred probability matrix and a blurred type matrix; the target blurred probability matrix includes the target probability that each pixel in the original blurred image is blurred; the blurred type matrix is ​​used to characterize the blurred type corresponding to each pixel in the original blurred image. The extraction module is further configured to input the blur probability matrix, the high-resolution image corresponding to the original blurred image, and the low-resolution image into the sparse attention module of the encoder, so as to extract the high-frequency detail feature map of the original blurred image from the high-resolution image based on the target pixel, and extract the low-frequency structural feature map of the original blurred image from the low-resolution image; the target pixel is a pixel selected from each pixel that has a higher blur level than a preset blur level; the high-frequency detail feature map represents the edges and textures contained in the original blurred image; the low-frequency structural feature map reflects the structure and semantic content of the original blurred image; The feature fusion module is used to input the high-frequency detail feature map and the low-frequency structural feature map into the feature fusion module of the encoder for feature fusion to obtain a fused feature map; The decoding module is used to input the fused feature map and the blur type matrix into the decoder, so as to upsample the fused feature map using the upsampling kernel generated by the blur type matrix, and obtain the enhanced clear image corresponding to the original blurred image.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for enhancing a blurred image as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method for enhancing a blurred image as described in any one of claims 1 to 7.

Images