A method for image blur classification by reconstructing blur kernels

Abstract

The application discloses an image blur classification method. The method generates a blur kernel and a mask first, and then classifies the blur kernel to further classify the blur type more accurately. When generating the mask, the idea of an image region classification model such as U-Net is used, and a skip connection is established with the corresponding layer, so that the postposition layer can directly perceive from the high-level features which regions are blur kernels, thereby making the generation of the image mask more accurate. The application creatively designs a method of adding image re-blurring loss and blur kernel loss in the loss function, which identifies the type of blur while inferring the shape of the blur kernel in different regions. Therefore, the method improves the performance in determining the subtle blur of the image.

Description

Technical Field

[0001] This invention belongs to the field of fuzzy category discrimination technology in computer vision, and particularly relates to a method for image fuzzy classification by reconstructing a fuzzy kernel. Background Technology

[0002] When performing contrast-detection autofocus or high-precision image acquisition, subtle blurring can occur. However, it's difficult to determine whether this blurring is caused by slight defocusing or slight motion blur. Most literature on contrast-detection autofocus uses still life as the subject, failing to consider that motion blur functions also affect many focus evaluation functions, including those using high-frequency information content detection and sharpness detection. Therefore, determining the presence of motion-blurred objects in an image and, based on that, whether they are out of focus, becomes a point that contrast-detection autofocus needs improvement. In scenarios requiring the automatic acquisition of multiple high-precision images, such as high-fidelity digital body reconstruction, having photographers manually check each image for slight blur and determine its type to deduce the cause or remove problematic images is a tedious and repetitive task that wastes a significant amount of the photographer's time. Furthermore, current literature on blur type identification is not ideal for subtle blur. It can only determine the type of fuzziness that is highly blurred, but not the type of fuzziness that is slightly blurred. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for image fuzzy classification through the reconstruction of a fuzzy kernel.

[0004] The specific technical solution of this invention is as follows: A method for image fuzzy classification by reconstructing a fuzzy kernel, obtained through the following steps:

[0005] (1) Create an artificially constructed dataset for training the neural network, which contains sharp images and corresponding blurred images generated by processing the sharp images;

[0006] (2) Construct a neural network model for image blur classification by reconstructing blur kernels. This model is used to generate different blur kernels and binary black and white images covering the corresponding blur regions for different regions of a blurry image, i.e., masks. The model selects the blur kernel corresponding to the region with the largest mask coverage area and distinguishes the blur type of the image by classifying the blur kernels.

[0007] (3) Use the dataset created in step (1) to train the neural network model in step (2) and iterate until the trained neural network model is obtained.

[0008] (4) Finally, the image to be detected is input into the neural network model trained in step (3) to complete the fuzzy classification of the images in the dataset.

[0009] Furthermore, step (1) specifically includes the following sub-steps:

[0010] (1.1) Take N clear images;

[0011] (1.2) Generate X blur kernels with different blur circle radii, treating the blur circle caused by camera defocusing as a uniform solid circle; to maintain energy conservation, the sum of the blur kernels is 1, therefore the formula for the defocus blur kernel is as follows:

[0012]

[0013] Where x is the x-coordinate of the image; y is the y-coordinate of the image; and r is the radius of the discrete circle;

[0014] (1.3) Obtain Y motion blur kernels. To maintain energy conservation, the blur kernels are normalized. The normalization formula is as follows:

[0015]

[0016] (1.4) Convolve N clear images with X blur kernels of different blur circle radii and Y motion blur kernels to generate N×(X+Y) images.

[0017] Furthermore, the neural network model for image fuzzy classification by reconstructing the fuzzy kernel in step (2) is specifically as follows:

[0018] (1) In the design of the model, each pixel in the blurred region of the image is blurred by generating the point spread function corresponding to the blurred image, and the blur in the image is classified by classifying the point spread function.

[0019] (2) Using the idea of ​​U-Net image region classification model, different blur kernels are generated for different regions of the image to eliminate the influence of different blur kernels in different regions; in the process of generating the mask, the U-Net image region classification model is used to establish skip connections with the corresponding layers, so that the subsequent layers can directly perceive the blur regions from the high-level features, thereby generating the image mask; the region division method avoids the influence of irrelevant regions on the blur kernel discrimination.

[0020] (3) The fuzzy kernel is obtained by direct generation, thereby simulating real fuzziness.

[0021] The present invention has the following beneficial effects:

[0022] This invention addresses this problem by generating a large training dataset and designing a deep learning model. Instead of directly feeding massive amounts of data into a neural network, this invention first generates blur kernels and masks, then classifies the blur kernels, thus achieving more accurate classification of blur types. When generating masks, it borrows ideas from image region classification models like U-Net, establishing skip connections with corresponding layers. This allows subsequent layers to directly perceive which regions are blurred from high-level features, resulting in more accurate image mask generation. This method creatively designs a loss function, incorporating re-blurring loss and blur kernel loss, making the model's behavior more aligned with human blur identification logic, guiding the model's training direction and optimizing its performance. Experimental data shows that this model represents a significant improvement over previous methods in identifying subtle blur kernels. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] Figure 1 This is a diagram summarizing the method of the present invention;

[0025] Figure 2 This is a structural diagram of the image subtle blur classification model of the present invention. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0027] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0028] The first step is to generate the training dataset. Since the L1 value in the loss function of this invention is the convolution result of the sharp image and the generated blur kernel, and the convolution result of the sharp image and the actual blur kernel, it is necessary to ensure that the baseline ground truth values ​​of the blurred image and the sharp image are only affected by the blur kernel and not by any other factors such as camera position movement or scene position movement. Therefore, the dataset used in this invention cannot be actually collected data but must be computer-generated data to ensure that the baseline ground truth values ​​of the blurred image and the sharp image are only affected by the blur kernel.

[0029] The method for generating data according to this invention is as follows;

[0030] The first step, a method for image fuzzy classification by reconstructing a fuzzy kernel, is obtained through the following steps:

[0031] Create an artificially constructed dataset for training the neural network, which contains sharp images and corresponding blurred images generated by processing the sharp images;

[0032] like Figure 1 As shown, specifically:

[0033] (1.1) Take N clear images.

[0034] (1.2) Generate X blur kernels with different radii of the blur circle. Based on extensive literature, it is known that the blur circle caused by camera defocusing can be approximated as a uniform solid circle. To maintain energy conservation, the sum of the blur kernels should be 1. Based on the above reasoning, the formula for the defocus blur kernel is shown in Equation 1.

[0035]

[0036] Where x is the horizontal coordinate of the image; y is the vertical coordinate of the image; and r is the radius of the discrete circle.

[0037] (1.3) Obtain Y motion blur kernels. In this invention, the motion blur kernels are generated empirically. To maintain energy conservation, the blur kernels are normalized. The normalization formula is shown in Equation 2.

[0038]

[0039] (1.4) Convolve N clear images with X blur kernels of different blur circle radii and Y motion blur kernels to generate N×(X+Y) images.

[0040] However, in order to demonstrate its practical application value, this invention did not use the generated dataset for judgment when evaluating the model's performance. Instead, it re-collected datasets with different fuzzy types for discrimination.

[0041] The second step is to construct a neural network model for image blur classification by reconstructing blur kernels. This model is used to generate different blur kernels and binary black and white images covering the corresponding blur regions of a blurred image, i.e., masks. The model selects the blur kernel corresponding to the region with the largest mask coverage area and distinguishes the blur type of the image by classifying the blur kernels.

[0042] Construct a network model for image fuzzy classification by reconstructing a fuzzy kernel, the structure of which is shown in the attached figure. Figure 2 .

[0043] To overcome the problem that traditional frequency domain and gradient methods are easily affected by other information contained in the image during the judgment process, this model applies a point spread function to each pixel in the blurred region of the image, uses deep learning methods to generate the point spread function, and then classifies the point spread function to detect motion blur in the image.

[0044] In blurred images, different regions experience varying degrees of blur; some areas are affected while others remain unaffected. For out-of-focus blur, all objects outside the depth of field in the entire image are blurred, and the degree of blur is related to the defocus distance. For motion blur, if the blur is caused by camera movement, the entire image will be affected; if only one object in the image is moving, other objects will not exhibit motion blur characteristics. Determining the blur type based on the entire image inevitably involves the influence of other regions. To address this issue, this invention borrows ideas from image region classification models such as U-Net, generating different image masks for different regions to eliminate the influence of different blur kernels in different regions. During mask generation, borrowing from U-Net and other image region classification models, skip connections are established with corresponding layers, allowing subsequent layers to directly perceive which regions are blurred from high-level features, thus making image mask generation more accurate. This region-based method avoids the influence of irrelevant regions on blur kernel discrimination.

[0045] Motion blur has its irregularities. Take, for example, motion blur in portrait photography caused by facial tremors and micro-expression changes in a model. This motion blur cannot be easily synthesized from several motion blur basis functions. The blur kernel is not synthesized from the basis blur kernels, but rather generated directly.

[0046] Based on the above principles, the deep learning model designed by the researchers of this invention is attached. Figure 2 As shown:

[0047] The model's input starts from the top left corner of the image, and the required results are a mask, a blur kernel, and the image category. Each differently colored module represents a different module, and the numbers in the diagram indicate the number of channels in that neural network layer. The dotted dashed boxes represent downsampling modules, containing two convolutional layers and one pooling layer; the dashed line segments represent upsampling modules, containing one bilinear upsampling layer and three convolutional layers; the dotted horizontal boxes represent skip connection modules designed to receive skip connection information from the downsampling section.

[0048] The loss function designed by the researchers of this invention is shown in equations (3)-(4):

[0049] LOSS=k1SUM(L1)+k2SUM(L2)+k3SUM(L3);

[0050]

[0051] L2 = |kernel - kernelgt| (3)

[0052] L3 = |class - classgt| (4)

[0053] Where LOSS is the loss function, L1 is the image re-blurring loss function, L2 is the blur kernel difference loss function, and L3 is the classifier loss function. k1, k2, and k3 are coefficients that balance the values ​​of different loss functions. In L1, CONV is the convolution symbol, piccl is the clear image before convolution, kernelli is the convolution kernel of the i-th region, mask is the mask corresponding to the i-th region, picgt is the ground truth value of the re-blurred image, kernelgt is the ground truth value of the convolution kernel of the i-th region, w is the image width in pixels, h is the image height in pixels, class is the classification result, and classgt is the ground truth value of the classification.

[0054] The third step is to use the dataset created in the first step to train the neural network model from the second step.

[0055] In the fourth step, when performing fuzzy image classification, the image to be detected is input into the neural network model trained in the third step to obtain the fuzzy classification of the image.

[0056] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0057] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for image fuzzy classification by reconstructing a fuzzy kernel, characterized in that, It is obtained through the following steps: (1) Create an artificially constructed dataset for training the neural network, which contains sharp images and corresponding blurred images generated by processing the sharp images; specifically including the following sub-steps: (1.1) Take N clear images; (1.2) Generate X blur kernels with different defocusing radii for the blur circles, and treat the blur circles caused by camera defocusing as uniform solid circles. To maintain energy conservation, the sum of the blur kernels is 1; therefore, the formula for the defocus blur kernel is as follows: ； in, The x-axis of the image; It is the vertical axis of the image; It is the radius of the discrete circle; (1.3) Obtain Y motion blur kernels. To maintain energy conservation, the blur kernels are normalized. The normalization formula is as follows: ； (1.4) Convolve N clear images with X images of different blur circles and Y images of motion blur kernels to generate N×(X+Y) images; (2) Construct a neural network model for image blur classification by reconstructing blur kernels. This model is used to generate different blur kernels and binary black-and-white images covering the corresponding blur regions of a blurred image, i.e., masks. The model selects the blur kernel corresponding to the region with the largest mask coverage area and classifies the blur kernels to distinguish the blur types of the image. The neural network model for image blur classification by reconstructing blur kernels is specifically as follows: (2.1) In the design of the model, each pixel in the blurred region of the image is blurred by generating the point spread function corresponding to the blurred image, and the blur in the image is classified by classifying the point spread function. (2.2) Using the idea of ​​U-Net image region classification model, different blur kernels are generated for different regions of the image to eliminate the influence of different blur kernels in different regions; in the process of generating the mask, the U-Net image region classification model is used to establish skip connections with the corresponding layers, so that the subsequent layers can directly perceive the blur regions from the high-level features, thereby generating the image mask; the region division method avoids the influence of irrelevant regions on the blur kernel discrimination. (2.3) The fuzzy kernel is obtained by direct generation, thereby simulating real fuzziness; (3) Use the dataset created in step (1) to train the neural network model in step (2), and iterate until the trained neural network model is obtained; (4) Finally, input the image to be detected into the neural network model trained in step (3) to complete the fuzzy classification of the images in the dataset.

Images