A U-Net-based image non-uniform blur kernel estimation method
Through the deep neural network model based on U-Net, combined with sparse distribution parameters, the problem of insufficient correlation of fuzzy kernels in the prior art is solved, and the estimation accuracy of image non-uniform fuzzy kernels is improved.
Patent Information
- Application Number
- CN202210640745.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-06-07
AI Technical Summary
The existing image non-uniform blur kernel estimation method based on deep learning failed to effectively consider the blur kernel correlation between adjacent image blocks, which affected the accuracy of blur kernel estimation.
Using the U-Net-based image non-uniform fuzzy kernel estimation method, a deep neural network model containing encoder, decoder and jump layer connection is constructed, and the entire image training model is used to design loss functions and optimization algorithms to optimize network parameters for estimation. Inhomogeneous fuzzy kernels, and sparse distribution is introduced when fuzzy kernels are generated to simulate motion consistency in neighboring regions.
The accuracy of image non-uniform blur kernel estimation is improved, and the overall image information is used to better learn the distribution of motion blur kernels, which enhances the estimation accuracy of blur kernels in different regions.
Smart Images

Figure CN115018726B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for estimating an image non-uniform blur kernel based on U-Net, and belongs to the technical field of image processing. Background Art
[0002] Image blur is a condition of image quality degradation that is often encountered in the process of image acquisition in the field of digital image processing. There are many reasons for image blur, among which motion blur is a typical reason. It is mainly caused by the relative motion between the camera and the scene or different objects in the scene, and the long exposure time of the camera. From the perspective of image generation, different degrees and types of relative motion will have different motion blur kernels, resulting in different degrees and types of blur in the captured images.
[0003] In addition, according to whether the blur in different areas of the image is the same, the image blur can be divided into two cases: uniform blur and non-uniform blur. Uniform blur means that the motion blur kernel of the entire image is consistent, which is relatively simple; non-uniform blur means that the blur kernel in different areas of the image is different, which is more complicated and more in line with the situation we encounter in practice.
[0004] The present invention mainly considers the situation of non-uniform blur and proposes a method for estimating a non-uniform blur kernel from a blurred image. The accuracy of image blur kernel estimation plays a vital role in restoring a clear image from a blurred image. However, in real life, there is often only a single blurred image, so estimating the motion blur kernel from a single blurred image is often a difficult problem.
[0005] In the prior art, there are mainly two types of methods that can be used to estimate blur kernels. One is the traditional iterative optimization method based on natural image statistical priors, and the other is a method based on deep learning that learns from a large amount of data. In recent years, algorithm models based on deep neural networks have been widely applied to various problems in the field of computer vision, and great progress has been made. In the problem of image deblurring, various deep network models have also been developed to estimate the kernel from blurred images, or to directly restore a clear image from a blurred image, achieving good results.
[0006] However, among the existing deep learning-based image non-uniform blur kernel estimation methods, some use image blocks to train and learn local blur kernels of images, while some methods use the entire image to learn but do not consider the correlation between blur kernels of adjacent image blocks, thus affecting the accuracy of image blur kernel estimation. Summary of the invention
[0007] The present invention proposes a method for estimating the non-uniform blur kernel of an image based on U-Net, which aims to overcome the above-mentioned shortcomings of the prior art and improve the accuracy of blur kernel estimation.
[0008] The technical solution of the present invention is a method for estimating an image non-uniform blur kernel based on U-Net, comprising the following steps:
[0009] Step S1: Obtain several clear natural images: Use these clear natural images to construct a data set for training the model.
[0010] Step S2: Generate a non-uniform blur kernel, and use the blur kernel to blur the clear image to obtain the corresponding blurred image, which constitutes the training data set: first generate a number of different non-uniform blur kernels through artificial sampling, and then generate the corresponding blurred image from the clear natural image through convolution operation, so as to constitute the data set required for the training model;
[0011] Step S3: Building a network model for estimating non-uniform blur kernel based on U-Net: Using a U-Net type network model to estimate the non-uniform blur kernel of a blurred image, the U-Net type network model includes an encoder module, a decoder module, and a skip layer connection, and its input is a 3-channel non-uniform blurred color image, and its output is the non-uniform blur kernel that causes the image to be blurred, which is a 1-channel grayscale image;
[0012] Step S4: Design the loss function for network model training: The loss function for model optimization is It is a cost function for a given blurred image. Its purpose is to use the estimated kernel to blur the corresponding clear image, and the resulting blurred image is as similar as possible to the original blurred image. Its calculation formula is:
[0013] where i is the position of the pixel in the clear image u, k i is the local blur kernel at pixel i, u i is a pixel centered on pixel i in the clear image and kernel k i The local area of the same size, v is the blurred image corresponding to the clear image u, and * is the convolution operator symbol;
[0014] Step S5: Select an optimization algorithm and use the training data set to optimize the network model: Use the gradient descent method to optimize the cost function, specifically use the Adam algorithm, which uses the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of the parameters, so that the update of the parameters is relatively stable; the learning rate is 0.0001, and after each training set iteration of 200 epochs, the learning rate is halved, and the training set is iterated for a total of 800 epochs;
[0015] Step S6: Input the blurred image to be processed into the model to obtain its non-uniform blur kernel.
[0016] Preferably, the encoder module in step S3 includes a convolution layer and a downsampling layer, which extracts features from the input image by gradually reducing the size of the feature map and increasing the number of channels, thereby encoding the input. The basic units and network structure parameters included in the encoder module are set as follows:
[0017] (1) Two convolutional layers, with a kernel size of 3×3, a stride of 1, and 16 channels;
[0018] (2) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0019] (3) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 32;
[0020] (4) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0021] (5) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;
[0022] (6) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0023] (7) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 128;
[0024] (8) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0025] (9) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 256;
[0026] The decoder module includes upsampling layers and convolution layers. By gradually increasing the size of the feature map and reducing the number of channels, the original resolution is gradually restored, so as to finally obtain the non-uniform blur kernel image corresponding to the input image. The basic units and network structure parameters contained in the decoder module are set as follows:
[0027] (1) An upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0028] (2) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 128;
[0029] (3) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0030] (4) Two convolutional layers, with a kernel size of 3 × 3, a stride of 1, and 64 channels;
[0031] (5) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0032] (6) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 32;
[0033] (7) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0034] (8) A convolutional layer with a kernel size of 3 × 3, a stride of 1, and 16 channels;
[0035] (9) A convolutional layer with a kernel size of 1×1, a stride of 1, and a channel number of 1;
[0036] The purpose of skip layer connection is to perform feature fusion, that is, in the process of upsampling of the decoder module, the feature map obtained in the process of downsampling at the same level of the encoder is fused with the current feature map, that is, the channels of the feature map are superimposed, and then the next step is performed; four basic operations of skip layer connection are used in this network:
[0037] (1) Connect the 16-channel feature map obtained by unit 1 in the encoder to the 32-channel input feature map of unit 8 in the decoder;
[0038] (2) Connect the 32-channel feature map obtained by unit 3 in the encoder to the 64-channel input feature map of unit 6 in the decoder;
[0039] (3) Connect the 64-channel feature map obtained by unit 5 in the encoder to the 128-channel input feature map of unit 4 in the decoder;
[0040] (4) Connect the 128-channel feature map obtained by unit 7 in the encoder to the 256-channel input feature map of unit 2 in the decoder;
[0041] In the entire network model, except for the last 1×1 convolution layer, the nonlinear activation functions used after all convolution layers are rectified linear units. In order to make the numerical range of the model output, i.e. the estimated blur kernel, be within [0,1], the activation function after the last 1×1 convolution layer is the Sigmoid function.
[0042] Preferably, in step S2, a non-uniform blur kernel is generated, and the blur kernel is used to blur the clear image to obtain a corresponding blurred image. The process of forming the training data set specifically includes:
[0043] Step S21: Construct a motion blur kernel generation model: Model the local blur kernel k of the image as a local motion vector, i.e. k = (k ι ,k θ ), where k ι k is the length of the motion vector, indicating the amplitude of the motion vector, which ranges from 0 to 20 pixels; θ is the angle of the motion vector, indicating the direction of the motion vector, ranging from 0 to 180 degrees, and the sampling interval is 15 degrees; in order to represent the blur kernel in the two-dimensional image area, it needs to be written in the form of a Cartesian coordinate system:
[0044] k x =k l cos(k θ ), k y =k l sin(k θ )
[0045] Among them, k x and k y are the x-axis and y-axis coordinates of vector k respectively;
[0046] Step S22: Sampling the parameters in the blur kernel generation model to generate different blur kernels in adjacent areas, and finally obtaining an overall non-uniform blur kernel; the blur kernel generation model in step S21 can only generate a single local kernel. Since the blur conditions of the entire image area are different, kernels of different areas need to be generated when constructing a training data set, and these kernels cannot be completely random. When sampling and generating blur kernels for the entire image area, it is necessary to constrain the parameters of these kernels to reflect the actual situation.
[0047] For two kernels k in adjacent regions m =(k ι m ,k θ m ) and k n =(k ι n ,kθ n ), when generating these synthetic kernels, the difference in their parameters ▽k ι =k ι m -k ι n , and ▽k θ =k θ m -k θ n , is obtained by sampling the Laplace distribution f(x|μ,b) with parameters μ=0 and b=1:
[0048]
[0049] This distribution is a sparse distribution. Most of the values generated by sampling are near 0, and a few values are larger. In this way, the parameter values of the blur kernels in most adjacent areas can be made close, and only the blur kernels in a few adjacent areas have significant changes.
[0050] Step S23: for a given clear image, generate a corresponding blurred image using a non-uniform blur kernel; use the non-uniform blur kernel k generated in step S22 to blur the clear image; given a clear image u, the pixel at the position of the corresponding blurred image is:
[0051] v i =u i *k i +n i
[0052] where k i is the local blur kernel at pixel i obtained in step S22, u i is a pixel centered on pixel i in the clear image and kernel k i Local areas of the same size, n i It is an additive noise with a Gaussian distribution with a mean of 0 and an intensity of 0.01. In a specific local small area, the blur kernel k at each pixel position is i are the same;
[0053] Step S24: The non-uniform blur kernel and the corresponding blurred image constitute a training data set; the blur kernel k generated in step S22, the corresponding blurred image v and the original clear image u constitute a training data set, and the clear images collected in step S1 are subjected to similar operations to obtain the entire training data set.
[0054] Preferably, in the steps S3, S4 and S5, PyTorch is used to build a network model, design a loss function of the model, and select an optimization algorithm to train the model.
[0055] The advantages of the present invention are as follows: 1) when modeling the motion vectors of adjacent image regions, the difference of their parameters obeys a sparse distribution, which can make the motion of the local region of the image relatively consistent, while the motion of different regions may vary greatly, which is more in line with the situation of non-uniform blur of the image in reality;
[0056] 2) Use the entire blurred image to estimate the blur kernel to learn the distribution of motion vectors over the entire image, rather than using individual image patches. This allows for better use of the overall information of the image when estimating the local motion kernel.
[0057] 3) U-Net is an effective classic network model in the field of image segmentation. Using this model to estimate the motion blur kernels of different regions can take advantage of its advantages in regional segmentation to better estimate the blur of different regions. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 It is a flow chart of the method for estimating the image non-uniform blur kernel based on U-Net of the present invention.
[0059] Figure 2 It is a schematic diagram of an embodiment of a non-uniformly blurred image.
[0060] Figure 3 yes Figure 1 Schematic diagram of the structure of the network model for estimating the non-uniform blur kernel based on U-Net in step S3.
[0061] Figure 4 yes Figure 1 Schematic diagram of the process of constructing a training data set in step S2.
[0062] Figure 5 It is a schematic diagram of an embodiment of a non-uniform blur kernel.
[0063] Figure 6 It is a schematic diagram of an embodiment of a blurred image and a blur kernel of a corresponding region. DETAILED DESCRIPTION
[0064] The present invention is further described in detail below with reference to examples and specific implementation modes.
[0065] In view of the problems existing in the prior art solutions, the present invention proposes a method for estimating the non-uniform blur kernel of an image based on U-Net. In order to better utilize the overall information of the blurred image, the U-Net deep neural network structure is adopted, and the entire blurred image is used as input to train and learn the blur kernel. U-Net is a classic model in the field of image segmentation. This model can better utilize the overall prior information of the image, which is conducive to learning the distribution of the motion blur kernel on the entire image. Moreover, using U-Net to estimate the motion blur kernel of different regions can utilize the ability of U-Net to segment different regions, so as to more accurately estimate the blur kernel of each region.
[0066] In addition, in order to better learn the correlation between blur kernels in adjacent areas of the image, when constructing training data, the vector parameters of the motion blur kernels in adjacent image block areas are made to obey a sparse distribution. This can make the motion of local areas of the image more consistent, while the motion of different areas may vary greatly. This can better model the situation of non-uniform blur kernels in real images and improve the accuracy of blur kernel estimation.
[0067] like Figure 1 As shown, a method for estimating a non-uniform blur kernel of an image based on U-Net includes the following steps:
[0068] Step S1: Obtain some clear natural images: These clear natural images are mainly used to construct a data set for training the model.
[0069] Step S2: Generate a non-uniform blur kernel, and use the blur kernel to blur the clear image to obtain the corresponding blurred image, which constitutes the training data set: There are many reasons for image blur, such as defocus blur, motion blur, etc. The image blur process can be mathematically represented as a clear image and a blur kernel obtained through convolution operation. If the blur kernels in different areas of the image are different, it is called a non-uniform blur kernel, which is also a common situation in reality. First, some different non-uniform blur kernels are generated by artificial sampling, and then the clear natural image is converted into a corresponding blurred image through convolution operation, thereby forming the data set required for the training model.
[0070] In a specific embodiment, a non-uniformly blurred image such as Figure 2 As shown, it can be seen that in this image, the blurring of the two sides is more serious, while the middle part is relatively clear, so the blurring kernels of different regions of the image are inconsistent. The purpose of the present invention is to estimate the non-uniform blur kernel that causes the blurring of the blurred image from the blurred image.
[0071] Step S3: Build a network model for estimating non-uniform blur kernel based on U-Net: U-Net is a classic fully convolutional deep neural network with a U-shaped structure based on encoder and decoder, which has been widely used in tasks such as image semantic segmentation and image restoration.
[0072] The U-Net network model is used to estimate the non-uniform blur kernel of the blurred image. The model structure is as follows: Figure 3 shown.
[0073] The network model is mainly composed of an encoder module on the left, a decoder module on the right, and skip-layer connections between them. Its input is a 3-channel non-uniform blurred color image, and its output is a non-uniform blur kernel that causes the image to be blurred, which is a 1-channel grayscale image.
[0074] The encoder module consists of basic units such as convolutional layers and downsampling layers. It extracts features from the input image by gradually reducing the size of the feature map and increasing the number of channels, thereby encoding the input. The basic units and network structure parameters are set as follows:
[0075] (1) Two convolutional layers, with a kernel size of 3×3, a stride of 1, and 16 channels;
[0076] (2) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0077] (3) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 32;
[0078] (4) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0079] (5) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;
[0080] (6) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0081] (7) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 128;
[0082] (8) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2;
[0083] (9) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 256;
[0084] The decoder module consists of basic units such as upsampling layers and convolutional layers. It gradually restores the original resolution by gradually increasing the size of the feature map and reducing the number of channels, thus finally obtaining a non-uniform blur kernel image corresponding to the input image. The basic units and network structure parameters are set as follows:
[0085] (1) An upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0086] (2) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 128;
[0087] (3) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0088] (4) Two convolutional layers, with a kernel size of 3 × 3, a stride of 1, and 64 channels;
[0089] (5) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0090] (6) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 32;
[0091] (7) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2;
[0092] (8) A convolutional layer with a kernel size of 3 × 3, a stride of 1, and 16 channels;
[0093] (9) A convolutional layer with a kernel size of 1×1, a stride of 1, and a channel number of 1.
[0094] The purpose of skip layer connection is to perform feature fusion, that is, during the upsampling process of the decoder module, the feature map obtained during the downsampling process of the encoder at the same level is fused with the current feature map, that is, the channels of the feature map are superimposed, and then the next step is performed. In this network, four basic skip layer connection operations are used:
[0095] (1) Connect the 16-channel feature map obtained by unit 1 in the encoder to the 32-channel input feature map of unit 8 in the decoder;
[0096] (2) Connect the 32-channel feature map obtained by unit 3 in the encoder to the 64-channel input feature map of unit 6 in the decoder;
[0097] (3) Connect the 64-channel feature map obtained by unit 5 in the encoder to the 128-channel input feature map of unit 4 in the decoder;
[0098] (4) Connect the 128-channel feature map obtained by unit 7 in the encoder to the 256-channel input feature map of unit 2 in the decoder;
[0099] In addition, in the entire network model, except for the last 1×1 convolution layer, the nonlinear activation function used after all convolution layers is the Rectified Linear Unit (ReLU). In order to make the numerical range of the model output, i.e. the estimated blur kernel, be within [0,1], the activation function after the last 1×1 convolution layer is the Sigmoid function.
[0100] Step S4: Design the loss function for network model training: The loss function for model optimization is It is a cost function for a given blurred image, and its purpose is to make the blurred image obtained by using the estimated kernel to blur the corresponding clear image as similar as possible to the original blurred image. Its calculation formula is:
[0101] where i is the position of the pixel in the clear image u, k i is the local blur kernel at pixel i, u i is a pixel centered on pixel i in the clear image and kernel k i The local area is of the same size, v is the blurred image corresponding to the clear image u, and * is the convolution operation symbol.
[0102] Step S5: Select an optimization algorithm and use the training data set to optimize the network model: Use the gradient descent method to optimize the cost function, specifically the Adam algorithm, which uses the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of the parameters, so that the parameter update is relatively stable. The learning rate is 0.0001, and after each training set iteration of 200 epochs, the learning rate is halved, and the training set is iterated for a total of 800 epochs.
[0103] Step S6: Input the blurred image to be processed into the model to obtain its non-uniform blur kernel.
[0104] In step S2, a non-uniform blur kernel is generated, and the blur kernel is used to blur the clear image to obtain a corresponding blurred image. The specific process of forming the training data set is as follows: Figure 4 As shown, including:
[0105] Step S21: Construct a motion blur kernel generation model: Model the local blur kernel k of the image as a local motion vector, i.e. k = (k ι ,k θ ), where k ιk is the length of the motion vector, indicating the amplitude of the motion vector, which ranges from 0 to 20 pixels; θ is the angle of the motion vector, indicating the direction of the motion vector, ranging from 0 to 180 degrees, and the sampling interval is 15 degrees. In order to represent the blur kernel in the two-dimensional image area, it needs to be written in the form of a Cartesian coordinate system:
[0106] k x =k l cos(k θ ), k y =k l sin(k θ )
[0107] Among them, k x and k y are the x-axis and y-axis coordinates of vector k respectively.
[0108] Step S22: Sampling the parameters in the blur kernel generation model to generate different blur kernels for adjacent regions, and finally obtaining an overall non-uniform blur kernel. The blur kernel generation model in the previous step can only generate a single local kernel. Since the blur conditions of the entire image area are different, kernels of different regions need to be generated when constructing the training data set, and these kernels cannot be completely random, because generally speaking, the blur conditions of adjacent regions are similar, while the blur conditions of different regions may be quite different. Therefore, when sampling and generating blur kernels for the entire image area, it is necessary to impose certain constraints on the parameters of these kernels to reflect the real situation as much as possible.
[0109] For two kernels k in adjacent regions m =(k ι m ,k θ m ) and k n =(k ι n ,k θ n ), when generating these synthetic kernels, the difference in their parameters ▽k ι =k ι m -k ι n , and ▽k θ =k θ m -k θ n , is obtained by sampling the Laplace distribution f(x|μ,b) with parameters μ=0 and b=1:
[0110]
[0111] This distribution is a sparse distribution, and most of the sampled values are near 0, with a few values being larger. In this way, the blur kernel parameter values of most adjacent regions can be made close, and only the blur kernels of a few adjacent regions have changed significantly. In a specific embodiment, a sample of a non-uniform blur kernel is Figure 5 As shown in the figure, it can be seen that the motion vectors representing the blur kernels of adjacent regions change slowly, while the blur kernels of different regions change significantly.
[0112] Step S23: For a given clear image, generate a corresponding blurred image using a non-uniform blur kernel. With the non-uniform blur kernel k generated in the previous step, it can be used to blur the clear image. Given a clear image u, the pixel at the position of the corresponding blurred image is:
[0113] v i =u i *k i +n i
[0114] where k i is the local blur kernel at pixel i obtained in the previous step, u i is a pixel centered on pixel i in the clear image and kernel k i Local areas of the same size, n i is an additive noise with a Gaussian distribution and a mean of 0 and an intensity of 0.01. It should be noted that in a specific local small area, the blur kernel k at each pixel position is i are the same.
[0115] Step S24: The non-uniform blur kernel and the corresponding blurred image constitute a training data set. The blur kernel k generated in the previous step, the corresponding blurred image v and the original clear image u constitute a training data set. The clear images collected in step S1 are subjected to similar operations to obtain the entire training data set.
[0116] In steps S3, S4 and S5, PyTorch is used to build the network model, design the model's loss function and select the optimization algorithm to train the model. PyTorch is an open source deep learning framework that is concise, flexible, fast and efficient. It can build deep neural networks of various structures with automatic differentiation functions. At the same time, it can use GPU to achieve computing acceleration, greatly shortening the training time of the model.
[0117] In a specific embodiment, after the network model is trained, Figure 2By inputting the blurred image shown in into the model, we can get its non-uniform blur kernel. Then, by drawing the estimated local blur kernel onto the corresponding area in the image, we can see the corresponding relationship between the blurred image and the kernel, as shown in Figure 6 As shown in the figure, it can be seen that the blur kernel estimated from the two side areas has a larger scale, while the blur kernel estimated from the middle area has a smaller scale, which is consistent with the observation Figure 2 The intuitive feeling of the blurred image shown is consistent, which shows that the proposed method achieves a relatively good effect of estimating the non-uniform blur kernel of the image. After obtaining the blur kernel of the image, some classic image deblurring algorithms can be used to restore a clear image.
[0118] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A method for estimating image non-uniform blur kernel based on U-Net, characterized in that: The following steps are involved: Step S1: Obtain several clear natural images: Use these clear natural images to construct a data set for training the model; Step S2: Generate a non-uniform blur kernel, and use the blur kernel to blur the clear image to obtain the corresponding blurred image, which constitutes the training data set: first generate a number of different non-uniform blur kernels through artificial sampling, and then generate the corresponding blurred image from the clear natural image through convolution operation, so as to constitute the data set required for the training model; Step S3: Building a network model for estimating non-uniform blur kernel based on U-Net: Using a U-Net type network model to estimate the non-uniform blur kernel of a blurred image, the U-Net type network model includes an encoder module, a decoder module, and a skip layer connection, and its input is a 3-channel non-uniform blurred color image, and its output is the non-uniform blur kernel that causes the image to be blurred, which is a 1-channel grayscale image; Step S4: Design the loss function for network model training: The loss function for model optimization is It is a cost function for a given blurred image. Its purpose is to use the estimated kernel to blur the corresponding clear image, and the resulting blurred image is as similar as possible to the original blurred image. Its calculation formula is: where i is the position of the pixel in the clear image u, k i is the local blur kernel at pixel i, u i is a pixel centered on pixel i in the clear image and kernel k i The local area of the same size, v is the blurred image corresponding to the clear image u, and * is the convolution operator symbol; Step S5: Select an optimization algorithm and use the training data set to optimize the network model: Use the gradient descent method to optimize the cost function, specifically use the Adam algorithm, which uses the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of the parameters, so that the update of the parameters is relatively stable; the learning rate is 0.0001, and after each training set iteration of 200 epochs, the learning rate is halved, and the training set is iterated for a total of 800 epochs; Step S6: inputting the blurred image to be processed into the model to obtain its non-uniform blur kernel; The encoder module in step S3 includes a convolution layer and a downsampling layer, which extracts features from the input image by gradually reducing the size of the feature map and increasing the number of channels, thereby encoding the input. The basic units and network structure parameters included in the encoder module are set as follows: (1) Two convolutional layers, with a kernel size of 3×3, a stride of 1, and 16 channels; (2) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2; (3) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 32; (4) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2; (5) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64; (6) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2; (7) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 128; (8) A downsampling layer using maximum pooling with a window size of 2×2 and a stride of 2; (9) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 256; The decoder module includes upsampling layers and convolution layers. By gradually increasing the size of the feature map and reducing the number of channels, the original resolution is gradually restored, so as to finally obtain the non-uniform blur kernel image corresponding to the input image. The basic units and network structure parameters contained in the decoder module are set as follows: (1) An upsampling layer using bilinear interpolation with an enlargement scale of 2×2; (2) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 128; (3) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2; (4) Two convolutional layers, with a kernel size of 3 × 3, a stride of 1, and 64 channels; (5) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2; (6) Two convolutional layers, the convolution kernel size is 3×3, the stride is 1, and the number of channels is 32; (7) an upsampling layer using bilinear interpolation with an enlargement scale of 2×2; (8) A convolutional layer with a kernel size of 3 × 3, a stride of 1, and 16 channels; (9) A convolutional layer with a kernel size of 1×1, a stride of 1, and a channel number of 1; The purpose of skip layer connection is to perform feature fusion, that is, in the process of upsampling of the decoder module, the feature map obtained in the process of downsampling at the same level of the encoder is fused with the current feature map, that is, the channels of the feature map are superimposed, and then the next step is performed; four basic operations of skip layer connection are used in this network: (1) Connect the 16-channel feature map obtained by unit 1 in the encoder to the 32-channel input feature map of unit 8 in the decoder; (2) Connect the 32-channel feature map obtained by unit 3 in the encoder to the 64-channel input feature map of unit 6 in the decoder; (3) Connect the 64-channel feature map obtained by unit 5 in the encoder to the 128-channel input feature map of unit 4 in the decoder; (4) Connect the 128-channel feature map obtained by unit 7 in the encoder to the 256-channel input feature map of unit 2 in the decoder; In the entire network model, except for the last 1×1 convolution layer, the nonlinear activation function used after all convolution layers is the rectified linear unit. In order to make the model output, i.e. the estimated blur kernel, have a value range of [0,1], the activation function after the last 1×1 convolution layer is the Sigmoid function. The step S2 generates a non-uniform blur kernel, and uses the blur kernel to blur the clear image to obtain a corresponding blurred image. The process of forming a training data set specifically includes: Step S21: Construct a motion blur kernel generation model: Model the local blur kernel k of the image as a local motion vector, i.e. k = (k ι ,k θ ), where k ι k is the length of the motion vector, indicating the amplitude of the motion vector, which ranges from 0 to 20 pixels; θ is the angle of the motion vector, indicating the direction of the motion vector, ranging from 0 to 180 degrees, and the sampling interval is 15 degrees; in order to represent the blur kernel in the two-dimensional image area, it needs to be written in the form of a Cartesian coordinate system: k x =k l cos(k θ ),k y =k l sin(k θ ) Among them, k x and ky are the x-axis and y-axis coordinates of vector k respectively; Step S22: Sampling the parameters in the blur kernel generation model to generate different blur kernels in adjacent areas, and finally obtaining an overall non-uniform blur kernel; the blur kernel generation model in step S21 can only generate a single local kernel. Since the blur conditions of the entire image area are different, kernels of different areas need to be generated when constructing a training data set, and these kernels cannot be completely random. When sampling and generating blur kernels for the entire image area, it is necessary to constrain the parameters of these kernels to reflect the actual situation. For two kernels k in adjacent regions m =(k ι m ,k θ m ) and k n =(k ι n ,k θ n ), when generating these synthetic kernels, the difference in their parameters ▽k ι =k ι m -k ι n , and ▽k θ =k θ m -k θ n , is obtained by sampling the Laplace distribution f(x|μ,b) with parameters μ=0 and b=1: This distribution is a sparse distribution. Most of the values generated by sampling are near 0, and a few values are larger. In this way, the parameter values of the blur kernels in most adjacent areas can be made close, and only the blur kernels in a few adjacent areas have significant changes. Step S23: for a given clear image, generate a corresponding blurred image using a non-uniform blur kernel; use the non-uniform blur kernel k generated in step S22 to blur the clear image; given a clear image u, the pixel at the position of the corresponding blurred image is: v i =u i *k i +n i where k i is the local blur kernel at pixel i obtained in step S22, u i is a pixel centered on pixel i in the clear image and kernel k i Local areas of the same size, n i It is an additive noise with a Gaussian distribution with a mean of 0 and an intensity of 0.
01. In a specific local small area, the blur kernel k at each pixel position is i are the same; Step S24: The non-uniform blur kernel and the corresponding blurred image constitute a training data set; the blur kernel k generated in step S22, the corresponding blurred image v and the original clear image u constitute a training data set, and the clear images collected in step S1 are subjected to similar operations to obtain the entire training data set.
2. The method for estimating a non-uniform blur kernel of an image based on U-Net as claimed in claim 1, characterized in that: In the steps S3, S4 and S5, PyTorch is used to build a network model, design a loss function of the model, and select an optimization algorithm to train the model.
