Denoising and non-uniform blurred image enhancement method based on multi-scale pyramid

Abstract

The invention provides a denoising and non-uniform blurring-removing image enhancement method based on a multi-scale pyramid, and the method comprises the steps: firstly constructing a multi-scale Gaussian pyramid image, determining a blurring layer and a mask through fuzzy segmentation in order to improve the accuracy, carrying out the layered estimation of a motion blurring layer, and determining a blurring matrix of each image through a non-blind method, the method comprises the following steps: firstly, removing low-frequency noise by using a Gaussian low-pass filter, removing high-frequency noise by using wavelet transform, then performing deconvolution deblurring in a Gaussian pyramid, fusing two processing results through weighted multi-scale, and finally, performing inverse operation through a multi-scale pyramid to synthesize a clear picture. According to the method, the fuzzy kernel is solved in a layering mode, the Gaussian pyramid is used for denoising and deblurring, meanwhile, the high-frequency noise problem which is prone to being ignored in a traditional algorithm is considered, adaptability to pictures of different sizes is enhanced through the pyramid, uneven blurring caused by camera movement is removed, and the PSNR is increased by about 2.74 d compared with the PSNR in a traditional method.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, especially image enhancement optimization based on computer vision, and particularly relates to a denoising and non-uniform blur image enhancement method based on a multi-scale pyramid. Background Art

[0002] In the context of the rapid development of computer vision technology, image enhancement, as one of the core research directions, is becoming a key technology to break through the bottleneck of visual information processing in multiple fields. Application scenarios such as medical image analysis based on computer vision, intelligent security systems, remote sensing monitoring platforms, and scientific visualization research generally face the problem of image quality degradation under dynamic shooting conditions, among which the compound motion blur problem caused by camera movement is particularly prominent. This type of non-uniform blur pattern not only significantly reduces the visual information entropy value but also systematically interferes with subsequent high-order visual tasks such as target recognition and feature extraction. Although traditional image enhancement methods have achieved certain results in dealing with uniform blur patterns, their algorithm frameworks based on static scenes are difficult to effectively model the complex blur kernels formed by the relative motion between the camera and the target in real scenes.

[0003] In response to the above problems, the literature "Image Deblurring Algorithm for Moving Objects" adopts a deblurring algorithm for moving objects, which better realizes the demand for deblurring non-moving regions and has good effects on synthetic images. However, this method has two significant limitations: First, using a fixed-size blur kernel for global deconvolution results in the algorithm lacking multi-scale adaptability; second, no noise suppression mechanism is constructed, and the residual noise after deblurring seriously affects the visual quality. The present invention deconvolves the obtained blur kernel for each layer of the pyramid based on a multi-scale pyramid and performs high-frequency and low-frequency denoising on the denoising problem ignored by this method, improving the calculation efficiency and being able to adapt to different sizes.

[0004] Although the literature "Real Image Denoising Method Based on Pyramid Attention and Edge Enhancement" has made breakthroughs in static image denoising through the pyramid attention mechanism, its technical route has two limitations: On the one hand, it does not form a collaborative optimization framework with the deblurring algorithm, and independent processing leads to high computational complexity; on the other hand, the high-frequency noise processing adopts a unified threshold strategy, making it difficult to cope with the non-uniform noise distribution characteristics in industrial scenarios. The multi-scale pyramid proposed by the present invention removes different frequency domain noises, accurately distinguishes image details and noise components through frequency domain analysis, and combines non-uniform blur modeling in the spatial domain, effectively improving the denoising effect.

[0005] The literature "Research on Blind Motion Blur Algorithm for Mobile Phone Photos" uses multi-frame Fourier domain fusion to effectively cope with camera shake blur. However, through experimental verification, it has a high artifact rate when dealing with non-rigid blur caused by six-degree-of-freedom camera motion. This literature is more applicable to the blur caused by camera shake rather than the blur caused by camera spatial motion. The present invention mainly considers the uneven blur caused by camera motion, and the applicable environment is different. Summary of the Invention

[0006] In view of the problems existing in the prior art, the present invention proposes a denoising and uneven blur removal image enhancement method based on a multi-scale pyramid. Fuzzy segmentation is used to determine the blur layer and the mask, and the motion blur layer is estimated layer by layer. A non-blind method is used to determine the blur matrix of each image. Then, a multi-scale Gaussian pyramid image is constructed, and both low-frequency and high-frequency noises are removed. After that, deconvolution is performed in the Gaussian pyramid to remove blur, and the two processing results are fused through weighted multi-scale.

[0007] To achieve the above objectives, the present invention adopts the following specific technical solutions to solve the problems:

[0008] A denoising and uneven blur removal image enhancement method based on a multi-scale pyramid includes the following steps:

[0009] S1: Considering that direct deconvolution for blur removal has poor adaptability to the picture scale, after converting the input original image into a grayscale image, a multi-scale feature pyramid is innovatively established, and deblurring and denoising processing are performed on the basis of the pyramid. The number of layers Q of the pyramid is determined by the image size and the scale of the feature points described, and its calculation formula is shown as follows:

[0010] Q = [log b (min(w, h)) - log b (patch_size · 4)] (1)

[0011] In the formula, b is the downsampling coefficient, w and h are the width and height of the image respectively, and patch_size is the scale for describing the feature points;

[0012] S2: The RGB image with the lowest resolution in the pyramid is segmented for the blur area through the method of natural image mask segmentation. The motion blur image can be segmented into a motion blur layer and a non-motion blur layer. The specific calculation formula is as follows:

[0013] B = KL f + (1 - Km)B b + n (2)

[0014] In the formula, B, L f and B bThey are an observable blurred image, a clear moving object region, and a non-moving object region respectively. m and n are the binary mask of the moving object region and noise, and K is the blur matrix;

[0015] S3: Implement adaptive mask segmentation based on foreground and background region division, and then estimate the non-binary mask m f , the motion blur layer B f and the non-motion blur layer B b , the spatially-varying blur model and the image restoration model can be respectively expressed as Equations (3) and (4):

[0016] B f = KL f + n (3)

[0017] L = L f +(1 - m)B b (4)

[0018] S3.1: By obtaining the image blur features in the blur kernel direction and the orthogonal dimension, the blur features of each pixel under the known blur kernel can be obtained;

[0019] S3.2: Use the TRWS algorithm to solve the pixel-level blur kernel label, realize the identification and classification of multi-directional blur regions, and record the blur length and angle information of each region. The blur length is 0 ≤ l ≤ l max , and the angle is -90° ≤ θ < 90°;

[0020] S3.3: Label the pixels with zero blur kernel labels as 0 and assign other values as 1. This is used as the initial blur region marking result. For the existing mislabeling phenomenon, adopt a multi-level optimization strategy, remove the moving object regions with a small number of pixels, and optimize through the methods of opening operation, closing operation, and guided filtering. Finally, the moving region marking result can be obtained;

[0021] S4: Estimate the linear blur kernel of the blurred image block. To avoid the blur kernel falling into a trivial solution during the initial iteration, a constraint term is set to guide the blur kernel estimation process. The objective function for blur kernel estimation is as shown in Equation (5):

[0022]

[0023] In the formula, L i is the clear image during the solution process, f(k fi ) is the linear constraint of the blur kernel, ||k fi || and γ are the constraint term and the constraint coefficient respectively;

[0024] S5: Considering the complexity of the theoretical solution of Equation (5), use a non-blind method to solve alternately, and the corresponding objective functions are as shown in Equations (6) and (7):

[0025]

[0026] S6: Based on the objective functions (6) and (7), an alternating optimization method is used to iteratively solve for the blur kernel and the intermediate latent image;

[0027] S6.1: First, with the blur kernel k fi held constant, the intermediate latent image L i is solved first. To this end, an auxiliary variable g = (g h , g v ) Τ is introduced to reconstruct Equation (6) to transform it into an optimization problem involving L i and g;

[0028]

[0029] S6.2: Subsequently, with the auxiliary variable g fixed, the least squares optimization method is used to iteratively solve for L i , and its mathematical model is as shown in Equation (9):

[0030]

[0031] S6.3: When solving for L i in Equation (9), the fast Fourier transform is used for the solution, and the corresponding closed-form solution is shown in Equation (10):

[0032]

[0033] In the formula, F() and F -1 () represent the fast Fourier transform and the inverse fast Fourier transform, respectively; denotes the complex conjugate operation; ▽ h and ▽ v represent the difference operations in the horizontal and vertical directions, respectively;

[0034] S6.4: When k fi is fixed, the variable g can be obtained by solving Equation (12). Equation (12) is a per-pixel minimization problem, and its solution can be derived through equation derivation, as shown in Equation (13):

[0035]

[0036] S7: To implement the alternating optimization strategy, the blur kernel k i is estimated with the intermediate latent image L fi fixed, which specifically includes the following steps:

[0037] S7.1: Initial estimation. First, without considering the linear constraint f(k fi ), the blur kernel kfi0 Make a preliminary estimate, and then apply the linearization constraint to k fi0 ; fi

[0038] S7.2: Unconstrained optimization modeling, the objective function to be solved without the linearization constraint f(k fi ) is shown in Equation (14). This equation is also a least squares problem, and k is solved using Equation (15) fi0 , but the k solved by Equation (14) fi0 may contain negative numbers. Therefore, negative numbers need to be set to 0, and k fi0 is normalized so that the sum of all elements of k fi0 is 1;

[0039]

[0040] S7.3: Modeling with constraints to obtain the fuzzy kernel k fi0 , and based on the weighted least squares method, the geometric feature parameters of the corresponding linear fuzzy kernel k fi0 are deduced, including the tilt angle θ fi and the length r i . Set the straight line corresponding to the fuzzy kernel as y = αx + β, where α = tanθ i , and in this process, the solution of the fuzzy kernel under the linear constraint condition follows Equation (16): i where X = [x, ones], x is the vector composed of the abscissas of each point of the fuzzy kernel, ones is the vector with the same number of elements as x and all taking the value of 1, y is the vector composed of the ordinates of each point of the fuzzy kernel, l is the vector [α, β]

[0041]

[0042] , and W is the diagonal matrix composed of the coordinate point values corresponding to the fuzzy kernel k T ; fi0

[0043] S7.4: Equation (16) is a convex quadratic optimization problem, and the global optimal solution can be obtained by Equation (17):

[0044] l = (X T WX) -1 W(X T WX) (17)

[0045] S7.5: For the solution of the fuzzy radius, project each point of the fuzzy kernel onto the preset straight line, and the projection length segment of the fuzzy kernel on the straight line is used as the length r of the fuzzy kernel i ;

[0046] ​​S8: Iterate the obtained initial blur kernel of the image block by repeating the above steps, and finally determine the blur matrix of each image block;

[0047] S8.1: Region parameter integration. After obtaining the blur kernels corresponding to each image block, integrate the blur kernels to form a blur matrix K according to the image blocks to which each pixel belongs;

[0048] S8.2: Matrix construction. To form the blur matrix, first input the image size information (M, N), and set a zero matrix with the same image size (M, N) as the current blur kernel matrix. Fill the corresponding blur kernel parameters according to the image block index to which pixel i belongs in the set of blur kernels {k f}, and solve the blur matrix K. It should be noted that the blur matrix K is sparse, which can effectively reduce the number of parameters and improve the efficiency;

[0049] S9: Establish a noise separation model that divides the noise n into two parts. Traditional deblurring often ignores this problem, so the denoising effect is poor. To improve this problem, solve the noise part;

[0050] n = n l + n h (18)

[0051] In the formula, n l and n h are low-frequency noise and high-frequency noise respectively;

[0052] S10: Smooth the noise of the low-frequency layer using Gaussian filtering, and remove the high-frequency noise of the high-frequency layer using wavelet transform;

[0053] S10.1: To achieve low-frequency noise suppression, construct a low-pass filter. The principle of the Gaussian low-pass filter is that the value of each pixel point in the image is replaced by the weighted average value of its neighboring pixels, and the weighting coefficient is determined by the Gaussian distribution. The specific formula of the Gaussian two-dimensional convolution kernel is shown in Equation (19):

[0054]

[0055] In the formula, δ is the standard deviation of the Gaussian distribution, which determines the degree of filtering smoothness, and (x, y) represents the relative coordinates to the center of the kernel;

[0056] S10.2: Parameter configuration. Select the kernel size of Gaussian filtering and set it to 5×5. At the same time, set the standard deviation δ to 3 and the center point to (2, 2);

[0057] S10.3: Calculate the weights through Equation (19), add up all the terms to get the sum S, and then divide each value by the sum S to perform normalization processing;

[0058] S11: While performing low - frequency filtering, use wavelet transform for high - frequency filtering, and select the appropriate wavelet basis function Haar function;

[0059] S11.1: Use discrete wavelet transform (DWT) to decompose the image into different frequency components, usually four parts (LL, LH, HL, HH), namely low - frequency - low - frequency, low - frequency - high - frequency, high - frequency - low - frequency, and high - frequency - high - frequency. After decomposition, process the components separately;

[0060] S11.2: Perform thresholding on the high - frequency components. Coefficients with values less than the threshold are directly set to 0, and the remaining coefficients are shrunk. The threshold formula is as follows in Equation (20):

[0061]

[0062] In the formula, σ is the standard deviation of the noise, and N is the total number of image pixels;

[0063] S11.3: Use inverse wavelet transform to reconstruct the modified wavelet components into an image with high - frequency noise removed;

[0064] S11.4: Use weighted multi - scale fusion to remove the image after high - frequency and low - frequency noise;

[0065] S12: Remove blurring layer by layer. Perform deconvolution operation on images at different scales, and perform de - blurring according to the already obtained blurring kernel matrix. Use Wiener filtering. The specific formula is as follows in Equation (21):

[0066]

[0067] In the formula, K * is the conjugate of the blurring kernel, and N / S is the power ratio of noise to signal;

[0068] S13: Through pyramid inverse synthesis operation, fuse the restoration results of each layer to generate the final high - definition image. The present invention has the following beneficial effects:

[0069] 1. Traditional algorithms have poor effects in dealing with non - uniform blurring. A denoising and non - uniform blurring image enhancement method based on multi - scale pyramid is proposed. Perform deconvolution on the image blurring kernel solved layer by layer based on the multi - scale pyramid. This method can still be well optimized when the picture has non - uniform blurring, and is more adaptable to images of different sizes. The PSNR of the images optimized by this invention in the GoPro dataset is 2.74 higher than that of the Deblur - GANv2 algorithm optimizing the images in this dataset.

[0070] 2. Denoising is also an important part of image deblurring. In the traditional algorithm for non-uniform blur removal, the influence of low-frequency and high-frequency noise is not considered. Therefore, considering the noise part, in the present invention, by introducing a Gaussian multi-scale pyramid, low-frequency and high-frequency denoising of the image is performed on the basis of the pyramid, optimizing the problem of poor denoising effect caused by only low-frequency denoising of noise in the traditional deblurring method. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0072] Figure 1 It is the overall flowchart of the solution of the present invention;

[0073] Figure 2 It is the PSNR comparison chart between the present invention and the Deblur-GANv2 method;

[0074] Figure 3 It is the actual experimental environment;

[0075] Figure 4 It is the data picture used in this simulation experiment;

[0076] Figure 5 It is the effect picture of deblurring using the GANv2 algorithm;

[0077] Figure 6 It is the deblurring effect picture of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0078] To make the above objects, features, and advantages of the present invention more obvious and understandable, a denoising and non-uniform blur removal image enhancement method based on a multi-scale pyramid, the overall structure is as Figure 1 shown, including the following steps:

[0079] S1: Considering that the direct deconvolution deblurring has poor adaptability to the picture scale, after converting the input original image into a grayscale image, a multi-scale feature pyramid is innovatively established, and deblurring and denoising processing are performed on the basis of the pyramid. The number of layers Q of the pyramid is determined by the image size and the scale of the description feature points, and its calculation formula is shown as follows:

[0080] Q = [log b (min(w, h)) - log b (patch_size · 4)] (1)

[0081] Wherein, b is the downsampling factor, w and h are the width and height of the image respectively, and patch_size is the scale for describing feature points;

[0082] S2: Segment the blurred region of the lowest-resolution RGB image in the pyramid by the method of natural image mask segmentation. The motion-blurred image can be segmented into a motion-blurred layer and a non-motion-blurred layer. The specific calculation formula is as follows:

[0083] B = KL f +(1 - Km)B b +n (2)

[0084] Wherein, B, L f and B b are the observable blurred image, the clear moving object region and the non-moving object region respectively, m and n are the binary mask of the moving object region and noise, and K is the blur matrix;

[0085] S3: Implement adaptive mask segmentation based on foreground and background region division, and then estimate the non-binary mask m f , the motion-blurred layer B f and the non-motion-blurred layer B b . The spatially-varying blur model and the image restoration model can be expressed as Equation (3) and Equation (4) respectively:

[0086] B f = KL f +n (3)

[0087] L = L f +(1 - m)B b (4)

[0088] S3.1: By obtaining the blur direction of the blur kernel and the image blur features in the orthogonal dimension, the blur features of each pixel under the known blur kernel can be obtained;

[0089] S3.2: Use the TRWS algorithm to solve the pixel-level blur kernel label, realize the identification and classification of multi-directional blur regions, and record the blur length and angle information of each region. The blur length is 0 ≤ l ≤ l max , and the angle is -90° ≤ θ < 90°;

[0090] S3.3: Label the pixels with zero blur kernel labels as 0 and the others as 1. This is used as the initial blur region marking result. For the existing mislabeling phenomenon, a multi-level optimization strategy is adopted to remove the moving object regions with a small number of pixels, and optimize through the methods of opening operation, closing operation and guided filtering. Finally, the moving region marking result can be obtained;

[0091] S4: Estimate the linear blur kernel of the blurred image block. To avoid the blur kernel falling into a trivial solution during the initial iteration, a constraint term is set to guide the blur kernel estimation process. The objective function for blur kernel estimation is as shown in Equation (5):

[0092]

[0093] In the formula, L i is the clear image during the solution process, f(k fi ) is the linear constraint of the blur kernel, ||k fi || and γ are the constraint term and the constraint coefficient respectively;

[0094] S5: Given the complexity of the theoretical solution of Equation (5), a non-blind method is used for alternative solution, and the corresponding objective functions are as shown in Equation (6) and Equation (7):

[0095]

[0096] S6: Based on the objective functions in Equation (6) and Equation (7), an alternative optimization method is used to iteratively solve the blur kernel and the intermediate latent image;

[0097] S6.1: First, on the premise that the blur kernel k fi remains unchanged, solve the intermediate latent image L i . To this end, introduce an auxiliary variable g = (g h , g v ) Τ Reconstruct Equation (6) to convert it into an optimization problem involving L i and g;

[0098]

[0099] S6.2: Subsequently, fix the auxiliary variable g and use the least squares optimization method to iteratively solve L i multiple times. Its mathematical model is as shown in Equation (9):

[0100]

[0101] S6.3: When solving L i in Equation (9), use the fast Fourier transform to solve, and the corresponding closed-form solution is as shown in Equation (10):

[0102]

[0103] In the formula, F() and F -1 () represent the fast Fourier transform and the inverse fast Fourier transform respectively; denotes the complex conjugate operation; ▽ h and ▽ v represent the difference operations in the horizontal and vertical directions respectively;

[0104] S6.4: When k is fixed fi In the case of , the variable g can be obtained by solving formula (12). Formula (12) is a pixel-by-pixel minimization problem, and its solution can be derived by equation, as shown in formula (13):

[0105]

[0106] S7: In order to realize the alternating optimization strategy, the intermediate latent image L is fixed. i In the case of blur kernel k fi The estimation process includes the following steps:

[0107] S7.1: Initial estimation, first ignore the linear constraints f(k fi ) for the blur kernel k fi0 Make a preliminary estimate and then fi0 Apply linearization constraint k fi ;

[0108] S7.2: Unconstrained optimization modeling, without linearization constraints f(k fi ) is shown in formula (14), which is also a least squares problem. Formula (15) is used to solve k fi0 , but k obtained by solving equation (14) fi0 It may contain negative numbers, so we need to set the negative numbers to 0 and fi0 Normalize so that k fi0 The sum of all elements is 1;

[0109]

[0110] S7.3: Modeling with constraints to obtain the fuzzy kernel k fi0 , k is calculated based on the weighted least squares method fi0 The corresponding linear blur kernel k fi The geometric characteristic parameters include the tilt angle θ i With length r i , set the straight line corresponding to the blur kernel to y = αx + β, where α = tanθ i In this process, the fuzzy kernel solution under linear constraints follows formula (16):

[0111]

[0112] Where X = [x, ones], x is the vector of the horizontal coordinates of the blur kernel points, ones is a vector with the same number of elements as x and all of them are 1, y is the vector of the vertical coordinates of the blur kernel points, and l is the vector [α, β] T, where \(W\) is the blur kernel \(k\) fi0 The diagonal matrix formed by the coordinate point values corresponding to

[0113] S7.4: Equation (16) is a convex quadratic optimization problem, and the global optimal solution can be obtained from Equation (17):

[0114] \(l=(X\) T \(WX)\) -1 \(W(X\) T \(WX)(17)\)

[0115] S7.5: For the solution of the blur radius, project each point of the blur kernel onto the preset line, and obtain the projection length segment of the blur kernel on the line as the length \(r\) of the blur kernel i ;

[0116] S8: Iterate the obtained initial blur kernel of the image block by repeating the above steps, and finally determine the blur matrix of each image block

[0117] S8.1: Region parameter integration. After obtaining the blur kernels corresponding to each image block, integrate the blur kernels to form the blur matrix \(K\) according to the image blocks to which each pixel belongs

[0118] S8.2: Matrix construction. To form the blur matrix, first input the image size information \((M, N)\), and set a zero matrix with the same image size \((M, N)\) as the current blur kernel matrix. Fill the corresponding blur kernel parameters according to the image block index to which pixel \(i\) belongs in the set of blur kernels \(\{k\) f \}\), and solve the blur matrix \(K\). It should be noted that the blur matrix \(K\) is sparse, which can effectively reduce the number of parameters and improve the efficiency

[0119] S9: Establish a noise separation model that divides the noise \(n\) into two parts. Traditional deblurring often ignores this problem, so the denoising effect is poor. To improve this problem, solve the noise part

[0120] \(n = n\) l \(+\ n\) h \((18)\)

[0121] In the formula, \(n\) l and \(n\) h are the low-frequency noise and high-frequency noise respectively

[0122] S10: Smooth the low-frequency layer with Gaussian filtering and remove the high-frequency noise from the high-frequency layer using wavelet transform

[0123] S10.1: To achieve low-frequency noise suppression, construct a low-pass filter. The principle of the Gaussian low-pass filter is that the value of each pixel point in the image is replaced by the weighted average value of its neighboring pixels, and the weighting coefficient is determined by the Gaussian distribution. The specific formula of the Gaussian two-dimensional convolution kernel is shown in Equation (19):

[0124]

[0125] In the formula, δ is the standard deviation of the Gaussian distribution, which determines the filtering smoothness, and (x, y) represents the relative coordinates to the kernel center;

[0126] S10.2: Parameter configuration, select the kernel size of Gaussian filtering and set it to 5×5, and at the same time set the standard deviation δ to 3 and the center point to (2, 2);

[0127] S10.3: Calculate the weights through formula (19), add up all the terms in the formula to get the sum S, and then divide each value by the sum S to perform the normalization process;

[0128] S11: While performing low-frequency filtering, use wavelet transform for high-frequency filtering, and select the appropriate wavelet basis function Haar function;

[0129] S11.1: Use the discrete wavelet transform (DWT) to decompose the image into different frequency components, usually four parts (LL, LH, HL, HH), namely low-frequency - low-frequency, low-frequency - high-frequency, high-frequency - low-frequency, and high-frequency - high-frequency. After decomposition, process the components separately;

[0130] S11.2: Perform thresholding on the high-frequency components. Coefficients with values less than the threshold are directly set to 0, and the remaining coefficients are shrunk. The threshold formula is as follows in formula (20):

[0131]

[0132] In the formula, σ is the standard deviation of the noise, and N is the total number of image pixels;

[0133] S11.3: Use the inverse wavelet transform to reconstruct the modified wavelet components into a denoised image;

[0134] S11.4: Use weighted multi-scale fusion to remove the image with high-frequency and low-frequency noises;

[0135] S12: Layer-by-layer deblurring, perform deconvolution operation on images at different scales, and deblur according to the already obtained blur kernel matrix, using Wiener filtering. The specific formula is as follows in formula (21):

[0136]

[0137] In the formula, K * is the conjugate of the blur kernel, and N / S is the power ratio of the noise to the signal;

[0138] S13: Through the pyramid reverse synthesis operation, fuse the restoration results of each layer to generate the final high-definition image.

[0139] Numerical verification of the algorithm in this time, including the effect verification of hierarchical deblurring pyramid denoising, the specific steps are as follows:

[0140] The experimental software environment is ubuntu18.04 + vscode + python, and the hardware environment is AMD Ryzen5 3600 + NVIDA GeForce RTX 1060 + 8GB memory. The comparison of the image performance evaluation of each algorithm in the GoPro dataset is shown in Table 1, and the comparison of the performance after optimizing the self-made pictures is shown in Table 2.

[0141] Table 1 Comparison of average performance evaluation after GoPro optimization

[0142]

[0143] Table 2 Comparison of performance evaluation after optimizing the test pictures

[0144]

[0145] In this experiment, 1756 pictures were taken by the manipulator of the unmanned vehicle equipped with a depth camera as shown in Figure 2 for verification. Figure 3 The line chart fully shows the comparison of deblurring data between the present invention and the DeblurGANv2 algorithm, and the present invention has a better improvement at the data level. Select a test picture as shown in Figure 4 , and the height and width of the picture are 1120 and 1494 respectively. At the same time, the improved effect of introducing the DeblurGANv2 method and hierarchical solution and adopting pyramid denoising is verified by means of a comparative experiment.

[0146] Figure 5 The effect of the DeblurGANv2 algorithm on image deblurring is shown. Figure 5 It shows that the deblurring effect of the DeblurGANv2 algorithm is poor and the PSNR value is small. The effect of the present invention is as shown in Figure 6 . After adopting the hierarchical deblurring and pyramid denoising methods, the deblurring effect is significantly enhanced. Through the detailed analysis of the red solid line frame and blue dotted line frame areas in Figure 4 , Figure 5 and Figure 6 , obvious differences in the deblurring effect of different algorithms can be clearly observed. The image part in the red solid line frame mainly contains the part with rich texture in the image, so there is a more obvious contrast in the effect of removing uneven blur. The deblurring effect of the present invention is better than that of the DeblurGANv2 algorithm. The image part in the blue dotted line frame is relatively smooth, and the comparison of the effect of removing noise is more obvious. The present invention has a significant improvement in the noise suppression effect, and the smooth area is cleaner.

[0147] The specific implementation manners described above further illustrate the invention object and technical solution of the present invention. The above embodiments are only used to illustrate the technical solution of the present invention, rather than limiting the protection scope of the present invention. Those of ordinary skill in the art should understand that any modification and equivalent replacement made to the technical solution of the present invention are included in the protection scope of the present invention.

Claims

1. A denoising and uneven blur removal image enhancement method based on a multi-scale pyramid, characterized in that It includes the following steps: S1: Considering that the direct deconvolution for defocusing has poor adaptability to the image scale, after converting the input original image into a grayscale image, a multi-scale feature pyramid is innovatively established, and defocusing and denoising processing are performed on the basis of the pyramid. The number of layers Q of the pyramid is determined by the image size and the scale of the feature points for description, and its calculation formula is shown as follows: Q = [log b (min(w, h)) - log b (patch_size·4)] (1) In the formula, b is the downsampling coefficient, w and h are the width and height of the image respectively, and patch_size is the scale for describing the feature points; S2: The RGB image with the lowest resolution in the pyramid is used to segment the defocused area by the method of natural image mask segmentation. The motion defocused image can be segmented into a motion defocused layer and a non-motion defocused layer. The specific calculation formula is as follows: B = KL f +(1 - Km)B b +n(2) where B, L f and B b are respectively the observable blurred image, the clear moving object region, and the non-moving object region, m and n are the binary mask and noise of the moving object region, and K is the blur matrix; S3: Implement adaptive mask segmentation based on foreground and background region division, and then estimate the non-binary mask m f , the motion blur layer B f and the non-motion blur layer B b , the spatially-varying blur model and the image restoration model can be respectively expressed as equations (3) and (4): B f = KL f + n(3) L = L f +(1 - m)B b (4) S3.1: By obtaining the defocus kernel direction and the image defocus features in the orthogonal dimension, the defocus features of each pixel under the known defocus kernel can be obtained; S3.2: Solve the pixel-level blur kernel labels using the TRWS algorithm to identify and classify multi-directional blurred regions, and record the blur length and angle information of each region, where the blur length is 0 ≤ l ≤ l max , and the angle is -90° ≤ θ < 90°; S3.3: Label the pixels with zero defocus kernel tags as 0, and assign other values as 1. This is used as the initial defocus area marking result. For the existing mislabeling phenomenon, a multi-level optimization strategy is adopted to remove the moving object areas with a small number of pixels, and it is optimized by the methods of opening operation, closing operation and guided filtering. Finally, the moving area marking result can be obtained; S4: Estimate the linear defocus kernel of the defocused image patch. In order to avoid the defocus kernel falling into the trivial solution during the initial iteration, a constraint term is set to guide the defocus kernel estimation process. The defocus kernel estimation objective function is as shown in Equation (5): where L i is the clear image in the solution process, f(k fi ) is the linear constraint of the blur kernel, ||k fi || and γ are the constraint term and the constraint coefficient, respectively; S5: Considering the complexity of the theoretical solution of Equation (5), a non-blind method is used for alternating solution, and the corresponding objective functions are as shown in Equation (6) and Equation (7): S6: Based on the objective functions of Equation (6) and Equation (7), an alternating optimization method is used to iteratively solve the defocus kernel and the intermediate latent image; S6.1: First, under the premise that the blur kernel k fi remains unchanged, solve the intermediate latent image L i . To this end, introduce an auxiliary variable g = (g h , g v ) Τ Reconstruct Equation (6) to transform it into an optimization problem involving L i and g; S6.2: Subsequently, fix the auxiliary variable g and use the least squares optimization method to iteratively solve for L multiple times i , and its mathematical model is as shown in Equation (9): S6.3: When solving for L in Equation (9), use the fast Fourier transform to solve, and the corresponding closed-form solution is shown in Equation (10): i when solving, use the fast Fourier transform, and the corresponding closed-form solution is shown in Equation (10): where F() and F -1 () represent the fast Fourier transform and the inverse fast Fourier transform, respectively; denotes the complex conjugate operation; and denote the difference operations in the horizontal and vertical directions, respectively; S6.4: When fixing k fi the variable g can be obtained by solving Equation (12), which is a per-pixel minimization problem and its solution can be derived through equation derivation, specifically shown in Equation (13): S7: To implement the alternating optimization strategy, estimate the blur kernel k i while fixing the intermediate latent image L fi This specifically includes the following steps: S7.1: Initial estimation. First, the initial estimation of the blur kernel k is performed without considering the linear constraint f(k fi ). After that, the linearization constraint k fi0 is imposed on k fi0 ; fi ​ S7.2: Unconstrained optimization modeling, without linearization constraint f(k fi ). The objective function to be solved is shown in Equation (14), which is also a least squares problem. Solve for k using Equation (15) fi0 , but the k solved by Equation (14) fi0 may contain negative numbers. Therefore, negative numbers need to be set to 0, and k fi0 is normalized so that the sum of all elements of k fi0 is 1; S7.3: Modeling with constraint terms to obtain the fuzzy kernel k fi0 , and k is deduced based on the weighted least squares method fi0 The corresponding linear fuzzy kernel k fi The geometric feature parameters of, including the tilt angle θ i and the length r i , set the straight line corresponding to the fuzzy kernel y = αx + β, where α = tanθ i , in this process, the solution of the fuzzy kernel under the linear constraint condition follows Equation (16): where \(X = [x, ones]\), \(x\) is the vector formed by the abscissas of each point of the blur kernel, \(ones\) is the vector with the same number of elements as \(x\) and all elements are 1, \(y\) is the vector formed by the ordinates of each point of the blur kernel, and \(l\) is the vector \([\alpha, \beta]\). T , \(W\) is the blur kernel \(k\). fi0 The diagonal matrix formed by the values of the corresponding coordinate points. S7.4: Equation (16) is a convex quadratic optimization problem, and the global optimal solution can be obtained from Equation (17): l = (X T WX) -1 W(X T WX) (17) S7.5: For the solution of the blur radius, project each point of the blur kernel onto the preset line, and use the projected length segment of the blur kernel on the line as the length r of the blur kernel i ; S8: Repeat the above steps iteratively for the obtained initial defocus kernel of the image patch, and finally determine the defocus matrix of each image patch; S8.1: Region parameter integration. After obtaining the defocus kernels corresponding to each image patch, the defocus kernels are integrated to form the defocus matrix K according to the image patches to which each pixel belongs; S8.2: Matrix construction. To form a fuzzy matrix, first input the image size information (M, N), and set a zero matrix with the same image size (M, N) as the current fuzzy kernel matrix. Fill the corresponding fuzzy kernel parameters of the fuzzy kernel set {k f} according to the image block index to which the pixel i belongs, and solve the fuzzy matrix K. It should be noted that the fuzzy matrix K is sparse, which can effectively reduce the number of parameters and improve the efficiency; S9: Establish a noise separation model that divides the noise n into two parts. Traditional defocusing often ignores this problem, so the denoising effect is poor. To improve this problem, the noise part is solved; n = n l +n h (18) where n l and n h are low-frequency noise and high-frequency noise, respectively; S10: Smooth the noise in the low-frequency layer using Gaussian filtering, and remove the high-frequency noise in the high-frequency layer using wavelet transform; S10.1: To achieve low-frequency noise suppression, a low-pass filter is constructed. The principle of the Gaussian low-pass filter is that the value of each pixel point in the image is replaced by the weighted average value of its neighboring pixels, and the weighting coefficient is determined by the Gaussian distribution. The specific formula of the Gaussian two-dimensional convolution kernel is shown as Equation (19): In the formula, δ is the standard deviation of the Gaussian distribution, which determines the filtering smoothness, and (x, y) represents the relative coordinates with respect to the center of the kernel; S10.2: Parameter configuration. Select the kernel size of Gaussian filtering and set it to 5×5, and at the same time set the standard deviation δ to 3, and the center point is (2, 2); S10.3: Calculate the weights through Equation (19), add up all terms to obtain the sum S, and then divide each value by the sum S for normalization processing; S11: While performing low-frequency filtering, use wavelet transform for high-frequency filtering, and select the appropriate wavelet basis function Haar function; S11.1: Use discrete wavelet transform (DWT) to decompose the image into different frequency components, usually four parts (LL, LH, HL, HH), namely low-frequency - low-frequency, low-frequency - high-frequency, high-frequency - low-frequency, and high-frequency - high-frequency. After decomposition, process the components separately; S11.2: Perform thresholding on the high-frequency components. Coefficients with values less than the threshold are directly set to 0, and the remaining coefficients are shrunk. The threshold formula is as follows in Equation (20): In the formula, σ is the standard deviation of the noise, and N is the total number of image pixels; S11.3: Use inverse wavelet transform to reconstruct the modified wavelet components into a denoised image; S11.4: Use weighted multi-scale fusion to remove the high-frequency and low-frequency noise from the image; S12: De-blur layer by layer, perform deconvolution operations on images at different scales, and de-blur according to the already obtained blur kernel matrix, using Wiener filtering. The specific formula is as in Equation (21): where K * is the conjugate of the blur kernel, and N / S is the power ratio of noise to signal; S13: Through pyramid reverse synthesis operation, fuse the restoration results of each layer to generate the final high-definition image.

Images