Edge-aware image processing method based on L2 regularized optimization model

Abstract

The present invention discloses an edge-aware image processing method based on an L2 regularized optimization model. Given an input image and setting parameters as required, the method first performs edge-aware gradient processing on the image to obtain a new image gradient, then substitutes the new image gradient into the L2 loss function, and finally minimizes the L2 loss function to obtain a reconstructed image. In order to perform edge perception on the image, the present invention proposes a set of mapping functions, which provide well-defined nonlinear mappings between input and output gradients, so that the proposed method is suitable for various applications in the field of computer graphics. Experimental results show that the method of the present invention suppresses halo, gradient reversal, and intensity drift artifacts while ensuring the image processing speed, and achieves good edge-aware image processing effects.

Description

Technical Field

[0001] The present invention belongs to the technical field of computational photography, and specifically is an edge-aware image processing method based on an L2 regularization optimization model. Background Art

[0002] Edge-aware image processing is an important topic in computational photography. Images can be divided into many different parts according to the edge position, giving us image information. Therefore, the edge is the most important for the image. The limitation of many image processing methods is that the edge structure will be destroyed when processing the image, resulting in incomplete image information. In order to solve this problem, edge-aware image processing was proposed. The purpose of edge-aware image processing is to retain the effective edge information of the image while effectively performing various processing on the image.

[0003] Edge-aware image processing methods are mainly divided into traditional methods and methods based on deep learning. Traditional methods include local methods and global methods. Local methods mainly output pixel values ​​by calculating pixel values ​​in a local window, such as bilateral filter BLF, guided filter GIF, local Laplace method LLF, etc. The advantage is that most local methods are very fast, but some methods have disadvantages such as gradient reversal and halo. Global methods usually use optimization frameworks to adjust image pixels, such as L0 method, weighted least squares filter WLS, etc. The advantage is that gradient reversal and halo phenomena are suppressed, but the disadvantage is that the speed is very slow and sometimes intensity drift artifacts appear. Deep learning methods achieve similar image effects to traditional methods by training machines, such as deep edge-aware filter DEAF. Deep methods are fast, but have the disadvantages of lack of flexibility and inability to make partial modifications. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention proposes an edge-aware image processing method based on an L2 regularized optimization model, which can effectively suppress the halo and gradient reversal artifacts of the local edge-aware image processing method while ensuring the running speed, and can effectively suppress the intensity drift artifacts of the existing global edge-aware image processing method, and is more flexible than the deep learning method. The present invention can not only quickly process images, but also well preserve the edge structure in the image, and effectively suppress the three artifacts of halo, gradient reversal and intensity drift.

[0005] The present invention proposes an edge-aware image processing method based on an L2 regularized optimization model, comprising the following steps:

[0006] Step 1, set the gradient processing coefficient γ, threshold σ and penalty coefficient β, and select the input image g;

[0007] Step 2: perform edge-aware gradient processing on the image, which is divided into two types: gradient compression and expansion, corresponding to HDR tone mapping and detail enhancement respectively;

[0008] Step 3, substitute the processed image gradient into the L2 loss function;

[0009] Step 4, minimize the L2 loss function, reconstruct the input image and output the image to obtain the final image processing result.

[0010] Furthermore, the mapping function used in the edge-aware gradient processing described in step 2 is:

[0011]

[0012]

[0013] in represents the gradient of the input image, Indicates the maximum gradient value in the input image. Indicates the positive or negative gradient. σ indicates the threshold value, which is used to divide the gradient processing range. γ indicates the degree of gradient processing. In γ, γ>1 is usually taken, and The γ in is usually set to γ<1.

[0014] Furthermore, two formulas are used to compress and expand the two image gradients respectively. Compressing and expanding the gradients are usually not performed at the same time, corresponding to two different applications of HDR tone mapping and detail enhancement.

[0015] Furthermore, when applying HDR tone mapping, when the image gradient value is less than the set threshold σ, using the formula Compress the image gradient; when the image gradient value If it is greater than or equal to the set threshold σ, the image gradient remains unchanged.

[0016] Furthermore, when doing detail enhancement applications, when the image gradient value is greater than the set threshold σ, using the formula Expand the image gradient; when the image gradient value If it is less than or equal to the set threshold σ, the image gradient remains unchanged.

[0017] Furthermore, the L2 loss function described in step 3 is:

[0018]

[0019] Where u represents the output image, g represents the input image, and s represents the specific pixel position. and Represent the gradient of the output image in the x and y directions respectively. and Respectively represent the gradients in the x and y directions after processing. β represents the penalty coefficient, which controls the data item (u s -g s ) 2 and penalty items When β is larger, the output image gradient is closer to the processed gradient. When β is zero, the input image is the output image.

[0020] Furthermore, the Fast Fourier Transform (FFT) method is used to solve the problem, converting all complex calculations in matrix form into simple point operations.

[0021] Furthermore, the method for calculating the minimum value of the L2 loss function is:

[0022] First, rewrite the L2 loss function as:

[0023]

[0024] Among them, F x and F y are the gradient operators in the x and y directions respectively. v 、u v Represents the vector form of input image pixels and output image pixels respectively. x u v 、F y u v Represents the matrix form of solving the gradient in the x-direction and y-direction of the output image respectively. It indicates the vector form of the gradient in the x and y directions after processing.

[0025] Then, taking the derivative of the formula and setting it to zero, we get u v The solution formula is:

[0026]

[0027] in and Respectively represent F x and F y The transposed matrix of , I represents the identity matrix.

[0028] Finally, rewrite it into the fast Fourier transform (FFT) solution form, and then get the solution formula of L2 loss function:

[0029]

[0030] Where F(·) and F -1(·) denote the fast Fourier transform (FFT) and inverse fast Fourier transform (IFFT) operators, respectively. represents the complex conjugate of F(·). F(1) represents the FFT of the Dirac function. Indicates the gradient solution in the x and y directions, g x , g y It means that the gradients in the x and y directions are obtained after processing.

[0031] The beneficial effects of the present invention are:

[0032] The present invention proposes an edge-aware image processing method based on an L2 regularization optimization model, wherein according to the sensitivity of the gradient to edge perception in the image, the present invention uses a set of mapping functions to change the image gradient. Even when processing large images, the present invention has a faster running speed than other global image processing methods, and is even faster than some local image processing methods and deep learning methods. For example, it only takes 0.46 seconds to process a 1000 pixel × 1000 pixel color RGB image on a machine with an Intel i7-6700 CPU, Nvidia Geforce 1070 GPU, and 16G memory. At the same time, in the two applications of image detail enhancement and HDR tone mapping, the processing effect is greatly improved compared to other local image processing methods, and has good performance in suppressing halos, gradient reversals, and reducing intensity drift artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention.

[0034] Figure 2 2(a) is the original image; 2(b) is the image processed by the present invention, with parameters σ=0, γ=0.65, β=100; 2(c) is the result of guided filter GIF image processing, with parameters window radius r=12, regularization parameter ∈=0.3 2 , we can see that there are obvious halo artifacts; 2(d) is the result of image processing by the weighted least squares filter WLS, with the parameters of mapping scale α=0.8 and penalty parameter λ=3 between data term and smoothness term, we can see that there are intensity drift artifacts; 2(e) is the result of image processing by the L0 method, with the parameters of λ=0.007, we can see that there are serious gradient reversal artifacts; 2(f) is the result of image processing by the multi-scale context aggregation network filter CAN, we can see that there are gradient reversal artifacts.

[0035] Figure 33(a) is the original image; 3(b) is the image processed by the present invention, with parameters σ=0.5, γ=2, β=500; 3(c) is the image processed by the bilateral filter BLF, with parameters σ s =8, range bandwidth σ r = 0.3, we can see obvious halo artifacts; 3(d) is the result of image processing by gradient domain guided image filter GDGIF, with parameters r = 8, ∈ = 0.3 2 , we can see that light and shadow artifacts appear; 3(e) is the result of bilateral filtering-least squares filter BLE-LS image processing, the parameter is σ s =8,σ r =0.006, λ=500, we can see the presence of halo artifacts; 3(f) is the result after processing by the multi-scale context aggregation network filter CAN, we can see the presence of redundant strip textures. DETAILED DESCRIPTION

[0036] The present invention proposes an edge-aware image processing method based on an L2 regularized optimization model. Given an input image and setting parameters as required, the method first performs edge-aware gradient processing on the image to obtain a new image gradient, then substitutes the new image gradient into the L2 loss function, and finally minimizes the L2 loss function to obtain a reconstructed image. In order to perform edge perception on the image, the present invention proposes a set of mapping functions that provide well-defined nonlinear mappings between input and output gradients, making the proposed method suitable for various applications in the field of computer graphics. Experimental results show that the method of the present invention suppresses halo, gradient reversal, and intensity drift artifacts while ensuring image processing speed, and achieves good edge-aware image processing effects.

[0037] In order to further highlight the purpose, technical solutions and advantages of the present invention, the present invention is further clearly and completely described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0038] like Figure 1 As shown, the edge-aware image processing method based on the L2 regularization optimization model proposed in the present invention has the following specific process:

[0039] Step 1: Set the gradient processing coefficient γ, threshold σ and penalty coefficient β, select the original image g to be processed, and calculate the gradient of the image in the x and y directions. and

[0040] Step 2: Process the image edge-aware gradient, which is divided into two types: gradient compression and expansion, corresponding to HDR tone mapping and detail enhancement respectively:

[0041] The mapping function of the edge-aware gradient processing method is:

[0042]

[0043]

[0044] in represents the gradient of the input image, Indicates the maximum gradient value in the input image. Indicates the positive or negative gradient. σ indicates the threshold value, which is used to divide the gradient processing range. γ indicates the degree of gradient processing. In γ, γ>1 is usually taken, and The γ in is usually set to γ<1,

[0045] For the above mapping function, two formulas are used to compress and expand the two image gradients. Compressing and expanding gradients are usually not performed at the same time, corresponding to two different applications: tone mapping and detail enhancement.

[0046] When doing HDR tone mapping application, when the image gradient value is less than the set threshold σ, using the formula Compress the image gradient; when the image gradient value If it is greater than or equal to the set threshold σ, the image gradient remains unchanged.

[0047] When doing detail enhancement application, when the image gradient value is greater than the set threshold σ, using the formula Expand the image gradient; when the image gradient value If it is less than or equal to the set threshold σ, the image gradient remains unchanged.

[0048] Step 3: Substitute the processed image gradient into the L2 loss function:

[0049] The L2 loss function is:

[0050]

[0051] Where u represents the output image, g represents the input image, and s represents the specific pixel position. and Represent the gradient of the output image in the x and y directions respectively. and Respectively represent the gradients in the x and y directions after processing. β represents the penalty coefficient, which controls the data item (us -g s ) 2 and penalty items When β is larger, the output image gradient is closer to the processed gradient. When β is zero, the input image is the output image.

[0052] Step 4: Minimize the L2 loss function and reconstruct the input image. The gradient value obtained after processing is used as and Substitute into the L2 regularization optimization model and control the data term (u s -g s ) 2 and penalty items The proportion of is solved, the minimum value of L2 loss function is obtained, and the output result image u is the reconstructed image.

[0053] The method to calculate the minimum value of the L2 loss function is:

[0054] First, rewrite the L2 loss function as:

[0055]

[0056] Among them, F x and F y are the gradient operators in the x and y directions respectively. v 、u v Represents the vector form of input image pixels and output image pixels respectively. x u v 、F y u v Represents the matrix form of solving the gradient in the x-direction and y-direction of the output image respectively. It indicates the vector form of the gradient in the x and y directions after processing.

[0057] Then, taking the derivative of the formula and setting it to zero, we get u v The solution formula is:

[0058]

[0059] in and Respectively represent F x and F y The transposed matrix of , I represents the identity matrix.

[0060] Finally, rewrite it into the fast Fourier transform (FFT) solution form, and then get the solution formula of L2 loss function:

[0061]

[0062] Where F(·) and F -1 (·) denote the fast Fourier transform (FFT) and inverse fast Fourier transform (IFFT) operators, respectively. represents the complex conjugate of F(·). F(1) represents the FFT of the Dirac function. Indicates the gradient solution in the x and y directions, g x , g y It means that the gradients in the x and y directions are obtained after processing.

[0063] In addition, two reference-free image quality assessment (IQA) models, ILNIQE and SSEQ, are used to evaluate the comparative experiments. The image quality assessment models used are all such that the lower the score obtained by the image, the better the quality. Table 1 and Table 2 are the quality assessment scores of the results of the image processing method applied to image detail enhancement and HDR tone mapping, respectively. It can be concluded that the present invention has good results in both image detail enhancement and HDR tone mapping applications.

[0064] Table 1

[0065] The present invention GIF WLS <![CDATA[L0]]> CAN SSEQ 11.39 15.11 19.57 22.75 29.57 ILNIQE 27.52 31.44 29.78 32.04 30.57

[0066] Table 2

[0067] The present invention BLF GDGIF BLE-LS CAN SSEQ 14.88 19.90 15.02 15.71 19.22 ILNIQE 22.78 23.16 24.49 22.86 22.94

[0068] In the experiment, it only takes 0.46 seconds to process a 1000 pixel × 1000 pixel color RGB image on a machine with Intel i7-6700 CPU, Nvidia Geforce 1070 GPU, and 16G memory, which can meet the computational efficiency requirements of edge-aware image processing methods in real applications and is superior to many edge-aware image processing methods.

[0069] In summary, the method of the present invention includes the following steps: 1) setting the gradient processing coefficient γ, the threshold σ and the penalty coefficient β, and selecting the input image g; 2) performing edge-aware gradient processing on the image, which is divided into two types: gradient compression and expansion, corresponding to HDR tone mapping and detail enhancement, respectively; 3) substituting the processed image gradient into the L2 loss function; 4) minimizing the L2 loss function, reconstructing the input image and outputting the image to obtain the final image processing result.

[0070] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. An edge-aware image processing method based on an L2 regularized optimization model, characterized in that: The steps include: S1, set the gradient processing coefficient γ, threshold σ and penalty coefficient β, and select the input image g; S2, performs edge-aware gradient processing on the image, which is divided into gradient compression and expansion, corresponding to HDR tone mapping and detail enhancement respectively; S3, substitute the processed image gradient into the L2 loss function; S4, minimizing the L2 loss function, reconstructing the input image and outputting the image to obtain the final image processing result; The mapping function established during the edge-aware gradient processing described in step S2 is: in represents the gradient of the input image, Represents the maximum gradient value in the input image. Indicates the positive or negative nature of the gradient, σ indicates the threshold value, which is used to divide the gradient processing range, and γ indicates the degree of gradient processing; The loss function established by S3 is: Where u represents the output image, g represents the input image, and s represents the specific pixel position. and Respectively represent the gradient of the output image in the x direction and y direction, and They represent the gradients in the x and y directions after processing, β represents the penalty coefficient, and the control data item (u s -g s ) 2 and penalty items When β is larger, the output image gradient is closer to the processed gradient, and when β is zero, the input image is the output image. The method for calculating the minimum value of the L2 loss function in S4 is: First, rewrite the L2 loss function as: Among them, F x and F y are the gradient operators in the x and y directions respectively, g v 、u v Represents the vector form of input image pixels and output image pixels respectively, F x u v 、F y u v Represents the matrix form of solving the gradient in the x-direction and y-direction of the output image; Indicates the vector form of the gradient in the x and y directions after processing; Then, taking the derivative of the formula and setting it to zero, we get u v The solution formula is: in and Respectively represent F x and F y The transposed matrix of , I represents the identity matrix; Finally, rewrite it into the fast Fourier transform solution form, and then get the solution formula of L2 loss function: Where F(·) and F -1 (·) represent the fast Fourier transform and inverse fast Fourier transform operators, respectively, represents the complex conjugate of F(·), F(1) represents the FFT of the Dirac function, Indicates the gradient solution in the x and y directions, g x , g y It means that the gradients in the x and y directions are obtained after processing; The method of S4 for reconstructing the input image is as follows: The gradient value obtained after processing is used as and Substitute into the L2 regularization optimization model and control the data term (u s -g s ) 2 and penalty items The proportion of is solved, the minimum value of L2 loss function is obtained, and the output image u obtained is the reconstructed image.

2. The edge-aware image processing method based on the L2 regularization optimization model according to claim 1, characterized in that: In γ, γ>1, and Let γ<1.

3. The edge-aware image processing method based on the L2 regularization optimization model according to claim 1, characterized in that: The S2 uses the formulas for edge-aware gradient processing. and formula Compression and expansion are performed, and the compression and expansion of gradients are not performed at the same time, corresponding to two different applications of tone mapping and detail enhancement: When doing HDR tone mapping application, when the image gradient value is less than the set threshold σ, using the formula Compress image gradients; When the image gradient value If it is greater than or equal to the set threshold σ, the image gradient remains unchanged; When doing detail enhancement application, when the image gradient value is greater than the set threshold σ, using the formula The image gradient is enlarged; when the image gradient value If it is less than or equal to the set threshold σ, the image gradient remains unchanged.

4. The edge-aware image processing method based on the L2 regularization optimization model according to claim 1, characterized in that: The loss function is solved by fast Fourier transform, which converts all complex calculations in matrix form into simple point operations.

Images