A video adaptive sharpening method and system based on Huber function

Abstract

A method and system for adaptive sharpening of a video based on Huber function, the method comprising: S1: training at least parameters a, b, c of Huber piecewise function #imgabs0# through a picture data set; S2: calculating gradient information g on the Y component of the preprocessed image YCbCr data, and then normalizing the gradient information in the value range [0,255] to the interval [0,c] to complete the mapping of the gradient information to the three ranges of a, b, and c; S3: automatically adjusting the sharpening intensity of the output image according to the gradient information. Using the L1loss effective supervised learning method, first calculate y g , then gt g , then idx, and finally the L1loss. The present invention is a video adaptive sharpening algorithm based on Huber function, which can adaptively process a variety of video scenes, can meet the requirements of real-time processing, and can better highlight the edges and details of the image.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a video adaptive sharpening method and system based on Huber function. Background Art

[0002] For video sharpening, there are two faster image sharpening algorithms: Contrast Adaptive Sharpening (contrast adaptive sharpening algorithm) and unsharp (USM algorithm).

[0003] The Contrast Adaptive Sharpening algorithm uses the 4-neighborhood and 8-domain of each pixel to calculate the weight and get the value of the current pixel, ultimately achieving the purpose of image sharpening. Figure 1 , Figure 1 Example diagrams of the current pixel and its 4-neighborhood and 8-neighborhood. Figure 1 In the figure, e represents the current pixel when sliding the window, b, h, d, f are the 4-neighborhood of the current pixel; a, b, c, d, f, g, h, i are the 8-neighborhood of the current pixel, and the sharpening result of the Contrast Adaptive Sharpening algorithm for the current pixel

[0004]

[0005] in,

[0006] w=amp / strength

[0007]

[0008]

[0009] strength=-[16+(16-4)·s]

[0010] Here, s is a parameter that needs to be set manually, and the size of the parameter set by s is directly related to the sharpening result.

[0011] Unsharp algorithm: This algorithm obtains the gradient grad of the image by subtracting the Gaussian blurred image of the current image from the current image, and then calculates the weighted src and gradient grad, ultimately achieving the improvement of image edges and details.

[0012] dst=src+[src-gaussianblur(src)*m]

[0013] Here m is a parameter that needs to be set manually, gaussianblur represents the image Gaussian blur operation, src is the input video frame, and dst is the unsharp sharpened output video frame. Similarly, m is a parameter that needs to be set manually, and the size of the parameter set by m is directly related to the sharpening result.

[0014] It should be noted that existing image sharpening algorithms all have steps that require manual parameter design. Setting the parameter size is not only directly related to the sharpening result, but also makes the result of the entire sharpening process uncertain, and similarly reduces the efficiency of the entire processing. Summary of the invention

[0015] The present invention discloses a video adaptive sharpening method and system based on Huber function, so as to solve the technical defect that the prior art cannot adaptively sharpen image information.

[0016] A video adaptive sharpening method based on Huber function, characterized by comprising:

[0017] S1: Training Huber piecewise function through image dataset At least parameters a, b, c;

[0018] S2: Calculate the gradient information g on the Y component of the preprocessed image YCbCr data, and then normalize the gradient information in the value range [0,255] to the interval [0,c] to complete the mapping of the gradient information to the three ranges of a, b, and c;

[0019]

[0020] Where x represents the Y component data of the image; G represents a Gaussian kernel of size 9; Represents the gradient amplitude of the Y component;

[0021] S3: Automatically adjust the sharpening intensity of the output image according to the gradient information

[0022]

[0023] x is the input image, y is the output image, the correction signal is the y component gradient amplitude g, and the factor controlling the intensity is a function of g: Said The calculation method is to make three piecewise functions based on the three parameters a, b, c of the training results of S1, and the segmented threshold range is a, b, c.

[0024]

[0025] Where y represents the result of sharpening the input image x; a, b, c are the parameters learned by the deep learning network step S1.

[0026] The present invention also includes adopting an effective supervised learning method of L1loss, first calculating y g , then gt g , then idx, and finally the L1loss,

[0027]

[0028] where y g is the gradient of the image after sharpening; gt g Represents the gradient of the GT image in the dataset; idx represents the location of the maximum gradient of the GT image.

[0029] The three parameters of the model training of the present invention: a, b, c are three intervals. We need to map all gradients into the three ranges of a, b, c, and perform different intensities of sharpening on the different intervals. By regressing the Huber function parameters through the deep learning algorithm, the present invention can realize real-time video sharpening on the CPU, enhance the details of the video, but do not introduce noise, and the algorithm has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the current pixel and its 4-neighborhood and 8-neighborhood;

[0031] Figure 2 The schematic diagram of the process of the video adaptive sharpening method based on Huber function. DETAILED DESCRIPTION

[0032] The present invention will be specifically described below with reference to the accompanying drawings. Many problems in the fields of computer vision and graphics can be reduced to image smoothing problems for processing. However, the requirements for smoothing features vary greatly or even extremely for different tasks. According to the required smoothing characteristics for different tasks, researchers have classified image smoothing into four categories. The first category requires the algorithm to smooth details while retaining strong edges, weakening the edge amplitude while avoiding blurring or sharpening. The representative tasks of this category are image detail enhancement and high-dynamic-range tone mapping. In these tasks, edge blurring will cause halos, while image sharpening is prone to gradient reversal. The second category is mainly used for removing artificial traces in clip art compression, image line extraction, and pencil sketch generation. Different from the first category, these tasks require sharpening strong edges on the basis of smoothing details, and at the same time, the intensity of the edges is not allowed to decay. This is mainly because image edges will be blurred due to compression, and edge sharpening is required during restoration. The sharpened edges are beneficial to the improvement of visual effects. The third category of tasks is guided image filtering methods, mainly including guided depth upsampling, no flash filtering, etc. The discontinuity in the structure between the guided image and the target image will cause edge blurring and artificial texture traces in the smoothed image, and this defect needs to be processed through specially designed smoothing operations. At the same time, the smoothed image also needs to sharpen the edges to adapt to the noise in the low-resolution depth map and non-flash images to handle the subsequent edge blurring. The fourth category of image smoothing tasks requires multi-scale operations, such as image texture removal is a representative of this category. This type of task requires smoothing small structures in the image (even if they contain strong edges), but needs to appropriately retain larger-scale edges (even if these edges are weak). This type of task is different from the previous three tasks that need to preserve strong edges.

[0033] To improve the adaptability of the smoothing algorithm, researchers from Shanghai Jiao Tong University, the University of Adelaide, and Dalian University of Technology proposed a non-convex and non-smooth optimization framework, which can achieve smoothing with various different functions and provide a unified solution for the above four types of smoothing requirements. By introducing a truncated Huber filter during the optimization process and changing its parameters, flexible processing of each smoothing requirement can be achieved (the name of the paper is: A Generalized Framework for Edge-preserving and Structure-preserving Image Smoothing). The definition of the Huber penalty function in the image smoothing method based on the truncated Huber function is as follows. It can be seen that it behaves as a quadratic function in the region where x < a, and as a linear function in the region greater than a.

[0034]

[0035] The truncated Huber, on the other hand, introduces a new parameter b to saturate the region where x > b, and all values exceeding it will be limited to the value of b - a / 2:

[0036]

[0037] The truncated Huber can sharpen edges through saturated truncation processing, that is, exceeding the truncation preset becomes a constant instead of continuing to increase.

[0038] Another Welsch penalty function is also plotted in the figure. It can preserve edges and hardly penalize strong edges at the same time. When the input is small, it approximates L2. While the truncated Huber approximates L1 when the input is small and can better preserve weak edges in the image.

[0039] The truncated Huber can flexibly exhibit different characteristics under different parameter settings. Assume that the intensity range of the input image is between [0, Im], then all edge intensities will also be within [0, Im]. If a = epsilon (~1e-7) is set, when b > Im, the truncated Huber will not saturate like the Huber function, and a small enough a also makes it exhibit L1-like characteristics for small inputs; if b < Im is set, the truncated Huber will saturate before the edge intensity reaches the maximum value Im, so that it penalizes weak edges while not penalizing strong edges, thus sharpening strong edges. This means that b can be used as a switch to adjust whether the truncated Huber function sharpens edges.

[0040] On the other hand, if a = b > Im or a = b < Im is set, the function exhibits L2 characteristics at this time, and a becomes the switch between the ordinary L2 loss and the truncated L2 loss. This is the flexibility that the Welsch loss function does not have. Such a flexible model can effectively construct an efficient objective function to achieve multifunctional smoothing processing. Given the input image f and the guidance image g, the output image u is the solution of the following objective function:

[0041]

[0042] Where the first term is the sum of the differences between the corresponding pixels in the output image and the corresponding pixels in the input image. The lambda before the second term is the global smoothing coefficient, and the interpolation between the corresponding pixels in the smoothed image in hT. w is the guidance weight calculated by the guidance image according to the following formula, and its exponent determines the sensitivity to edges

[0043]

[0044] The application of truncated Huber function makes the model have excellent flexibility, and it can be applied to different image smoothing tasks such as edge-preserving smoothing or structure-preserving smoothing under different parameter settings.

[0045] To this end, on this basis, our company has developed a video adaptive sharpening method based on the Huber function, such as Figure 2 shown.

[0046] S110: Training Huber piecewise function with image dataset At least parameters a, b, c. When training AI, we input some image data. The data acquisition method uses the open source REDS The deblur data set, the trained model is a piecewise function, the three parameters of the piecewise function are a, b, c, the theoretical model of this piecewise function comes from the aforementioned truncated Huber function, the specific paper is https: / / arxiv.org / pdf / 2107.07058.pdf), the g1(x, y), g2(x, y), g3(x, y) mentioned therein are these three segment functions, these three functions are applied in three threshold ranges respectively, this theory is the same as the theory in the patent, the difference is that the three thresholds are determined differently, c is not mentioned in the paper, the scenario of applying g3(x, y) is when >= b, in this patent, we use the gradient amplitude to determine the threshold range, for each frame of image data, the gradient amplitude must have its maximum value, so there must be a c, 0, a, b, c four points, with a starting point and an end point, to determine the three line segments, that is, the application range of these three functions, the first two functions form a continuous curve, and the last function is a straight line.

[0047] S120: Calculate the gradient information g on the Y component of the preprocessed image YCbCr data, and then normalize the gradient information in the value range [0, 255] to the interval [0, c], so as to complete the mapping of the gradient information to the three ranges of a, b, and c;

[0048]

[0049] Where x represents the Y component data of the image; G represents a Gaussian kernel of size 9; Indicates the gradient magnitude of the Y component.

[0050] That is, if the received image is not YCbCr data, the preprocessed image is first converted into YCbCr data. Then the gradient information g is calculated first, and then the calculated gradient information in the range of [0,255] is normalized to the interval [0,c], because the three parameters of the above model training: a, b, c are 3 intervals, we need to map all gradients to the three ranges of a, b, c, and perform different intensities of sharpening on these different intervals.

[0051] S130: Automatically adjust the sharpening intensity of the output image according to the gradient information.

[0052]

[0053] x is the input image, y is the output image, the correction signal is the y component gradient amplitude g, and the factor controlling the intensity is a function of g: Said The calculation method is to make three piecewise functions based on the three parameters a, b, c of the training results of S1, and the segmented threshold range is a, b, c.

[0054]

[0055] Where y represents the result of sharpening the input image x; a, b, c are the parameters learned by the deep learning network step S1.

[0056] That is, here is the general algorithm for the unsharp masking algorithm:

[0057] Y(n,m)=x(n,m)+λz(n,m)

[0058] Where x(n,m) is the input image, y(n,m) is the output image, and z(n,m) is the correction signal, which is generally obtained by high-pass filtering x. λ is a scaling factor used to control the enhancement effect.

[0059] The sharpening formula of this algorithm is as follows:

[0060]

[0061] x is the input image, y is the output image, the correction signal here is the y component gradient amplitude: g, and the factor controlling the intensity is a function of g: In this way, the sharpening intensity can be automatically adjusted according to the gradient amplitude. That is, different ranges are determined according to different gradient amplitudes, and adjustments are made according to different ranges.

[0062] In addition, the present invention can also adopt an effective supervised learning method using L1loss, first calculating y g , then gt g, then idx, and finally the L1loss,

[0063]

[0064] where y g is the gradient of the image after sharpening; gt g Indicates the gradient of the GT image in the dataset; idx indicates the location of the maximum gradient of the GT image. That is, how to evaluate the effectiveness of learning during training, that is, when we think that AI learning has reached its goal, we use the L1loss indicator to evaluate the effective supervised learning method, so as to adjust the training adaptively.

[0065] Application Examples

[0066] An application example of a video adaptive sharpening method based on Huber function is presented.

[0067] Step 0: This is about a theory of machine learning: L1loss, which talks about how to evaluate the effectiveness of learning during training. That is, when we think that AI learning has reached its goal, we use the L1loss indicator to evaluate. An effective supervised learning method uses L1loss. The following is the calculation steps of this indicator. First, calculate y g , then gt g , then idx, and finally the L1loss,

[0068]

[0069] where y g is the gradient of the image after sharpening; gt g Represents the gradient of the GT image in the REDS deblur dataset; idx indicates the location of the maximum gradient of the GT image.

[0070] Huber function is the Huber Loss function. The loss function is used to measure the inconsistency between the model's predicted value f(x) and the true value y. Our goal is to minimize the loss function and make f(x) and y as close as possible.

[0071] Step 1: When training AI, we input some image data. We use the open source REDS deblur dataset to obtain the data. The training model is a piecewise function. The three parameters of the piecewise function are a, b, and c. The theoretical model of this piecewise function comes from a paper. Here is an approach to piecewise function: https: / / arxiv.org / pdf / 2107.07058.pdf

[0072] The result of training is to output three parameters a, b, and c.

[0073] The theory of three-segment functions comes from the above-mentioned paper. The g1(x,y), g2(x,y), and g3(x,y) mentioned therein are these three-segment functions. These three functions are applied in three threshold ranges respectively. This theory is the same as the theory in the patent. The difference is that the three thresholds are determined differently. There is no mention of c in the paper. The scenarios for applying g3(x,y) are all when >=b. In our patent, we use the gradient amplitude to determine this threshold range. For each frame of image data, the gradient amplitude must have its maximum value. Therefore, there must be a c, 0, a, b, c four points, with a starting point and an end point, to determine the three line segments, which is the application range of these three functions. The first two functions form a continuous curve, and the last function is a straight line.

[0074] When we train the model, the model algorithm logic is as follows:

[0075] We first define three model parameters a, b, and c. For any Y component data x,

[0076] Calculate the Gaussian blur value: blur, and then calculate the gradient value

[0077] The second step calculates three piecewise functions:

[0078]

[0079]

[0080]

[0081] Define three threshold flags: flag_a, flag_b, flag_c, so that these three flags meet the following conditions:

[0082]

[0083] Then for any gradient magnitude, we can get the piecewise function expression:

[0084]

[0085] The third step is to calculate the Y component after sharpening.

[0086]

[0087] The fourth step is to perform Huber Loss evaluation, continue data training, and iteratively update the a, b, and c model parameters until the evaluation function produces the best effect.

[0088] Step 2: Start sharpening the image. The image encoding format here is yuv, and only the y component is operated here.

[0089] (1) First, calculate the gradient information g on the Y component:

[0090]

[0091] Where x represents the Y component data of the image; G represents a Gaussian kernel of size 9; Indicates the gradient magnitude of the Y component.

[0092] Next, normalize the calculated gradient information in the range of [0,255] to the interval [0,c]. Because the three parameters of the above model training: a, b, c are 3 interval ranges, we need to map all gradients to the three ranges of a, b, and c, and perform different intensities of sharpening on these different intervals.

[0093] Here is the general algorithm of the unsharp masking algorithm:

[0094] Y(n,m)=x(n,m)+λz(n,m)

[0095] Where x(n,m) is the input image, y(n,m) is the output image, and z(n,m) is the correction signal, which is usually obtained by subtracting Gaussian blur from the original image. λ is a scaling factor used to control the enhancement effect.

[0096] The sharpening formula of this algorithm is as follows:

[0097]

[0098] x is the input image, y is the output image, g is the correction signal, which is consistent with the unsharp mask algorithm and is obtained by subtracting Gaussian blur from the original image; Enhancement factor, this enhancement factor is a piecewise function of the gradient amplitude, which can automatically adjust the sharpening intensity according to the image gradient amplitude.

[0099] Below is The calculation method is as follows: according to the three parameters a, b, and c of the above training results, they correspond to the three threshold ranges of the piecewise function respectively.

[0100]

[0101] Where y represents the result of sharpening the input image x; a, b, and c are the three parameters obtained from the model training in Step 1.

[0102] A video adaptive sharpening system based on Huber function.

[0103] Training model: Training Huber piecewise function through image dataset At least parameters a, b, c;

[0104] Gradient calculation unit: used to calculate the gradient information g on the Y component of the preprocessed image YCbCr data; then normalize the gradient information in the value range [0,255] to the interval [0,c] to complete the mapping of the gradient information to the three ranges of a, b, and c;

[0105]

[0106] Where x represents the Y component data of the image; G represents a Gaussian kernel of size 9; Represents the gradient amplitude of the Y component;

[0107] Sharpening intensity adjustment unit: used to automatically adjust the sharpening intensity of the output image according to the gradient information

[0108]

[0109] x is the input image, y is the output image, the correction signal is the y component gradient amplitude g, and the factor controlling the intensity is a function of g: Said The calculation method is to make three piecewise functions based on the three parameters a, b, c of the training results of S1, and the segmented threshold range is a, b, c.

[0110]

[0111] Where y represents the result of sharpening the input image x; a, b, c are the parameters learned by the deep learning network step S1.

[0112] Supervised learning unit: used for supervised learning with L1loss, first calculate y g , then gt g , then idx, and finally the L1loss,

[0113]

[0114] where y g is the gradient of the image after sharpening; gt g Represents the gradient of the GT image in the dataset; idx represents the location of the maximum gradient of the GT image.

[0115] The invention discloses a video adaptive sharpening algorithm based on Huber function, which can adaptively process various video scenes, meet the requirements of real-time processing, and better highlight the edges and details of images.

[0116] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they still fall within the protection scope of the present invention.

Claims

1. A video adaptive sharpening method based on Huber function, characterized in that: include: S1: Training Huber piecewise function through image dataset At least three parameters a, b, c; S2: Calculate the gradient information of the preprocessed image YCbCr data on the Y component , and then normalize the gradient information in the value range [0, 255] to the interval [0, c] to complete the mapping of the gradient information to the three ranges of a, b, and c; in Represents the Y component data of the image; represents a Gaussian kernel of size 9; Represents the gradient amplitude of the Y component; S3: Automatically adjust the sharpening intensity of the output image according to the gradient information is the input image, y is the output image, and the correction signal is the gradient information of the Y component , the factor controlling the intensity is A function of: , The calculation method is: according to the training results of step S1, three parameters a, b, c are used to make three piecewise functions. The threshold range of the segment is a, b, c, Where y represents the input image The result after sharpening; a, b, c are the parameters learned by the deep learning network step S1.

2. The method according to claim 1, characterized in that Also includes: use An effective supervised learning method is to first calculate ,Then , followed by , and finally came to this , in is the gradient of the image after sharpening; Represents the gradient of the GT image in the dataset; Indicates the location of the maximum gradient of the GT image; is the initial value of the GT image.

3. The method according to claim 1 or 2, characterized in that The image dataset uses the REDS deblur dataset.

4. The method according to claim 1 or 2, characterized in that: Step S2 also includes converting the pre-processed image into YCbCr data.

5. A video adaptive sharpening system based on Huber function, characterized in that: Training model: Training Huber piecewise function through image dataset At least three parameters a, b, c; Gradient calculation unit: used to calculate the gradient information of the preprocessed image YCbCr data on the Y component ; Then normalize the gradient information in the value range [0, 255] to the interval [0, c] to complete the mapping of the gradient information to the three ranges of a, b, and c; in Represents the Y component data of the image; represents a Gaussian kernel of size 9; Represents the gradient amplitude of the Y component; Sharpening intensity adjustment unit: used to automatically adjust the sharpening intensity of the output image according to the gradient information is the input image, y is the output image, and the correction signal is the gradient information of the Y component , the factor controlling the intensity is A function of: , The calculation method is: according to the training results of step S1, three parameters a, b, c are used to make three piecewise functions. The threshold range of the segment is a, b, c, Where y represents the input image The result after sharpening; a, b, c are the parameters learned by the deep learning network step S1.

6. The system of claim 5, further comprising: Supervised learning unit: used to An effective supervised learning method is to first calculate ,Then , followed by , and finally came to this , in is the gradient of the image after sharpening; Represents the gradient of the GT image in the dataset; Indicates the location of the maximum gradient of the GT image; is the initial value of the GT image.

Images