A method for low-light enhanced task dark region denoising

Abstract

The present application provides a kind of method for low light enhancement task dark area denoising, the method comprises: S1. Network is constructed spatial attention structure: for the ability of enhancement model adaptive attention light and dark area, for neural network adds spatial attention structure;S2. Loss function is concerned for light and dark: since this attention is as weight proportion synthesis in loss function, so the value range of this weight is between [0,1], using (bright map-dark map) / bright map obtains a 0 to 1 feature, this feature is positively correlated with brightness, and the feature of attention dark area is obtained by flipping: S3. Merge with mean square error function. The attention of loss function plays a good role;The estimation of noise is more accurate, only need to select appropriate training data, effectively handle the denoising function in low light enhancement task;Effectively reduce the computational complexity, so that the engineering feasibility rises.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of low-light enhancement in deep learning neural network low-level vision tasks, and particularly relates to a method for dark area denoising in a low-light enhancement task. BACKGROUND

[0002] In the existing supervised deep learning low-light enhancement task, the denoising part mainly relies on model self-adaptive training denoising. For the part of loss function processing noise, the mean absolute error or the mean square error is generally used to achieve the denoising effect. The basic implementation of denoising is to use the smoothing effect of the neural network on the image. This operation is actually highly similar to the idea of the traditional algorithm mean filter. However, for the image low-light enhancement task, how to simply concatenate the denoising and the enhancement process is a difficult problem. In particular, denoising before enhancement will cause blurring, and enhancement before denoising will cause noise amplification. Most schemes are to randomly add noise such as Gaussian distribution and Poisson distribution on the dark image on the data set, rely on model self-adaptation, and use the mean absolute error or the mean square error or some smoothing function to achieve the denoising effect.

[0003] However, the current approach does not actually associate the low-light enhancement and the denoising task in a friendly manner. Using the absolute error loss function often cannot well remove the dark area noise. Using the smoothing loss or the mean square error loss can remove the noise in the bright area, but often causes the image in the dark area to lose a lot of details, making the image become excessively smooth and blurred.

[0004] In addition, the common terms in the prior art are as follows:

[0005] Image Signal Processing (ISP): mainly used to process the output signal of the front-end image sensor unit to match different manufacturers' image sensors.

[0006] Low-level vision (low-level version): mainly focuses on extracting the characteristics of the image from the retinal image, and does not involve the task of object attribute.

[0007] Mean filter: is a typical linear filter algorithm. It refers to giving a target pixel a template on the image, which includes the surrounding adjacent pixels (such as the surrounding 8 pixels centered on the target pixel, forming a filter template, including the target pixel itself), and replacing the original pixel value with the average value of all pixels in the template. The mean filter itself has inherent defects, that is, it cannot well protect the details of the image. While denoising the image, it also destroys the details of the image, making the image blurred and unable to well remove the noise points.

[0008] Mean Squared Error (MSE): MSE refers to the average of the square of the distance between the model prediction value f(x) and the sample true value y.

[0009] Mean Absolute Error (MAE): MAE refers to the average of the distance between the model prediction value f(x) and the sample true value y. SUMMARY

[0010] To solve the above problems, the purpose of the present application is to optimize the loss function, for the part of denoising, to associate noise with brightness, to enhance the attention to dark noise, to solve the problem of losing a lot of details due to denoising, to solve the problem of excessive blurring in dark areas, and to combine the denoising task with the low-light enhancement task well.

[0011] Specifically, the present application provides a method for low-light enhancement task dark area denoising, which increases the attention to dark areas, that is, actively pays attention to dark areas in the loss function, and adaptively denoises according to the distribution of noise estimated by bright and dark area, the adaptation means that the input image contains noise and the label image does not contain noise, and the model learns the difference between the input and the label through back propagation in model training, comprising the following steps:

[0012] S1. Constructing a spatial attention structure in the network:

[0013] To enhance the ability of the model to adaptively pay attention to bright and dark areas, a spatial attention structure is added to the neural network;

[0014] S2. Attention to bright and dark in the loss function:

[0015] Since this attention is synthesized as a weight ratio in the loss function, the value range of this weight is between [0, 1], and using (light image-dark image) / light image obtains a feature of 0 to 1, which is positively correlated with brightness, and the feature of paying attention to dark areas is obtained by flipping, that is, the operation of directly subtracting 1 in the following formula (2), the original weight value range is [0, 1], the larger the value, the brighter the brightness of the area, after using 1 to subtract, the value range is still [0, 1], but the larger the value, the darker the brightness of the area:

[0016]

[0017] S3. Merge with mean squared error function or absolute error loss function.

[0018] The structure in step S1 is:

[0019] S = sigma(Conv(Avg(F), Max(F)]) formula (1)

[0020] Wherein, F is an input feature map, Conv is a convolution, Avg is an average, Max is a maximum value, and sigma is a sigmoid function; under the joint action of such a structure and a loss function, the model can more easily learn relevant features.

[0021] The step S2 comprises: performing Gaussian blur on the bright image and the dark image, and then subtracting the absolute value of the dark image from the bright image subjected to Gaussian blur to obtain a ratio of the absolute value to the bright image.

[0022] The step S2 further comprises:

[0023] S2.1, performing Gaussian smoothing on the bright image and the dark image;

[0024] S2.2, using (bright image-dark image) / bright image to obtain a feature in the range of [0, 1];

[0025] S2.3, calculating according to formula (1), so that formula (1) becomes:

[0026]

[0027] Wherein, gauss() represents Gaussian blur operation, F light represents a data matrix of the bright image, and F dark represents a data matrix of the dark image.

[0028] In the step S3, the mean square error function and the absolute error function of the loss function are optional, formula (2) calculates a weight, and the weight is directly multiplied on the selected loss function; if the mean square error is used in the step S2, the step S3 is combined with the mean square error; if the absolute error is used in the step S2, the step S3 is combined with the absolute error function.

[0029] In the step S3, if the mean square error function is combined, the feature is directly multiplied on the feature map for calculating the mean square error as a weight:

[0030]

[0031] Wherein, A is obtained according to formula (3), MSE is a mean square error loss function, and formula (4) without the parameter A is a calculation formula of the mean square error, wherein y is a result output by the model, is the label image.

[0032] Therefore, the application has the following advantages:

[0033] 1. The low-light enhancement noise degree has a strong correlation with the light and dark areas of the image. Most existing methods do not consider this, or use deeper network structures to actively learn, which increases the computational complexity. The method used in this paper uses the adaptive ability of the network, and the attention of the loss function can play a good role;

[0034] 2. The noise estimation is more accurate, and only needs to select appropriate training data to effectively handle the denoising function in the low-light enhancement task;

[0035] 3. Effectively reduce the computational complexity, and make the engineering feasibility rise. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings used to provide further understanding of the present application, constitute a part of this application, and do not constitute a limitation of the present application.

[0037] Figure 1 is a flowchart of the method of the present application.

[0038] Figure 2 is a block diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to more clearly understand the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the drawings.

[0040] The present application relates to a method for improving the denoising of low-light enhancement task, as shown in Figure 1 To realize the flow, the specific implementation idea of the method is stated as follows:

[0041] Adopting dark area attention:

[0042] In practical application, it can be found that the noise degree has a strong correlation with the light and dark areas of the image. The noise in the bright area is generally less, and the noise in the dark area is generally more, so for denoising, the attention degree to the dark area should be increased. Image noise is easily confused with image texture, and after applying a simple denoising method, unnecessary blurring effect will be caused. According to the estimation of the noise distribution of the light and dark areas, and adaptive denoising can reduce this influence.

[0043] Method of dark area attention:

[0044] The purpose can be achieved by actively focusing on the dark area on the loss function and the adaptive denoising method:

[0045] Step S1. Constructing a spatial attention structure in the network

[0046] To enhance the ability of the model to adaptively focus on the light and dark areas, a spatial attention structure is added to the neural network, which is self-adaptive because the input image contains noise and the label image does not contain noise. During model training, the model learns the difference between the input and the label through back propagation. The spatial attention structure is common, such as:

[0047] S = σ (Conv ([Avg (F), Max (F)])) Formula (1)

[0048] F is the input feature map, Conv is the convolution, Avg is the mean, Max is the maximum, and sigma is the sigmoid function.

[0049] Depending on the joint action of such structure and loss function, the model can easily learn the relevant features;

[0050] Step S2. Attention to light and dark in loss function

[0051] Since this attention is synthesized as a weight ratio in loss functions such as absolute error or mean square error, the value range of this weight is most suitable between [0, 1]. The loss function is optional, generally absolute error or mean square error, and the mean square error loss function is used in implementation.

[0052] It can be simply thought that using (light map-dark map) / light map can get a 0-1 feature, which is positively correlated with brightness. This feature can be obtained by some flipping method, that is, the operation of directly subtracting 1 in formula (2) below. The original weight value range is [0, 1], and the larger the value, the brighter the area brightness. After using 1 to subtract, the value range is still [0, 1], but the larger the value, the darker the area brightness:

[0053]

[0054] But this calculation method is too rough, every pixel is considered, and it is not suitable for the regional attention required, so the light map and dark map can be first smoothed by Gaussian, and then calculated in the above manner. Then the above formula (2) becomes:

[0055]

[0056] Where, gauss() refers to the Gaussian blur operation, F light represents the data matrix of the light map, and F dark represents the data matrix of the dark map.

[0057] In summary, the step S2 further comprises:

[0058] S2.1, Gaussian smoothing is first performed on the light map and the dark map;

[0059] S2.2, using (light map - dark map) / light map can get a [0, 1] feature;

[0060] S2.3, according to formula (1) manner calculation, then formula (1) becomes formula (3).

[0061] Step S3. The embodiment adopts the combination of the mean square error function

[0062] This attention feature can be directly multiplied as a weight on the feature map for calculating the mean square error.

[0063]

[0064] Wherein, A is obtained by formula (3), MSE is the mean square error loss function, formula (4) removes the parameter A, that is, the calculation formula of the mean square error, wherein y is the output result of the model, The above label image.

[0065] As shown in the above, as Figure 2 As shown in the above, as

[0066] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application can be variously changed and modified. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for low-light enhanced task dark region de-noising, the method comprising: The method increases the attention to dark areas, that is, actively pays attention to dark areas on the loss function, and adaptively denoises according to the distribution of noise estimated by light and dark area, the adaptivity is that the input image contains noise, the label image does not contain noise, and the model learns the difference between the input and the label through back propagation in model training, including the following steps: S1. Constructing a spatial attention structure in the network: To enhance the ability of the model to adaptively pay attention to light and dark areas, a spatial attention structure is added to the neural network; S2. Attention to light and dark in the loss function: Since this attention is synthesized as a weight ratio in the loss function, the value range of this weight is between [0, 1], and using (light image - dark image) / light image obtains a feature of 0 to 1, which is positively correlated with brightness, and the feature of paying attention to dark areas is obtained by flipping, that is, the operation of directly subtracting 1 in formula (2), the original weight value range is [0, 1], and the greater the value represents the brighter the area brightness, after using 1 to subtract, the value range is still [0, 1], but the greater the value represents the darker the area brightness: Equation (2); F light Data matrix representing the light pattern, F dark Data matrix representing the dark pattern; S3. Merge with mean square error function or absolute error function; the merging is directly multiplied on the selected loss function, that is, if step S2 uses mean square error, step S3 merges with mean square error; if step S2 uses absolute error, step S3 merges with absolute error function.

2. The method for low-light enhanced task dark region de-noising according to claim 1, characterized in that, The structure in step S1 is: Formula (1); wherein, F is an input feature map, Conv is a convolution, Avg is a mean value, Max is a max value, is a sigmoid function; depending on the joint action of the structure and the loss function, the model can learn the relevant features.

3. The method for low-light enhanced task dark region de-noising according to claim 1, wherein, Step S2 includes, Gaussian blur is performed on the light image and the dark image, and then the absolute value of the light image after Gaussian blur is subtracted from the dark image is calculated, and the ratio of the absolute value to the light image is calculated.

4. The method for low-light enhanced task dark region de-noising according to claim 3, wherein, The step S2 further includes: S2.1, Gaussian smoothing is performed on the light image and the dark image; S2.2, using (light image - dark image) / light image can obtain a feature of [0, 1]; S2.3, calculate according to formula (2), then formula (2) becomes: Equation (3) Wherein, gauss() refers to the operation of Gaussian blur.

5. The method for low-light enhanced task dark region de-noising according to claim 1, wherein, In step S3, if the mean square error function is merged, the attention feature is directly multiplied as a weight on the feature map for calculating the mean square error: Equation (4) Wherein, A is obtained from formula (3), MSE is mean square error loss function, formula (4) is the calculation formula of mean square error by removing the parameter A, wherein y is the output result of the model, is the label image described above.

Images