A supervised image enhancement method based on the Zero-DCE network structure

Abstract

The present invention provides a method for supervised image enhancement based on the Zero-DCE network structure. The method constructs a convolutional neural network for supervised learning, uses contrast enhancement loss, improves the loss function's adaptability to supervised learning, and uses backpropagation to allow the model to adaptively learn the mapping function through training. Because the visual effects of existing Zero-DCE-enhanced images are not good enough in real scenes, such as contrast loss and unclear images, the method of this application can produce images with higher fidelity and quality. The model is lightweight and highly efficient, with only a few convolutions and downsampling steps. The model is adaptive and trained using real-world training data. The actual results meet engineering requirements and are more suitable for real-world scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement in low-level visual tasks of deep learning neural networks, and in particular to a method for image enhancement based on an improved Zero-DCE network structure. Background Art

[0002] Unsupervised learning low-light enhancement learns its own function mapping to map pixel values ​​to a higher value range, thereby achieving the purpose of low-light enhancement. Existing supervised deep learning low-light enhancement essentially learns the function mapping from the value range of dark-light images to the bright-light image domain, and learns the mapping between the two from paired data.

[0003] The original Zero-DCE is an unsupervised learning method that learns the function mapping of its own pixels and maps the pixel values ​​to a higher value range to achieve the purpose of low-light enhancement. The loss functions used are spatial consistency loss, exposure control loss, color constancy loss, and lighting smoothness loss.

[0004] However, while the original Zero-DCE method requires fewer datasets than supervised learning methods, its focus on pixel mapping often makes it difficult to process real-world scenes, resulting in unsatisfactory visual effects. For example, the brightness boost can be overly balanced, resulting in reduced image contrast and clarity. Such flaws make it unsuitable for real-world applications.

[0005] In addition, the commonly used terms in the prior art are as follows:

[0006] Image contrast: This refers to the measurement of the brightness levels between the brightest white and darkest black in an image. It also refers to the degree of grayscale contrast in an image. Generally speaking, images with high contrast appear clearer, while images with low contrast appear blurry.

[0007] Supervised learning: A function (model parameters) is learned from a given training data set. When new data arrives, the results can be predicted based on this function. The training set for supervised learning is required to include inputs and outputs, or features and targets. The targets in the training set are labeled by humans. Supervised learning is the most common classification (attention and clustering) problem. Existing training samples (that is, known data and its corresponding outputs) are used to train an optimal model (this model belongs to a set of functions, and optimal means it is the best under a certain evaluation criterion). This model is then used to map all inputs to corresponding outputs, and simple judgments are made on the outputs to achieve the purpose of classification. This means that the computer has the ability to classify unknown data. The goal of supervised learning is often to let the computer learn the model we have already created.

[0008] Unsupervised learning: Input data is unlabeled, and there are no definitive outcomes. Sample data categories are unknown, and the dataset must be classified (clustered) based on similarities, attempting to minimize intra-class differences and maximize inter-class differences. In practical applications, in many cases, sample labels are not known in advance, meaning there are no corresponding categories for training samples. Therefore, classifier design must be learned from a pre-existing dataset of unlabeled samples.

[0009] Convolutional Neural Network (CNN): A convolutional neural network (CNN) consists of a feature extractor consisting of convolutional layers and subsampling layers (pooling layers). In a convolutional layer of a CNN, a neuron is connected only to some neurons in adjacent layers. A convolutional layer of a CNN typically contains several feature maps, each composed of a rectangular array of neurons. Neurons in the same feature map share weights, which are the convolution kernel. Subsampling, also known as pooling, typically takes two forms: mean subsampling and max subsampling. Subsampling can be considered a special convolution process. Convolution and subsampling greatly simplify model complexity and reduce model parameters. Summary of the Invention

[0010] In order to solve the above problems, the purpose of this application is to combine the ideas of supervised learning algorithms with the ideas of Zero-DCE network structure to construct a set of supervised learning convolutional neural networks, use contrast enhancement loss, improve the adaptation of loss function to supervised learning, and through backpropagation, let the model adaptively learn mapping functions through training, solve the problem of poor performance of the original Zero-DCE network, and obtain an image enhancement algorithm with better visual effects in real scenes.

[0011] Specifically, the present invention provides a method for image enhancement based on an improved Zero-DCE network structure for supervised learning. The method constructs a convolutional neural network for supervised learning, uses contrast enhancement loss, improves the adaptability of the loss function to supervised learning, and allows the model to adaptively learn the mapping function through training through backpropagation. The method further includes the following steps:

[0012] S1, using supervised learning to replace the original unsupervised design of Zero-DCE:

[0013] S1.1, Original Zero-DCE Design: The original Zero-DCE model is trained as unsupervised learning, meaning that only input data is available during training, without a labeled dataset. The exposure loss function used in the original Zero-DCE model training is to set a brightness range, and the model is trained to ensure that the brightness of the output is close to this range.

[0014] S1.2, Improved Zero-DCE Design: The Zero-DCE model is modified to supervised learning. Paired data is provided in the dataset. The training data are low-light images, and the labeled data are normal, well-exposed images. The modified model is supervised learning, that is, both input data and labeled data are present during training, and the learning process is the function mapping from input data to labeled data.

[0015] S2, modify the loss function: modify the loss function, for exposure loss, change the original given brightness range to adaptive learning mapping from training data to label data; the brightness range of the modified model exposure loss function is derived from the label data;

[0016] The modifying exposure loss further comprises:

[0017] The original exposure loss is given as the target exposure and the absolute value loss is calculated:

[0018]

[0019] Among them, Y k is the mean value of the k pixels in the RGB channel of the output image, E is the given target value between 0.5 and 0.7, and M is the total number of pixels in the image;

[0020] where Y k is the mean of the RGB channels, which is modified in this step to

[0021]

[0022] Among them, P k is the grayscale value of pixel k in the output image, T k is the grayscale value of the label image, and M is the total number of pixels in the image;

[0023] Among them, P k is the grayscale value of the output image, and the calculation formula is:

[0024] P=R*0.299+G*0.587+B*0.114

[0025] Among them, R, G, B are the values ​​of the R, G, B channels of the RGB image at the pixel position, and P is P k ;T k is the grayscale value of the label image;

[0026] S3, add structure loss SSMI and contrast loss:

[0027] The loss function calculations for contrast and exposure are based on each block rather than each pixel. That is, when calculating the loss function, the input image and the label image are divided into blocks of the same size, and then the loss function is calculated for each block separately. This can better express the correlation of the local information of the model. That is, the modified model adds structural similarity loss to supervise and improve the quality of the enhanced image; it also adds contrast enhancement loss to supervise and improve the contrast of the enhanced image.

[0028] S4 uses bright images with good exposure, i.e., the scene details are well preserved and there are no overexposed or underexposed areas, as labels, and uses low-light images as input.

[0029] The step S1.1 further comprises:

[0030] The design of the Zero-DCE network uses CNN to estimate the high-order mapping equation of each pixel in the image as an enhancement. Its basic idea is to estimate the transformation curve of the pixels given an image. The function must meet three conditions:

[0031] a. Scope [0,1], value range [0,1];

[0032] b. The curve is differentiable;

[0033] c. The curve is monotonic, ensuring that the relative size relationship of pixels remains unchanged after enhancement.

[0034] The three conditions further include:

[0035] Let the pixel value be x, and the transformation function used is the quadratic equation:

[0036] f(x)=x+αx(1-x) Formula 1

[0037] In order to improve the expressiveness of the curve, the quadratic equation is iterated:

[0038] f(x) n =f(x) n-1 +α n *f(x) n-1 (1-f(x) n-1 ) Formula 2

[0039] Given α∈[-1,1], we can derive f′(x)=1+α-2αx. For x∈[0,1], we can continue to derive f″(x)=-2α. Once α is determined, the sign of f″(x) is also determined, and we can get f′(x) to be monotonic. Substituting the two ends of the range of x, we can further get that f′(x) takes values ​​between 1-α and 1+α. Given α∈[-1,1], we can get f′(x)≥0. Therefore, f(x) is monotonically increasing. Substituting the two ends of the range of x, we can get that the range of f(x) is [0,1].

[0040] Since image enhancement is a one-to-many relationship, there should be changes in different areas of the image. In order to solve this problem, α in the above formula n Change to matrix A with the same size as the image n , which is the amount to be estimated by the original Zero-DCE model; for each pixel of the model, a mapping is made through a different monotonically increasing function.

[0041] In the step S3:

[0042] The added structural loss SSMI loss:

[0043] The calculation formula of SSMI loss is:

[0044]

[0045] Among them, x is the input data, y is the label data, μ x is the mean of x, μ y is the mean of y, σ x is the variance of x, σ y is the variance of y, σ xy is the covariance of x and y, c1=(k1l) 2 ,c2=(k2l) 2 are two constants to avoid division by zero, and l is the range of pixel values;

[0046] The increased contrast loss:

[0047] The standard deviation reflects the discreteness of the image grayscale relative to the average grayscale. It is used to evaluate the size of the image contrast and calculate the image standard deviation.

[0048]

[0049]

[0050] in Represents the standard deviation of the training image and the label image in the i-th block, respectively. The two images are n blocks each. The right side of the equation is the standard deviation calculation formula, x j and are the pixel values ​​at position j in the i-th block of the training image and the label image, respectively. and are the mean values ​​of pixels in the training image and the label image in the i-th block, respectively, and m is the total number of pixels in the i-th block;

[0051] Use L2 loss to calculate the loss of each standard deviation, sum and average;

[0052]

[0053] The calculated L is used as the loss function of contrast.

[0054] In the calculation formula of the SSMI loss, the common RGB image [0, 255] is used, and k1 = 0.01 and k2 = 0.03 are the default values.

[0055] Therefore, the advantages of this application are:

[0056] 1. Since the visual effect of the original Zero-DCE enhanced image is not good enough in real scenes, such as contrast reduction and unclear image, the use of this patented method can obtain a more realistic and higher-quality image;

[0057] 2. The model is lightweight and computationally efficient, with only a few convolutions and downsampling steps.

[0058] 3. Model adaptation: using training data from real scenarios for training, the actual effect meets engineering requirements and is more applicable to real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0060] Figure 1 This is a schematic diagram of the original Zero-DCE.

[0061] Figure 2 Schematic diagram of the improved algorithm of the method of the present invention.

[0062] Figure 3 It is a schematic flow diagram of the method of the present invention. DETAILED DESCRIPTION

[0063] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail with reference to the accompanying drawings.

[0064] The present invention relates to a method for improving supervised learning based on Zero-DCE network structure. Figure 1 Schematic diagram of the original Zero-DCE, Figure 2 A schematic diagram of the algorithm is shown in Figure 3 As shown, the specific ideas and logic of the algorithm are stated as follows:

[0065] S1. Use supervised learning instead of the original Zero-DCE unsupervised:

[0066] S1.1.Zero-DCE Design

[0067] CNN is used to estimate the high-order mapping equation for each pixel in the image as an enhancement.

[0068] The basic idea of ​​the original Zero-DCE is to estimate the pixel transformation curve given an image. The function needs to meet the following three conditions:

[0069] a. Scope [0,1], value range [0,1];

[0070] b. The curve is sufficiently simple and differentiable (a relative concept, and in actual implementation it can be represented by a function of approximately 8th-order Taylor expansion);

[0071] c. The curve is monotonic, ensuring that the relative size relationship of pixels remains unchanged after enhancement;

[0072] Let the pixel value be x, and the transformation function used is the quadratic equation:

[0073] f(x)=x+αx(1-x) Formula 1

[0074] In order to improve the expressiveness of the curve, the quadratic equation is iterated:

[0075] f(x) n =f(x) n-1 +α n *f(x) n-1 (1-f(x) n-1 ) Formula 2

[0076] Given α∈[-1,1], we can differentiate to get f′(x)=1+α-2αx, and x∈[0,1]. Continuing to differentiate, we can get f″(x)=-2α. When α is determined, the sign of f″(x) is also determined, and we can get f′(x) is monotonic. Substituting into the two ends of the range of x, we can further get that f′(x) takes values ​​between 1-α and 1+α. Given α∈[-1,1], f′(x)≥0, so f(x) is monotonically increasing. Substituting into the two ends of the range of x, we can get that the range of f(x) is [0,1]. It is easy to see that the above three conditions are met.

[0077] Since image enhancement is a one-to-many relationship, there should be changes in different areas of the image. To solve this problem, replace α in the above formula with n Change to matrix A with the same size as the image n , which is the quantity that the model wants to estimate.

[0078] To sum up, a mapping is made for each pixel of the model through a different monotonically increasing function.

[0079] S1.2. Improvement ideas, modified Zero-DCE:

[0080] As attached Figure 2, the Zero-DCE model is changed to supervised learning. The model includes network structure, data loading and training. Paired data is provided on the dataset. The training data is low-light images, and the label data is normal well-exposed images. The loss function is modified. For exposure loss, the original given brightness interval is changed to adaptive learning mapping from training data to label data. The structural loss SSMI is added, and the contrast loss is added.

[0081] S2. Modify the loss function: Modify the loss function. For exposure loss, the original given brightness range is changed to adaptive learning mapping from training data to labeled data. The brightness range of the modified model exposure loss function is derived from the labeled data.

[0082] The modifying exposure loss further comprises:

[0083] The original exposure loss is given as the target exposure and the absolute value loss is calculated:

[0084]

[0085] Among them, Y k is the mean value of the k pixels in the RGB channel of the output image, E is the given target value between 0.5 and 0.7, and M is the total number of pixels in the image;

[0086] where Y k is the mean of the RGB channels, which is modified in this step to

[0087]

[0088] Among them, P k is the grayscale value of pixel k in the output image, T k is the grayscale value of the label image, and M is the total number of pixels in the image;

[0089] Among them, P k is the grayscale value of the output image, and the calculation formula is:

[0090] P=R*0.299+G*0.587+B*0.114

[0091] Among them, R, G, B are the values ​​of the R, G, B channels of the RGB image at the pixel position, and P is P k ;T k is the grayscale value of the label image;

[0092] S3. Add structure loss SSMI and contrast loss:

[0093] The added structural loss SSMI loss:

[0094] The calculation formula of SSMI loss is:

[0095]

[0096] The increased contrast loss:

[0097] The standard deviation reflects the discreteness of the image grayscale relative to the average grayscale. It can be used to evaluate the size of the image contrast and calculate the image standard deviation.

[0098]

[0099] m is the number of pixels, Std f2 Same definition as above.

[0100] Use L2 loss to calculate the loss of each standard deviation, sum and average;

[0101]

[0102] The calculated L is used as the loss function of contrast.

[0103] S4. Use bright images with better exposure as labels and low-light images as input.

[0104] In summary, combined with the corresponding description of the original Zero-DCE, this method makes the following changes to the original Zero-DCE:

[0105] 1. The original Zero-DCE model is trained as unsupervised learning, meaning that only input data is used during training, but no labeled dataset is available. The modified model is trained as supervised learning, meaning that both input data and labeled data are used during training, and the learning objective is to map the function from input data to labeled data.

[0106] 2. The exposure loss function of the original Zero-DCE model training sets a brightness range, and the model is trained to make the brightness of the output close to this range; the brightness range of the exposure loss function of the modified model is derived from the label data;

[0107] 3. In addition, the modified model also adds structural similarity loss to supervise and improve the quality of the enhanced image.

[0108] 4. Contrast enhancement loss is added to supervise the improvement of the contrast of the enhanced image.

[0109] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for image enhancement based on a Zero-DCE network structure improved to a supervised one, characterized in that: The method constructs a convolutional neural network for supervised learning, uses contrast enhancement loss, improves the adaptability of the loss function to supervised learning, and allows the model to adaptively learn the mapping function through training through back propagation. The method further includes the following steps: S1, using supervised learning to replace the original unsupervised design of Zero-DCE: S1.1, Original Zero-DCE Design: The original Zero-DCE model is trained as unsupervised learning, meaning that only input data is available during training, without a labeled dataset. The exposure loss function used in the original Zero-DCE model training is to set a brightness range, and the model is trained to ensure that the brightness of the output is close to this range. S1.2, Improved Zero-DCE Design: The Zero-DCE model is modified to supervised learning. Paired data is provided in the dataset. The training data are low-light images, and the labeled data are normal, well-exposed images. The modified model is supervised learning, that is, both input data and labeled data are present during training, and the learning process is the function mapping from input data to labeled data. S2, modify the loss function: modify the loss function, for exposure loss, change the original given brightness range to adaptive learning mapping from training data to label data; the brightness range of the modified model exposure loss function is derived from the label data; The modifying exposure loss further comprises: The original exposure loss is given as the target exposure and the absolute value loss is calculated: Among them, Y k is the mean value of the k pixels in the RGB channel of the output image, E is the given target value between 0.5 and 0.7, and M is the total number of pixels in the image; where Y k is the mean of the RGB channels, which is modified in this step to Among them, P k is the grayscale value of pixel k in the output image, T k is the grayscale value of the label image, and M is the total number of pixels in the image; Among them, P k is the grayscale value of the output image, and the calculation formula is: P=R*0.299+G*0.587+B*0.114 Among them, R, G, B are the values ​​of the R, G, B channels of the RGB image at the pixel position, and P is P k ;T k is the grayscale value of the label image; S3, add structure loss SSMI and contrast loss: The loss function calculations for contrast and exposure are based on each block rather than each pixel. That is, when calculating the loss function, the input image and the label image are divided into blocks of the same size, and then the loss function is calculated for each block separately. This can better express the correlation of the local information of the model. That is, the modified model adds structural similarity loss to supervise and improve the quality of the enhanced image; it also adds contrast enhancement loss to supervise and improve the contrast of the enhanced image. S4 uses bright images with good exposure, i.e., the scene details are well preserved and there are no overexposed or underexposed areas, as labels, and uses low-light images as input.

2. The method for image enhancement based on the Zero-DCE network structure improved to a supervised one according to claim 1, characterized in that: The step S1.1 further comprises: The design of the Zero-DCE network uses CNN to estimate the high-order mapping equation of each pixel in the image as an enhancement. Its basic idea is to estimate the transformation curve of the pixels given an image. The function must meet three conditions: a. Scope [0,1], value range [0,1]; b. The curve is differentiable; c. The curve is monotonic, ensuring that the relative size relationship of pixels remains unchanged after enhancement.

3. The method for image enhancement based on the Zero-DCE network structure improved to a supervised one according to claim 2, characterized in that: The three conditions further include: Let the pixel value be x, and the transformation function used is the quadratic equation: f(x)=x+αx(1-x) Formula 1 In order to improve the expressiveness of the curve, the quadratic equation is iterated: f(x) n =f(x) n-1 +α n *f(x) n-1 (1-f(x) n-1 ) Formula 2 Given α∈[-1,1], we can derive f′(x)=1+α-2αx. For x∈[0,1], we can continue to derive f″(x)=-2α. Once α is determined, the sign of f″(x) is also determined, and we can get f′(x) to be monotonic. Substituting the two ends of the range of x, we can further get that f′(x) takes values ​​between 1-α and 1+α. Given α∈[-1,1], we can get f′(x)≥0. Therefore, f(x) is monotonically increasing. Substituting the two ends of the range of x, we can get that the range of f(x) is [0,1]. Since image enhancement is a one-to-many relationship, there should be changes in different areas of the image. In order to solve this problem, α in the above formula n Change to matrix A with the same size as the image n , which is the amount to be estimated by the original Zero-DCE model; for each pixel of the model, a mapping is made through a different monotonically increasing function.

4. The method for image enhancement based on the Zero-DCE network structure improved to a supervised one according to claim 1, characterized in that: In the step S3: The added structural loss SSMI loss: The calculation formula of SSMI loss is: Among them, x is the input data, y is the label data, μ x is the mean of x, μ y is the mean of y, σ x is the variance of x, σ y is the variance of y, σ xy is the covariance of x and y, c1=(k1l) 2 ,c2=(k2l) 2 are two constants to avoid division by zero, and l is the range of pixel values; The increased contrast loss: The standard deviation reflects the discreteness of the image grayscale relative to the average grayscale. It is used to evaluate the size of the image contrast and calculate the image standard deviation. in Represents the standard deviation of the training image and the label image in the i-th block, respectively. The two images are n blocks each. The right side of the equation is the standard deviation calculation formula, x j and are the pixel values ​​at position j in the i-th block of the training image and the label image, respectively. and are the mean values ​​of pixels in the training image and the label image in the i-th block, respectively, and m is the total number of pixels in the i-th block; Use L2 loss to calculate the loss of each standard deviation, sum and average; The calculated L is used as the loss function of contrast.

5. The method for image enhancement based on the Zero-DCE network structure improved to a supervised one according to claim 4, characterized in that: In the calculation formula of the SSMI loss, the common RGB image [0, 255] is used, and k1 = 0.01 and k2 = 0.03 are the default values.

Images