A low-light image enhancement method based on multi-scale network structure

Abstract

The present invention proposes a low-light image enhancement method based on a multi-scale network structure. The enhancement method includes a loss function for calculating a mean square error. The color offset of corresponding pixel points on the low-light image and the high-light image is controlled within a threshold range by controlling the mean square error. The loss function calculates the mean square error by calculating the ratio between the pixel values ​​on three RGB channels of the low-light image and the high-light image. The low-light image enhancement model adopted by the enhancement method is a network model based on a multi-scale network structure. The model first performs a gamma transform on the grayscale image of the low-light image, then obtains a grayscale image of the highlight through a branch of the network, and finally weights the low-light image to guide the restoration of the low-light image. The present invention can solve the problem of color distortion caused by the enhancement of brightness in the low-light enhancement process; and can also solve the problem that the detailed texture of the extreme low-light area is difficult to restore during the enhancement process.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to a low-light image enhancement method based on a multi-scale network structure. Background Art

[0002] Nowadays, digital images are widely used. However, factors such as insufficient light source brightness, improper lighting angle, and short camera exposure time can significantly affect the image quality. For example, images taken at night are dark overall due to insufficient lighting, and portraits in photos are not well lit due to backlighting, which affects people's perception. Therefore, low-light enhancement algorithms are a research direction with great value.

[0003] The traditional low-light enhancement algorithms are as follows:

[0004] The first method is to adjust the distribution interval of pixel values ​​through histogram equalization to increase the contrast of the image. The pixel values ​​of low-light images are distributed in a smaller interval, so the contrast of the pixel values ​​is smaller, resulting in unclear images. Histogram equalization increases the dynamic range between pixels by redistributing the pixel values ​​evenly, thereby improving the contrast of the image and enhancing the brightness of low-light images.

[0005] The second method is a low-light image enhancement algorithm based on the Retinex theory. The Retinex theory is a brightness and color perception model of human vision proposed by Land in 1963. It proposes that the color and brightness of an object are determined by the object's reflective ability, and the color of the object itself is not affected by the non-uniformity of illumination. Based on this theory, researchers have proposed many methods of low-light enhancement algorithms, such as SSR, MSRCR, etc.

[0006] Image enhancement algorithms based on deep learning have become a hot topic in recent years. They aim to enhance images using prior knowledge of images and artificial intelligence technology. Combining deep learning with Retinex theory, RetinexNet was proposed in 2018. This model decomposes the original image into two parts: the illumination image and the reflection image. Based on the color consistency of Retinex, only the brightness of the illumination image needs to be enhanced.

[0007] The third method is a low-light image enhancement algorithm based on a generative adversarial network (GAN). EnlightenGAN proposes an unsupervised GAN model to address the problem of difficulty in obtaining paired data. The model can be trained even without paired low-light and high-light images, and for the first time proposes a dual discriminator to balance global and local enhancements.

[0008] The above algorithms have all made certain contributions to low-light enhancement, but none of them solves the two main problems in low-light enhancement: the first is the color distortion caused by the enhancement of brightness during the enhancement process; the second is that the detailed texture of extreme low-light areas is difficult to restore during the enhancement process. Summary of the invention

[0009] The present invention proposes a low-light image enhancement method based on a multi-scale network structure, which can solve the color distortion problem caused by the enhancement of brightness in the low-light enhancement process; it can also solve the problem that the detailed texture of extreme low-light areas is difficult to restore during the enhancement process.

[0010] The present invention adopts the following technical solutions.

[0011] A low-light image enhancement method based on a multi-scale network structure, the enhancement method includes a loss function for calculating a mean square error, and the color offset of corresponding pixel points on the low-light image and the high-light image is controlled within a threshold range by controlling the mean square error. The loss function calculates the mean square error by calculating the ratio between the pixel values ​​on the three RGB channels of the low-light image and the high-light image; the low-light image enhancement model adopted by the enhancement method is a network model based on a multi-scale network structure, which first performs a gamma transform on the grayscale image of the low-light image, then obtains the grayscale image of the highlight through a branch of the network, and finally weights the low-light image to guide the restoration of the low-light image.

[0012] The training of the low-light image enhancement model includes the following steps:

[0013] Step S1, obtaining pairs of low-light images and high-light images for training a low-light image enhancement model;

[0014] Step S2, gamma transform is performed on the grayscale image of the image. Through nonlinear transformation, the gap between pixels with smaller pixel values ​​in the image is amplified, thereby increasing the contrast in the extreme low-light area of ​​the image and increasing the high-frequency information of the image, so as to restore the detail texture in the image enhancement process; the formula used for gamma transform is as follows:

[0015] E=α·I γ Formula 1;

[0016] Step S3: Divide the image into three scales using a multi-scale network structure, use a dense residual connection module in the network, and add a dilated spatial convolution pooling pyramid module during downsampling of the network to obtain information under different receptive fields, thereby helping to enhance pixel values;

[0017] Step S4: adding a grayscale image branch on the basis of the multi-scale network structure, enhancing the grayscale image of the low-light image separately to obtain more texture information and high-frequency information, and merging it into the feature map of the RGB channel network through the information fusion module;

[0018] Step S5, constructing a color loss function, the specific method is: first, by calculating the pixel value ratio between the three RGB channels in the image to obtain a ratio matrix, respectively obtain the ratio matrix of the low-light image and the enhanced low-light image, and after Sigmoid function processing, compress the ratio to between 0 and 1; finally, the error between the two matrices is calculated by the MSE loss function, and the formula is as follows:

[0019] ColorLoss=MSE[Sigmoid(low),Sigmoid(high)] Formula 2;

[0020] Among them, low represents the ratio matrix of the low-light image, and high represents the ratio matrix of the enhanced low-light image. The color error between the two images is obtained through this loss function;

[0021] Step S6: input the gamma-transformed grayscale image and low-light image into the multi-scale network respectively, and derive the low-light enhanced image and enhanced grayscale image through forward propagation; back-propagate through the color loss function and the MSE loss function, use the gradient descent optimization algorithm to update the convolution weight parameters and bias parameters, and use the cosine annealing strategy to update the learning rate until the preset number of training cycles is completed to obtain the final result.

[0022] In step S1, the method for obtaining a pair of low-light images and high-light images is: cropping the original image of 600x400 pixels into m small images of 256x256 pixels, where m is the result of rounding up the area of ​​the image divided by 256x256.

[0023] In step S5, the color loss function is divided into the following two parts, which can be expressed as

[0024] Loss 灰度图 =MSELoss formula three;

[0025] Loss RGB =MSELoss+α×ColorLoss Formula 4;

[0026] Where α is the weight coefficient.

[0027] In step S6, the gradient descent optimization algorithm is used to update the convolution weight parameters and bias parameters on the image RGB channel backbone network and the grayscale image branch network respectively, and the cosine annealing strategy is used to gradually reduce the learning rate to 0 during the training process. Until the preset number of training cycles is completed, the final result is obtained.

[0028] In step S2, when performing gamma transformation on the grayscale image of the image, the grayscale image of the low-light image is first normalized to transform the pixel values ​​to between 0 and 1, and then the normalized grayscale image of the low-light image is subjected to gamma transformation to improve the contrast of small pixel values ​​in the image by utilizing its nonlinear characteristics.

[0029] In step S3, a multi-scale network structure is constructed to divide the image into three scales: original size, size after one downsampling, and size after two downsamplings; the original size part supplements the shallow information required for enhancing the image, and the two downsampled size parts obtain the deep information of the image; at the same time, a dense residual connection module is used to extract image features.

[0030] The advantages of the solution described in the present invention are:

[0031] First, through a new loss function, the colors of corresponding pixels in low-light images and enhanced images are constrained to not produce significant distortion during the training process.

[0032] Second, the grayscale image of the low-light image is first gamma transformed to improve the contrast of the extreme low-light area in the grayscale image of the low-light image and increase the high-frequency information of the image. After passing through the dense residual connection module, the information is fused with the image on the RGB channel to guide the enhancement of the low-light image.

[0033] Through the above two technical features, the present invention solves the two major problems of color distortion and difficulty in enhancing texture in extreme low-light areas in low-light image enhancement algorithms, and the generated image has fine texture and low color distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0035] Attached Figure 1 It is a structural schematic diagram of a low-light image enhancement algorithm model based on a multi-scale network structure of the present invention;

[0036] Attached Figure 2 It is a structural schematic diagram of some modules in a low-light image enhancement algorithm model based on a multi-scale network structure of the present invention;

[0037] Attached Figure 3 It is a flowchart of a low-light image enhancement algorithm method based on a multi-scale network structure of the present invention;

[0038] Attached Figure 4 This is a schematic diagram of some results of a low-light image enhancement algorithm method based on a multi-scale network structure of the present invention (the upper layer is the input image, and the lower layer is the image after low-light enhancement). DETAILED DESCRIPTION

[0039] As shown in the figure, a low-light image enhancement method based on a multi-scale network structure is provided. The enhancement method includes a loss function for calculating a mean square error. The color offset of corresponding pixels on the low-light image and the high-light image is controlled within a threshold range by controlling the mean square error. The loss function calculates the mean square error by calculating the ratio between the pixel values ​​on the three RGB channels of the low-light image and the high-light image. The low-light image enhancement model adopted by the enhancement method is a network model based on a multi-scale network structure. The model first performs a gamma transform on the grayscale image of the low-light image, then passes it through a branch of the network to obtain a grayscale image of the highlight, and finally weights the low-light image to guide the restoration of the low-light image.

[0040] The training of the low-light image enhancement model includes the following steps:

[0041] Step S1, obtaining pairs of low-light images and high-light images for training a low-light image enhancement model;

[0042] Step S2, gamma transform is performed on the grayscale image of the image. Through nonlinear transformation, the gap between pixels with smaller pixel values ​​in the image is amplified, thereby increasing the contrast in the extreme low-light area of ​​the image and increasing the high-frequency information of the image, so as to restore the detail texture in the image enhancement process; the formula used for gamma transform is as follows:

[0043] E=α·I γ Formula 1;

[0044] Step S3: Divide the image into three scales using a multi-scale network structure, use a dense residual connection module in the network, and add a dilated spatial convolution pooling pyramid module during downsampling of the network to obtain information under different receptive fields, thereby helping to enhance pixel values;

[0045] Step S4: adding a grayscale image branch on the basis of the multi-scale network structure, enhancing the grayscale image of the low-light image separately to obtain more texture information and high-frequency information, and merging it into the feature map of the RGB channel network through the information fusion module;

[0046] Step S5, constructing a color loss function, the specific method is: first, by calculating the pixel value ratio between the three RGB channels in the image to obtain a ratio matrix, respectively obtain the ratio matrix of the low-light image and the enhanced low-light image, and after Sigmoid function processing, compress the ratio to between 0 and 1; finally, the error between the two matrices is calculated by the MSE loss function, and the formula is as follows:

[0047] ColorLoss=MSE[Sigmoid(low),Sigmoid(high)] Formula 2;

[0048] Among them, low represents the ratio matrix of the low-light image, and high represents the ratio matrix of the enhanced low-light image. The color error between the two images is obtained through this loss function;

[0049] Step S6: input the gamma-transformed grayscale image and low-light image into the multi-scale network respectively, and derive the low-light enhanced image and enhanced grayscale image through forward propagation; back-propagate through the color loss function and the MSE loss function, use the gradient descent optimization algorithm to update the convolution weight parameters and bias parameters, and use the cosine annealing strategy to update the learning rate until the preset number of training cycles is completed to obtain the final result.

[0050] In step S1, the method for obtaining a pair of low-light images and high-light images is: cropping the original image of 600x400 pixels into m small images of 256x256 pixels, where m is the result of rounding up the area of ​​the image divided by 256x256.

[0051] In step S5, the color loss function is divided into the following two parts, which can be expressed as

[0052] Loss 灰度图 =MSELoss formula three;

[0053] Loss RGB =MSELoss+α×ColorLoss Formula 4;

[0054] Where α is the weight coefficient.

[0055] In step S6, the gradient descent optimization algorithm is used to update the convolution weight parameters and bias parameters on the image RGB channel backbone network and the grayscale image branch network respectively, and the cosine annealing strategy is used to gradually reduce the learning rate to 0 during the training process. Until the preset number of training cycles is completed, the final result is obtained.

[0056] In step S2, when performing gamma transformation on the grayscale image of the image, the grayscale image of the low-light image is first normalized to transform the pixel values ​​to between 0 and 1, and then the normalized grayscale image of the low-light image is subjected to gamma transformation to improve the contrast of small pixel values ​​in the image by utilizing its nonlinear characteristics.

[0057] In step S3, a multi-scale network structure is constructed to divide the image into three scales: original size, size after one downsampling, and size after two downsamplings; the original size part supplements the shallow information required for enhancing the image, and the two downsampled size parts obtain the deep information of the image; at the same time, a dense residual connection module is used to extract image features.

Claims

1. A low-light image enhancement method based on a multi-scale network structure, characterized by: The enhancement method includes a loss function for calculating a mean square error, and the color offset of corresponding pixels on the low-light image and the high-light image is controlled within a threshold range by controlling the mean square error. The loss function calculates the mean square error by calculating the ratio between the pixel values ​​on the three RGB channels of the low-light image and the high-light image. The low-light image enhancement model adopted by the enhancement method is a network model based on a multi-scale network structure. The training of the low-light image enhancement model includes the following steps: Step S1, obtaining pairs of low-light images and high-light images for training a low-light image enhancement model; Step S2, gamma transform is performed on the grayscale image of the image. Through nonlinear transformation, the gap between pixels with smaller pixel values ​​in the image is amplified, thereby increasing the contrast in the extreme low-light area of ​​the image and increasing the high-frequency information of the image, so as to restore the detail texture in the image enhancement process; the formula used for gamma transform is as follows: E=α·I γ Formula 1; Step S3: Divide the image into three scales using a multi-scale network structure, use a dense residual connection module in the network, and add a dilated spatial convolution pooling pyramid module during downsampling of the network to obtain information under different receptive fields, thereby helping to enhance pixel values; Step S4: adding a grayscale image branch on the basis of the multi-scale network structure, enhancing the grayscale image of the low-light image separately to obtain more texture information and high-frequency information, and merging it into the feature map of the RGB channel network through the information fusion module; Step S5, constructing a color loss function, the specific method is: first, by calculating the pixel value ratio between the three RGB channels in the image to obtain a ratio matrix, respectively obtain the ratio matrix of the low-light image and the enhanced low-light image, and after Sigmoid function processing, compress the ratio to between 0 and 1; finally, the error between the two matrices is calculated by the MSE loss function, and the formula is as follows: ColorLoss=MSE[Sigmoid(low),Sigmoid(high)] Formula 2; Among them, low represents the ratio matrix of the low-light image, and high represents the ratio matrix of the enhanced low-light image. The color error between the two images is obtained through this loss function; Step S6: input the gamma-transformed grayscale image and low-light image into the multi-scale network respectively, and derive the low-light enhanced image and enhanced grayscale image through forward propagation; back-propagate through the color loss function and the MSE loss function, use the gradient descent optimization algorithm to update the convolution weight parameters and bias parameters, and use the cosine annealing strategy to update the learning rate until the preset number of training cycles is completed to obtain the final result.

2. The low-light image enhancement method based on a multi-scale network structure according to claim 1, characterized in that: In step S1, the method for obtaining a pair of low-light images and high-light images is: cropping the original image of 600x400 pixels into m small images of 256x256 pixels, where m is the result of rounding up the area of ​​the image divided by 256x256.

3. The low-light image enhancement method based on a multi-scale network structure according to claim 1, characterized in that: In step S5, the color loss function is divided into the following two parts, which can be expressed as Loss 灰度图 =MSELoss Formula 3; Loss RGB = MSELoss + α × ColorLoss Formula 4; Where α is the weight coefficient.

4. The low-light image enhancement method based on a multi-scale network structure according to claim 1, characterized in that: In step S6, the gradient descent optimization algorithm is used to update the convolution weight parameters and bias parameters on the image RGB channel backbone network and the grayscale image branch network respectively, and the cosine annealing strategy is used to gradually reduce the learning rate to 0 during the training process; until the preset number of training cycles is completed, the final result is obtained.

5. The low-light image enhancement method based on a multi-scale network structure according to claim 1, characterized in that: In step S2, when performing gamma transformation on the grayscale image of the image, the grayscale image of the low-light image is first normalized to transform the pixel values ​​to between 0 and 1, and then the normalized grayscale image of the low-light image is subjected to gamma transformation to improve the contrast of small pixel values ​​in the image by utilizing its nonlinear characteristics.

6. The low-light image enhancement method based on a multi-scale network structure according to claim 1, characterized in that: In step S3, a multi-scale network structure is constructed to divide the image into three scales: original size, size after one downsampling, and size after two downsampling; The original size part supplements the shallow information needed to enhance the image, and the two size parts after downsampling obtain the deep information of the image; at the same time, the dense residual connection module is used to extract image features.

Images