Method for optimizing halo problem by introducing PPM structure into low light enhancement network
By designing a deep learning neural network structure based on RAW domain in a low-light environment, introducing the PPM structure optimization halo problem, solving the problem of improving low-light image quality, achieving effective improvement in contrast, color and detail recovery, and maintaining low computing power requirements during terminal deployment.
Patent Information
- Application Number
- CN202311547089.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-20
- Publication Date
- 2025-05-20
AI Technical Summary
In low-light environments, the signal-to-noise ratio of images is low, and traditional image enhancement methods cannot effectively improve the quality of low-light images. Deep learning methods have problems in contrast, color, and detail recovery.
Design a deep learning neural network structure based on RAW domain, and introduce the halo problem generated during low-light enhancement process of PPM structure optimization. The network adopts a Unet structure with jump connection, combines a simplified Resnet module and a Resnet module with attention mechanism, uses PixelShuffle and its inverse operations to achieve upsampling and downsampling, and realizes global multi-scale context information interaction through a pyramid pooling module.
It has achieved improvement in image quality in low-light environments, especially in contrast, color and detail recovery. At the same time, due to the structural design, the network has less computing power when deploying terminals, which is suitable for practical applications.
Smart Images
Figure CN120020864A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of neural networks, and particularly relates to a method for optimizing the halo problem by introducing a PPM structure into a low-light enhancement network. Background Art
[0002] With the development of technology, the consumer demand for intelligent devices is increasing day by day. The introduction of vision technology into intelligent devices has become an essential weapon for competition in major markets. Vision technology is inseparable from the imaging quality of images, and the imaging quality in extreme scenarios is particularly crucial. The low-light scene image enhancement technology is one of the links. In a low-light environment, the signal-to-noise ratio of an image is low; short-exposure images have a lot of noise and significant detail loss, while long-exposure images are prone to overexposure, being blurry and unrealistic. How to obtain high-quality low-light images or low-light enhanced images is a technical problem currently faced. When traditional image enhancement methods cannot effectively improve the quality of low-light images, scholars have proposed different deep learning-based methods for enhancing low-light images, opening up a new idea for this field. However, deep learning-based methods for enhancing low-light images still face some problems, such as contrast improvement, detail restoration, and color restoration.
[0003] Low-light image enhancement (LLIE) aims to improve the perceptual quality of images captured in low-light environments. The recent progress in this field is mainly dominated by deep learning methods (including different learning strategies, network architectures, loss functions, training data, etc.). Learning strategies include unsupervised, self-supervised, etc.; network architectures are also innovative, with those based on CNN, GAN, and those combining CNN with Retinex theory; loss functions include L1, L2, SSIM, MS_SSIM, perceptual loss, smooth loss, etc.; many scholars have also released some low-light datasets, such as LOL, SCIE, SID, etc.
[0004] However, traditional image enhancement methods, such as global histogram equalization and gamma transformation, have poor enhancement effects on low-light images. And some existing deep learning methods may have their own problems in terms of contrast, color, detail restoration, model structure complexity, dataset, etc. Summary of the Invention
[0005] To solve the above problems, the purpose of the present application is to provide a deep learning neural network structure based on the RAW domain for the low-light enhancement field. This network is convenient for terminal deployment in terms of structure and requires less computing power; and a pyramid pooling module is added on the basis of this network to further optimize the halo problem generated during the low-light enhancement process.
[0006] Specifically, the present invention provides a method for optimizing the halo problem by introducing a PPM structure into a low-light enhancement network, and the method includes:
[0007] S1. Build a low-light enhancement network introducing the PPM structure:
[0008] S1.1. Overall, use the Unet structure with skip connections to achieve context information interaction;
[0009] S1.2. In the first 4 layers, use the simplified Resnet module to reduce the number of network layers and network computational complexity;
[0010] S1.3. In the last 3 layers, use the Resnet module with attention mechanism to enable the network to better focus on the global and local brightness information of the image during training;
[0011] S1.4. Use PixelShuffle and its inverse operation to achieve the upsampling and downsampling functions of the network;
[0012] S1.5. Introduce the pyramid pooling module PPM structure to achieve global multi-scale context information interaction; among them, the output channels of the first and second PPM modules are 256, the internal convolution channels are 32, and the multi-scale parameters are [1, 2, 3, 6] respectively; the output channels of the third PPM module are 24, the internal convolution channels are 8, and the multi-scale parameters are [1, 6] respectively; perform pooling operations on the original feature map at 4 different scales to obtain 4 feature maps of different sizes; then perform upsampling operations on these 4 feature maps of different sizes to restore to the size of the original feature map, and finally splice them in the channel dimension to obtain the final composite feature map;
[0013] S1.6. The network output is a curve graph iterated N times. The final prediction graph is obtained by performing curve enhancement iteration on the input graph N times. The weight coefficient can be set for the curve graph to control the final brightness effect;
[0014] S2. Train the network. The loss function used for network training is shown in the following formula (1):
[0015] loss = w 1 *MAE_loss + w 2 *TV_loss Formula (1)
[0016] Among them, MAE_loss represents the mean absolute error loss function, as shown in formula (2);
[0017] TV_loss represents the gradient smoothing loss function, as shown in formula (3);
[0018] w 1 、w 2 are the weights of the corresponding loss functions respectively, and can be set to w 1 = 20, w 2 = 5;
[0019]
[0020] Among them, Ω represents the position domain of all pixel points of the image, (i, j) is a position on the i-th row and j-th column of the image, x is the prediction result map obtained by training the input image through the network, and y is the GroundTruth image corresponding to the input image; M represents the total number of pixel points of the image;
[0021]
[0022] Among them, N is the number of iterations, and are the horizontal and vertical gradient operators respectively;
[0023] S3. Use the paired low-light training set, where each pair of data contains two images, one original low-light raw format image and one brightened image processed by artificial synthesis and screening operations, and perform training iterations on the proposed network. The training set is RAW format images, the initial learning rate of network training is set to le-4, and the decay coefficient of each epoch is 0.95.
[0024] In the step S1.1, for the Unet structure with skip connections, the overall network structure of the Unet is U-shaped and can fuse shallow features and deep features, including:
[0025] conv_first is a single-layer convolution; the convolution kernel is 3x3, the number of output channels is 8, and the activation function is Relu;
[0026] conv_last is a convolution module with an attention mechanism:
[0027] The number of output channels of the first convolution is 16, the activation function is Relu, and the convolution kernel is 1x1;
[0028] The result of the first convolution passes through the channel attention module and the spatial attention module in sequence to obtain the output result of the conv_last module. Among them, the number of output channels of the first convolution of the channel attention module is 4, and the number of output channels of the second convolution is 16; among them, the number of output channels of the first convolution of the spatial attention module is 16, and the number of output channels of the second convolution is 1.
[0029] In the step S1.2, the simplified Resnet module includes:
[0030] The number of output channels of the first convolution is 32, the activation function is Relu, and the convolution kernel is 1x1;
[0031] The number of output channels of the second convolution is 8, the activation function is Relu, and the convolution kernel is 3x3;
[0032] The number of output channels of the 3rd convolution is 32, the activation function is Relu, and the convolution kernel is 1x1;
[0033] The residual part is the sum of the first convolution output and the third convolution output.
[0034] The step S1.3 further includes:
[0035] S1.3.1, Layers 5 and 6 use the Resnet module with an attention mechanism. The Resnet module is the residual network module, and its structure is as described in step S1.2, which consists of 3 convolutions. The residual part is the sum of the first convolution output and the third convolution output to output the final result of the module. Specifically, the number of output channels of the 1st convolution is 128, the activation function is Relu, and the convolution kernel is 1x1; the number of output channels of the 2nd convolution is 32, the activation function is Relu, and the convolution kernel is 3x3; the number of output channels of the 3rd convolution is 128, the activation function is Relu, and the convolution kernel is 1x1; the residual part is the sum of the first convolution output and the third convolution output;
[0036] The attention mechanism before the Resnet structure is that the feature map is input into the channel_attention and spatial_attention modules in sequence. The structure of the channel attention module channel_attention is as follows: the input is subjected to global average pooling and then enters the first convolution, with the number of output channels being 16, the convolution kernel being 1x1, and the activation function being Relu; the second convolution has 64 output channels, a 1x1 convolution kernel, and a sigmoid activation function, and the output is the product of the input and the second convolution; the structure of the spatial attention module spatial_attention is as follows: the input enters the first convolution, with 64 output channels, a 1x1 convolution kernel, and a Relu activation function; then it enters the second convolution, with 1 output channel, a 1x1 convolution kernel, and a sigmoid activation function, and the output is the product of the input and the second convolution;
[0037] S1.3.2, Layer 7 has the same structure as that in S1.3.1 above. Specifically, all convolution channels in the Resnet module are 32, and other settings are the same as those in S.1.3.1 above.
[0038] In the step S1.6,
[0039] The quadratic function for each iteration refers to the Zero-DCE network, specifically as shown in formula (0);
[0040] LE n (x) = LE n-1 (x) + A n LE n-i (x)(1 - LEn-1 (x)) Formula (0)
[0041] Among them, A n is the n-th channel curve graph output by the network; LE n is the final low-light enhancement prediction graph.
[0042] The method is set to N = 8.
[0043] Therefore, the advantages of this application are as follows:
[0044] 1. Overall, the Unet structure with skip connections is used (as shown in the attached Figure 2 ) to achieve context information interaction;
[0045] 2. The first 4 layers use a simplified Resnet module (as shown in the attached Figure 3 ) to reduce the number of network layers and network computation;
[0046] 3. The last 3 layers use a Resnet module with an attention mechanism (as shown in Figure 4(a) of the attached drawing), enabling the network to better focus on more important information during the training process;
[0047] 4. PixelShuffle and its inverse operation are used to achieve the upsampling and downsampling functions of the network, without information loss or introduction of non-original information;
[0048] 5. The PPM (Pyramid Pooling Module) structure is introduced, and its structure is as shown in the attached Figure 5 to achieve global multi-scale context information interaction;
[0049] 6. The network output is a curve graph iterated N times. The final prediction graph is obtained by performing curve enhancement iteration on the input graph multiple times. The weight coefficient can be set for the curve graph to control the final brightness effect.
[0050] 7. RAW format data is selected for image enhancement. RAW domain data contains more color gamuts and a higher dynamic range. Therefore, the depth model based on RAW data can reconstruct clearer details, higher contrast, and has better color information. Description of the Drawings
[0051] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not limit the present invention.
[0052] Figure 1 is a schematic structural diagram of a low-light enhancement network incorporating the PPM structure constructed in this application.
[0053] Figure 2 is a schematic diagram of the Unet structure with skip connections in this application.
[0054] Figure 3 It is a schematic diagram of the simplified Resnet module in this application.
[0055] Figure 4(a) is a schematic diagram of the Resnet module with an attention mechanism used in this application.
[0056] Figure 4(b) and Figure 4(c) are schematic diagrams of the attention mechanism before the Resnet structure in this application. Figure 4(b) is the channel attention module (channel_attention), and Figure 4(c) is the spatial attention module (spatial_attention).
[0057] Figure 5 It is a schematic diagram of the structure introducing the Pyramid Pooling Module (PPM) structure in this application. Detailed implementation manners
[0058] In order to more clearly understand the technical content and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.
[0059] The solution of this application belongs to the technical field of low-light enhancement in deep learning neural network vision tasks, and relates to a method for improving a low-light enhancement network to eliminate halos. Specifically, a method for introducing a PPM structure into a low-light enhancement network to optimize the halo problem is provided, and the main content of the method is introduced as follows:
[0060] S1. Construct a low-light enhancement network introducing the PPM structure, and its structural schematic diagram is as shown in the appendix; S1.1. Overall, use a Unet structure with skip connections (as shown in the appendix Figure 1 ) The overall network structure of the Unet is U-shaped, which can fuse shallow features and deep features. It is a relatively excellent network structure in the current image field, with the characteristics of feature fusion and light weight, and realizes context information interaction; among them, conv_first is a single-layer convolution. The convolution kernel is 3x3, the number of output channels is 8, and the activation function is Relu; conv_last is a convolution module with an attention mechanism: Figure 2 The number of output channels of the first convolution is 16, the activation function is Relu, and the convolution kernel is 1x1;
[0061] The result of the first convolution passes through the channel attention module and the spatial attention module in sequence to obtain the output result of the conv_last module. Among them, the number of output channels of the first convolution of the channel attention module is 4, and the number of output channels of the second convolution is 16; among them, the number of output channels of the first convolution of the spatial attention module is 16, and the number of output channels of the second convolution is 1;
[0062]
[0063] S1.2. The first 4 layers use a simplified Resnet module (as shown in the appendixFigure 3 As shown in the figure, reduce the number of network layers and network computation; among them, the number of output channels of the first convolution is 32, the activation function uses Relu, and the convolution kernel is 1x1; the number of output channels of the second convolution is 8, the activation function uses Relu, and the convolution kernel is 3x3; the number of output channels of the third convolution is 32, the activation function uses Relu, and the convolution kernel is 1x1; the residual part is the sum of the first convolution output and the third convolution output.
[0064] S1.3. In the last three layers, use the Resnet module with attention mechanism to enable the network to better focus on the global and local brightness information of the image during training; the network proposed in this application is used for low-light enhancement, and the attention module can better focus on the global and local brightness information of the image; the local brightness information can ensure that the dark areas are effectively brightened, while the bright areas are moderately brightened or not brightened; the global information can ensure the overall consistency of the brightness of the whole image, and it is not easy to cause the dark areas to be brightened too much and exceed the bright areas of the original image; further including:
[0065] S1.3.1. In Layer 5 and 6, use the Resnet module with attention mechanism (as shown in Figure 4(a)). The Resnet module is the residual network module, and its structure is as described in step S1.2, which consists of three convolutions. The residual part is the sum of the first convolution output and the third convolution output to output the final result of the module. The first two layers in Figure 4(a) are the attention module, and the subsequent structure is the Resnet module; specifically set as:
[0066] The number of output channels of the first convolution is 128, the activation function uses Relu, and the convolution kernel is 1x1; the number of output channels of the second convolution is 32, the activation function uses Relu, and the convolution kernel is 3x3; the number of output channels of the third convolution is 128, the activation function uses Relu, and the convolution kernel is 1x1; the residual part is the sum of the first convolution output and the third convolution output;
[0067] Before the Resnet structure, the attention mechanism is that the feature map is successively input into the channel_attention and spatial_attention modules. The structure of the channel attention module channel_attention (Figure 4(b)) is as follows: The input passes through global average pooling and enters the first convolution with an output channel number of 16, a convolution kernel of 1x1, and a ReLU activation function; the second convolution has an output channel number of 64, a convolution kernel of 1x1, and a sigmoid activation function, and the output is the product of the input and the second convolution. The structure of the spatial attention module spatial_attention (Figure 4(c)) is as follows: The input enters the first convolution with an output channel number of 64, a convolution kernel of 1x1, and a ReLU activation function; then it enters the second convolution with an output channel number of 1, a convolution kernel of 1x1, and a sigmoid activation function, and the output is the product of the input and the second convolution.
[0068] S1.3.2, The structure of Layer 7 is the same as that of the above content S1.3.1. Specifically, all convolution channels in the Resnet module are 32, and other settings are the same as those in the above content S1.3.1.
[0069] S1.4, Use PixelShuffle and its inverse operation to implement the upsampling and downsampling functions of the network; PixelShuffle (Sub-Pixel Convolutional Neural Network) is a classic upsampling method proposed in "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network" for dealing with the problem of image super-resolution, which can effectively enlarge the downsampled feature map.
[0070] S1.5, Introduce the PPM (Pyramid Pooling Module) structure, as shown in the appendix Figure 5 to achieve global multi-scale context information interaction; the output channel numbers of the first and second PPM modules are 256, the internal convolution channel numbers are 32, and the multi-scale parameters are [1, 2, 3, 6] respectively; the output channel number of the third PPM module is 24, the internal convolution channel number is 8, and the multi-scale parameters are [1, 6] respectively. As Figure 5 shown, perform pooling operations on the original feature map at 4 different scales to obtain 4 feature maps of different sizes; then perform upsampling operations on these 4 feature maps of different sizes to restore them to the size of the original feature map, and finally concatenate them in the channel dimension to obtain the final composite feature map.
[0071] S1.6. The network output is a curve graph after N iterations. The quadratic function for each iteration refers to the Zero-DCE network, specifically as shown in formula (0);
[0072] LE n (x) = LE n-1 (x) + A n LE n-1 (x)(1 - LE n-1 (x)) Formula (0)
[0073] Among them, A n is the curve graph of the nth channel output by the network; LE n is the final low-light enhancement prediction graph.
[0074] S2. Train the network. The loss function used for network training is as shown in formula (1) below:
[0075] loss = w 1 *MAE_loss + w 2 *TV_loss Formula (1)
[0076] Among them, MAE_loss represents the mean absolute error loss function (as shown in formula (2)); TV_loss represents the gradient smoothing loss function (as shown in formula (3)); w 1 , w 2 are the weights of the corresponding loss functions, which can be set to w 1 = 20, w 2 = 5;
[0077]
[0078] Among them, Ω represents the position domain of all pixel points of the image, (i, j) is a position on the i-th row and j-th column of the image, x is the prediction result graph obtained by training the input image through the network, and y is the GroundTruth image corresponding to the input image; M represents the total number of pixel points of the image;
[0079]
[0080] Among them, N is the number of iterations (after multiple experimental verifications, the best effect is achieved when N = 8, and this method is set to N = 8), and are the horizontal and vertical gradient operators respectively.
[0081] S3. Use the paired low-light training set to perform training iterations on the above network. The training set is RAW format images. The initial learning rate for network training is set to le-4, and the decay coefficient for each epoch is 0.95.
[0082] The paired low-light training set, where each pair of data contains two images, one original low-light raw format image and one brightened image processed by artificial synthesis and screening operations, is used to perform training iterations on the network proposed in this application (such as Figure 1 shown). The specific method for making the dataset can adopt existing methods and will not be elaborated again.
[0083] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A method for optimizing the halo problem by introducing a PPM structure into a low-light enhancement network, characterized in that: The method comprises: S1, build a low-light enhancement network introducing the PPM structure: S1.1, overall, a Unet structure with skip connections is used to achieve context information interaction; S1.2, the first 4 layers use simplified Resnet modules to reduce the number of network layers and the amount of network calculation; S1.3, the last three layers use the Resnet module with attention mechanism, which enables the network to better focus on the global and local brightness information of the image during training; S1.4, use PixelShuffle and its inverse operation to implement the upsampling and downsampling functions of the network; S1.5, introduces the pyramid pooling module PPM structure to realize global multi-scale context information interaction; the first and second PPM modules have 256 output channels, 32 internal convolution channels, and multi-scale parameters [1, 2, 3, 6] respectively; the third PPM module has 24 output channels, 8 internal convolution channels, and multi-scale parameters [1, 6] respectively; the original feature map is pooled at 4 different scales to obtain 4 feature maps of different sizes; then the 4 feature maps of different sizes are upsampled to restore them to the original feature map size, and finally spliced in the channel dimension to obtain the final composite feature map; S1.6, the network output is a curve graph with N iterations. The input image is iterated through N curve enhancement iterations to obtain the final prediction image. The weight coefficient can be set for the curve graph to control the final brightness effect. S2. Train the network. The loss function used in network training is shown in the following formula (1): loss=w1*MAE_loss+w2*TV_loss formula (1) Wherein, MAE_loss represents the mean absolute error loss function, as shown in formula (2); TV_loss represents the gradient smoothing loss function, as shown in formula (3); w1 and w2 are the weights of the corresponding loss functions, which can be set to w1=20 and w2=5 respectively; Where Ω represents the location domain of all pixels in the image, (i, j) is a position in the i-th row and j-th column on the image, x is the predicted result image obtained by network training, and y is the GroundTruth image corresponding to the input image; M represents the total number of pixels in the image; Where N is the number of iterations, and are the horizontal and vertical gradient operators respectively; S3. Use the paired low-light training set, where each pair of data contains two images, one original low-light RAW image and one brightened image after artificial synthesis and filtering operations, to train the proposed network iteratively. The training set is RAW format images, the initial learning rate of the network training is set to le-4, and the decay coefficient of each epoch is 0.
95.
2. According to claim 1, a method for optimizing the halo problem by introducing a PPM structure into a low-light enhancement network is characterized in that: In step S1.1, the Unet structure with jump connections has a U-shaped overall network structure that can fuse shallow features and deep features, including: conv_first is a single-layer convolution; the convolution kernel is 3x3, the number of output channels is 8, and the activation function is Relu; conv_last is a convolution module with attention mechanism: The first convolution has 16 output channels, uses Relu as activation function, and has a convolution kernel of 1x1. The first convolution result passes through the channel attention module and the spatial attention module in turn to obtain the output result of the conv_last module, where the first convolution of the channel attention module outputs channel 4, and the second convolution outputs channel 16; the first convolution of the spatial attention module outputs channel 16, and the second convolution outputs channel 1.
3. According to claim 1, a method for optimizing the halo problem by introducing a PPM structure into a low-light enhancement network is characterized in that: In step S1.2, the simplified Resnet module includes: the first convolution output channel number is 32, the activation function adopts Relu, and the convolution kernel is 1x1; The second convolution has 8 output channels, uses Relu as the activation function, and has a convolution kernel of 3x3. The third convolution has 32 output channels, uses Relu as activation function, and has a convolution kernel of 1x1. The residual part is the sum of the first convolution output and the third convolution output.
4. According to claim 1, a method for optimizing halo problem by introducing PPM structure into low-light enhancement network is characterized in that: The step S1.3 further comprises: S1.3.1, Layer 5 and 6 use the Resnet module with attention mechanism. The Resnet module is a residual network module. Its structure is as described in step S1.
2. It consists of 3 convolutions. The residual part is the sum of the first convolution output and the third convolution output to output the final result of the module; the specific settings are: the first convolution output channel number is 128, the activation function uses Relu, and the convolution kernel is 1x1; the second convolution output channel number is 32, the activation function uses Relu, and the convolution kernel is 3x3; the third convolution output channel number is 128, the activation function uses Relu, and the convolution kernel is 1x1; the residual part is the sum of the first convolution output and the third convolution output; The attention mechanism before the Resnet structure is that the feature map is input into the channel_attention and spatial_attention modules in turn. The channel_attention module has the following structure: the input is globally averaged and enters the first convolution. The output channel number is 16, the convolution kernel is 1x1, and the activation function is Relu; the second convolution output channel number is 64, the convolution kernel is 1x1, the activation function is sigmoid, and the output is the product of the input and the second convolution; the spatial attention module has the following structure: the input enters the first convolution, the output channel number is 64, the convolution kernel is 1x1, and the activation function is Relu; then enters the second convolution, the output channel number is 1, the convolution kernel is 1x1, the activation function is sigmoid, and the output is the product of the input and the second convolution; S1.3.2, Layer7 has the same structure as S1.3.1 above. In the specific Resnet module, all convolution channels are 32, and other settings are consistent with those in S.1.3.1 above.
5. According to claim 1, a method for optimizing the halo problem by introducing a PPM structure into a low-light enhancement network is characterized in that: In step S1.6, The quadratic function of each iteration refers to the Zero-DCE network, as shown in formula (0); LE n (x) = LE n-1 (x) + A n LE n-1 (x)(1 - LE n-1 (x)) Formula (0) Among them, A n That is the nth channel curve graph of the network output; LE n The final low-light enhancement prediction map.
6. According to claim 1, a method for optimizing halo problem by introducing PPM structure into low-light enhancement network is characterized in that: The method is set as N=8.