Extremely low-light image enhancement method based on multistage convergence block and comprehensive loss function system

Abstract

An extremely low-light image enhancement method based on a multistage convergence (MSC) block and a comprehensive loss function system, comprising an MSC block and a comprehensive loss function system. The MSC block comprises a selection block and a denoising block. By using an attention mechanism, the MSC block can efficiently extract and recover fine-grained image details in an extremely low illumination environment, and can reduce noise caused by insufficient sensitivity while restoring real colors. Extremely low-light image enhancement can provide a visual effect having brightness and clarity similar to those of sunlight in a highly dark environment, and plays a key role in both civil and military applications.

Description

Low-light image enhancement method based on multi-stage convergence block and comprehensive loss function system TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and specifically relates to a low-light image enhancement method based on a multi-stage convergence block and a comprehensive loss function system. BACKGROUND

[0002] Under extremely low light conditions, due to limited light flux, images often have low brightness, low contrast and a large amount of noise. This results in unclear image details and makes it difficult for the human eye to distinguish meaningful information. To solve this problem, low-light image enhancement technology aims to improve image clarity, enhance details, reduce noise, and improve brightness and contrast. This technology has a wide range of applications in night shooting, surveillance, military reconnaissance and medical imaging.

[0003] Since 2017, low-light image enhancement methods based on deep learning, represented by LLNet, have become increasingly popular. These solutions use different learning strategies on RAW or RGB format datasets to try to solve the problem of low-light image enhancement. For example, MBLLEN, Retinex-Net, LPNet, DSLR based on supervised learning, DeepExposure based on reinforcement learning, EnlightenGAN based on unsupervised learning, ExCNet, ZERO-DCE and DRBN based on semi-supervised learning.

[0004] However, these models are trained using LOL datasets, and their image lighting is still different from real low-light environments. In addition, Transformer models lack the ability to effectively model spatial information and have a large number of parameters, making them less suitable for processing high-resolution images.

[0005] SUMMARY

[0006] To address the above issues of unsatisfactory datasets and unsuitable network models, the purpose of the present application is

[0007] To provide a specialized dataset for low-light images and a multi-stage convergence block for low-light image enhancement.

[0008] Technical solution: The present application is a multi-stage convergence (MSC) block and a comprehensive loss function system for low-light image enhancement. The MSC block includes a selection block and a denoising block. By combining attention mechanisms, this block can effectively extract and restore fine-grained image details in low-light environments, reducing noise caused by insufficient light sensitivity while restoring true colors.

[0009] The very low light image enhancement system of the method is shown in Figure 1, which adopts a U-Net network structure, which is known for its efficiency in extracting and processing multi-scale features. In the U-Net structure, given an RGB image, a feature extraction block is used at each layer to enhance details through internal attention mechanisms and suppress noise through external attention mechanisms. In addition, down-sampling or up-sampling operations are introduced after each feature extraction block to capture deeper feature information. To further address the problem of detail loss, a multi-level convergence block is introduced. This block takes feature maps from the first n (n = 3 in the method) output layers and the final output layer of the network, and performs a convergence operation. The result is a three-channel RGB image as the final output.

[0010] In order to better simulate the very low light environment and take advantage of the self-attention mechanism in the Transformer, the present application provides a special dataset for very low light images and proposes MRASN (Multi-Residual Attention Shrinkage Network) as an advanced method for enhancing very low light images.

[0011] The structure of the multi-level convergence block is shown in Figures 2-3, which includes a selection block and a denoising block. The feature maps of the first n (n = 3) layers and the output layer of the U-shaped network are sent to the selection block. First, global pooling and multi-layer perceptron (MLP) are used to calculate channel weights in space to capture channels containing rich and complex information, and then spatial weights are obtained by using global pooling in the channels to find edge information in the feature maps. The output is obtained by multiplying the three and convolution operation

[0012] M = Conv (I + I · F c · F s )

[0013] = Conv (I + I · ReLU (MLP (Pool s (I)))

[0014] · ReLU (Conv (Pool c (I)))

[0015] where Pools and Poolc represent global pooling in space and channels, respectively.

[0016] Since the additional information also increases the noise, it is necessary to denoise the Input into the denoising block, calculate the absolute value of all features in the feature map, obtain a one-dimensional vector after global average pooling, and then pass it to the two-layer FC network to calculate the threshold Finally subtract the threshold and activate to obtain the output The formula is as follows:

[0017] O = ReLU (M - tau)

[0018] Where z c is the feature at the cth neuron, i, j and c are the width, height and channel of the feature map respectively.

[0019] The loss function body selects the mean absolute error loss (L1) to make the network converge quickly and realize the overall enhancement of the image. Secondly, in order to enrich the color information and reduce the error between the image and the real color, the application designs a color recovery loss function to calculate the color average error of each channel in the three channels. The color recovery loss function is as follows:

[0020] Where G j and Y j represent the average color value of the p channel in the real image and the output image respectively.

[0021] In order to eliminate color shift and solve the problem of edge blur of the recovered image, the application adds an average structure loss (L ssim ), which divides the image into smaller parts, calculates the structure loss in the local range, and then takes the average value of the whole. The total loss function is as follows:

[0022] L = alpha L1 + beta L c + (1-alpha-beta) L ssim

[0023] Where alpha and beta are the weights of the loss function.

[0024] Beneficial effects: (1) A new method combining global attention mechanism and CNN model is proposed to enhance extremely low light images. The method introduces multi-level convergence block and comprehensive loss function system, which can improve the brightness, detail recovery and color recovery ability of the image. It can effectively restore fine-grained image details and real colors while reducing noise, and the effect surpasses previous models. (2) In addition, compared with other models, the model used in the application shows significantly faster processing speed and can smoothly process extremely low light videos. BRIEF DESCRIPTION OF DRAWINGS

[0025] Fig. 1 is a flowchart of the extremely low light image enhancement method according to the application.

[0026] Figure 2 is a structural diagram of a selection block in the multi-level convergence block of the present application.

[0027] Figure 3 is a structural diagram of a denoising block in the multi-level convergence block of the present application.

[0028] The reference signs in the drawings are as follows: 1 - ReLU activation function; 2 - matrix multiplication; 3 - element addition. DETAILED DESCRIPTION

[0029] The technical solutions of the present application will be further described in detail below in conjunction with the drawings of the specification.

[0030] Figure 1 is a flowchart of the method described in the present application, which adopts a U-Net network structure. In the U-Net structure, given an RGB image, a feature extraction block is used at each layer to enhance details through internal attention mechanisms and suppress noise through external attention mechanisms. In addition, down-sampling or up-sampling operations are introduced after each feature extraction block to capture deeper feature information. In order to further solve the problem of detail loss, a multi-level convergence block is introduced. This block obtains feature maps from the first n (n = 3 in this embodiment) output layers and the final output layer of the network, and performs convergence operations. The result is a three-channel RGB image as the final output. In addition, a new loss function system has been developed to enhance the ability of the block to extract complex details and accurately restore colors, thereby avoiding color shift phenomena. Figure 1 is the network model proposed, and the loss function is used when training the model.

[0031] The training and testing procedures were performed on a Dell small computing workstation with the following specifications: an Intel(R) Xeon(R) Gold 5218R CPU and four NVIDIA GeForce 3090 GPUs. The framework was implemented using PyTorch and Python 3.7. During training, the images in the training set were randomly cropped into small blocks of size 512x512. The data was further augmented through random flipping, rotation, and scaling techniques. The batch size was set to 8, and the network was initialized using default parameters. During training, the network was trained for a total of 70 epochs using the Adam optimizer. For the first 20 epochs, the initial learning rate was set to 1e-3, and then it was reduced to 1e-4 for the remaining epochs. The loss function hyperparameters a and b were set to 0.5 and 0.2, respectively, to achieve the required balance between the different components of the loss function.

[0032] Based on the above, the method of the present application is compared with the most advanced algorithms for low-light image enhancement and very low-light image enhancement proposed in recent years. Specifically, the method of the present application is evaluated for MBLLEN, DSLR, Zero-DCE, MRASN based on CNN and IAT and LLFormer based on Transformer. In order to evaluate the performance of different methods on different data sets, two widely used data sets, SID, LOL, and the proposed MRASN are selected. These data sets can test the processing effect, generalization ability, noise suppression ability and detail representation ability of each method. It is found through calculation that the LOL data set has moderate noise, the SID data set has a darker environment, and the MRASN shows lower brightness and higher noise level.

[0033] The key parameters commonly used for image evaluation are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). PSNR is a reliable indicator of the quality of the processed image, and the higher the PSNR value, the lower the image noise. Since image details and noise are both high-frequency components, SSIM provides a complementary measure of the degree of detail preservation, including brightness, contrast, color and detail features, etc. After processing each data set, the PSNR and SSIM of the above-mentioned methods and the method of the present application are calculated.

[0034] According to the results in Table 1, the method of the present application is excellent in terms of SSIM and PSNR, and reaches the highest value in both indicators. Higher PSNR value ensures the reduction of distortion and the improvement of fidelity of the reconstructed image, and at the same time, compared with other methods, higher SSIM value means significant improvement in brightness, contrast, color and detail features.

[0035] Table 1

[0036] In addition, runtime tests are also conducted on various data sets to compare the processing time of the method of the present application with other methods, while ensuring the consistency of the platform configuration. The LOL data set contains images with a size of 600x400, while the SID data set images are converted to PNG format with a size of 1920x1080. Similarly, the data set used by the present application also consists of images with a size of 1920x1080. Each method is thoroughly tested on each image in each data set, and the average running time results are recorded as shown in Table 2. In summary, compared with other methods, the method of the present application has superior computational efficiency, and has significant stability in efficiency when applied to various data sets.

[0037] Table 2

[0038] To verify the effectiveness of each component in the method described in the present application, separate tests were also conducted to evaluate the effect when only the selection block and only the denoising block were used, which performed worse compared to merging the entire MSC block. In addition, to determine the coefficients of the loss function, various tests were also conducted, including the separate use of L1 loss, the combination of L1 and L c ssim These tests demonstrated the effectiveness of each loss function. Finally, the effect of the number of layers on compensating for missing details was also tested. These specific data are shown in Table 3.

[0039] Table 3

[0040] In the present application, a new method combining global attention mechanism and CNN model is studied to enhance extremely low light images. The method introduces a multi-level convergence block and a comprehensive loss function system, which can improve the brightness, detail recovery and color recovery ability of the image. It can effectively restore fine-grained image details and real colors while reducing noise. Compared with other models, the model used in the present application exhibits significantly faster processing speed and can smoothly process extremely low light videos.

[0041] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above embodiments. Any equivalent modifications or changes made by those of ordinary skill in the art based on the disclosed content of the present application shall be included in the protection scope of the claims.​

Claims

1. A method for enhancing extremely low light images based on multi-stage convergent block and comprehensive loss function system, characterized in that: The method comprises a U-Net network structure using multi-stage convergent MSC block and a comprehensive loss function system; The MSC block comprises a selection block and a denoising block, and by combining the attention mechanism, the MSC block can extract and restore fine-grained image details in extremely low lighting environment, reduce noise caused by insufficient sensitivity while restoring true colors; Given an RGB image, a feature extraction block is used at each layer of the U-Net network structure, the details are enhanced through the internal attention mechanism, and the noise is suppressed through the external attention mechanism; in addition, a down-sampling or up-sampling operation is introduced after each feature extraction block to capture deep feature information; The multi-stage convergent block obtains feature maps from the first n output layers and the final output layer of the U-Net network structure, n = 3, and performs a convergent operation to obtain a three-channel RGB image as the final output.

2. The method of very low light image enhancement based on multi-level converging block and integrated loss function system according to claim 1, characterized in that: The first n layers and the output layer of the U-Net network structure The feature map of the input is sent to the selection block, first using global pooling and multi-layer perceptron MLP to calculate channel weights in space To capture channels containing information, then obtain spatial weights by using global pooling in the channels To find edge information in the feature map; the output is obtained by multiplication and convolution operation of the three Where H, W, C represent height, width and channel number respectively; M=Conv(I+I·F c ·F s )=Conv(I+I·ReLU(MLP(Pool s (I)))·ReLU(Conv(Pool c (I))) where Pool s and Pool c denote global pooling in the spatial and channel dimensions, respectively.

3. The method of very low light image enhancement based on multi-level converging block and integrated loss function system according to claim 2, characterized in that: The Input into the denoising block, calculate the absolute value of all features in the feature map, get a one-dimensional vector after global average pooling, and then pass it to the two-layer FC network to calculate the threshold Finally subtract the threshold and activate to get the output The formula is as follows: O = ReLU (M - τ) where z c is the feature at the cth neuron, i, j and c are the width, height and channel of the feature map, respectively.

4. The method of claim 1, wherein the method is based on a multi-level converging block and a comprehensive loss function system. The loss function body selects the mean absolute error loss L1, so that the network converges quickly and the overall enhancement of the image is realized; a color recovery loss function is designed to calculate the color average error of each channel in the three channels, and the color recovery loss function is as follows: where G j and Y j represent the average color values of the p-th channel in the real and output images, respectively.

5. The method of claim 1, wherein the method is based on a multi-level converging block and a comprehensive loss function system. The comprehensive loss function system contains an average structure loss L ssim , which is used to eliminate color deviation and solve the problem of edge blur in the recovered image; it divides the image into smaller parts, calculates the structure loss in the local range, and then takes the average value of the whole; the total loss function is as follows: L=αL1+βL c +(1-α-β)L ssim Wherein α and β are the weights of the loss function.

Images