Lightweight dark-light image enhancement method based on residual dense block

Abstract

This invention discloses a lightweight low-light image enhancement method based on residual dense blocks, comprising the following steps: acquiring a paired low-light image dataset; constructing a conditional generative adversarial network (GAN) model, including designing a lightweight generator network based on residual dense blocks and channel attention mechanisms, and a fully convolutional discriminator network; determining a multimodal loss function based on global similarity loss, structural similarity loss, content similarity loss, color similarity loss, and local texture loss; and training and testing the GAN model. This invention can process 400*600 low-light images at 36 frames per second on an RTX 2080 Ti graphics card. The low-light images enhanced by this invention are more consistent with human visual perception and exhibit superior performance on common image evaluation metrics such as peak signal-to-noise ratio and structural similarity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of dark light image enhancement, and particularly to a lightweight dark light image enhancement method based on residual dense blocks. BACKGROUND

[0002] In the process of dark light imaging, affected by environmental factors, equipment factors and human factors, the collected dark light images will present a series of problems such as low brightness, color cast, low contrast and poor visibility, which is not conducive to subsequent machine vision tasks such as scene understanding and target tracking. The traditional method of increasing the ISO of the camera can make the image brighter, but it will also increase the noise and produce color difference. The method of prolonging the exposure time to obtain better dark light images is only suitable for shooting still scenes, and the imaging quality is sensitive to camera shaking, which requires a stable platform, otherwise blurring will inevitably occur. Therefore, the dark light image enhancement technology which can improve the quality of dark light images at a low shooting cost has become one of the important research directions in the field of image processing, and has important practical application value.

[0003] In recent decades, in order to solve the problems of low brightness, color distortion, high noise level and loss of scene texture details in dark light images, researchers have conducted extensive research on dark light image enhancement algorithms, which can be generally divided into two categories: traditional method-based and deep learning-based dark light image enhancement algorithms. The dark light image enhancement algorithm based on traditional method is mostly realized according to the physical model. Ren et al. designed a joint enhancement and denoising (JED) method based on the Retinex model, which enhances the dark light image by estimating the piecewise smooth illumination map and noise suppressed reflectivity map in turn (Ren X, Li M, Cheng WH, et al. Joint enhancement and denoising method via sequential decomposition [C] / / 2018IEEE international symposium on circuits and systems (ISCAS). IEEE, 2018: 1-5.). This method can obtain good denoising and contrast enhancement results in some cases, but the applicability of the algorithm is poor, and the processing speed is slow.

[0004] Dark-light image enhancement algorithms based on deep learning rely on the powerful data fitting ability of neural networks and play an increasingly important role in the field of dark-light image enhancement. Chen et al. proposed a deep visual neural network Retinex-Net based on the Retinex model, including Decom-Net for decomposition and Enhance-Net for illumination adjustment (Wei C, Wang W, Yang W, et al. Deep retinex decomposition for low-light enhancement [J]. arXiv preprint arXiv:1808.04560, 2018.). This method achieves satisfactory dark-light enhancement quality in vision, but the image noise is not effectively removed. Zhang et al. proposed a network model named KinDNet, which denoises and corrects the color of dark-light images (Zhang Y, Zhang J, Guo X. Kindling the darkness: A practical low-light image enhancer [C] / / Proceedings of the 27th ACM international conference on multimedia. 2019:1632-1640.). This method performs well in image denoising, color correction, and other dark-light image enhancement tasks, but it lacks detail preservation. Wu et al. proposed a deep unfolding network URetinex-Net based on the Retinex model, which contains three learning-based modules (Wu W, Weng J, Zhang P, et al. Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:5901-5910.). This method can achieve noise suppression and detail preservation for dark-light images, but the processing speed needs to be improved.Xu et al. proposed a SNR network model using signal noise perception transformer and convolution model (Xu X, Wang R, Fu C W, et al. SNR-aware low-light image enhancement [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 17714-17724.). The enhanced image has superior perceptual quality, but the image details are lost. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the present application provides a lightweight low-light image enhancement method based on residual dense block to improve the deficiencies of traditional low-light image enhancement algorithms in noise suppression, detail preservation, and processing speed.

[0006] The technical solution for achieving the purpose of the present application is: a lightweight low-light image enhancement method based on residual dense block, comprising the following steps:

[0007] Step 1, obtaining a paired low-light image dataset, the paired low-light image dataset is composed of a low-light image and its corresponding normal light image;

[0008] Step 2, constructing a conditional generative adversarial network model, the conditional generative adversarial network model includes a generator network and a discriminator network, wherein the generator network is a lightweight network based on residual dense block and channel attention mechanism, and the discriminator network is a fully convolutional network;

[0009] Step 3, determining a multi-modal loss function for measuring the difference between the predicted value and the true value of the model, the loss function is composed of global similarity loss, structural similarity loss, content similarity loss, color similarity loss and local texture loss;

[0010] Step 4, using the training images in the paired low-light image dataset to conduct adversarial training on the conditional generative adversarial network model, and obtaining the loss value for optimizing the network through the multi-modal loss function until a generator network model with good prediction performance is obtained;

[0011] Step 5, inputting the collected low-light image into the trained generator network model to obtain the enhanced image.

[0012] Preferably, the generator network comprises an input layer for inputting a dark-light image of RGB three channels, a hidden layer for feature extraction of the input image through convolution operation, and an output layer for outputting a processing result; the hidden layer comprises three residual dense blocks; the specific structure of the generator hidden layer is as follows:

[0013] Convolution layer 1: taking the M*N*3 image input by the input layer as input, after 32 step 1 3*3 convolution kernel convolution and LeakyReLU activation function activation, an M*N feature map with 32 output channels is output, and M and N are the length and width of the input image respectively;

[0014] Convolution layer 2: taking the M*N*32 feature map output by the convolution layer 1 as input, after 32 step 1 3*3 convolution kernel convolution and LeakyReLU activation function activation, an M*N feature map with 32 output channels is output;

[0015] Residual dense block 1: taking the M*N*32 feature map output by the convolution layer 2 as input, after 3 dense connection layers, 1 local feature fusion layer, 1 channel attention layer and 1 local residual learning layer, an M*N feature map with 32 output channels is output;

[0016] Residual dense block 2: taking the M*N*32 feature map output by the residual dense block 1 as input, after 3 dense connection layers, 1 local feature fusion layer, 1 channel attention layer and 1 local residual learning layer, an M*N feature map with 32 output channels is output;

[0017] Residual dense block 3: taking the M*N*32 feature map output by the residual dense block 2 as input, after 3 dense connection layers, 1 local feature fusion layer, 1 channel attention layer and 1 local residual learning layer, an M*N feature map with 32 output channels is output;

[0018] Connection layer 1: inputting three feature maps processed by the residual dense block 1, the residual dense block 2 and the residual dense block 3, after connection layer connection, an M*N feature map with 96 output channels is output;

[0019] Channel attention layer 1: inputting the M*N*96 feature map output by the connection layer, after introducing channel attention mechanism to distinguish and pay attention to the feature map from the channel, an M*N feature map with 96 output channels is output;

[0020] Convolution layer 3: taking the M*N*96 feature map output by the channel attention layer 1 as input, after 32 step 1 3*3 convolution kernel convolution, an M*N feature map with 32 output channels is output;

[0021] Convolutional layer 4: taking the M*N*32 feature map output by the convolutional layer 3 as input, performing convolution through 32 3*3 convolution kernels with a step size of 1 and activating through a LeakyReLU activation function, and outputting an M*N feature map with a channel number of 32;

[0022] Residual learning layer 1: taking the two feature maps processed by the convolutional layer 1 and the convolutional layer 4 as input, performing pixel-by-pixel addition operation, and outputting an M*N feature map with a channel number of 32;

[0023] Convolutional layer 5: taking the M*N*32 feature map output by the residual learning layer 1 as input, performing convolution through 3 3*3 convolution kernels with a step size of 1 and activating through a sigmoid activation function, and outputting an M*N feature map with a channel number of 3.

[0024] Preferably, the discriminator comprises an input layer, a hidden layer and an output layer, the input layer is used for inputting a dark-light picture of RGB three channels and a picture to be discriminated of RGB three channels, the hidden layer is used for convolution calculation on the input image, and the output layer is used for outputting a discrimination result, the discriminator hidden layer is composed of six convolutional layers, specifically:

[0025] Convolutional layer 1: taking the M1*N1*6 feature map obtained by connecting the two pictures input by the input layer in the channel dimension as input, performing convolution through 32 3*3 convolution kernels with a step size of 2 and activating through a relu activation function, and outputting a (M1 / 2)*(N1 / 2) feature map with a channel number of 32, the two pictures input by the input layer include a dark-light picture of M1*N1*3 size and a picture to be discriminated of M1*N1*3 size, M1 and N1 are the length and width of the image respectively, and M1 and N1 are integer multiples of 16;

[0026] Convolutional layer 2: taking the (M1 / 2)*(N1 / 2)*32 feature map output by the convolutional layer 1 as input, performing convolution through 64 3*3 convolution kernels with a step size of 2 and activating through a relu activation function, and outputting a (M1 / 4)*(N1 / 4) feature map with a channel number of 64;

[0027] Convolutional layer 3: taking the (M1 / 4)*(N1 / 4)*64 feature map output by the convolutional layer 2 as input, performing convolution through 128 3*3 convolution kernels with a step size of 2 and activating through a relu activation function, and outputting a (M1 / 8)*(N1 / 8) feature map with a channel number of 128;

[0028] Convolutional layer 4: taking the 32*32*128 feature map output by the convolutional layer 3 as input, performing convolution through 256 3*3 convolution kernels with a step size of 2 and activating through a relu activation function, and outputting a (M1 / 16)*(N1 / 16) feature map with a channel number of 256;

[0029] Convolution layer 5: taking the (M1 / 8)*(N1 / 8)*256 feature map output by the convolution layer 4 as input, after convolution by 1 3*3 convolution kernel with a step of 1, output a (M1 / 16)*(N1 / 16) feature map with a channel number of 1;

[0030] Convolution layer 6: taking the (M1 / 16)*(N1 / 16)*1 feature map output by the convolution layer 5 as input, after convolution by 1 3*3 convolution kernel with a step of 1, output a (M1 / 16)*(N1 / 16) feature map with a channel number of 1, that is, an information distribution matrix.

[0031] Preferably, the loss function in step 3 is:

[0032]

[0033] In the formula, L cGAN is the discriminator loss calculated by the discriminator, which is considered as a local texture loss here, λ1, λ S , λ C and λ P are hyperparameters for adjusting the proportion, L1 is a global similarity loss, L S is a structural similarity loss, L C is a content similarity loss, L P is a color similarity loss, G is a generator mapping, and D is a discriminator mapping.

[0034] Preferably, the local texture loss calculated by the discriminator is specifically:

[0035] L cGAN (G,D)=E X,Y [logD(Y)]+E X,Y [log(1-D(X,G(X,Z)))]

[0036] Wherein, X and Y represent the dark-light image to be enhanced and the corresponding normal-light image, respectively, Z represents the input random noise, E X,Y (*) represents the average value calculated after pixel-by-pixel operation with X and Y as independent variables, G is a generator mapping, and D is a discriminator mapping.

[0037] Preferably, the global similarity loss is specifically:

[0038]

[0039] Wherein, and have the same function mapping relationship, take c={r,g,b}, then ||*||1 represents 1-norm operation, ω r , ω g and ωb The calculation method is the same, by and The three are calculated, also take c={r, g, b}, then is and The sum of the three.

[0040] Preferably, the structural similarity loss is specifically:

[0041] L S (G) = 1-SSIM(Y, G(X, Z))

[0042] Where SSIM(*) represents the structural similarity calculation function, X and Y represent the dark light image and the corresponding normal light image to be enhanced, respectively, and Z represents the input random noise image.

[0043] Preferably, the content similarity loss is specifically:

[0044]

[0045] Where Θ(·) represents the feature extraction function corresponding to the block5_conv2 layer in the VGG-19 pre-trained model, E X,Y,Z [*] represents the average value of the matrix obtained after the matrix variables X, Y, Z are calculated according to the formula.

[0046] Preferably, the color similarity loss is specifically:

[0047] L P (G) = delta_E(Y, G(X, Z))

[0048] Where delta_E(*) represents the color difference calculation function based on weighted Euclidean distance, X and Y represent the dark light image and the corresponding normal light image to be enhanced, respectively, and Z represents the input random noise.

[0049] Preferably, the specific steps of training the conditional generative adversarial network model in step 4 are:

[0050] Assign values to the hyperparameters required in the training process;

[0051] Load the training image pair in step 1;

[0052] Initialize the generator optimizer and the generator network parameters;

[0053] The generator generates an enhanced image according to the input dark light image and random noise;

[0054] Initialize the discriminator optimizer and the discriminator network parameters;

[0055] The discriminator generates an information matrix according to the input dark-light image and the image to be discriminated; when the image to be discriminated is the image enhanced by the generator, the expected output of the discriminator is a zero matrix; and when the image to be discriminated is the normal-light image corresponding to the dark-light image, the expected output of the discriminator is a one matrix;

[0056] The generator optimizer updates the generator network parameters according to the generator loss function value, so as to minimize the generator loss function;

[0057] The discriminator optimizer updates the discriminator network parameters according to the discriminator loss function value, so as to maximize the discriminator loss function;

[0058] The above steps are repeated by loading a new training image pair and using the updated generator and discriminator network parameters; and the cycle is ended when the generator and discriminator models obtained by training meet the target function condition;

[0059] The model that makes the generator loss function reach the minimum value is saved as the final dark-light image enhancement model;

[0060] The target function of the lightweight dark-light image enhancement model based on the residual dense block is:

[0061]

[0062] In the formula, G is the mapping obtained by training the generator network, and D is the mapping obtained by training the discriminator network.

[0063] Compared with the prior art, the present application has the following characteristics: (1) the present application takes the conditional generative adversarial network as the main structure, builds a generator based on the residual dense block structure to strengthen feature transmission, better preserves image details, introduces an attention mechanism in the generator to realize dynamic adjustment of feature weights, improves image processing efficiency, and uses a PatchGan discriminator to limit attention to different scale local regions of the image to sharpen local region details of the image; (2) the present application designs a multi-modal loss function for measuring the difference between the predicted value and the true value of the model, which is composed of global similarity loss, structural similarity loss, content similarity loss, color similarity loss and local texture loss, and can better guide the model training direction; (3) the average processing frame number of the present application on an RTX 2080Ti graphics card is about 36 frames (400*600 size picture), and the present application performs well in qualitative and quantitative experiments. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 It is a flowchart of the dark-light image enhancement method based on the residual dense block.

[0065] Figure 2A conditional generative adversarial network generator and discriminator network structure diagram is generated for the designed condition.

[0066] Figure 3 A test result schematic diagram of 15 groups of test images in the LOL dark light image data set, wherein from left to right, (a) is an original dark light image, (b) is a linear lightening image, (c) is a JED method processing result image, (d) is a Retinex-Net method processing result image, (e) is a KinDNet method processing result image, (f) is a URetinex-Net method processing result image, (g) is an SNR method processing result image, (h) is a method processing result image of the application, and (i) is a normal light image. Specific embodiments

[0067] The application proposes a lightweight dark light image enhancement method based on residual dense blocks to improve the quality of dark light images and make them conform to the observation habits of the human eye, facilitating subsequent machine vision tasks.

[0068] To more stably solve the noise and color deviation in the dark light image, the application selects a conditional generative adversarial network model conducive to improving the robustness of the network as the network main structure. The conditional generative adversarial network is composed of a generator network and a discriminator network, wherein the generator network is used to complete image enhancement tasks such as brightness adjustment, color correction, noise suppression, and scene texture detail restoration, and the discriminator network is used to identify the authenticity of the image to adjust the training direction of the generator network. In order to avoid the loss of feature information in the generator network during transmission, the application selects a residual dense block with strong feature propagation and encourages feature reuse performance as the basic structure of the lightweight generator network. In addition, in order to select useful channel information to further improve the processing efficiency of the generator network, the application also introduces a channel attention mechanism in the generator network. In addition, the application uses a PatchGAN discriminator network that can limit attention to different scale local regions of the image to identify the authenticity of the local region of the image, thereby improving the ability of the generator network to sharpen the details of the local region.

[0069] The application also designs a multi-modal loss function for measuring the difference between the predicted value and the true value of the model, which is composed of a global similarity loss, a structure similarity loss, a content similarity loss, a color similarity loss, and a local texture loss.

[0070] The lightweight dark light image enhancement method based on residual dense blocks proposed by the application includes the following steps:

[0071] Step 1, in order to enable the designed conditional generative adversarial network model to complete the task of dark light image enhancement, the present application uses the LOL data set containing 500 pairs of real image data made by Ren et al. to complete the training of the network model. Ren et al. realize the collection of dark light images and normal light images in the same scene by fixing other parameters of the camera and changing the exposure time and ISO sensitivity. The 500 pairs of images in the data set are divided into 485 pairs of training images and 15 pairs of test images according to the purpose, the present application uses the training images to train the model, and uses the test images to test the trained model.

[0072] Step 2, constructing a conditional generative adversarial network model, the conditional generative adversarial network model comprising a generator network and a discriminator network, wherein the generator network is a lightweight network based on a residual dense block and a channel attention mechanism, and the discriminator network is a fully convolutional network.

[0073] Step 2.1, the generator network comprises an input layer, a hidden layer and an output layer, the generator input layer is used for inputting an RGB three-channel dark light image, the generator hidden layer extracts and restores features of the input image through convolution operation, and the generator output layer is used for outputting a processing result. Taking the input of the generator input layer as an example, the dark light image with a size of 256*256*3 is input, the specific structure of the generator hidden layer is as follows:

[0074] Convolution layer 1: taking the image with a size of 256*256*3 input by the input layer as input, after being convolved by 32 3*3 convolution kernels with a step of 1 and activated by a LeakyReLU activation function, a feature map with a size of 256*256 and a channel number of 32 is output;

[0075] Convolution layer 2: taking the feature map with a size of 256*256*32 output by the convolution layer 1 as input, after being convolved by 32 3*3 convolution kernels with a step of 1 and activated by a LeakyReLU activation function, a feature map with a size of 256*256 and a channel number of 32 is output;

[0076] Residual dense block 1: taking the feature map with a size of 256*256*32 output by the convolution layer 2 as input, after being processed by 3 dense connection layers, 1 local feature fusion layer, 1 channel attention layer and 1 local residual learning layer, a feature map with a size of 256*256 and a channel number of 32 is output;

[0077] Residual dense block 2: taking the feature map with a size of 256*256*32 output by the residual dense block 1 as input, after being processed by 3 dense connection layers, 1 local feature fusion layer, 1 channel attention layer and 1 local residual learning layer, a feature map with a size of 256*256 and a channel number of 32 is output;

[0078] Residual dense block 3: taking the 256*256*32 feature map output by the residual dense block 2 as input, after passing through 3 dense connection layers, 1 local feature fusion layer, 1 channel attention layer and 1 local residual learning layer, a 256*256 feature map with 32 channels is output;

[0079] Connection layer 1: input 3 feature maps processed by residual dense block 1, residual dense block 2 and residual dense block 3, after connection in the channel dimension, a 256*256 feature map with 96 channels is output;

[0080] Channel attention layer 1: input the 256*256*96 feature map output by the connection layer, after introducing the channel attention mechanism to distinguish and pay attention to the feature map in the channel, a 256*256 feature map with 96 channels is output;

[0081] Convolution layer 3: taking the 256*256*96 feature map output by the channel attention layer 1 as input, after convolution by 32 3*3 convolution kernels with a step of 1, a 256*256 feature map with 32 channels is output;

[0082] Convolution layer 4: taking the 256*256*32 feature map output by the convolution layer 3 as input, after convolution by 32 3*3 convolution kernels with a step of 1 and LeakyReLU activation function activation, a 256*256 feature map with 32 channels is output;

[0083] Residual learning layer 1: input 2 feature maps processed by convolution layer 1 and convolution layer 4, after pixel-by-pixel addition operation, a 256*256 feature map with 32 channels is output;

[0084] Convolution layer 5: taking the 256*256*32 feature map output by the residual learning layer 1 as input, after convolution by 3 3*3 convolution kernels with a step of 1 and sigmoid activation function activation, a 256*256 feature map with 3 channels is output;

[0085] At this time, the output of the generator output layer is a 256*256*3 dark-light image enhancement map.

[0086] The generator network structure diagram is as shown in Figure 2 The convolution layer is 3*3 by default.

[0087] Step 2.2, the discriminator network structure also includes an input layer, a hidden layer, and an output layer. The input layer of the discriminator is used to input the dark-light picture of RGB three channels and the picture to be discriminated of RGB three channels. The hidden layer of the discriminator is used to perform convolution calculation on the input picture. The output layer of the discriminator is used to output the discrimination result. Taking the input of the input layer of the discriminator as an example, the dark-light picture of 256*256*3 size and the picture to be discriminated of 256*256*3 size, the specific structure of the hidden layer of the discriminator is as follows:

[0088] Convolution layer 1: taking the 256*256*6 feature map obtained by connecting the two pictures input by the input layer in the channel dimension as the input, performing convolution through 32 3*3 convolution kernels with a step of 2 and activating the relu activation function, and outputting a 128*128 feature map with a channel number of 32;

[0089] Convolution layer 2: taking the 128*128*32 feature map output by the convolution layer 1 as the input, performing convolution through 64 3*3 convolution kernels with a step of 2 and activating the relu activation function, and outputting a 64*64 feature map with a channel number of 64;

[0090] Convolution layer 3: taking the 64*64*64 feature map output by the convolution layer 2 as the input, performing convolution through 128 3*3 convolution kernels with a step of 2 and activating the relu activation function, and outputting a 32*32 feature map with a channel number of 128;

[0091] Convolution layer 4: taking the 32*32*128 feature map output by the convolution layer 3 as the input, performing convolution through 256 3*3 convolution kernels with a step of 2 and activating the relu activation function, and outputting a 16*16 feature map with a channel number of 256;

[0092] Convolution layer 5: taking the 16*16*256 feature map output by the convolution layer 4 as the input, performing convolution through 1 3*3 convolution kernel with a step of 1, and outputting a 16*16 feature map with a channel number of 1;

[0093] Convolution layer 6: taking the 16*16*1 feature map output by the convolution layer 5 as the input, performing convolution through 1 3*3 convolution kernel with a step of 1, and outputting a 16*16 feature map with a channel number of 1, that is, the information distribution matrix.

[0094] At this time, the output of the output layer of the discriminator is a 16*16*1 prediction information matrix based on the to-be-tested picture. Each number in the matrix represents the probability that the corresponding local region is true.

[0095] The discriminator network structure diagram is as shown in Figure 2 .

[0096] Step 3, a multi-modal loss function for measuring the difference between the predicted value of the model and the true value is designed, and the loss function is composed of global similarity loss, structure similarity loss, content similarity loss, color similarity loss and local texture loss.

[0097] The loss function designed in the application is:

[0098]

[0099] In the formula, L cGAN is the discriminator loss calculated by the discriminator, which is considered as the local texture loss here, λ1, λ S , λ C and λ P are hyperparameters for adjusting the proportion, which are respectively taken as 0.35, 0.30, 0.30 and 0.05 according to the test process, L1 is the global similarity loss, L S is the structure similarity loss, L C is the content similarity loss, L P is the color similarity loss, G is the generator mapping, and D is the discriminator mapping.

[0100] The local texture loss calculated by the discriminator is specifically:

[0101] L cGAN (G,D)=E X,Y [logD(Y)]+E X,Y [log(1-D(X,G(X,Z)))]

[0102] Wherein, X and Y represent the dark light image and the corresponding normal light image to be enhanced respectively, and Z represents the input random noise, E X,Y (*) represents the average value calculated after pixel-by-pixel operation with X and Y as independent variables, G is the generator mapping, and D is the discriminator mapping.

[0103] The global similarity loss is specifically:

[0104]

[0105] Wherein,

[0106] The structure similarity loss is specifically:

[0107] L S (G)=1-SSIM(Y,G(X,Z))

[0108] Wherein, SSIM(*) represents the structure similarity calculation function.

[0109] The content similarity loss is specifically:

[0110] LC (G) = E X,Y,Z [||Θ(Y)-Θ(G(X,Z))||2]

[0111] Where, Θ(·) represents the feature extraction function corresponding to the block5_conv2 layer in the VGG-19 pre-training model, which is used to extract the content feature information of the image to be tested.

[0112] The color similarity loss is specifically:

[0113] L P (G) = delta_E(Y,G(X,Z))

[0114] Where, delta_E(*) represents a color difference calculation function based on weighted Euclidean distance.

[0115] Step 4, using the training image pairs in the paired low-light image dataset to conduct adversarial training on the conditional generative adversarial network model, and obtaining the loss value for optimizing the network through the multi-modal loss function, until a generator network model with good prediction performance is obtained. The training process is as follows:

[0116] First, the hyperparameters required in the training process are assigned values, and the selection of the hyperparameters depends on past experimental experience and the experimental process. The values of the hyperparameters in the loss function are as shown in step 3. In addition, in order to improve the stability of the conditional adversarial generative network model during training, the Adam optimizer is selected to optimize the network model. It is worth noting that, in order to avoid the problem of mode collapse, the generator network and the discriminator network each use an Adam optimizer with a different initial learning rate. The initial learning rate of the generator network optimizer is 1x10-4, and the initial learning rate of the discriminator network optimizer is 9x10-7.

[0117] During training, the LOL training dataset is used to train a total of 4000 rounds, and the batch size is set to 4. Before loading the training image pairs, the training image pairs need to be preprocessed: the training images are randomly cropped into image blocks with a size of 256x256, and each image block is subjected to data augmentation operations such as random rotation and flipping.

[0118] Before starting adversarial training, the generator network parameters and the discriminator network parameters need to be initialized. In order to ensure the effective flow of information in the forward and backward propagation, the He initialization method is used to initialize the neural generator and discriminator network parameters.

[0119] The initialized generator generates an enhanced image according to the input low-light image and random noise.

[0120] The initialized discriminator generates an information matrix according to the input dark-light image and the to-be-discriminated image. When the to-be-discriminated image is the image enhanced by the generator, the expected output of the discriminator is a zero matrix. When the to-be-discriminated image is the normal-light image corresponding to the dark-light image, the expected output of the discriminator is a one matrix.

[0121] The generator optimizer updates the generator network parameters according to the generator loss function value, so as to minimize the generator loss function, wherein the generator loss function value is obtained by calculating the enhanced image output by the generator and the information matrix output by the discriminator according to the multi-modal loss function designed in the application.

[0122] The discriminator optimizer updates the discriminator network parameters according to the discriminator loss function value, so as to maximize the discriminator loss function, wherein the discriminator loss function value is obtained by calculating the information matrix output by the discriminator according to the discriminator loss function, i.e. the local texture loss, in the above-mentioned multi-modal loss function.

[0123] Load a new training image pair, and repeat the above steps using the updated generator and discriminator network parameters until the loop ends.

[0124] Save the model that makes the generator loss function reach the minimum value as the final dark-light image enhancement model.

[0125] In summary, the objective function of the lightweight dark-light image enhancement model based on the residual dense block is:

[0126]

[0127] In the formula, G is the mapping obtained by training the generator network, and D is the mapping obtained by training the discriminator network.

[0128] Step 5, input the test image in the dark-light image data set into the trained generator network model, and the enhanced image can be obtained.

[0129] Embodiment

[0130] The method proposed in the present application is compared with the traditional method JED and four deep learning methods Retinex-net, KinDNet, URetinex-Net and SNR in the LOL low-light image dataset mentioned in step 1. Among them, the JED method is a joint enhancement and denoising method based on the Retinex model, which can enhance the low-light image by decomposing the segmented smooth illumination map and noise suppressed reflectivity map in sequence. The Retinex-Net method is based on the Retinex model and designs two subnetworks Decom-Net for decomposition and Enhance-Net for illumination adjustment to enhance the low-light image. The KinDNet method is also based on the Retinex model and designs two branches for processing reflectivity and illumination respectively to denoise and color correct the low-light image. The deep unfolding network URetinex-Net realizes noise suppression and detail preservation of the low-light image through three learning-based modules: initialization module, unfolding optimization module and illumination adjustment module. The SNR method is based on the signal noise perception transformer and convolution model, which improves the perceptual quality of the enhanced low-light image. The test results of each method are shown in Figure 3 Fig. 6, where from left to right, (a) is the original low-light image, (b) is the linear brightening image, (c) is the JED method processing result image, (d) is the KinDNet method processing result image, (e) is the Retinex-Net method processing result image, (f) is the SNR method processing result image, (g) is the present application method processing result image, and (h) is the normal light image. The JED method can obtain good denoising and contrast enhancement results in some cases, but the algorithm has poor applicability and there is still noise in some test images. The Retinex-Net method achieves satisfactory low-light enhancement quality in vision, but the image noise is not effectively removed. The KinDNet method performs well in image denoising, color correction and other low-light image enhancement tasks, but has some deficiencies in detail preservation. The URetinex-Net method can realize noise suppression and detail preservation of the low-light image, but the color correction needs to be improved. The SNR method enhances the image in perceptual quality, but the image details are lost. In addition, the commonly used image evaluation indicators, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), are used to evaluate the above test images. At the same time, the frame rate of each method is also tested. The test results are shown in Table 1. Among them, PSNR is used to evaluate the image quality reference value between the maximum signal value and the background noise of the image, the larger the value, the less the image distortion. SSIM is used to evaluate the structural similarity between the processed image and the reference image, the larger the value, the better the structural similarity. Frame rate is used to evaluate the image processing speed, the larger the frame rate, the faster the image processing speed. Obviously, the method proposed in the present application has the best enhancement effect and the fastest processing speed compared with other comparative algorithms.

[0131] Table 1: Evaluation index results

[0132]

[0133]

[0134] In summary, compared with other dark light image enhancement methods, the light dark light image enhancement method based on residual dense block can obtain a dark light enhancement image with lower noise, less color deviation and higher contrast in a shorter processing time.

Claims

1. A lightweight dark-light image enhancement method based on residual dense block, characterized in that, The method comprises the following steps: Step 1, obtaining a paired dark-light image dataset, the paired dark-light image dataset being composed of a dark-light image and a corresponding normal-light image; Step 2, constructing a conditional generative adversarial network model, the conditional generative adversarial network model comprising a generator network and a discriminator network, wherein the generator network is a lightweight network based on a residual dense block and a channel attention mechanism, and the discriminator network is a fully convolutional network; the generator network comprises an input layer, a hidden layer and an output layer, the input layer being used for inputting a dark-light image of an RGB three-channel, the hidden layer being used for performing feature extraction on the input image through convolution operation, and the output layer being used for outputting a processing result; the hidden layer comprises three residual dense blocks; and a specific structure of the generator hidden layer is as follows: Convolution layer 1: taking an image of M*N*3 input by the input layer as input, performing convolution through 32 step 1 3*3 convolution kernels and LeakyReLU activation function activation, and then outputting a feature map of M*N with a channel number of 32, wherein M and N are respectively a length and a width of the input image; Convolution layer 2: taking the feature map of M*N*32 output by the convolution layer 1 as input, performing convolution through 32 step 1 3*3 convolution kernels and LeakyReLU activation function activation, and then outputting a feature map of M*N with a channel number of 32; Residual dense block 1: taking the feature map of M*N*32 output by the convolution layer 2 as input, performing convolution through three dense connection layers, one local feature fusion layer, one channel attention layer and one local residual learning layer, and then outputting a feature map of M*N with a channel number of 32; Residual dense block 2: taking the feature map of M*N*32 output by the residual dense block 1 as input, performing convolution through three dense connection layers, one local feature fusion layer, one channel attention layer and one local residual learning layer, and then outputting a feature map of M*N with a channel number of 32; Residual dense block 3: taking the feature map of M*N*32 output by the residual dense block 2 as input, performing convolution through three dense connection layers, one local feature fusion layer, one channel attention layer and one local residual learning layer, and then outputting a feature map of M*N with a channel number of 32; Connection layer 1: inputting three feature maps processed by the residual dense block 1, the residual dense block 2 and the residual dense block 3, performing connection through the connection layer, and then outputting a feature map of M*N with a channel number of 96; Channel attention layer 1: inputting the feature map of M*N*96 output by the connection layer, distinguishing and paying attention to the feature map from the channel by introducing a channel attention mechanism, and then outputting a feature map of M*N with a channel number of 96; Convolution layer 3: taking the feature map of M*N*96 output by the channel attention layer 1 as input, performing convolution through 32 step 1 3*3 convolution kernels, and then outputting a feature map of M*N with a channel number of 32; Convolution layer 4: taking the feature map of M*N*32 output by the convolution layer 3 as input, performing convolution through 32 step 1 3*3 convolution kernels and LeakyReLU activation function activation, and then outputting a feature map of M*N with a channel number of 32; Residual learning layer 1: input two feature maps processed by convolution layer 1 and convolution layer 4, and output an M*N feature map with 32 channels after pixel-by-pixel addition operation; Convolution layer 5: take the M*N*32 feature map output by the residual learning layer 1 as input, and output an M*N feature map with 3 channels after 3*3 convolution kernel convolution with a step of 1 and sigmoid activation function activation; Step 3, determine a multi-modal loss function for measuring the difference between the predicted value of the model and the true value, the loss function is composed of global similarity loss, structure similarity loss, content similarity loss, color similarity loss and local texture loss; Step 4, use the training image pair in the paired dark light image dataset to conduct adversarial training on the conditional generative adversarial network model, and obtain the loss value for optimizing the network through the multi-modal loss function, until a generator network model with good prediction performance is obtained; Step 5, input the collected dark light image into the trained generator network model to obtain the enhanced image.

2. The residual dense block-based lightweight dim light image enhancement method according to claim 1, characterized in that, The discriminator includes an input layer, a hidden layer and an output layer, the input layer is used for inputting a dark light picture of RGB three channels and a picture to be discriminated of RGB three channels, the hidden layer is used for convolution calculation on the input image, and the output layer is used for outputting a discrimination result, the discriminator hidden layer is composed of six convolution layers, specifically: Convolution layer 1: take the M1*N1*6 feature map obtained by connecting the two pictures input by the input layer in the channel dimension as input, and output a (M1 / 2)*(N1 / 2) feature map with 32 channels after 32 3*3 convolution kernel convolution with a step of 2 and relu activation function activation, the two pictures input by the input layer include a dark light picture of M1*N1*3 size and a picture to be discriminated of M1*N1*3 size, M1 and N1 are the length and width of the image respectively, and M1 and N1 are integer multiples of 16; Convolution layer 2: take the (M1 / 2)*(N1 / 2)*32 feature map output by convolution layer 1 as input, and output a (M1 / 4)*(N1 / 4) feature map with 64 channels after 64 3*3 convolution kernel convolution with a step of 2 and relu activation function activation; Convolution layer 3: take the (M1 / 4)*(N1 / 4)*64 feature map output by convolution layer 2 as input, and output a (M1 / 8)*(N1 / 8) feature map with 128 channels after 128 3*3 convolution kernel convolution with a step of 2 and relu activation function activation; Convolution layer 4: take the 32*32*128 feature map output by convolution layer 3 as input, and output a (M1 / 16)*(N1 / 16) feature map with 256 channels after 256 3*3 convolution kernel convolution with a step of 2 and relu activation function activation; Convolution layer 5: take the (M1 / 8)*(N1 / 8)*256 feature map output by convolution layer 4 as input, and output a (M1 / 16)*(N1 / 16) feature map with 1 channel after 1 3*3 convolution kernel convolution with a step of 1; Convolutional layer 6: taking the (M1 / 16)*(N1 / 16)*1 feature map output by the convolutional layer 5 as input, after convolution by a 3*3 convolution kernel with a step size of 1, a (M1 / 16)*(N1 / 16) feature map with a channel number of 1, i.e. an information distribution matrix, is output.

3. The residual dense block-based lightweight dim light image enhancement method according to claim 1, characterized in that, The loss function in step 3 is: where L cGAN is the discriminator loss computed by the discriminator, here considered as local texture loss, λ1, λ S , λ C and λ P are hyperparameters used to adjust the weights, L1is the global similarity loss, L S is the structural similarity loss, L C is the content similarity loss, L P is the color similarity loss, G is the generator mapping, and D is the discriminator mapping.

4. The residual dense block-based lightweight dim light image enhancement method according to claim 3, characterized in that, The local texture loss calculated by the discriminator is specifically: L cGAN (G,D) = E X,Y [logD(Y)] + E X,Y [log(1-D(X,G(X,Z)))] where X and Y represent the dark-light image and the corresponding normal-light image to be enhanced, respectively, Z represents the input random noise, E X,Y (*) denotes the average value calculated after operating pixel by pixel with X and Y as arguments, G is the generator mapping, and D is the discriminator mapping.

5. The residual dense block-based lightweight dim light image enhancement method according to claim 3, characterized in that, The global similarity loss is specifically: wherein, and have the same function mapping relationship, taking c = {r, g, b}, then ||*||1 represents the 1-norm operation, ω r , ω g , and ω b are calculated in the same way, from and , and similarly taking c = {r, g, b}, then is the sum of and .

6. The residual dense block-based lightweight dim light image enhancement method according to claim 3, characterized in that, The structural similarity loss is specifically: L S (G) = 1 - SSIM(Y, G(X,Z)) Wherein, SSIM(*) represents the structural similarity calculation function, X and Y represent the dark-light image to be enhanced and the corresponding normal-light image respectively, and Z represents the input random noise image.

7. The residual dense block-based lightweight dim light image enhancement method according to claim 3, characterized in that, The content similarity loss is specifically: L C (G) = E X,Y,Z [||Θ(Y)-Θ(G(X,Z))||2] where Θ(·) represents the feature extraction function corresponding to the block5_conv2 layer in the VGG-19 pre-trained model, E X,Y,Z [*] represents the average value of the matrix obtained after the matrix variables X, Y, Z are calculated according to the * formula.

8. The residual dense block-based lightweight dim light image enhancement method according to claim 3, characterized in that, The color similarity loss is specifically: L P (G) = delta_E(Y, G(X,Z)) Wherein, delta_E(*) represents the color difference calculation function based on weighted Euclidean distance, X and Y represent the dark-light image to be enhanced and the corresponding normal-light image respectively, and Z represents the input random noise.

9. The residual dense block-based lightweight dim light image enhancement method of claim 1, wherein, The specific steps for training the conditional generative adversarial network model in step 4 are: Assign values to the hyperparameters required in the training process; Load the training image pairs in step 1; Initialize the generator optimizer and the generator network parameters; The generator generates an enhanced image according to the input dark-light image and random noise; Initialize the discriminator optimizer and the discriminator network parameters; The discriminator generates an information matrix according to the input dark-light image and the image to be discriminated. When the image to be discriminated is the enhanced image generated by the generator, the expected output of the discriminator is a zero matrix. When the image to be discriminated is the normal-light image corresponding to the dark-light image, the expected output of the discriminator is a one matrix; The generator optimizer updates the generator network parameters according to the generator loss function value, which is used to minimize the generator loss function; The discriminator optimizer updates the discriminator network parameters according to the discriminator loss function value, which is used to maximize the discriminator loss function; Load new training image pairs and repeat the above steps using the updated generator and discriminator network parameters. When the training obtained generator and discriminator model meet the objective function condition, the loop is ended; Save the model that minimizes the generator loss function as the final dark-light image enhancement model; Wherein, the objective function of the lightweight dark-light image enhancement model based on the residual dense block is: In the formula, G is the mapping obtained by training the generator network, and D is the mapping obtained by training the discriminator network.

Images