Weak Light Image Enhancement Method Based on Event Camera

Abstract

The present invention discloses a low-light image enhancement method based on an event camera. First, a data set with normal-light images and event streams is selected, and it is required that the normal-light images and event streams are spatially paired. Then, a gradient image and a noisy low-light image are generated from the normal-light images, and the event stream is preprocessed to obtain an event pseudo-image with good edge information. Then, the gradient image is reconstructed and enhanced, a feature fusion module is designed, a conditional discriminator is added, and the neural network built for the gradient branch and the low-light image enhancement branch is trained, and the model of the neural network is saved. Finally, the model is tested and the enhanced image is output. The present invention can guide the enhancement of low-light images in the image domain through the gradient map reconstructed by events, and a normal-light image with rich edge information can be obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer digital image processing, and particularly relates to a low-light image enhancement method based on an event camera. Background Art

[0002] With the advent of the information age, taking pictures has become an indispensable part of people's lives. Therefore, digital image processing has developed rapidly, and the demand for digital image processing has gradually increased.

[0003] In terms of low light, there is not only a need to take pictures under normal light, but also night scene photography has become an important selection criterion for mobile phones. For photos taken under low light, if the shooting time is short, the photos generally have a large amount of noise. If the exposure time is extended, the photos are prone to blur due to movement. Using hardware methods will increase the cost of the imaging system. At the same time, with the rapid development of deep learning, software method enhancement can also achieve very good results.

[0004] Most early low-light image enhancement methods were enhanced using traditional methods. For example, histogram equalization or gamma correction is prone to cause local overexposure of the image while other regions are under-enhanced, resulting in insufficient overall contrast. With the significant improvement of computer computing power and the rapid development of deep learning, using deep learning methods can often obtain better experimental results than traditional methods. For relatively weak low-light image enhancement, single-image enhancement can be used. However, for extremely low-light situations, since noise seriously damages the structural information of the image, single-image enhancement often cannot obtain good results. Event cameras record changes in intensity values and have the ability of high dynamic range, low latency, and no motion blur. Even under low-light conditions, good event streams can be obtained, which can synthesize event pseudo-images with good edge information structures, supplement the dark area structures that cannot be captured by traditional cameras, and finally obtain normal light images with rich texture details. Summary of the Invention

[0005] The purpose of the present invention is to provide a low-light image enhancement method based on an event camera, which can guide the enhancement of low-light images in the image domain through the gradient map reconstructed by events, and can obtain normal light images with rich edge information.

[0006] The technical solution adopted by the present invention is that the low-light image enhancement method based on an event camera is specifically implemented according to the following steps:

[0007] Step 1, dataset synthesis: Select a dataset with normal light images and event streams, requiring that the normal light images and event streams are paired in space. Then, generate gradient images and noisy low-light images from the normal light images, and preprocess the event streams to obtain event pseudo-images with good edge information;

[0008] Step 2: Use the UNet gradient branch to reconstruct the event pseudo-image obtained in Step 1 into a gradient image;

[0009] Step 3: Use the UNet low-light image enhancement branch to enhance the low-light image obtained in Step 1 to obtain an enhanced image;

[0010] Step 4: Design a feature fusion module, which is the CBAM module based on channel and spatial attention, and fuse the information contained in the gradient image in Step 2 into the low-light enhancement branch in Step 3;

[0011] Step 5: Add a conditional discriminator, with the event pseudo-image and the gradient image as the conditions, to generate a more realistic enhanced image;

[0012] Step 6: Train the neural network built by the gradient branch in Step 2 and the low-light image enhancement branch in Step 3 for 300 epochs, verify the training results, and save the model of the neural network;

[0013] Step 7: Test the model of the neural network saved in Step 6 and output the enhanced image.

[0014] The features of the present invention also lie in that

[0015] Step 1 is specifically as follows:

[0016] Step 1.1: Select a dataset with normal illumination images and event streams, requiring that the normal illumination images and event streams are paired spatially, where the representation of the event is , where represents an event, x and y represent the coordinate positions of the pixel points, t represents the timestamp, p represents the polarity of the event, j represents the nth event. Assume that the time length of the event stream is , during which the event camera returns n gray frames at the same time. For the pixel value of the synthesized pseudo-image, each event interval is represented as , is obtained by adding the polarity values of the events to obtain an event pseudo-image;

[0017] Step 1.2: For the reference image GT, add noise to the reference image GT, using Gaussian blind noise. The specific method is to generate Gaussian blind noise with a standard deviation range to replace the Gaussian noise generated with a single standard deviation, and then perform low-light scene simulation and gamma correction, as shown in Formula (1):

[0018] (1)

[0019] wherein represents the corrected image, represents the image before correction, represents the scaling strength of the pixel value;

[0020] After that, the data is linearly normalized as shown in Formula 2,

[0021] (2)

[0022] wherein represents the normalized image, X represents the coordinate at the pixel value, represents the minimum pixel value of image x, represents the maximum pixel value of the image, and a simulated low-light image with noise is obtained;

[0023] Step 1.3: Edge extraction is performed on the normal illumination image obtained in Step 1.1 using the sobel operator. The specific calculation methods are shown in Formulas (3), (4), and (5): For the assumed image A:

[0024] (3)

[0025] (4)

[0026] (5)

[0027] represents the first-order difference in the horizontal direction, represents the first-order difference in the vertical direction, and G represents the gradient map, wherein represents the convolution operation to obtain G as the gradient image.

[0028] Step 2 is specifically as follows:

[0029] Step 2.1: First, use the dataset class of the deep learning framework PyTorch, and perform a transform operation on the images in this class. First, convert the images to the tensor format, perform a normalization operation, and then perform a standardization process as shown in Formula (6):

[0030] (6)

[0031] wherein represents the output of the c th channel, represents the c th channel input, represents the mean value of the c th channel, Indicates c The variance of each channel is calculated, and then the data is packaged through DataLoader;

[0032] Step 2.2: For the packed data, first extract the features to obtain the feature map, select UNet as the backbone network of the gradient branch, replace the maximum pooling operation with the convolution operation with a step size of 2, then adjust the data range with the batchnorm algorithm, and then use the ReLU activation function for activation, perform seven downsampling, then perform upsampling with the deconvolution operation with a step size of 2, use the batchnorm algorithm to adjust the data range and use the ReLU activation function, and perform seven upsampling. Finally, reconstruct the gradient map of the upsampled feature map to obtain the reconstructed gradient image, perform padding operation, and introduce the information of the low-light image obtained in step 1.2 into the gradient branch. Specifically, as shown in the convolution formulas (7) and (8) and the deconvolution formulas (9) and (10):

[0033] (7)

[0034] (8)

[0035] (9)

[0036] (10)

[0037] in Indicates the height of the output image. Indicates the width of the output image, letters Indicates the height of the input image. p Indicates the padding size. k represents the convolution kernel size, s represents the step length, Indicates the width of the input image;

[0038] Step 2.3: For the reconstructed gradient map and gradient reference map output by the gradient branch, use L1 loss, as shown in formula (11) (12):

[0039] (11)

[0040] (12)

[0041] Where L represents the sum of all pixel loss values. x , y Represent the coordinates of the pixels, Represent pixel values, nrepresents the number of pixel points, represents the loss value. Here, the loss value is calculated in the mean way, and the gradient branch parameters are updated using the loss value to obtain the updated gradient branch.

[0042] Step 3 is specifically as follows:

[0043] Step 3.1: For the packed data input to the image enhancement branch in Step 2.1, UNet is selected as the backbone network of the image enhancement branch. The downsampling operation is replaced by a convolutional operation with a stride of 2, followed by batch norm and ReLU operations, which are repeated seven times. Then, the upsampling operation is completed using a transposed convolution with a stride of 2, followed by batch norm and ReLU operations, which are repeated seven times to obtain a feature map. Then, a convolution and ReLU operation are performed on the feature map to obtain an output image with 3 channels;

[0044] Step 3.2: Calculate the L1 loss for the output map of the image enhancement branch and the normal light image, and determine which of the output image and the normal light image is true, using the MSE loss, as shown in formulas (13) and (14).

[0045] (13)

[0046] (14)

[0047] where L represents the sum of the loss values of all pixel points, respectively represent the values of the corresponding blocks, represents the loss value, x , y represents the horizontal and vertical coordinates of the patch. Here, the mean method is used, and the image enhancement branch parameters are updated using the loss value to obtain the updated image enhancement branch.

[0048] Step 4 is specifically as follows:

[0049] Step 4.1: Construct a feature fusion block, using the CBAM module based on channel and spatial attention. The specific operation of CBAM is as follows: First, perform channel attention operations on the output feature maps of Steps 2 and 3, perform max pooling and average pooling operations spatially respectively, then input to the shared multi-layer perceptron MLP, and then perform addition and pass through the tanh function, as shown in formula (15).

[0050] (15)

[0051] where represents the activated output value, x represents the input value, and exp represents the natural logarithm.

[0052] After that, perform spatial attention operation, conduct max pooling and average pooling on each channel, and then perform a concatenation operation. Similarly, after passing through the sigmoid function, send the output feature map to the image enhancement branch in Step 3;

[0053] Step 4.2: First, perform a concatenation operation on the feature maps from Step 2 and Step 3, then perform convolution to obtain a new feature map. For the new feature map, follow it with a CBAM module, then perform batch norm operation and ReLU operation, and add a dropout operation to randomly mask half of the neurons in the feature fusion block in Step 4.1. Repeat the concatenation operation until the neuron masking operation is performed twice. A total of nine fusion modules are used in the gradient branch and the low-light enhancement branch to obtain nine feature fusion modules.

[0054] Step 5 is as follows:

[0055] Adopt the relativistic generative adversarial neural network RGAN, and in the way of using a conditional discriminator, add conditions to control the output of the discriminator. The added conditions are the event pseudo-image in Step 1, the reconstructed gradient image, and the low-light image. When calculating the probability that the picture is real, use the patchGAN discriminator to judge the probability that parts of the image are true or false, and adopt the sigmoid activation function, as shown in formula (16),

[0056] (16)

[0057] where represents the activated value, x represents the input value, exp represents the natural logarithm. The specific operation of the discriminator is to first perform a concatenation on the feature map and extract features through convolution operations, then perform instance norm and ReLU operations, repeat four times, and judge the probability that the output image in Step 3 is a normal illumination image;

[0058] Step 6 is as follows:

[0059] Step 6.1: For the neural network composed of Step 2 and Step 3, select the ADAM optimizer for the network, set the initial learning rate to 0.0002, and the scheduler strategy to the multi-step decay strategy. The decay steps are 25 and 100 respectively, and each time it is decayed by half. A total of 300 epochs are trained. During the training, observe the psnr, as shown in formulas (17) and (18),

[0060] (17)

[0061] (18)

[0062] Among them, MSE represents the mean square error, m and n respectively represent the length and width of the image, i and j represent the coordinate position of the pixel point, represents the image output by the network in step 3, represents the normal illumination image obtained in step 1, PSNR represents the peak signal-to-noise ratio, represents the maximum pixel value of the image;

[0063] ssim is as shown in formula (23). Formula (22) contains three parts, the luminance comparison , as shown in formula (19), the structure comparison , as shown in formula (21), the contrast comparison , as shown in formula (20), and respectively represent x and y the means of and respectively represent x and y the variances of represents x and y the covariance of where and simplify formula (22) to formula (23):

[0064] (19)

[0065] (20)

[0066] (21)

[0067] (22)

[0068] (23)

[0069] Dynamically adjust the hyperparameters by observing the two indicators of PSNR and SSIM: the learning rate lr, the balance parameter , and the number of training epochs epoch;

[0070] Step 6.2: Use the SummaryWriter of the third-party Python library tensorboard to output the reference metrics of loss, PSNR, and SSIM during the training process to tensorboard, and test the results on the validation set. After that, save the model, saving the neural network parameters trained in Step 6.1, the current training epoch, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0071] Step 7 is as follows:

[0072] Load the network model trained in Step 6, input the test set into the trained network model, and then save the test results to obtain the enhanced images.

[0073] The beneficial effect of the present invention is a low-light image enhancement method based on an event camera. First, an event stream is synthesized into an event pseudo-image, and then the event pseudo-image is reconstructed into a gradient map. To reduce the number of parameters in the reconstruction, information from some low-light enhancement branches is introduced into the gradient branch, and then the gradient information is used to guide the low-light enhancement branches. The fusion block uses the CBAM module to better focus on the information that the network needs to learn. To increase the robustness of the network, L1 loss is used as the loss function. Finally, to improve the authenticity of the generated images, a conditional discriminator is added. The experimental results prove that the present invention has a good low-light enhancement effect. Description of the Drawings

[0074] Figure 1 is the overall structural schematic diagram of the low-light enhancement method based on an event camera of the present invention;

[0075] Figure 2 is an example of the dataset for training, including low-light images, event pseudo-images, normal-light images, and gradient images;

[0076] Figure 3 is the network schematic diagram for reconstructing the event pseudo-image into a gradient image;

[0077] Figure 4 is the network schematic diagram for enhancing the low-light image to a normal-light image;

[0078] Figure 5(a) is the network schematic diagram of the fusion module;

[0079] Figure 5(b) is the network schematic diagram of the channel and spatial attention module;

[0080] Figure 6(a) shows the change of the evaluation index psnr during the training process;

[0081] Figure 6(b) shows the change of the evaluation index ssim;

[0082] Figure 6 (c) shows the decrease in the loss value;

[0083] Figure 7 It is the experimental result diagram of the present invention. Specific embodiments

[0084] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0085] The low-light image enhancement method based on an event camera of the present invention has a flowchart as Figure 1 shown, and is specifically implemented according to the following steps:

[0086] Step 1, dataset synthesis. Select a dataset with normal-light images and event streams, requiring that the normal-light images and event streams are paired spatially. Then generate gradient images and noisy low-light images from the normal-light images, and preprocess the event streams to obtain event pseudo-images with good edge information;

[0087] Step 1 is specifically as follows:

[0088] The low-light image enhancement process based on an event camera is as follows:

[0089] For an event , first sum the polarities of the events to obtain an event pseudo-image, then extract features from the event pseudo-image and perform gradient image reconstruction. During the reconstruction process, to reduce the computational amount of parameters, introduce the feature map of the low-light image into the gradient branch (GB), and at the same time use a fusion block to introduce gradient information into the low-light enhancement branch, complete the enhancement of the image through the low-light enhancement branch, and finally judge whether the generated image is real through a conditional discriminator.

[0090] First, combine the events into an event pseudo-image, then obtain a simulated low-light image by adding noise and gamma correction to the normal-light image, perform sobel edge extraction operation on the normal-light image to obtain a gradient map, and divide it into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0091] Step 1.1, select a dataset with normal-light images and event streams, requiring that the normal-light images and event streams are paired spatially, where the representation of an event is , where represents an event, x and y represent the coordinate positions of pixel points, t represents the timestamp, p represents the polarity of the event, j represents the nth event. Assume that the time length of the event stream is , during which the event camera simultaneously returns n gray frames. For the pixel values of the synthesized pseudo-image , each event interval is represented as , obtained by adding the polarity values of the events, to obtain an event pseudo-image;

[0092] Step 1.2: For the reference image GT, in order to obtain a more realistic low-light image, noise is added to the reference image GT. For the authenticity of the dataset, Gaussian blind noise is used. The specific method is to generate Gaussian blind noise with a standard deviation range to replace generating Gaussian noise with a single standard deviation, and then perform simulation of the low-light scene and gamma correction, as shown in Formula 1,

[0093] (1)

[0094] where represents the corrected image, represents the image before correction, represents the scaling strength of the pixel value;

[0095] After that, the data is linearly normalized, as shown in Formula 2,

[0096] (2)

[0097] where represents the normalized image, X represents the coordinate at the pixel value, represents the minimum pixel value of image x, represents the maximum pixel value of the image, to obtain a simulated noisy low-light image;

[0098] Step 1.3: In order to obtain a gradient image, edge extraction is performed on the normal illumination image obtained in Step 1.1 using the sobel operator. The specific calculation method is shown in Formulas (3), (4), and (5): For the assumed image A:

[0099] (3)

[0100] (4)

[0101] (5)

[0102] represents the first-order difference in the horizontal direction, represents the first-order difference in the vertical direction, G represents the gradient map, where represents the convolution operation, and G obtained is the gradient image.

[0103] Step 2: Use the UNet gradient branch to reconstruct the event pseudo-image obtained in Step 1 into a gradient image;

[0104] Step 2 is as follows:

[0105] Step 2.1: First, use the dataset class of the deep learning framework PyTorch and perform transform operations on the images in this class. First, convert the images into tensor format and perform normalization operations. In order to enable deep learning to better learn data, standardization processing is then performed, as shown in formula (6):

[0106] (6)

[0107] in Indicates c The output of each channel, Indicated c Channel input, Indicates c The mean of the channels, Indicates c The variance of each channel is calculated, and then the data is packaged through DataLoader;

[0108] Step 2.2: For the packed data, first extract features to obtain feature maps. The UNet network has a good feature extraction structure. UNet is selected as the backbone network of the gradient branch. The maximum pooling operation is replaced by a convolution operation with a step size of 2. Then the data range is adjusted by the batchnorm algorithm, followed by activation using the ReLU activation function. Downsampling is performed seven times. Then upsampling is performed by a deconvolution operation with a step size of 2. The data range is adjusted using the batchnorm algorithm and the ReLU activation function is used. Upsampling is performed seven times. Finally, the upsampled feature map is reconstructed by the gradient map to obtain a reconstructed gradient image. In order to ensure the size of the feature map, a padding operation is performed. In order to reduce the number of parameters, the information of the low-light image obtained in step 1.2 is introduced into the gradient branch. Specifically, as shown in the convolution formulas (7) and (8) and the deconvolution formulas (9) and (10):

[0109] (7)

[0110] (8)

[0111] (9)

[0112] (10)

[0113] in Indicates the height of the output image. Indicates the width of the output image, letters represents the height of the input image, p represents the padding size, k represents the convolutional kernel size, s represents the stride, represents the width of the input image;

[0114] Step 2.3: For the reconstructed gradient map and the gradient reference map output by the gradient branch, use L1 loss, as shown in formulas (11) and (12),

[0115] (11)

[0116] (12)

[0117] where L represents the sum of the loss values of all pixel points, x , y represent the coordinates of the pixel respectively, represent the pixel values respectively, n represents the number of pixel points, represents the loss value. Here, the loss value is calculated in the form of the mean, and the loss value is used to update the parameters of the gradient branch to obtain the updated gradient branch.

[0118] Step 3: Use the UNet low-light image enhancement branch to enhance the low-light image obtained in Step 1 to obtain the enhanced image;

[0119] Step 3 is specifically as follows:

[0120] Step 3.1: For the packed data input image enhancement branch in Step 2.1, select UNet as the backbone network of the image enhancement branch, replace the downsampling operation with a convolutional operation with a stride of 2, then perform batch norm and ReLU operations, repeat seven times, then complete the upsampling operation with a transposed convolution with a stride of 2, then perform batch norm and ReLU operations, repeat seven times, obtain the feature map, and perform a convolution and ReLU operation on the feature map to obtain the output image with 3 channels;

[0121] Step 3.2: Calculate the L1 loss for the output map of the image enhancement branch and the normal light image, and judge which of the output image and the normal light image is true. Here, the MSE loss is used, as shown in formulas (13) and (14),

[0122] (13)

[0123] (14)

[0124] where L represents the sum of the loss values of all pixel points, respectively represent the values of the corresponding blocks, represent the loss value, x , y represent the horizontal and vertical coordinates of the patch. Here, the mean value method is adopted, and the loss value is used to update the parameters of the image enhancement branch to obtain the updated image enhancement branch.

[0125] Step 4: To help the gradient map better guide the enhancement of low-light images, a feature fusion module is designed. The feature fusion module is a module CBAM based on channel and spatial attention, which fuses the information contained in the gradient image in Step 2 into the low-light enhancement branch in Step 3;

[0126] The specific steps of Step 4 are as follows:

[0127] Step 4.1: Construct a feature fusion block, using the module CBAM based on channel and spatial attention. The specific operation of CBAM is as follows: First, perform channel attention operations on the output feature maps of Step 2 and Step 3, perform max-pooling and average-pooling operations spatially respectively, then input them into a shared multi-layer perceptron MLP, and then perform addition and pass through the tanh function, as shown in formula (15).

[0128] (15)

[0129] where represents the activated output value, x represents the input value, and exp represents the natural logarithm.

[0130] Then perform spatial attention operations, perform max-pooling and average-pooling on each channel, and then perform a concatenation operation. Similarly, after passing through the sigmoid function, the output feature map is sent to the image enhancement branch in Step 3;

[0131] Step 4.2: First, perform a concatenation operation on the feature maps of Step 2 and Step 3, then perform convolution to obtain a new feature map. For the new feature map, follow a CBAM module, then perform batch norm operation and ReLU operation, and add a dropout operation to randomly mask half of the neurons in the feature fusion block in Step 4.1. Repeat the concatenation operation until the operation of masking half of the neurons is performed twice. A total of nine fusion modules are used in the gradient branch and the low-light enhancement branch to obtain nine feature fusion modules.

[0132] Step 5: Add a conditional discriminator, with the conditions being the event pseudo-image and the gradient image, to generate a more realistic enhanced image;

[0133] The specific steps of Step 5 are as follows:

[0134] To increase the authenticity of the generated images, the relativistic generative adversarial neural network RGAN is adopted, and in the way of using a conditional discriminator, adding conditions can better control the output of the discriminator. The added conditions are the pseudo-image of the step 1 event, the reconstructed gradient image, and the low-light image. When calculating the probability that the image is real, it is not represented by a single value, but by using a patchGAN discriminator to judge the probability that parts of the image are real or fake, and the sigmoid activation function is used, as shown in formula (16).

[0135] (16)

[0136] where represents the activated value, x represents the input value, exp represents the natural logarithm. The specific operation of the discriminator is to first splice the feature maps and extract features through convolutional operations, and then perform instance norm and ReLU operations, repeating four times to judge the probability that the output image of step 3 is a normal illumination image;

[0137] Step 6: Train the neural network built by the gradient branch of step 2 and the low-light image enhancement branch of step 3 for 300 epochs, verify the training results, and save the model of the neural network;

[0138] Step 6 is specifically as follows:

[0139] Step 6.1: For the neural network composed of step 2 and step 3, select the ADAM optimizer for the network, set the initial learning rate to 0.0002, and the scheduler strategy to the multi-step decay strategy, with the decay steps being 25 and 100 respectively, and each time it decays by half, and train for a total of 300 epochs. During the training, by observing the psnr, as shown in formulas (17) and (18).

[0140] (17)

[0141] (18)

[0142] where MSE represents the mean square error, m and n respectively represent the length and width of the image, i and j represent the coordinate positions of the pixel points, represents the image output by the step 3 network, represents the normal illumination image obtained in step 1, and PSNR represents the peak signal-to-noise ratio, represents the maximum pixel value of the image;

[0143] ssim is as shown in formula (23), and formula (22) contains three parts, the brightness comparison , as shown in formula (19), structure comparison , as shown in formula (21), contrast comparison , as shown in formula (20), and respectively represent x and y means of and respectively represent x and y variances of represents x and y covariance of, where represents a constant to avoid a zero denominator. By setting and formula (22) is simplified to formula (23):

[0144] (19)

[0145] (20)

[0146] (21)

[0147] (22)

[0148] (23)

[0149] Dynamically adjust hyperparameters by observing two metrics, PSNR and SSIM: learning rate lr, balance parameter , and number of training epochs epoch;

[0150] Step 6.2: To better visualize the training results, use the SummaryWriter of the third-party Python library tensorboard to output the reference metrics of loss, PSNR, and SSIM during the training process to tensorboard, and test the results on the validation set. After that, save the model. To facilitate continued training and hyperparameter adjustment, save the neural network parameters trained in Step 6.1, the current training epoch, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0151] Step 7: Test the neural network model saved in Step 6 and output the enhanced image.

[0152] Step 7 is specifically as follows:

[0153] Load the network model trained in Step 6, input the test set into the trained network model, and then save the test results to obtain the enhanced image.

[0154] As Figure 1 shown, the embodiments of the present invention include:

[0155] For low-light image enhancement based on an event camera, an event stream and a normal light image are first extracted. An event pseudo-image is synthesized from the event stream, and a gradient image and a low-light noisy image are generated from the normal light image. Then, the event pseudo-image is input into a gradient branch to be reconstructed into a gradient image, and the low-light image is input into a low-light enhancement branch to obtain a normal light image. To reduce the parameter calculation amount and use the gradient image to guide the enhancement of the low-light image, a module based on channel and spatial attention mechanisms is adopted. To increase the authenticity of the generated low-light image, a discriminator network with added conditions is adopted. Experiments prove that the present invention has excellent enhancement effects.

[0156] The low-light enhancement method based on an event camera is specifically implemented according to the following steps:

[0157] Step 1: First, the event stream is combined into an event pseudo-image. Then, a simulated low-light image is obtained by adding noise and gamma correction to the normal light image. The sobel edge extraction operation is performed on the normal light image to obtain a gradient map, which is divided into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0158] Step 1.1: The representation of an event is , assuming that the time length of the event stream is , during which the event camera simultaneously returns n gray frames. For the pixel value of the synthesized pseudo-image, each event interval is represented as , which is obtained by adding the polarity values of the events and is the input of the gradient map reconstruction branch. Using this stacking method based on the time interval is likely to cause excessive superposition and sparsity of events. Therefore, better results can be obtained by using a fixed number of events, and the input of the gradient branch can be controlled by changing the value and number of ;

[0159] Step 1.2: For the reference image (GT), to obtain a more realistic low-light image, noise is added to the reference image. To increase the authenticity of the data set, Gaussian blind noise can be used. The specific method is to generate Gaussian blind noise within a standard deviation range to replace generating Gaussian noise with a single standard deviation, and then perform the simulation of the low-light scene and gamma correction, formula (1), where represents the corrected image, represents the image before correction, represents the scaling strength of the pixel value. Then, the data is linearly normalized, formula (2), whereX Represents the pixel value at the coordinate, represents the minimum pixel value of the image x, represents the maximum pixel value of the image, thus obtaining the input of the low-light image enhancement branch;

[0160] (1)

[0161] (2)

[0162] Step 1.3, in order to obtain the reference image for reconstructing the gradient map, edge extraction is performed on the normal light image using the sobel operator. The specific calculation method, formulas (3), (4), and (5) for the assumed image A, represents the first-order difference in the horizontal direction, represents the first-order difference in the vertical direction, G represents the gradient map, where * represents the convolution operation, then G is the reference image of the gradient map. Dataset examples are as Figure 2 , the first and fifth rows represent the low-light noisy image, the second and sixth rows represent the pseudo-image synthesized by events, the third and seventh rows represent the gradient image extracted from the normal light image, and the fourth and eighth rows represent the normal light image.

[0163] (3)

[0164] (4)

[0165] (5)

[0166] The specific operation of Step 2 is as follows:

[0167] Step 2.1, first define the dataset class, and perform transform operations on the images in this class. First, convert the images to tensor format and perform normalization operations. In order to enable deep learning to better learn the data, standardization processing is then performed, formula (6), where represents the output of the c th channel, represents the c th channel input, represents the mean of the c th channel, represents the variance of the c th channel. Next, the data is packed through DataLoader;

[0168] (6)

[0169] Step 2.2: For the packed data, first perform feature extraction to obtain a feature map. The UNet network has a good feature extraction structure. Select UNet as the backbone network of the gradient branch. As Figure 3 , replace the max pooling operation with a convolutional operation with a stride of 2. Then, adjust the data range through the batchnorm algorithm, followed by the ReLU activation function. Here, perform seven downsamplings. Then, perform upsampling through a transposed convolutional operation with a stride of 2, adjust the data range using the batch norm algorithm, and use the ReLU activation function. Similarly, perform upsampling several times. Finally, reconstruct the gradient map for the upsampled feature map. To ensure the size of the feature map, perform a padding operation. To reduce the number of parameters, introduce the feature map of the low-light image enhancement branch into the gradient branch. The specific convolutional formulas (7), (8) and transposed convolutional formulas (9), (10) are as follows, where the letter represents the height of the input image, p represents the padding size, k represents the convolutional kernel size, s represents the stride, represents the height of the output image, represents the width of the input image, represents the width of the output image;

[0170] (7)

[0171] (8)

[0172] (9)

[0173] (10)

[0174] Step 2.3: For the reconstructed gradient map and the gradient reference map output by the gradient branch, use L1 loss, formulas (11), (12), where x , y respectively represent the coordinates of the pixel, n represents the number of pixel points. Here, calculate the loss in an average manner.

[0175] (11)

[0176] (12)

[0177] Step 3 is specifically as follows:

[0178] Step 3.1, for the packed data input to the image enhancement branch in Step 2.1, similarly, replace the downsampling operation with a convolutional operation with a stride of 2, then perform batch norm and ReLU operations, repeat seven times, then complete the upsampling operation with a transposed convolution with a stride of 2, then perform batch norm and ReLU operations, repeat seven times, and finally perform a convolution and ReLU operation on its feature map to obtain an output map with 3 channels. The schematic diagram of the network structure is as shown in Figure 4 ;

[0179] Step 3.2, calculate the L1 loss for the output map of the image enhancement branch and the normal light image, and determine which of the output image and the normal light image is true. Here, the MSE loss is used, as shown in Formulas (13) and (14), where x , y represents the value of the n th patch, and the mean value method is used here.

[0180] (13)

[0181] (14)

[0182] Step 4 is specifically as follows:

[0183] Step 4.1, for the feature fusion block, here a module based on channel and spatial attention (CBAM) is used. The specific operation of CBAM is as follows: First, perform channel attention on the input feature map F, perform max pooling and average pooling operations spatially respectively, then input to the shared multi-layer perceptron MLP, then perform addition and pass through the tanh function, as shown in Formula (15), and then perform spatial attention, perform max pooling and average pooling on each channel, and then perform a concatenation operation. Similarly, pass through the sigmoid function;

[0184] (15)

[0185] Step 4.2, for the feature maps to be fused, first perform a concatenation operation, then perform convolution to obtain a feature map. For the fused feature map, follow a CBAM module, then perform batch norm operation and ReLU operation, and a dropout operation is added to randomly mask out half of the neurons. Repeat this operation twice. A total of nine fusion modules are used in the gradient branch and the low-light enhancement branch. The network structure of the fusion module refers to Figure 5(a) shown, and the schematic diagram of the channel and spatial attention mechanism module refers to Figure 5(b) shown;

[0186] Step 5 is specifically as follows:

[0187] To increase the authenticity of the generated images, the Relativistic Generative Adversarial Network (RGAN) is adopted here, and a conditional discriminator is used. By adding conditions, the output of the generator can be better controlled. The added conditions are event pseudo-images, reconstructed gradient images, and low-light images. When calculating the probability that an image is real, instead of using a single value, PatchGAN is used to judge the probability that parts of the image are real or fake. Here, the sigmoid activation function, formula (16), is used. The specific operation of the discriminator's network module is to first splice the feature maps and extract features through convolutional operations, and then perform instance normalization and ReLU operations, repeating this step four times.

[0188] (16)

[0189] Step 6 is specifically as follows:

[0190] In step 6.1, select the ADAM optimizer for the network, set the initial learning rate to 0.0002, and the scheduler to a multi-step decay strategy with decay steps of 25 and 100, each time decaying by half, and train for a total of 300 epochs. During training, by observing psnr, formulas (17) and (18), where represents the maximum pixel value of the image, and the ssim formula (23), where represents the luminance comparison, represents the contrast comparison, represents the structure comparison, and respectively represent x and y the mean values of and respectively represent x and y the variances of represents x and y the covariance of represents a constant to avoid a denominator of 0. By setting and the formula (22) can be simplified to formula (23). These two metrics dynamically adjust some hyperparameters, such as the learning rate lr, the balance parameter , and the number of training epochs epoch, etc.

[0191] (17)

[0192] (18)

[0193] (19)

[0194] (20)

[0195] (21)

[0196] (22)

[0197] (23)

[0198] Step 6.2. To better visualize the training results, use SummaryWriter to output reference metrics such as loss, PSNR, and SSIM during the training process to TensorBoard, and test the results on the validation set. The training results are shown in Figure 6, where Figure 6(a) shows the change in PSNR for each epoch during the training phase, Figure 6(b) shows the change in SSIM for each epoch during the training phase, and Figure 6(c) shows the decrease in loss during the training phase. After that, save the model. To facilitate continued training and hyperparameter adjustment, the network parameters, the current training epoch, the optimizer, and the scheduler can be saved. At the same time, prepare for the next test.

[0199] Step 7 is specifically as follows:

[0200] Load the trained model, input the test set into the trained network model, then save the test results, and calculate the PSNR and SSIM metrics for the test results. The objective metrics are that the PSNR is approximately 25.5 and the SSIM is 0.82. The subjective experimental results are as Figure 7 shown. The first row is the low-light image, the second row is the event pseudo-image, the third row is the reconstructed gradient map, the fourth row is the gradient map corresponding to the normal-light image, the fifth row is the result after enhancing the low-light image, and the sixth row is the normal-light image.

[0201] The present invention enhances the low-light image and guides the enhancement of the low-light image through the gradient image reconstructed from the event stream. The enhanced result obtains a high value in objective metrics and achieves a good enhancement effect in subjective experiments, and can remove most of the noise in the low-light image and reconstruct a normal-light image with good details.

Claims

1. A low-light image enhancement method based on an event camera, characterized in that, The specific implementation steps are as follows: Step 1, dataset synthesis: Select a dataset with normal-light images and event streams, requiring that the normal-light images and event streams are paired spatially. Then generate gradient images and noisy low-light images from the normal-light images, and preprocess the event stream to obtain an event pseudo-image with good edge information; Step 2, use the UNet gradient branch to reconstruct the event pseudo-image obtained in Step 1 into a gradient image; The specific steps of Step 2 are as follows: Step 2.1, first use the dataset class of the deep learning framework PyTorch, and perform a transform operation on the images in this class. First, convert the images into tensor format and perform a normalization operation, and then perform a standardization process, as shown in formula (6): （6） Among them represents the output of the c th channel, represents the c inputs of the channels, represents the mean of the c th channel, represents the variance of the c th channel. Next, the data is packed through DataLoader; Step 2.2, for the packed data, first perform feature extraction to obtain a feature map. Select UNet as the backbone network of the gradient branch, replace the max-pooling operation with a convolutional operation with a stride of 2, and then adjust the data range through the batchnorm algorithm. Immediately use the ReLU activation function for activation. Perform seven downsamplings here, and then perform upsampling through a transposed convolutional operation with a stride of 2. Use the batch norm algorithm to adjust the data range and use the ReLU activation function. Similarly, perform seven upsamplings. Finally, perform gradient map reconstruction on the upsampled feature map to obtain the reconstructed gradient image, and perform a padding operation to introduce the information of the low-light image obtained in Step 1.2 into the gradient branch, specifically as shown in convolutional formulas (7), (8) and transposed convolutional formulas (9), (10): （7） （8） （9） （10） wherein represents the height of the output image, represents the width of the output image, represents the height of the input image, p represents the padding size, k represents the convolution kernel size, s represents the stride, represents the width of the input image; Step 2.3, for the reconstructed gradient map and the gradient reference map output by the gradient branch, use L1 loss, as shown in formulas (11) and (12): （11） （12） Where L represents the sum of the loss values of all pixel points, x , y respectively represent the coordinate positions of the pixel points, respectively represent the pixel values, n represents the number of pixel points, represents the loss value. Here, the loss value is calculated by the mean method, and the gradient branch parameters are updated using the loss value to obtain the updated gradient branch; Step 3, use the UNet low-light image enhancement branch to enhance the low-light image obtained in Step 1 to obtain an enhanced image; The specific steps of Step 3 are as follows: Step 3.1, for the packed data input to the image enhancement branch in Step 2.1, select UNet as the backbone network of the image enhancement branch, replace the downsampling operation with a convolutional operation with a stride of 2, and then perform batch norm and ReLU operations, repeating seven times. Then complete the upsampling operation with a transposed convolutional operation with a stride of 2, and then perform batch norm and ReLU operations, repeating seven times, to obtain a feature map, and perform a convolution and ReLU operation on the feature map to obtain an output image with 3 channels; Step 3.2, calculate the L1 loss for the output image of the image enhancement branch and the normal-light image, and determine which of the output image and the normal-light image is true, using the MSE loss, as shown in formulas (13) and (14): （13） （14） where L represents the sum of the loss values of all pixel points, respectively represent the pixel values, represents the loss value, x , y represents the coordinate position of the pixel point. Here, in the way of taking the mean value, the loss value is used to update the parameters of the image enhancement branch, and the updated low-light image enhancement branch is obtained; Step 4, design a feature fusion module, that is, use the CBAM module based on channel and spatial attention to fuse the information contained in the gradient image in Step 2 into the low-light image enhancement branch in Step 3; Step 5, add a conditional discriminator, with the conditions being the event pseudo-image and the gradient image, to generate a more realistic enhanced image; Step 6: Train the neural network constructed in Step 2 (gradient branch) and Step 3 (low-light image enhancement branch) for 300 epochs, verify the training results, and save the model of the neural network; Step 7: Test the model of the neural network saved in Step 6 and output the enhanced image.

2. The method for low-light image enhancement based on an event camera according to claim 1, wherein The specific operations of Step 1 are as follows: Step 1.1: Select a dataset with normal light images and event streams, where the normal light images and event streams are spatially paired. The representation of an event is , where represents an event,[[]] x and y represent the coordinate position of a pixel point,[[]] t represents the timestamp,[[]] p represents the polarity of the event,[[]] j represents the nth event. Assume that the time length of the event stream is , during which the event camera simultaneously returns n grayscale frames. For the pixel values of the synthesized pseudo-image , each event interval is expressed as , obtained by adding the polarity values of the events, and an event pseudo-image is obtained; Step 1.2: Add noise to the reference image GT. Gaussian blind noise is used. Specifically, instead of generating Gaussian noise with a single standard deviation, Gaussian blind noise is generated within a standard deviation range, and then weak light scene simulation and gamma correction are performed as shown in Formula 1. （1） Among them represents the corrected image, represents the image before correction, represents the scaling strength of pixel values; After that, the data is linearly normalized as shown in Formula 2. （2） Among them represents the normalized image, X represents the coordinate at the pixel value, represents the minimum pixel value of image x, represents the maximum pixel value of the image, and a simulated low-light image with noise is obtained; Step 1.3: Extract the edges of the normal illumination image obtained in Step 1.1 using the sobel operator. The specific calculation methods are shown in Formulas (3), (4), and (5). For the assumed image A: （3） （4） （5） represents the first-order difference in the horizontal direction, represents the first-order difference in the vertical direction, G represents the gradient map, where represents the convolution operation, and obtaining G is the gradient image.

3. The method for enhancing low-light images based on an event camera according to claim 2, wherein The specific operations of Step 4 are as follows: Step 4.1: Construct a feature fusion block using the channel and spatial attention-based module CBAM. The specific operations of CBAM are as follows: First, perform channel attention operations on the output feature maps of Step 2 and Step 3, perform max pooling and average pooling operations spatially respectively, then input them into a shared multi-layer perceptron MLP, and then perform addition and pass through the tanh function as shown in Formula (15). （15） Among them represents the output value after activation, x represents the input value, and exp represents the natural logarithm; After that, perform spatial attention operations, perform max pooling and average pooling on each channel, and then perform a concatenation operation. Similarly, after passing through the sigmoid function, the output feature map is sent to the low-light image enhancement branch of Step 3. Step 4.2: First, perform a concatenation operation on the feature maps of Step 2 and Step 3, then perform convolution to obtain a new feature map. A CBAM module follows the new feature map, and then batch norm operation and ReLU operation are performed, and a dropout operation is added to randomly mask half of the neurons in the feature fusion block in Step 4.

1. Repeat the concatenation operation until the neuron masking operation is performed twice. A total of nine fusion modules are used in the gradient branch and the low-light image enhancement branch to obtain nine feature fusion modules.

4. The method for enhancing low-light images based on an event camera according to claim 3, wherein The specific operations of Step 5 are as follows: Adopt the relativistic generative adversarial neural network RGAN and use the conditional discriminator method to add conditions to control the output of the discriminator. The added conditions are the event pseudo-image in Step 1, the reconstructed gradient image, and the low-light image. When calculating the probability that the image is real, use the patchGAN discriminator to judge the probability that part of the image is real or fake, and use the sigmoid activation function as shown in Formula (16). （16） Among them represents the activated value x represents the input value, exp represents the natural logarithm. The specific operation of the discriminator is to first splice the feature maps and extract features through convolution operations, and then perform instance norm and ReLU operations, repeating four times to judge the probability that the output image in step 3 is a normal illumination image.

5. The method for low-light image enhancement based on an event camera according to claim 4, wherein The specific operations of Step 6 are as follows: Step 6.1: For the neural network composed of Step 2 and Step 3, select the ADAM optimizer for the network, set the initial learning rate to 0.0002, and the scheduler strategy to the multi-step decay strategy with decay steps of 25 and 100 respectively, and each time it decays by half. A total of 300 epochs are trained. During the training, observe the psnr as shown in Formulas (17) and (18). （17） （18） Among them, MSE represents the mean square error, m and n respectively represent the length and width of the image, i and j represent the coordinate position of the pixel point, represents the image output by the network in step 3, represents the normal illumination image obtained in step 1, and PSNR represents the peak signal-to-noise ratio, represents the maximum pixel value of the image; The ssim is as shown in formula (23). Formula (22) consists of three parts: luminance comparison , as shown in formula (19), structure comparison , as shown in formula (21), contrast comparison , as shown in formula (20), and respectively represent x and y the means of and respectively represent x and y the variances of represents x and y the covariance of, where represents a constant to avoid a zero denominator. By setting and formula (22) is simplified to formula (23): （19） （20） （21） （22） （23） Dynamically adjust hyperparameters by observing two metrics, PSNR and SSIM: learning rate lr, balance parameter , and number of training epochs epoch; Step 6.2: Use the SummaryWriter of the third-party Python library tensorboard to output the reference metrics of loss, PSNR, and SSIM during the training process to tensorboard, and test the results on the validation set. Then save the model, saving the neural network parameters trained in Step 6.1, the current number of training epochs, the optimizer ADAM, and the scheduler to obtain the trained network model.

6. The method for low-light image enhancement based on an event camera according to claim 5, wherein The specific steps of Step 7 are as follows: Load the network model trained in Step 6, input the test set into the trained network model, and then save the test results to obtain the enhanced image.

Images