Event stream and color image fusion based non-uniform illumination image enhancement method

Abstract

The application discloses a non-uniform illumination image enhancement method based on event stream and color image fusion, and belongs to the technical field of computer vision image enhancement.The application aims at the problem that events output by an event camera cannot provide color information of a scene.The application comprises the following steps: acquiring an event stream E of an event camera and a corresponding color image frame S of a standard camera;reconstructing the event stream E into an intensity image I1 by adopting an E2VID method;based on a Retinex theory, decomposing the color image frame S into a reflection image R, an illumination image I2 and a noise image N by adopting a convolutional neural network;adjusting the brightness of the intensity image I1 and the illumination image I2 by adopting a gamma transformation to generate two artificial multi-exposure image sequences;calculating the weight maps of the exposure images in the two artificial multi-exposure image sequences, mixing the weight maps with a pyramid fusion strategy to obtain a fusion image of the artificial multi-exposure image sequence; and based on the Retinex theory, converting the fusion image into a fused color image in combination with the illumination image I2 and the noise image N.The application is used for non-uniform illumination image enhancement.

Description

Technical Field

[0001] This invention relates to a non-uniform illumination image enhancement method based on event stream and color image fusion, belonging to the field of computer vision image enhancement technology. Background Technology

[0002] Standard cameras have a much smaller dynamic range than natural scenes. Due to the influence of artificial light sources, images captured by standard cameras are unevenly illuminated, containing both low-light and high-light areas, thus reducing visibility. This affects the accuracy of tasks in non-uniformly lit environments such as segmentation, detection, and tracking. Therefore, improving the visibility of non-uniformly lit images by enhancing contrast and restoring detail is an important task.

[0003] In recent years, many image enhancement methods have been proposed to improve the quality of non-uniformly illuminated images, including histogram equalization (HE), Retinex theory, and high dynamic range (HDR) imaging techniques. HE-based methods are global image enhancement approaches, but the enhanced images often suffer from severe noise, overexposed bright areas, and some loss of detail. Retinex-based methods use various priors and assumptions to estimate illumination, then utilize the estimated illumination map to restore image contrast. While Retinex-based methods improve image contrast, they cannot handle lighting effects or recover details in bright areas. HDR imaging methods can suppress lighting effects and preserve details in dark areas, but still cannot recover details in overexposed areas.

[0004] Event cameras, a type of bio-inspired sensor, operate very differently from standard cameras. Each pixel of an event camera can asynchronously detect changes in scene brightness and output them individually as events. Therefore, their output is not an intensity image, but a stream of asynchronous events. Event cameras are a novel type of sensor with high dynamic range, offering a potential solution for enhancing images with non-uniform lighting, particularly for detail recovery in bright areas. However, the events output by event cameras do not provide color information about the scene. Summary of the Invention

[0005] To address the issue that events output by event cameras cannot provide color information about the scene, this invention provides a non-uniform lighting image enhancement method based on the fusion of event streams and color images.

[0006] The present invention provides a non-uniform illumination image enhancement method based on event stream and color image fusion, comprising:

[0007] Step 1: Acquire the event stream E from the event camera and the corresponding color image frame S from the standard camera;

[0008] Step 2: Using the E2VID method, the event stream E is reconstructed into an intensity image I1; at the same time, based on Retinex theory, a convolutional neural network is used to decompose the color image frame S into a reflection image R, an illumination image I2, and a noise image N;

[0009] Step 3: Use gamma transform to adjust the brightness of intensity image I1 and illumination image I2 to generate two artificial multi-exposure image sequences;

[0010] Step 4: Calculate the weight map of each exposure image in the two artificial multi-exposure image sequences, and mix all weight maps with the pyramid fusion strategy to obtain the fused image of the artificial multi-exposure image sequences;

[0011] Step 5: Based on Retinex theory, and combining the illumination map I2 and the noise map N, the fused image is transformed into a fused color image.

[0012] The non-uniform illumination image enhancement method based on event stream and color image fusion according to the present invention,

[0013] In step two, the convolutional neural network is a three-branch convolutional neural network, and each branch is decomposed from the color image frame S to obtain the reflection map R, the illumination map I2, and the noise map N;

[0014] A three-branch convolutional neural network includes CNN1 branch, CNN2 branch, and CNN3 branch;

[0015] The CNN1 branch includes four ReLU+Conv layers, a Conv layer, and an activation function Sigmod, which are used to obtain the reflection map R of the color image frame S.

[0016] The CNN2 branch consists of four ReLU+Conv layers, a Conv layer, and an activation function Sigmod, which are used to obtain the illumination map I2 of the color image frame S.

[0017] The CNN3 branch consists of four ReLU+Conv layers, a Conv layer, and an activation function Tanh, which are used to obtain the noise map N of the color image frame S.

[0018] According to the non-uniform illumination image enhancement method based on event stream and color image fusion of the present invention, the loss function L0 of the three-branch convolutional neural network is:

[0019] L0 = L recon +λ1L ic +λ2L ref +λ3L is +λ4L n ,

[0020] Where L recon To rebuild the losses, L icFor the loss of lighting consistency, L ref For reflectivity loss, L is For the loss of lighting smoothness, L n λ1 is the weighting factor for the lighting consistency loss term, λ2 is the weighting factor for the reflectivity loss term, λ3 is the weighting factor for the lighting smoothness loss term, and λ4 is the weighting factor for the noise estimation loss term.

[0021] The non-uniform illumination image enhancement method based on event stream and color image fusion according to the present invention,

[0022] Each loss term is defined as follows:

[0023] L recon =||S-(R⊙I2+N)||1,

[0024] L ic =||I2-I0||1,

[0025]

[0026]

[0027] L n =||I2⊙N|| F ,

[0028] In the formula, ||·||1 represents the norm, I0 is the initial estimate of illumination, and ω r For first-order difference operators Weights, first-order difference operator Including level operators and vertical operator ω h For the level operator weights, ω v These are the vertical operator weights; ||·|| F is the Frobenius norm, and ⊙ represents element-wise multiplication;

[0029]

[0030] In the formula, normalize represents minimum-maximum normalization.

[0031]

[0032]

[0033] In the formula, G is a Gaussian filter, * denotes the convolution operator, and S g This is a grayscale representation of a color image frame.

[0034] According to the non-uniform illumination image enhancement method based on event stream and color image fusion of the present invention, the method for generating two artificial multi-exposure image sequences in step three is as follows:

[0035] By adjusting the contrast of intensity image I1 and illumination image I2 using gamma correction, two artificial multi-exposure image sequences J are obtained. j (x,y):

[0036]

[0037] In the formula, j represents the two inputs of gamma correction, j = 1, 2; j = 1 corresponds to intensity image I1, j = 2 corresponds to illumination image I2;

[0038] In the formula, x and y are pixel coordinates, α is the first positive constant, and γ is the second positive constant.

[0039] According to the non-uniform illumination image enhancement method based on event stream and color image fusion of the present invention, the method for calculating the weight map of each exposure image in two artificial multi-exposure image sequences in step four is as follows:

[0040] Two artificial multi-exposure image sequences J j (x,y) are integrated into a single artificial multi-exposure image sequence;

[0041] According to contrast W c Goodness W e and average brightness W b Construct the weight map W for each artificially multi-exposure image k (x,y):

[0042]

[0043] In the formula, k represents the sequence number of the image in the integrated artificial multi-exposure image sequence, k = 1, 2, ..., n, and n is the total number of images in the integrated artificial multi-exposure image sequence;

[0044]

[0045]

[0046]

[0047] In the formula, L represents the Laplace operator;

[0048] J k To represent the k-th image in the integrated multi-exposure image sequence, σ is a constant, set to 0.2. Let be the average brightness of the k-th image.

[0049] According to the non-uniform illumination image enhancement method based on event stream and color image fusion of the present invention, the method for obtaining the fused image of an artificial multi-exposure image sequence is as follows:

[0050] For the weighted graph W k Normalize (x,y) to obtain the normalized weight map.

[0051]

[0052] Normalized weight map By combining with a pyramid fusion strategy, a fused image is obtained.

[0053] In the formula U d This indicates an upsampling operation, where l represents the pyramid level. Represents the normalized weights graph L-series Gaussian pyramid decomposition, L l {J k (x,y)} is the input k-th image J k Laplace pyramid decomposition of (x,y) in l-series.

[0054] According to the non-uniform illumination image enhancement method based on event stream and color image fusion of the present invention, the method for obtaining the fused color image in step five is as follows:

[0055] Calculate the noise-free reflectance map of color image frame S.

[0056]

[0057] Combined fused images and noise-free reflection diagram Obtain the fused color image

[0058] In the non-uniform illumination image enhancement method based on event stream and color image fusion according to the present invention, λ1, λ2 and λ3 are taken as 1; λ4 is taken as 5000.

[0059] According to the non-uniform illumination image enhancement method based on event stream and color image fusion of the present invention, the initial illumination estimate I0 is taken as the maximum value of the RGB three-channel intensity of the color image frame S.

[0060] The beneficial effects of this invention are as follows: This invention proposes a non-uniform lighting nighttime visibility enhancement method based on event and color image fusion. It utilizes the complementarity of event cameras and standard cameras to integrate valuable information from event streams and color images, enhancing the visibility of non-uniform lighting images, increasing the brightness of dark areas, and simultaneously restoring details in bright areas while preserving details in dark areas, resulting in a fused image with natural colors. Quantitative and qualitative experimental results show that the fused color image obtained by the method of this invention has richer details, higher dynamic range, and better visual effects. Attached Figure Description

[0061] Figure 1 This is a flowchart of the non-uniform illumination image enhancement method based on event stream and color image fusion described in this invention;

[0062] Figure 2 This is a diagram of the convolutional neural network architecture for decomposing color image frame S;

[0063] Figure 3 The images show the results of enhancing five sets of event streams and color images acquired in a non-uniform lighting environment using different methods; where a is the original image; b is the event reconstructed image; c is the Zero-DCE method; d is the BIMEF method; e is the SRIE method; and f is the method of this invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0066] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0067] Specific Implementation Method 1: Combination Figures 1 to 3 As shown, this invention provides a method for enhancing non-uniform illumination images based on event stream and color image fusion, including:

[0068] Step 1: Acquire the event stream E from the event camera and the corresponding color image frame S from the standard camera;

[0069] Step 2: Using the E2VID (Events-to-Video) method, the event stream E is reconstructed into an intensity image I1; at the same time, based on the Retinex theory, a convolutional neural network is used to decompose the color image frame S into a reflection image R, an illumination image I2, and a noise image N;

[0070] Step 3: Use gamma transform to adjust the brightness of intensity image I1 and illumination image I2 to generate two artificial multi-exposure image sequences;

[0071] Step 4: Calculate the weight map of each exposure image in the two artificial multi-exposure image sequences, and mix all weight maps with the pyramid fusion strategy to obtain the fused image of the artificial multi-exposure image sequences;

[0072] Step 5: Based on Retinex theory, and combining the illumination map I2 and the noise map N, the fused image is transformed into a fused color image.

[0073] In this implementation, the E2VID method employs a convolutional recurrent network to learn intensity image reconstruction from asynchronous event data. First, a new recurrent network is constructed based on the UNet architecture. Then, a large amount of simulated event data is generated using an Event Simulator (ESIM) to train the network. Next, the continuous event stream is divided into a series of non-overlapping spatiotemporal windows and transformed into event tensors. Finally, the event sequences in each spatiotemporal window are sequentially input into the trained network to reconstruct the intensity image.

[0074] Furthermore, such as Figure 2 As shown, in step two, the convolutional neural network is a three-branch convolutional neural network. Based on Retinex theory, each branch is decomposed from the color image frame S to obtain the reflection map R, the illumination map I2, and the noise map N.

[0075] A three-branch convolutional neural network includes CNN1 branch, CNN2 branch, and CNN3 branch;

[0076] The CNN1 branch includes four ReLU+Conv layers, a Conv layer, and an activation function Sigmod, which are used to obtain the reflection map R of the color image frame S.

[0077] The CNN2 branch consists of four ReLU+Conv layers, a Conv layer, and an activation function Sigmod, which are used to obtain the illumination map I2 of the color image frame S.

[0078] The CNN3 branch consists of four ReLU+Conv layers, a Conv layer, and an activation function Tanh, which are used to obtain the noise map N of the color image frame S.

[0079] In this implementation, CNN1 and CNN2 are used for brightness and reflectivity estimation, and CNN3 is used for noise estimation. CNN1, CNN2, and CNN3 are almost identical in structure except for their last layer. To reduce the intensity in the range [0,1], the last layer of CNN1 and CNN2 uses an sigmoid layer. The difference is that the last layer of CNN3 has a tanh layer to ensure that the noise value decreases in the range [-1,1].

[0080] The loss function L0 of a three-branch convolutional neural network consists of five components:

[0081] L0 = L recon +λ1L ic +λ2L ref +λ3L is +λ4L n ,

[0082] Where L recon To rebuild the losses, L ic For the loss of lighting consistency, L ref For reflectivity loss, L is For the loss of lighting smoothness, L n λ1 is the weighting factor for the lighting consistency loss term, λ2 is the weighting factor for the reflectivity loss term, λ3 is the weighting factor for the lighting smoothness loss term, and λ4 is the weighting factor for the noise estimation loss term.

[0083] Each loss term is defined as follows:

[0084] L recon =||S-(R⊙I2+N)||1,

[0085] L ic =||I2-I0||1,

[0086]

[0087]

[0088] L n =||I2⊙N|| F ,

[0089] In the formula, ||·||1 represents the norm, I0 is the initial estimate of illumination, and ω r For first-order difference operators Weights, first-order difference operator Including level operators and vertical operator ω h For the level operator weights, ω v These are the vertical operator weights; ||·|| Fis the Frobenius norm, and ⊙ represents element-wise multiplication;

[0090]

[0091] In the formula, normalize represents minimum-maximum normalization.

[0092]

[0093]

[0094] In the formula, G is a Gaussian filter, * denotes the convolution operator, and S g This is a grayscale representation of a color image frame.

[0095] In the formula, S represents a color image frame, R represents a reflectance map, and N represents a noise map.

[0096] Furthermore, the method for generating two artificially multi-exposure image sequences in step three is as follows:

[0097] By adjusting the contrast of intensity image I1 and illumination image I2 using gamma correction, two artificial multi-exposure image sequences J are obtained. j (x,y):

[0098]

[0099] In the formula, j represents the two inputs of gamma correction, j = 1, 2; j = 1 corresponds to intensity image I1, j = 2 corresponds to illumination image I2;

[0100] In the formula, x and y are pixel coordinates, α is the first positive constant, and γ is the second positive constant.

[0101] The method for calculating the weight map of each exposure image in the two artificial multi-exposure image sequences in step four is as follows:

[0102] Two artificial multi-exposure image sequences J j (x,y) are integrated into a single artificial multi-exposure image sequence;

[0103] According to contrast W c Goodness W e and average brightness W b Construct the weight map W for each artificially multi-exposure image k (x,y):

[0104]

[0105] In the formula, k represents the sequence number of the image in the integrated artificial multi-exposure image sequence, k = 1, 2, ..., n, and n is the total number of images in the integrated artificial multi-exposure image sequence;

[0106]

[0107]

[0108]

[0109] In the formula, L represents the Laplace operator;

[0110] J k To represent the k-th image in the integrated multi-exposure image sequence, σ is a constant, set to 0.2. Let be the average brightness of the k-th image.

[0111] Methods for obtaining fused images from artificially multi-exposure image sequences include:

[0112] To ensure that the sum of the weights of each pixel is 1, the weight map W of the artificially multi-exposure image is... k Normalize (x,y) to obtain the normalized weight map.

[0113]

[0114] Normalized weight map By combining with a pyramid fusion strategy, a fused image is obtained.

[0115] In the formula U d This indicates an upsampling operation, where l represents the pyramid level. Represents the normalized weights graph L-series Gaussian pyramid decomposition, L l {J k (x,y)} is the input k-th image J k Laplace pyramid decomposition of (x,y) in l-series.

[0116] Furthermore, the method for obtaining the fused color image in step five is as follows:

[0117] Calculate the noise-free reflectance map of color image frame S.

[0118]

[0119] Combined fused images and noise-free reflection diagram Obtain the fused color image

[0120] As an example, λ1, λ2, and λ3 are all 1; λ4 is 5000.

[0121] Furthermore, the initial illumination estimate I0 is taken as the maximum value of the RGB three-channel intensity of the color image frame S.

[0122] Experimental verification: To verify the feasibility and effectiveness of the method of the present invention, five sets of event streams and color images were recorded in five different scenes using an event camera DAVIS346. The Zero-DCE, BIMEF, SRIE methods and the method of the present invention were used for image enhancement respectively.

[0123] To evaluate the effectiveness of each method, average gradient (AG) was used to assess image texture detail; a higher value indicates richer detail. The BRISQUE (Bare-Range Image Quality) non-referenced image spatial quality evaluator was used to assess image naturalness; a lower value indicates a more natural image. The AG comparison results are shown in Table 1, and the BRISQUE comparison results are shown in Table 2.

[0124] Table 1 AG Comparison Results

[0125]

[0126] Table 2 Comparison Results of BRISQUE

[0127]

[0128] As shown in Tables 1 and 2, the method of the present invention achieves the best performance on both AG and BRISQUE. From all aspects of performance, the non-uniform illumination nighttime image enhancement method of the present invention retains more detail and better visual effects.

[0129] from Figure 3 As can be seen, Zero-DCE and BIMEF both achieved good results in dark areas. However, these methods over-enhance bright areas, amplify noise, leading to halos and loss of some details, and failing to recover details in bright areas. SRIE can effectively reduce noise and suppress halos, but it cannot recover details in bright areas, resulting in lower overall brightness of the enhanced image and poor visibility in dark areas. The method of this invention improves image visibility and increases dynamic range through event stream and color image fusion, preserving details in dark areas and recovering details in bright areas, achieving good results in both bright and dark areas.

[0130] In summary, visual evaluation verifies that the images generated by the method of this invention have richer details, higher dynamic range, and better visual effects, which is consistent with the quantitative analysis results.

[0131] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A non-uniform illumination image enhancement method based on event stream and color image fusion, characterized in that Comprising, Step one: obtaining an event stream E of an event camera and a color image frame S of a corresponding standard camera; Step two: using an E2VID method to reconstruct the event stream E into an intensity image I1; and using a convolutional neural network to decompose the color image frame S into a reflectance map R, an illumination map I2 and a noise map N based on a Retinex theory; Step three: using a gamma transform to adjust the brightness of the intensity image I1 and the illumination map I2 to generate two artificial multi-exposure image sequences; Step four: calculating weight maps of each exposure image in the two artificial multi-exposure image sequences, mixing all the weight maps with a pyramid fusion strategy to obtain a fusion image of the artificial multi-exposure image sequences; Step five: converting the fusion image into a fused color image based on the Retinex theory and in combination with the illumination map I2 and the noise map N; The method for calculating the weight maps of each exposure image in the two artificial multi-exposure image sequences in step four is: combining two artificial multiple-exposure image sequences J j (x,y) into an artificial multiple-exposure image sequence; where j represents the two inputs of the gamma correction, j = 1,2; where x, y are the pixel coordinate positions; According to the contrast W c , the goodness W e and the average brightness W b of each artificial multi-exposure image, the weight map W k (x, y) of each artificial multi-exposure image is constructed (x, y) : In the formula, k represents the serial number of the image in the integrated artificial multi-exposure image sequence, k = 1, 2, …, n, and n is the total number of images in the integrated artificial multi-exposure image sequence; In the formula, L represents a Laplacian operator; J k For integrating the k-th image in the artificial multi-exposure image sequence after integration, σ is a constant, which is set to 0.2, is the average brightness of the k-th image. The method for obtaining the fusion image of the artificial multi-exposure image sequences is: normalizing the weight map W k (x,y) to obtain a normalized weight map : normalizing the weight map mixing with the pyramid fusion strategy to obtain a fused image : where U d denotes an up-sampling operation, and l denotes the pyramid level, denotes the normalized weight map of the l-level Gaussian pyramid decomposition, L l {J k (x,y) is the l-level Laplacian pyramid decomposition of the k-th input image J k (x,y). The method for obtaining the fused color image in step five is: Computing a noise-free reflectance map of a color image frame S : Combining the fused image and the noise-free reflectogram to obtain a fused color image :

2. The non-uniform illumination image enhancement method based on event stream and color image fusion according to claim 1, characterized in that, In step two, the convolutional neural network is a three-branch convolutional neural network, and each branch is obtained by decomposing the color image frame S into a reflectance map R, an illumination map I2 and a noise map N, respectively; The three-branch convolutional neural network comprises a CNN1 branch, a CNN2 branch and a CNN3 branch; The CNN1 branch comprises, in sequence, four ReLU+Conv layers, a Conv layer and an activation function Sigmod, and is used to obtain the reflectance map R of the color image frame S; The CNN2 branch comprises, in sequence, four ReLU+Conv layers, a Conv layer and an activation function Sigmod, and is used to obtain the illumination map I2 of the color image frame S; The CNN3 branch comprises, in sequence, four ReLU+Conv layers, a Conv layer and an activation function Tanh, and is used to obtain the noise map N of the color image frame S.

3. The event stream and color image fusion based non-uniform illumination image enhancement method according to claim 2, characterized in that, The loss function L0 of the three-branch convolutional neural network is: L0 = L recon + λ1L ic + λ2L ref + λ3L is + λ4L n , where L recon is a reconstruction loss, L ic is an illumination consistency loss, L ref is an albedo loss, L is is an illumination smoothness loss, L n is a noise estimation loss; λ1 is an illumination consistency loss term weight factor, λ2 is an albedo loss term weight factor, λ3 is an illumination smoothness loss term weight factor, and λ4 is a noise estimation loss term weight factor.

4. The non-uniform illumination image enhancement method based on event stream and color image fusion according to claim 3, characterized in that, Each loss term is defined as: L recon = ||S - (R ⊙ I2 + N) ||1, L ic = ||I2 - I0||1, L n = || I2 0 N || F , where || · ||1denotes the l1norm, I0is the initial estimate of the illumination, ω r is the weight of the first-order difference operator , the first-order difference operator includes a horizontal operator and a vertical operator ω h is the weight of the horizontal operator, ω v is the weight of the vertical operator; || · || F is the Frobenius norm, and is the element-wise multiplication. In the formula, normalize represents a minimum-maximum value normalization, where G is a Gaussian filter, * denotes a convolution operator, S g is a grayscale image of the color image frame.

5. The event stream and color image fusion based non-uniform illumination image enhancement method according to claim 4, characterized in that, The method for generating the two artificial multi-exposure image sequences in step three is: The contrast of the intensity image I1 and the illumination map I2 is adjusted by gamma correction to obtain two artificial multi-exposure image sequences J j (x,y): When j = 1, it corresponds to the intensity image I1, and when j = 2, it corresponds to the illumination map I2; a is a first normal number, and γ is a second normal number.

6. The event stream and color image fusion based non-uniform illumination image enhancement method according to claim 5, characterized in that, λ1, λ2 and λ3 take a value of 1; and λ4 takes a value of 5000.

7. The event stream and color image fusion based non-uniform illumination image enhancement method according to claim 6, characterized in that, The initial estimate I0 of the illumination is the maximum value of the RGB three-channel intensity of the color image frame S.

Images