Low-light image enhancement method based on image-event bidirectional guidance

Abstract

The invention relates to the related technical field of computer vision, in particular to a low-light image enhancement method based on image-event two-way guidance, which comprises the following specific steps: step 1, acquiring a low-light image and a paired event tensor as input, step 2, carrying out feature enhancement processing on the event tensor to suppress a stroboscopic effect and global low-frequency noise, and carrying out image-event two-way guidance on the low-light image and the event tensor. The method comprises the following steps of: 1, acquiring a low-light image, 2, enhancing the event tensor, 3, carrying out bidirectional cross attention fusion on the low-light image and the enhanced event tensor to suppress local noise and structural breakpoints, and 4, realizing image reconstruction through up-sampling and a decoder, and outputting the enhanced low-light image. According to the low-light image enhancement method based on image-event bidirectional guidance, through bidirectional cross attention fusion and frequency domain filtering technologies, the problems of noise and color cast of an existing method under an extremely low light condition are effectively solved, the image quality is remarkably improved, and a high-quality data set is constructed to verify the superior performance of the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a low-light image enhancement method based on image-event bidirectional guidance. BACKGROUND

[0002] In many real-world scenarios such as autonomous driving and unmanned aerial vehicles, low-light phenomena inevitably exist. Due to insufficient light, the image quality is significantly degraded, showing problems such as increased noise, lost details, blurred edges, and reduced contrast, which leads to reduced performance of many computer downstream tasks including object detection and semantic segmentation. Therefore, achieving high-quality low-light image enhancement has always been an important and challenging task in the field of computer vision. In recent years, with the rapid development of deep learning, low-light image enhancement methods based on frames have made significant progress. These methods improve image quality by addressing key issues such as noise, artifacts, and color distortion. However, under extreme low-light conditions, traditional frame-based cameras face problems such as loss of details and image blurring, which severely restrict the performance of existing methods and make it difficult to reconstruct clear natural light images.

[0003] In contrast, event cameras have unique advantages such as high dynamic range and microsecond-level temporal resolution, providing new solutions for low-light image enhancement. By fusing image modalities with event data, these methods have achieved significant performance improvements. However, existing fusion methods mainly use event guidance strategies to compensate for the lack of structural information in images. However, the differential sensitivity of event data to brightness changes makes it susceptible to global light fluctuations, leading to flicker artifacts. Moreover, due to the asynchronous independent imaging principle of event cameras, the generated event data is spatially sparse. This characteristic leads to the incompleteness of event-guided structural information, which is easily disturbed by breakpoints and local noise, resulting in structural breaks in the reconstructed image. In addition, existing low-light image-event datasets are limited by lower resolution and signal-to-noise ratio, and the normal light reference images captured have obvious noise and color bias, which severely affects the performance ceiling of the model. SUMMARY

[0004] The present application aims to provide a low-light image enhancement method based on image-event bidirectional guidance to solve the problem of poor image enhancement effect under extreme low-light conditions as described in the background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solution: a low-light image enhancement method based on image-event bidirectional guidance, the specific steps are as follows:

[0006] Step one, obtain a low-light image and a paired event tensor as input;

[0007] Step two, feature enhancement processing is performed on the event tensor to suppress the stroboscopic effect and global low-frequency noise;

[0008] Step three, bidirectional cross-attention fusion is performed on the low-light image and the enhanced event tensor to suppress local noise and structural breakpoints;

[0009] Step four, image reconstruction is realized through upsampling and decoder to output the enhanced low-light image.

[0010] Preferably, the step of feature enhancement processing on the event tensor comprises:

[0011] S1, two-dimensional discrete Fourier transform is performed on the input event feature to convert it from spatial domain to frequency domain;

[0012] S2, frequency domain translation is performed on the frequency domain feature, and a Gaussian high-pass filter is used for frequency domain filtering;

[0013] S3, two-dimensional discrete inverse Fourier transform is performed on the filtered frequency domain feature to convert it back to the spatial domain to obtain the enhanced event feature.

[0014] Preferably, the step of bidirectional cross-attention fusion comprises:

[0015] First stage: focusing on global structure compensation from events to images, projecting image features into queries through convolution, projecting event features into keys and values, calculating a global context matrix and multiplying it with the query, and injecting event features into image features through a residual connection;

[0016] Second stage: focusing on local noise suppression from images to events, projecting event features into queries, projecting updated image features into keys and values, and performing attention calculation to suppress local noise and compensate for structural breakpoints;

[0017] The outputs of the two stages are spliced and fused to obtain the final fused features.

[0018] Preferably, it further comprises the step of constructing a high-quality data set, which contains strictly registered low-light images, low-light events and normal light images.

[0019] Preferably, the step of constructing a high-quality data set comprises:

[0020] S1, a plurality of unsupervised low-light enhancement methods are used to enhance the reference image to generate a plurality of sets of enhanced images;

[0021] S2, volunteers are invited to compare each set of images and select the image with the best visual effect;

[0022] S3, white balance correction is performed on the candidate true value, and the corrected image is used as the true value of the data set.

[0023] Preferably, the low-light image enhancement device based on image-event bidirectional guidance comprises:

[0024] an input module configured to acquire a low-light image and a paired event tensor;

[0025] an event feature enhancement module configured to perform feature enhancement processing on the event tensor;

[0026] a bidirectional cross-attention fusion module configured to perform bidirectional cross-attention fusion on the low-light image and the enhanced event tensor;

[0027] an image reconstruction module configured to perform image reconstruction through upsampling and a decoder, and output an enhanced low-light image.

[0028] Preferably, the low-light image enhancement device based on image-event bidirectional guidance comprises:

[0029] a Fourier transform unit configured to perform two-dimensional discrete Fourier transform on input event features;

[0030] a frequency domain filtering unit configured to perform frequency domain translation and Gaussian high-pass filtering on the frequency domain features;

[0031] an inverse Fourier transform unit configured to perform two-dimensional discrete inverse Fourier transform on the filtered frequency domain features.

[0032] Preferably, the bidirectional cross-attention fusion module comprises:

[0033] a first stage fusion unit configured to focus on global structure compensation from events to images;

[0034] a second stage fusion unit configured to focus on local noise suppression from images to events;

[0035] a feature splicing unit configured to splice and fuse the outputs of the two stages.

[0036] Preferably, a computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the low-light image enhancement method based on image-event bidirectional guidance according to any one of the above.

[0037] Preferably, a computer device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the low-light image enhancement method based on image-event bidirectional guidance according to any one of the above when executing the computer program.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] 1、Proposed a low-light image enhancement framework based on image-event bidirectional guidance (BiLIE), with frequency loss and color consistency loss to generate output with less noise and higher color fidelity, effectively solving the noise problem caused by event guidance strategy and the color cast problem introduced by camera limitations.

[0040] 2、Proposed a bidirectional cross-attention fusion (BCAF) mechanism, which effectively suppresses local noise and structure breakpoints in sparse events through bidirectional guidance mechanism, with the help of image structure consistency and local smoothness; and introduced an event feature enhancement (EFE) module based on frequency domain filtering to alleviate the global low-frequency noise introduced by dynamic lighting.

[0041] 3、Built a high-quality dataset (RELIE) containing 2217 strictly registered low-light images, low-light events and normal light images.

[0042] 4、Achieved the best performance on the event camera-based low-light enhancement dataset LIE and RELIE, and realized high-quality image reconstruction in extremely low-light environments. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The overall architecture of the low-light image enhancement method based on image-event bidirectional guidance of the present application, including the calculation process diagram of the event feature enhancement (EFE) module based on frequency domain filtering and the bidirectional cross-attention fusion (BCAF) module;

[0044] Figure 2 The construction process diagram of the high-quality dataset (RELIE) of the present application;

[0045] Figure 3 The visualization diagram of the low-light enhancement result of the present application;

[0046] Figure 4 The comparison diagram before and after each module of the event feature enhancement (EFE) module and the bidirectional cross-attention fusion (BCAF) mechanism of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] Please refer to Figures 1-4 The present application provides a technical solution: a low-light image enhancement method based on image-event bidirectional guidance, the specific steps are as follows:

[0049] Step one, obtain low-light images and paired event tensors as input;

[0050] Step two, feature enhancement processing is performed on the event tensor to suppress stroboscopic effect and global low-frequency noise;

[0051] Step three, bidirectional cross-attention fusion is performed on the low-light image and the enhanced event tensor to suppress local noise and structure breakpoints;

[0052] Step four, image reconstruction is realized by upsampling and decoder, and the enhanced low-light image is output.

[0053] Further, the step of feature enhancement processing of the event tensor includes:

[0054] S1, two-dimensional discrete Fourier transform is performed on the input event feature to convert it from spatial domain to frequency domain, and the formula is as follows:

[0055]

[0056] Where M and N are the number of rows and columns of the image respectively, (x, y) is the spatial domain coordinate, (u, v) is the frequency domain coordinate,

[0057] S2, frequency domain translation is performed by fftshift function to obtain F' shift (u,v), Figure 1 (b) The more obvious center of the highlighted bright spot represents the low-frequency component, and the gradually weakened wavy pattern in the periphery reflects the edge and high-frequency detail component. Next, a Gaussian high-pass filter is used to multiply F' shift (u,v) element by element to realize frequency domain filtering, which has smooth cutoff characteristics and can effectively avoid the ringing effect caused by ideal high-pass filter, and the formula is as follows:

[0058]

[0059] F” event (u,v)=High(u,v)⊙F' shift (u,v),

[0060] Where u c , v c is the frequency domain center, (u,v) to the frequency domain center (u c ,v c ), and σ is a parameter to control the bandwidth of the filter, which is set to 12, and the filter performance is optimal at this time;

[0061] S3, filtered feature map F”event The ripple pattern of the peripheral region (u, v) becomes more apparent and concentrated, indicating that the low-frequency component is effectively weakened, and finally, the frequency domain image F shift (u, v) is subjected to two-dimensional discrete inverse Fourier transform, converting it back to the spatial domain to obtain the output F E (x, y), which is the original event representation F event In contrast, F E The overall brightness is reduced, the visual effect is more coordinated and natural, and the edge and high-frequency details are significantly enhanced, and the formula is as follows:

[0062]

[0063] Further, the steps of bidirectional cross-attention fusion include:

[0064] The first stage: Figure 1 (c) is the structure of BCAF, which adopts a two-stage cross-attention mechanism, the first stage focuses on global structure compensation from events to images, and the image vector F I ∈R H×W×C is projected into the query (Q), and the event vector F E ∈R H×W×C is projected into the key (K) and the value (V), and an efficient attention with linear complexity is adopted, first calculating K^T·V to generate a global context matrix to extract global structure information in event features, then performing matrix multiplication with the query (Q), and injecting these information into image features through residual connection, in this way, the image can obtain clearer structure clues and dynamic details from the event, making up for the lack of details in low-light scenes, the whole process can be described as:

[0065] CA1(F I ,F E )=ρ q (Q)(ρ k (K) T V),

[0066] Q=F I W q , K=F E W k , V=F E W v ,

[0067] F' I =F I +CA1(F I ,F E ),

[0068] where F 'I is the updated image feature, ρ q , ρ k is the normalization function of query and key features, Wq, W k , W v ∈C×(C / h) are three learnable parameter matrices, h is the number of heads of multi-head cross attention, h is set to [2, 4, 4, 4, 6] from low to high. F' I The original image information is preserved through the residual connection, and is supplemented and enhanced by the event feature;

[0069] The second stage: the second stage focuses on local noise suppression of image to event, and the fused representation of the first stage is modified again by means of the structural consistency and local smoothness of the image, to suppress local noise in the event and make up for the structure breakpoint, so as to ensure that the fused representation is smoother. Specifically, the original event vector F E ∈R H×W×C is projected into the query (Q), and the updated image vector F ' I ∈R H×W×C is projected into the key (K) and the value (V), and attention calculation is performed in the same way:

[0070] CA2(F E ,F' I )=ρ q (Q)(ρ k (K) T V),

[0071] Q=F E W q , K=F' I W k , V=F' I W v ,

[0072] F' E =F E +CA2(F E ,F' I )。

[0073] The outputs F' I and F' E of the two stages are spliced and fused to obtain the final fused feature:

[0074]

[0075] Where i = 1, 2, …, 5 represents five scales, through the bidirectional guidance mechanism, BCAF effectively suppresses the noise and structure breakpoint introduced by the event while preserving the high-frequency structure, ensuring that the reconstructed image has less noise and the overall visual effect is smoother and more natural.

[0076] Further, the step of constructing a high-quality dataset includes strictly registered low-light images, low-light events, and normal-light images.

[0077] Further, the step of constructing a high-quality dataset includes:

[0078] S1, 2217 reference images of the LIE dataset are enhanced by five advanced unsupervised low-light enhancement methods, including zero-reference depth curve estimation Zero-DCE, self-calibration illumination learning framework SCI, NeRCo based on implicit neural representation, and PairLIE and Zero-IG based on Retinex theory, finally, six groups of enhanced results are generated for each reference image, Zero-IG provides two groups of results before and after denoising, a total of 6x2217 enhanced images are generated;

[0079] S2, invite 11 volunteers with preliminary image processing experience to compare seven images including the original true value in each group, the volunteers need to consider noise, contrast, color cast and artifacts, and select the best visual effect from them, the image closest to the true situation, Figure 2 The generation process of high-quality true value and the percentage of images generated by each method selected as the best image are given;

[0080] S3, the gray world algorithm is used to correct the white balance of the candidate true value, and the processing result is as Figure 2 As shown in the last column, the color of the corrected image is more natural and true, and this image as the true value of the constructed new dataset is more in line with the visual effect of the human eye.

[0081] The low-light image enhancement device based on image-event bidirectional guidance comprises:

[0082] The input module is used to acquire low-light images and paired event tensors;

[0083] The event feature enhancement module is used to perform feature enhancement processing on the event tensor;

[0084] The bidirectional cross-attention fusion module is used to perform bidirectional cross-attention fusion on the low-light image and the enhanced event tensor;

[0085] The image reconstruction module is used to realize image reconstruction through upsampling and decoder, and output the enhanced low-light image.

[0086] Further, the event feature enhancement module comprises:

[0087] The Fourier transform unit is used to perform two-dimensional discrete Fourier transform on the input event feature;

[0088] a frequency domain filtering unit configured to perform frequency domain translation and Gaussian high-pass filtering on the frequency domain feature;

[0089] an inverse Fourier transform unit configured to perform two-dimensional discrete inverse Fourier transform on the filtered frequency domain feature.

[0090] Further, the bidirectional cross-attention fusion module comprises:

[0091] a first stage fusion unit configured to focus on global structure compensation from events to images;

[0092] a second stage fusion unit configured to focus on local noise suppression from images to events;

[0093] a feature concatenation unit configured to concatenate and fuse the outputs of the two stages.

[0094] A computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the low-light image enhancement method based on image-event bidirectional guidance according to any one of the above.

[0095] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the low-light image enhancement method based on image-event bidirectional guidance according to any one of the above.

[0096] Embodiment one, comparative experiments are performed on LIE and constructed RELIE datasets, using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and learned perceptual image patch similarity (LPIPS) as evaluation indexes, on the two datasets, the method proposed in the application is compared with an event-based method E2VID, three image-based methods SNR-Net, Retinexformer and Zero-IG, and two image and event fusion-based methods ELIE and EvLight for comparative analysis.

[0097] Table 1 performance comparison results on RELIE and LIE datasets

[0098]

[0099] The results are shown in Table 1. The application achieves the most advanced performance on both data sets. On the RELIE data set, the PSNR and LPIPS are improved by 0.96 dB and 0.03, respectively. On the LIE data set, the three indicators are improved by 0.33 dB, 0.121 and 0.005, respectively. The event-based method (E2VID) performs better on RELIE than LIE, but still lags far behind the frame-based supervised method and the fusion method, because it lacks important color information. The performance of the unsupervised Zero-IG is much lower than that of the supervised method.

[0100] Figure 3 The visualization results of the indoor scene of RELIE are shown. The event-based method (E2VID) severely lacks color information and has poor visual effects. The frame-based method (SNR-Net, Retinexformer, Zero-IG) has obvious artifacts and insufficient detail recovery. In contrast, the image and event fusion-based method (ELIE, EvLight and Ours) reconstructs a relatively clear and complete scene structure in the dark area. The application accurately restores the clear edges and scene colors in the dark area, effectively suppresses the noise and jagged spots introduced during the true value acquisition process due to unsupervised enhancement, and has smooth and natural overall visual effects without obvious artifacts and color deviation.

[0101] In addition, Figure 4 The feature maps before and after EFE and BCAF are qualitatively shown. Compared with F event The overall brightness is reduced, the noise is reduced, and the edges are clearer. Compared with the feature F E ' I Compared with F effectively compensates for the structure breakpoint at the edge, further verifying the effectiveness of the application.

[0102] Although embodiments of the application have been shown and described, it will be understood by those having ordinary skill in the art that various changes, modifications, alternatives and variations can be made thereto without departing from the principles and spirit of the application, the scope of which is defined by the appended claims and their equivalents.​

Claims

1. A low-light image enhancement method based on image-event bidirectional guidance, characterized in that: The specific steps are as follows: Step 1: Get the low-light image and paired event tensor as input; Step 2: performing feature enhancement processing on the event tensor to suppress stroboscopic effects and global low-frequency noise; Step 3: performing bidirectional cross-attention fusion on the low-light image and the enhanced event tensor to suppress local noise and structural discontinuities; Step 4: Image reconstruction is achieved through upsampling and decoder, and the enhanced low-light image is output.

2. The low-light image enhancement method based on image-event bidirectional guidance according to claim 1, characterized in that: The step of performing feature enhancement processing on the event tensor includes: S1. Perform a two-dimensional discrete Fourier transform on the input event features to convert them from the spatial domain to the frequency domain; S2, frequency domain shift the frequency domain features and use Gaussian high-pass filter to perform frequency domain filtering; S3. Perform a two-dimensional discrete inverse Fourier transform on the filtered frequency domain features, convert them back to the spatial domain, and obtain the enhanced event features.

3. The low-light image enhancement method based on image-event bidirectional guidance according to claim 1, characterized in that: The steps of bidirectional cross attention fusion include: The first stage focuses on global structure compensation from events to images. It projects image features into queries through convolution, projects event features into keys and values, calculates the global context matrix and multiplies it with the query, and injects event features into image features through residual connections. The second stage focuses on local noise suppression from image to event, projects event features into the query, projects updated image features into keys and values, and performs attention calculations to suppress local noise and fill structural breakpoints. The outputs of the two stages are concatenated and fused to obtain the final fusion features.

4. The low-light image enhancement method based on image-event bidirectional guidance according to claim 1, characterized in that: The method also includes the step of constructing a high-quality dataset comprising rigorously registered low-light images, low-light events, and normal-light images.

5. The low-light image enhancement method based on image-event bidirectional guidance according to claim 4, characterized in that: The steps of constructing a high-quality dataset include: S1. Use multiple unsupervised low-light enhancement methods to enhance the reference image and generate multiple sets of enhanced images; S2. Invite volunteers to compare each set of images and select the image with the best visual effect; S3. Perform white balance correction on the candidate true value and use the corrected image as the true value of the dataset.

6. A low-light image enhancement device based on image-event bidirectional guidance, characterized in that: include: Input module, which is used to obtain low-light images and paired event tensors; Event feature enhancement module, used to perform feature enhancement processing on event tensors; Bidirectional cross-attention fusion module, which is used to perform bidirectional cross-attention fusion on low-light images and enhanced event tensors; The image reconstruction module is used to reconstruct the image through upsampling and decoder, and output the enhanced low-light image.

7. The low-light image enhancement device based on image-event bidirectional guidance according to claim 6, characterized in that: The event feature enhancement module includes: A Fourier transform unit, used for performing a two-dimensional discrete Fourier transform on the input event features; Frequency domain filtering unit, used to perform frequency domain shift and Gaussian high-pass filtering on frequency domain features; The inverse Fourier transform unit is used to perform a two-dimensional discrete inverse Fourier transform on the filtered frequency domain features.

8. The low-light image enhancement device based on image-event bidirectional guidance according to claim 6, characterized in that: The bidirectional cross attention fusion module includes: The first stage fusion unit is used to focus on global structure compensation from events to images; The second stage fusion unit is used to focus on local noise suppression from image to event; The feature splicing unit is used to splice and fuse the outputs of the two stages.

9. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the low-light image enhancement method based on image-event bidirectional guidance according to any one of claims 1 to 5 is implemented.

10. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the low-light image enhancement method based on image-event bidirectional guidance as claimed in any one of claims 1 to 5 is implemented.

Images