Low-light enhancement method based on event and image bidirectional collaborative guidance

Abstract

The application discloses a low-light enhancement method based on event and image bidirectional collaborative guidance, and belongs to the technical field of computer vision.The application contains an event feature extraction branch and an image-event fusion branch, and an image-guided event enhancement module and an image-event fusion module based on Mamba are respectively arranged; the two modules realize cross-modal interaction through bidirectional collaborative guidance; the image-guided event enhancement module suppresses event noise and compensates for context loss by using semantic context information of the image; the image-event fusion module based on Mamba adjusts global illumination and contrast of the image by using the global representation ability and linear space-time complexity characteristics of Mamba, and combines the signal-to-noise ratio map to fuse event data to the image to supplement structural information.The application realizes enhancement and fusion of image and event information through a bidirectional collaborative guidance mechanism, and solves the enhancement problem of image structure loss and illumination contrast degradation in a low-light scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a low-light enhancement method based on bidirectional collaborative guidance of events and images. Background Technology

[0002] Images, as a crucial carrier of visual information, are ubiquitous in human production and daily life. On one hand, people can record and share their daily lives through images. On the other hand, in special working conditions such as industrial production, healthcare, autonomous driving, and military reconnaissance, images can accurately measure and depict detailed information about various scenarios, thus providing strong support for various technological applications in related fields. However, in low-light scenarios, due to the limitations of the photosensitive elements and dynamic range of image signal processors in traditional RGB cameras, captured images often suffer from low illumination, low contrast, sensor noise disturbances, and insignificant or even missing structural information. This not only affects visual perception but also has adverse effects on the aforementioned related fields. Therefore, developing a low-light image enhancement method that accurately recovers illumination, color, and structural information has become an urgent need in the field of computer vision.

[0003] Currently, significant progress has been made in low-light image enhancement methods for single images. Traditional methods mainly fall into three categories: histogram equalization, gamma correction, and methods based on human retinal imaging theory. However, these methods still have limitations, specifically: they require carefully designed manual priors for specific scenes, which significantly affects cross-scene generalization; they neglect sensor noise processing, resulting in noise being retained or amplified during enhancement, affecting the overall quality of the enhanced image; and they require complex optimization and hyperparameter tuning, leading to long runtimes. In recent years, with the development of deep learning, a large number of deep learning-based RGB low-light image enhancement methods have emerged. These methods learn the latent mapping between low-light and clear images through a data-driven approach, offering faster speeds and better performance compared to traditional methods. For example, Xu Ke et al. designed a frequency-domain decoupling prior, constructing a two-stage frequency decoupling network to achieve noise suppression and brightness restoration in the low-frequency domain and detail restoration in the high-frequency domain. However, due to the severe degradation of structural information in low-light images, it is still difficult to accurately recover structural information using only low-light images.

[0004] Event cameras are a novel type of bio-inspired visual sensor that records intensity changes for each pixel. Compared to traditional RGB cameras, event cameras offer advantages such as high temporal resolution (HTR) and high dynamic range (HDR), enabling them to capture structural information that is difficult for RGB cameras to perceive in low-light scenes. Therefore, recent research has attempted to fuse event information into images to restore their structural information. For example, Jiang Yu et al. constructed a residual fusion module to directly fuse event features into RGB cues element-wise. However, event information is noisy in low-light scenes, which can be disruptive; secondly, event information is sparse and lacks semantic context. Image-event fusion-based enhancement methods do not address the inherent noise degradation of events or the lack of semantic context due to sparsity, resulting in artifacts, unrealistic detail restoration, and color casts in the enhanced images. In contrast, RGB images contain rich semantic context information, effectively supplementing the semantic context missing in event data. Therefore, it is crucial to study effective noise reduction of event information and how to supplement the missing semantic context information of images. This is of great importance for researching low-light image enhancement methods for event- and image fusion.

[0005] While event information can provide valuable structural clues, low-light images are often accompanied by global degradation in both illumination and contrast. Existing methods struggle to simultaneously balance the structural representation of event information with the restoration of overall image brightness and contrast. Specifically, illumination and contrast degradation typically manifests as a weakening of low-frequency information, while event information primarily reflects high-frequency structural changes; the two are naturally complementary in the frequency domain. Therefore, designing an effective image-event fusion mechanism to collaboratively restore information from different frequency domains while introducing long-range dependencies to model global brightness degradation is crucial for researching low-light image enhancement methods oriented towards event-image fusion. Summary of the Invention

[0006] To overcome the problems of poor structural information recovery and inaccurate color recovery in existing methods, this invention proposes a low-light enhancement method based on bidirectional collaborative guidance of events and images. By setting up an image-guided event enhancement module and an image-event fusion module based on Mamba, high-quality restoration of low-light images can be achieved.

[0007] The technical solution of this invention is as follows:

[0008] A low-light enhancement method based on bidirectional collaborative guidance of events and images includes the following steps:

[0009] Step 1: Acquire low-light image-event data and its paired normal-light image-event data, and preprocess the event data to obtain the image-event dataset;

[0010] Step 2: Construct BGLIENet, a low-light enhancement network guided by bidirectional collaboration between events and images. This low-light enhancement network includes a preprocessing network, an image-guided event enhancement module, a feature pyramid encoder, and an image-event fusion branch. The preprocessing network converts the low-light image into a coarse-enhanced image and extracts its shallow feature information and obtains a signal-to-noise ratio (SNR) map based on the coarse-enhanced image. The image-guided event enhancement module removes spatiotemporal noise from the event data and uses the shallow feature information of the coarse-enhanced image to guide the supplementation of event data to obtain event features containing fine-grained semantic context. The feature pyramid encoder extracts the multi-scale features of the event features containing fine-grained semantic context supplemented by the event data. The image-event fusion branch, based on the Mamba model and using the SNR map as guidance, fuses the event features containing fine-grained semantic context and their multi-scale features with the shallow feature information of the coarse-enhanced image to obtain the enhanced image.

[0011] Step 3: Use the image-event dataset obtained in Step 1 to train the low-light enhancement network guided by the bidirectional collaboration of events and images in Step 2;

[0012] Step 4: Deploy the BGLIENet network to edge devices in low-light scenes, and use the trained BGLIENet network to enhance images captured in low-light scenes in real time.

[0013] Furthermore, in step 1, the preprocessing includes:

[0014] The low-light image-event data and the normal-light image-event data are sorted and paired according to timestamps, and the training set and test set are divided according to different shooting scenarios;

[0015] Given N discrete input event data ,in This represents the pixel position corresponding to the i-th event. It is polar. For timestamps; input the event. Convert to event voxel grid:

[0016] (1)

[0017] Where B represents the number of event slices in the event voxel grid, which is set to 32; Indicates the timestamp of the starting event; Indicates the timestamp of the termination event; Represents the maximum value operator; Represents the discrete index of the voxel grid in the temporal dimension; Represents the spatial pixel coordinates of the event voxel grid.

[0018] Furthermore, the preprocessing network converts the low-light image into a coarse-enhanced image and extracts its shallow feature information based on the coarse-enhanced image, as follows:

[0019] Using a filter of size The convolutional layer, LeaklyReLU activation function, and one filter of size 1 The initial illuminance distribution is estimated by a cascaded block composed of convolutional layers. :

[0020] (2)

[0021] in, For low-light images, For the LeaklyReLU operator with a leakage coefficient of 0.02, The filter size is Convolutional layers;

[0022] The initial illuminance distribution map is multiplied element-wise by the low-light image, and expressed as:

[0023] (3)

[0024] in, For coarse enhancement of the image, It is an element-wise multiplication operator;

[0025] Using filter size Convolutional layers extract coarse-enhanced images Shallow feature information:

[0026] (4)

[0027] in, This provides shallow feature information for coarse enhancement of the image.

[0028] Furthermore, the preprocessing network obtains a signal-to-noise ratio map based on the coarsely enhanced image, specifically including:

[0029] Obtaining coarse-enhanced images using Gaussian filtering Low-frequency components:

[0030] (5)

[0031] in, To coarsely enhance the low-frequency components of the image, The filter size is Gaussian filter, It is a convolution operator;

[0032] calculate and The residuals are used to estimate noise information:

[0033] (6)

[0034] in, Noise information;

[0035] calculate and The signal-to-noise ratio (SNR) map is obtained by comparing the absolute values ​​of the ...

[0036] (7)

[0037] in, For absolute value operators, This is a signal-to-noise ratio diagram.

[0038] Furthermore, the image-guided event enhancement module uses the shallow feature information of the coarsely enhanced image obtained from the preprocessing network. With the event voxel grid As input, the aim is to reduce noise in event data caused by low-light conditions, while utilizing the rich semantic context information contained in the image to supplement the missing semantic context information in the event information. This effectively alleviates problems such as artifacts in enhanced images, unrealistic detail restoration, and color cast caused by noise pollution and missing semantic context in event data. The specific process is as follows:

[0039] Using a filter size of The convolutional layer, NAFBlock, layer normalization operator, and filter size are... The size of the convolutional layer and filter is The depth can be used to separate the cascaded blocks of convolutional layers to obtain denoised event features. :

[0040] (8)

[0041] (9)

[0042] in, event voxel grid The corresponding shallow features, The filter size is Convolutional layers, For NAFBlock, For layer normalization operators, The filter size is Convolutional layers, The filter size is The depth of the separable convolutional layer;

[0043] Denoising event features Shallow feature information of coarsely enhanced images and event voxel grid Corresponding shallow features The input consists of a cross-space modulation module, a cross-channel modulation module, and a filter of size [missing information]. The concatenated blocks of convolutional layers yield event features containing coarse-grained semantic context. :

[0044] (10)

[0045] in, For cross-space modulation modules, It is a cross-channel modulation module;

[0046] Event features that include coarse-grained semantic context and shallow feature information of coarsely enhanced images Input to a normalization layer with a filter size of The size of the convolutional layer, cross-space modulation module, and filter is The concatenated blocks of convolutional layers yield event features containing fine-grained semantic context. :

[0047] (11).

[0048] Furthermore, the cross-spatial modulation module aims to leverage image features to enhance the spatial contextual semantic information of event features in the spatial dimension, given image features. Event characteristics Then the event features after spatial context semantic enhancement It can be represented as:

[0049] (12)

[0050] in, This refers to the event features enhanced with spatial semantic context, i.e., the output of the cross-spatial modulation module.

[0051] Furthermore, the cross-channel modulation module aims to utilize image features to enhance the temporal contextual semantic information of event features in the channel dimension, given image features. Event characteristics Then the event features after temporal context semantic enhancement It can be represented as:

[0052] (13)

[0053] in, This refers to the event features enhanced with temporal semantic context, i.e., the output of the cross-channel modulation module; This is the global average pooling operator.

[0054] Furthermore, the feature pyramid encoder consists of two groups, each group comprising a pixel reconstruction layer (PixelUnShuffle) and a filter size of [missing information]. The convolutional layers and NAFBlock are used to extract event features containing fine-grained semantic context. Multiscale features :

[0055] (14)

[0056] in, For scale numbering, For pixel recombination layer operators, For the first The filter size at each scale is Convolutional layers, For the first NAFBlock at various scales.

[0057] Furthermore, the image-event fusion branch is based on the UNet architecture, including an encoder and a decoder;

[0058] The encoder consists of three Mamba-based image-event fusion modules and two bilinear downsampling operators, and the Mamba-based image-event fusion modules and the bilinear downsampling operators are alternately arranged in sequence;

[0059] The shallow feature information of the coarsely enhanced image The event features containing fine-grained semantic context and its multi-scale features The input is fed into the encoder to extract the fused multi-scale features. :

[0060] (15)

[0061] in, For bilinear interpolation operators with a downsampling factor of 2, This is a Mamba-based image-event fusion module.

[0062] The decoder consists of three Mamba-based image-event fusion modules, two bilinear upsampling operators, and a filter size of [missing information]. The convolutional layers constitute the image-event fusion module based on Mamba, and the bilinear upsampling operator is alternately set in sequence; the bilinear upsampling operator consists of a concat operator and a... It consists of a convolutional layer and a bilinear interpolation operator with an upsampling factor of 2; where, scale 2 ( ) and scale 1 ( The features after fusion are as follows:

[0063] (16)

[0064] in, This is a bilinear interpolation operator with an upsampling factor of 2;

[0065] Scale 1 fusion features Event features containing fine-grained semantic context The input is fed into a Mamba-based image-event fusion module and then filtered by a filter of size [size missing]. The convolutional layer yields residual components, which are then combined with the coarsely enhanced image. The images are added together to obtain the final enhanced image. :

[0066] (17)

[0067] Furthermore, the overall architecture of the Mamba-based image-event fusion module follows the design approach of the VisionMamba module, with its main contribution being the state scan space model guided by the signal-to-noise ratio map, as detailed below:

[0068] Given image features Event characteristics Signal-to-noise ratio diagram The signal-to-noise ratio (SNR) map-guided state scan space model aims to model... The long-range dependency relationship is used to repair global illumination degradation and utilize the signal-to-noise ratio map. Guiding fusion event characteristics Repair structural information:

[0069] (18)

[0070] (19)

[0071] (20)

[0072] (twenty one)

[0073] in, An operator for splitting features along the channel dimension; is a fully connected layer; SiLU(.) is the SiLU activation function; VSSM(.) is the visual state space model of VMamba; and for Two features split along the channel dimension; This refers to global illumination restoration features for images after long-distance dependency modeling. Image and event fusion features guided by signal-to-noise ratio maps; For the The modulated image and event fusion features are the output of the Mamba-based image-event fusion module.

[0074] Furthermore, in step 3 of the training process, the optimizer used is the Adam optimizer with a learning rate of 0.00015, and the acquired low-light images and event voxel grids are... The resolution was randomly cropped, and a total of 80 batches were trained. The loss functions used in the training process included smoothing L1 loss, peak signal-to-noise ratio loss, multi-scale structural similarity loss, and peak signal-to-noise ratio star loss. The overall description of the loss functions is as follows:

[0075] (twenty two)

[0076] in, For the overall loss function, This represents the true value, i.e., the normal lighting image corresponding to the low-light image; To smooth out the L1 loss; This represents the peak signal-to-noise ratio loss. For peak signal-to-noise ratio star loss; This is a multi-scale structural similarity loss.

[0077] The beneficial effects of this invention are as follows: This invention fuses image and event information through a bidirectional interactive guidance strategy. Specifically, it uses image information to enhance event information that suffers from noise degradation and missing contextual semantics. Simultaneously, the enhanced event information can more effectively guide the restoration of structural information in low-light images, thereby improving the quality of the enhanced image. This invention suppresses event noise and compensates for missing context through an image-guided event enhancement module; it restores structural information by fusing the enhanced event information through a Mamba-based image-event fusion module; and it utilizes the global context modeling capabilities of the Mamba model to repair the image's illumination and contrast information, effectively improving the quality of the restored image. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0079] Figure 2 The overall framework diagram of BGLIENet, a low-light enhancement network guided by bidirectional collaboration between events and images.

[0080] Figure 3 Network architecture for the REE event enhancement module, which is guided by images.

[0081] Figure 4 The network architecture for MREA, a Mamba-based image-event fusion module. Detailed Implementation

[0082] The embodiments of the present invention are implemented under the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0083] This embodiment uses Windows 11 as the development environment, Ubuntu 22.04 as the deployment environment, C++ and Python as the development languages, and PyTorch as the deep learning framework. It employs the low-light enhancement method based on event and image bidirectional collaborative guidance described in this invention to deploy the low-light enhancement program. The overall process is as follows: Figure 1 As shown; the specific steps are as follows:

[0084] Step 1: Synchronously acquire low-light image-event data and its paired normal-light image-event data, and preprocess the event data.

[0085] Step 1.1: Turn on the DAVIS346 Color camera, a color image-event synchronization acquisition device, and acquire low-light image-event data and its paired normal-light image-event data. The specific process is as follows:

[0086] Step 1.1.1: Install the DAVIS346 Color camera onto the robotic arm, predefine the robotic arm's motion trajectory, and use a hardware triggering strategy to ensure that the robotic arm and the DAVIS346 Color camera start synchronously.

[0087] Step 1.1.2: Install the ND8 filter onto the lens of the DAVIS346 Color camera. Based on the predefined robotic arm motion trajectory and hardware triggering strategy in Step 1.1.1, simultaneously turn on the robotic arm and the event camera to collect low-light images and event data.

[0088] Step 1.1.3: Remove the ND8 filter from the lens of the DAVIS346 Color camera. Based on the predefined robotic arm motion trajectory and hardware triggering strategy in Step 1.1.1, simultaneously turn on the robotic arm and the event camera to acquire normal lighting image-event data that is paired with the low-light image-event data acquired in Step 1.1.2.

[0089] Step 1.1.4: Sort and pair the low-light image-event data collected in Step 1.1.2 with the normal-light image-event data collected in Step 1.1.3 according to timestamps, and divide the training set and test set according to different shooting scenarios.

[0090] Step 1.2: Given N discrete input event data ,in This represents the pixel position corresponding to the i-th event. It is polar. For timestamps; input the event. Convert to event voxel grid:

[0091] (1)

[0092] Where B represents the number of event slices in the voxel mesh, which is set to 32; Indicates the timestamp of the starting event; Indicates the timestamp of the termination event; Represents the maximum value operator; Represents the discrete index of the voxel grid in the temporal dimension; Represents the spatial pixel coordinates of the voxel grid.

[0093] Step 2: Construct BGLIENet, a low-light enhancement network guided by bidirectional collaboration between events and images. The BGLIENet network includes a preprocessing network, an image-guided event enhancement module, a feature pyramid encoder, and an image-event fusion branch.

[0094] Step 2.1: The preprocessing network converts the low-light image into a coarse-enhanced image and extracts its shallow feature information based on the coarse-enhanced image, as follows:

[0095] Use a filter with a size of The convolutional layer, LeaklyReLU activation function, and one filter size are A cascaded block of convolutional layers is used to estimate the initial illuminance distribution map. :

[0096] (2)

[0097] in, The low-light image obtained in step 1.1 For the LeaklyReLU operator with a leakage coefficient of 0.02, The filter size is Convolutional layers;

[0098] Initial illuminance distribution map Element-wise multiplication with the low-light image is expressed as:

[0099] (3)

[0100] in, For coarse enhancement of the image, It is an element-wise multiplication operator;

[0101] Using filter size The convolutional layer extracts shallow feature information from the coarsely enhanced image:

[0102] (4)

[0103] in, This provides shallow feature information for coarse enhancement of the image.

[0104] Step 2.2: Obtain the low-frequency components of the coarsely enhanced image using Gaussian filtering, which can be expressed as:

[0105] (5)

[0106] in, To coarsely enhance the low-frequency components of the image, The filter size is Gaussian filter, It is a convolution operator;

[0107] Through calculation and The residuals are used to estimate noise information:

[0108] (6)

[0109] in, Noise information;

[0110] calculate and The signal-to-noise ratio (SNR) map is obtained by comparing the absolute values ​​of the ...

[0111] (7)

[0112] in, For absolute value operators, This is a signal-to-noise ratio diagram.

[0113] Step 2.3: Extract the shallow feature information from the coarsely enhanced image obtained in Step 2.1. The event voxel grid obtained in step 1.2 The data is input to an image-guided event enhancement module to suppress spatiotemporal noise in the event data and enhance the spatial context of the effective signal; the specific steps are as follows:

[0114] Step 2.3.1: Use a filter with a size of The convolutional layer, NAFBlock, layer normalization operator, and filter size are... The convolutional layer and filter size are The depth can be used to separate the cascaded blocks of convolutional layers, thus obtaining the denoised event features. :

[0115] (8)

[0116] (9)

[0117] in, event voxel grid The corresponding shallow features, The filter size is Convolutional layers, For NAFBlock, For layer normalization operators, The filter size is Convolutional layers, The filter size is The depth of the convolutional layer can be separated.

[0118] Step 2.3.2: Extract the shallow feature information from the coarsely enhanced image obtained in Step 2.1. and the event voxel grid obtained in step 2.3.1 shallow features Denoising event features The input consists of a cross-space modulation module, a cross-channel modulation module, and a filter size of [missing information]. The concatenated blocks of convolutional layers yield event features containing coarse-grained semantic context. :

[0119] (10)

[0120] in, For cross-space modulation modules, It is a cross-channel modulation module.

[0121] Step 2.3.3: Analyze the event features obtained in Step 2.3.2 that contain coarse-grained semantic context. and the shallow feature information of the coarsely enhanced image obtained in step 2.1 Input to a normalization layer with a filter size of The convolutional layer, cross-space modulation module, and filter size are The concatenated blocks of convolutional layers yield event features containing fine-grained semantic context. :

[0122] (11).

[0123] Step 2.4: Use two sets of pixel reassembly layers (PixelUnShuffle) with a filter size of [missing value]. The feature pyramid encoder, composed of convolutional layers and NAFBlock, extracts the event features containing fine-grained semantic context obtained in step 2.3. Multiscale features :

[0124] (12)

[0125] in, For scale numbering, For pixel recombination layer operators, For the first The filter size at each scale is Convolutional layers, For the first NAFBlock at various scales.

[0126] Step 2.5: Extract the shallow feature information from the coarsely enhanced image obtained in Step 2.1. Step 2.3 yields event features containing fine-grained semantic context. The multi-scale features obtained in step 2.4 The input is fed into the UNet-based image-event fusion branch, as follows:

[0127] Step 2.5.1: The encoder consists of three Mamba-based image-event fusion modules and two bilinear downsampling operators, and the Mamba-based image-event fusion modules and the bilinear downsampling operators are alternately set in sequence;

[0128] Will , and The input is fed into the encoder to extract the fused multi-scale features. :

[0129] (13)

[0130] in, This is a bilinear interpolation operator with a downsampling factor of 2; This is a Mamba-based image-event fusion module.

[0131] Step 2.5.2: The decoder consists of three Mamba-based image-event fusion modules, two bilinear upsampling operators, and a filter with a size of [missing information]. The convolutional layers constitute the image-event fusion module based on Mamba, and the bilinear upsampling operator is alternately set in sequence; the bilinear upsampling operator consists of a concat operator and a... It consists of a convolutional layer and a bilinear interpolation operator with an upsampling factor of 2; the features after fusing scale 2 and scale 1 are as follows:

[0132] (14)

[0133] in, This is a bilinear interpolation operator with an upsampling factor of 2;

[0134] Scale 1 fusion features Step 2.3 yields event features containing fine-grained semantic context. The input is fed into a Mamba-based image-event fusion module and then filtered by a filter of size [size missing]. The convolutional layer yields residual components, which are then combined with coarse-grained image enhancement. The images are added together to obtain the final enhanced image. :

[0135] (15)

[0136] The overall architecture of the Mamba-based image-event fusion module follows the design approach of the VisionMamba module. Its main contribution is the state scan space model guided by the signal-to-noise ratio map, as detailed below:

[0137] Given image features Event characteristics Signal-to-noise ratio diagram The signal-to-noise ratio (SNR) map-guided state scan space model aims to model... The long-range dependency relationship is used to repair global illumination degradation and utilize the signal-to-noise ratio map. Guiding fusion event characteristics Repair structural information:

[0138] (16)

[0139] (17)

[0140] (18)

[0141] (19)

[0142] in, An operator for splitting features along the channel dimension; is a fully connected layer; SiLU(.) is the SiLU activation function; VSSM(.) is the visual state space model of VMamba; and for Two features split along the channel dimension; This refers to global illumination restoration features for images after long-distance dependency modeling. Image and event fusion features guided by signal-to-noise ratio map; For the The modulated image and event fusion features are the output of the Mamba-based image-event fusion module.

[0143] Step 3: Use the image-event dataset collected in Step 1 to train the BGLIENet network constructed in Step 2.

[0144] Step 3.1: Train the event and image bidirectional collaborative low-light enhancement network BGLIENet built in Step 2 using the training set partitioned in Step 1.1.4. The optimizer used is the Adam optimizer with a learning rate of 0.00015. The acquired low-light images and event voxel grids are... The resolution was randomly cropped, and a total of 80 batches were trained. The loss functions used in the training process included smoothing L1 loss, peak signal-to-noise ratio loss, multi-scale structural similarity loss, and peak signal-to-noise ratio star loss. The overall description of the loss functions is as follows:

[0145] (20)

[0146] in, For the overall loss function, This represents the true value, i.e., the normal lighting image corresponding to the low-light image; To smooth out the L1 loss; This represents the peak signal-to-noise ratio loss. For peak signal-to-noise ratio star loss; This is a multi-scale structural similarity loss.

[0147] Step 3.2: Save the weight file trained according to the training strategy in Step 3.1.

[0148] Step 3.3: Read the weight file from Step 3.2 into BGLIENet and test it using the test set divided in Step 1.1.4; the test results are as follows: peak signal-to-noise ratio: 23.09 dB, peak signal-to-noise ratio score: 25.56 dB, structural similarity: 0.7769.

[0149] Step 4: Deploy the BGLIENet network to edge devices in low-light scenes, and use the trained BGLIENet network to enhance images captured in low-light scenes in real time.

[0150] Step 4.1: The hardware uses an integrated edge computing platform based on ARM or x86 architecture. The DAVIS 346 Color camera is connected to the edge computing platform via a USB 3.0 interface.

[0151] Step 4.2: Read the weight file from Step 3.2 and load the trained BGLIENet network.

[0152] Step 4.3: Construct a producer-consumer queue. The queue is a blocking queue, process-safe, and supports multi-threaded concurrency. The producer is the acquisition thread, which obtains the event stream and image stream of DAVIS346 Color based on the API interface of the libcaer library. The acquired event stream is converted into an event voxel grid according to step 1.2 and stored in the queue. The consumer is the low-light enhancement network BGLIENet, which preloads the weight file provided in step 3.2 and is responsible for processing the image-event voxel grid data stored in the queue into normal lighting images.

[0153] Based on the above steps, this invention is compared with low-light image enhancement methods based on the Mamba architecture, low-light image enhancement methods based on Retinex theory and Transformer architecture, and end-to-end low-light image enhancement methods based on events and images. Table 1 shows that the method proposed in this invention outperforms other methods in terms of peak signal-to-noise ratio (PSNR), PNR score, and structural similarity index on the SDE indoor low-light dataset.

[0154] Table 1. Performance comparison of various methods on the SDE indoor low-light dataset.

[0155] method Peak signal-to-noise ratio Structural information similarity Low-light image enhancement method based on Mamba architecture 21.15 0.6950 Low-light image enhancement method based on Retinex theory and Transformer architecture 21.26 0.7139 An end-to-end low-light image enhancement method based on events and images 22.44 0.7697 A Low-Light Enhancement Method Based on Bidirectional Collaborative Guidance of Events and Images (This Invention) 23.09 0.7769

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A low-light enhancement method based on bidirectional collaborative guidance of events and images, characterized in that, include: Step 1: Acquire low-light image-event data and its paired normal-light image-event data, and preprocess the event data to obtain the image-event dataset; Step 2: Construct BGLIENet, a low-light enhancement network guided by bidirectional collaboration between events and images. This low-light enhancement network includes a preprocessing network, an image-guided event enhancement module, a feature pyramid encoder, and an image-event fusion branch. The preprocessing network converts low-light images into coarse-enhanced images and extracts shallow feature information and obtains a signal-to-noise ratio map based on the coarse-enhanced images. The image-guided event enhancement module removes spatiotemporal noise from the event data and uses the shallow feature information of the coarse-enhanced images to guide the supplementation of event data to obtain event features containing fine-grained semantic context. The feature pyramid encoder extracts multi-scale features of event features containing fine-grained semantic context from the event data; the image-event fusion branch is based on the Mamba model and uses the signal-to-noise ratio map as a guide to fuse the event features containing fine-grained semantic context and their multi-scale features with the shallow feature information of the coarse enhancement image to obtain the enhanced image. The image-guided event enhancement module will extract the shallow feature information of the coarsely enhanced image obtained from the preprocessing network. Compared with the event voxel mesh obtained in step 1 As input, the specific process is as follows: event voxel grid Using a filter size of The convolutional layer, NAFBlock, layer normalization operator, and filter size are... The size of the convolutional layer and filter is The depth can be used to separate the cascaded blocks of convolutional layers to obtain denoised event features. ; Denoising event features Shallow feature information of coarsely enhanced images and event voxel grid Corresponding shallow features The input consists of a cross-space modulation module, a cross-channel modulation module, and a filter of size [missing information]. The concatenated blocks of convolutional layers yield event features containing coarse-grained semantic context. ; Event features that include coarse-grained semantic context and shallow feature information of coarsely enhanced images Input to a normalization layer with a filter size of The size of the convolutional layer, cross-space modulation module, and filter is The concatenated blocks of convolutional layers yield event features containing fine-grained semantic context. ; The image-event fusion branch is based on the UNet architecture and includes an encoder and a decoder. The encoder consists of three Mamba-based image-event fusion modules and two bilinear downsampling operators, and the Mamba-based image-event fusion modules and the bilinear downsampling operators are alternately set in sequence. The shallow feature information of the coarsely enhanced image The event features containing fine-grained semantic context and the feature pyramid encoder pair Extracted multi-scale features The input is fed into the encoder to extract the fused multi-scale features. ; The decoder consists of three Mamba-based image-event fusion modules, two bilinear upsampling operators, and a filter size of [missing information]. The convolutional layers constitute the image-event fusion module based on Mamba, and the bilinear upsampling operator is alternately set in sequence; the bilinear upsampling operator consists of a concat operator and a... It consists of a convolutional layer and a bilinear interpolation operator with an upsampling factor of 2; where the features after the fusion of scale 2 and scale 1 are respectively represented as... , ; Scale 1 fusion features Event features containing fine-grained semantic context The input is fed into a Mamba-based image-event fusion module and then filtered by a filter of size [size missing]. The convolutional layer produces residual components, which are then added to the coarsely enhanced image to obtain the final enhanced image. ; Step 3: Train the BGLIENet network from Step 2 using the image-event dataset obtained in Step 1; Step 4: Deploy the BGLIENet network to edge devices in low-light scenes, and use the trained BGLIENet network to enhance images captured in low-light scenes in real time.

2. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 1, characterized in that, In step 1, the preprocessing includes: The low-light image-event data and the normal-light image-event data are sorted and paired according to timestamps, and the training set and test set are divided according to different shooting scenarios; Given N discrete input event data ,in This represents the pixel position corresponding to the i-th event. It is polar. For timestamps; input the event. Convert to event voxel grid: （1） Where B represents the number of event slices in the event voxel grid; Indicates the timestamp of the starting event; Indicates the timestamp of the termination event; Represents the maximum value operator; Represents the discrete index of the voxel grid in the temporal dimension; Represents the spatial pixel coordinates of the event voxel grid.

3. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 2, characterized in that, The preprocessing network converts the low-light image into a coarse-enhanced image and extracts its shallow feature information based on the coarse-enhanced image, as follows: Using a filter of size The convolutional layer, LeaklyReLU activation function, and one filter of size 1 The initial illuminance distribution is estimated by a cascaded block composed of convolutional layers. : （2） in, For low-light images, For the LeaklyReLU operator with a leakage coefficient of 0.02, The filter size is Convolutional layers; The initial illuminance distribution map is multiplied element-wise by the low-light image, and expressed as: （3） in, For coarse enhancement of the image, It is an element-wise multiplication operator; Using filter size Convolutional layers extract coarse-enhanced images Shallow feature information: （4） in, This provides shallow feature information for coarse enhancement of the image.

4. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 3, characterized in that, The preprocessing network obtains a signal-to-noise ratio map based on the coarsely enhanced image, specifically including: Obtaining coarse-enhanced images using Gaussian filtering Low-frequency components: （5） in, To coarsely enhance the low-frequency components of the image, The filter size is Gaussian filter, It is a convolution operator; calculate and The residuals are used to estimate noise information. : （6） calculate and The signal-to-noise ratio (SNR) map is obtained by comparing the absolute values ​​of the ... （7） in, For absolute value operators, This is a signal-to-noise ratio diagram.

5. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 3, characterized in that, In the image-guided event enhancement module, denoising event features The formula is: （8） （9） in, event voxel grid The corresponding shallow features, The filter size is convolutional layers, For NAFBlock, For layer normalization operators, The filter size is convolutional layers, The filter size is The depth of the separable convolutional layer; The event features containing coarse-grained semantic context The formula is: （10） in, For cross-space modulation modules, It is a cross-channel modulation module; The event features containing fine-grained semantic context The formula is: （11）。 6. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 5, characterized in that, The calculation methods for the cross-space modulation module and the cross-channel modulation module are respectively expressed as follows: （12） （13） in, This is the output of the cross-space modulation module; This is the output of the cross-channel modulation module; Image features; Characteristics of the event; This is the global average pooling operator.

7. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 5, characterized in that, The feature pyramid encoder consists of two groups, each group comprising a pixel reconstruction layer and a filter size of [missing information]. The convolutional layers and NAFBlock are used to extract event features containing fine-grained semantic context. Multiscale features : （14） in, For scale numbering, For pixel recombination layer operators, For the first The filter size at each scale is convolutional layers, For the first NAFBlock at various scales.

8. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 7, characterized in that, In the image-event fusion branch, the fused multi-scale features The formula is as follows: （15） in, For bilinear interpolation operators with a downsampling factor of 2, This is a Mamba-based image-event fusion module. Features after 2-scale fusion Features after fusion with scale 1 They are as follows: （16） in, This is a bilinear interpolation operator with an upsampling factor of 2; Final enhanced image The formula is as follows: （17）。 9. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 8, characterized in that, The overall architecture of the Mamba-based image-event fusion module follows the design approach of the VisionMamba module. Its main contribution is the state scan space model guided by the signal-to-noise ratio map, as detailed below: Given image features Event characteristics Signal-to-noise ratio diagram The signal-to-noise ratio (SNR) map-guided state scan space model aims to model... The long-range dependency relationship is used to repair global illumination degradation and utilize the signal-to-noise ratio map. Guiding fusion event characteristics Repair structural information: （18） （19） （20） （21） in, An operator for splitting features along the channel dimension; is a fully connected layer; SiLU(.) is the SiLU activation function; VSSM(.) is the visual state space model of VMamba; and for Two features split along the channel dimension; This refers to global illumination restoration features for images after long-distance dependency modeling. Image and event fusion features guided by signal-to-noise ratio maps; For the The modulated image and event fusion features are the output of the Mamba-based image-event fusion module.

10. The low-light enhancement method based on bidirectional collaborative guidance of events and images according to claim 1, characterized in that, In the training process of step 3, the loss function includes smoothing L1 loss, peak signal-to-noise ratio loss, multi-scale structural similarity loss, and peak signal-to-noise ratio star loss. The overall description of the loss function is as follows: （22） in, For the overall loss function, This represents the true value, i.e., the normal lighting image corresponding to the low-light image; To smooth out the L1 loss; This represents the peak signal-to-noise ratio loss. For peak signal-to-noise ratio star loss; This is a multi-scale structural similarity loss.

Images