Event-image fusion-based ultra-low light imaging methods, systems, and devices

Abstract

This invention belongs to the field of image processing technology, and discloses an ultra-low illumination imaging method, system, and apparatus based on event-image fusion. The ultra-low illumination imaging method includes: acquiring frame images and event sets; extracting illumination priors and image features F from the frame images. I A spatiotemporal filtering method guided by illumination priors is used to denoise the event set. This method includes: for each event to be processed, taking all events within the preceding ε seconds as a comparison event set; if the number of events in the comparison event set whose Manhattan distance to the corresponding event to be processed is less than σ is less than ξ, then the corresponding event to be processed is judged as noise and filtered out. σ is inversely correlated with light intensity, and ξ is positively correlated with light intensity. After denoising, event features F are extracted. e Based on illumination priors for F I and F e Weighted fusion is performed to obtain fused features, which are then input into a feature decoder to obtain a reconstructed image. Based on the above method, high-precision imaging can be achieved in extremely low-light environments.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to an ultra-low light imaging method, system and device based on event-image fusion. Background Technology

[0002] Extremely low-light imaging has immeasurable value in fields such as lunar polar exploration and cave exploration. However, frame-based imaging systems face a trade-off between exposure time and long exposures in low-light dynamic scenes. Long exposures result in motion blur, while short exposures lead to excessively low signal-to-noise ratios. Both of these severely degrade the visual quality of the images, affecting the results of higher-level tasks such as object detection and semantic segmentation. Event cameras, a novel neuromorphic sensor, feature asynchronous triggering of pixels and possess high temporal resolution and a wide dynamic range, making them highly valuable for applications in extremely low-light imaging.

[0003] While event cameras possess the potential for imaging in extremely low light conditions, their ability to capture relative light intensity changes makes it difficult to reconstruct the absolute light intensity of a scene. Therefore, event-image dual-mode fusion is often used to leverage the complementary advantages of both methods and achieve high-quality imaging in low-light environments. Existing techniques typically fuse features from both modes directly through channel merging (LIE, ELEDNet, etc.) or by weighting the image and event modes based on the image signal-to-noise ratio (EvLight, etc.). These methods often overlook the fact that event signals also exhibit significant noise in extremely low light conditions, leading to noise superposition between the two modes during reconstruction and limiting the quality of fused imaging in low-light environments. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an ultra-low illumination imaging method, system and device based on event-image fusion, the purpose of which is to fuse image and event modalities and reduce data noise, thereby improving the quality of fused imaging.

[0005] To achieve the above objectives, according to a first aspect of the present invention, an extremely low-light imaging method based on event-image fusion is provided, comprising:

[0006] Acquire frame images captured by a frame camera and a set of events synchronously captured by an event camera in the same scene;

[0007] Extract the illumination prior that reflects the relative intensity of illumination in different pixel regions from the frame image, and extract the image features F of the frame image using the first image encoder. I ;

[0008] The event set is denoised using a spatiotemporal filtering method guided by the aforementioned illumination prior. The spatiotemporal filtering method includes: traversing each event in the event set and treating it as an event to be processed; for each event to be processed, taking all events within the previous ε seconds as a comparison event set; if the number of events in the comparison event set with a Manhattan distance less than σ to the corresponding event to be processed is less than ξ, then the corresponding event to be processed is determined to be noise and filtered out; otherwise, it is a signal and retained. ε, σ, and ξ are all preset values, and events in pixel regions with different light intensities correspond to different σ and ξ. σ is inversely correlated with light intensity, and ξ is positively correlated with light intensity.

[0009] The events in the denoised event set are voxels converted into three-dimensional tensors, and the event features F of the three-dimensional tensors are extracted by a second image encoder. e ;

[0010] Based on the illumination prior, the image features F I and the event feature F e A weighted fusion is performed, whereby the fusion focuses more on image features in pixel regions with higher light intensity and more on event features in pixel regions with lower light intensity. The fusion process yields a fused feature F. m ;

[0011] The fusion feature F m Input the feature decoder to obtain the reconstructed image for the corresponding scene.

[0012] Optionally, the illumination prior of the frame image can be obtained using Retinex theory;

[0013] The operation of extracting illumination prior using Retinex theory includes: performing Gaussian convolution on the frame image, taking the maximum value of each channel in the convolved image, and obtaining the corresponding illumination prior.

[0014] Optionally, the formula for weighted fusion is:

[0015] ;

[0016] In the formula, The operator represents the Hadamard product, and M represents the normalized illumination prior.

[0017] Optionally, the feature decoder is based on a U-Net neural network to perform feature decoding.

[0018] According to a second aspect of the present invention, an extremely low-light imaging system based on event-image fusion is provided, comprising:

[0019] The data acquisition module is used to acquire frame images captured by the frame camera and event sets synchronously captured by the event camera in the same scene.

[0020] The illumination prior module is used to extract the illumination prior of the frame image, which reflects the relative intensity of illumination in different pixel regions.

[0021] A first image encoder is used to extract image features F of the frame image. I ;

[0022] A spatiotemporal filtering module is used to denoise the event set using a spatiotemporal filtering method guided by the illumination prior. The spatiotemporal filtering method includes: traversing each event in the event set and treating it as an event to be processed; for each event to be processed, taking all events within the previous ε seconds as a comparison event set; if the number of events in the comparison event set with a Manhattan distance less than σ to the corresponding event to be processed is less than ξ, then the corresponding event to be processed is determined to be noise and filtered out; otherwise, it is a signal and retained. ε, σ, and ξ are all preset values, and events in pixel regions with different light intensities correspond to different σ and ξ. σ is inversely correlated with light intensity, and ξ is positively correlated with light intensity.

[0023] The voxelization module is used to voxelize the events in the denoised event set into three-dimensional tensors;

[0024] A second image encoder is used to extract the event features F of the three-dimensional tensor. e ;

[0025] The feature fusion module is used to perform image feature F based on the illumination prior. I and the event feature F e A weighted fusion is performed, whereby the fusion focuses more on image features in pixel regions with higher light intensity and more on event features in pixel regions with lower light intensity. The fusion process yields a fused feature F. m ;

[0026] Feature decoder, used for the fused feature F m Decode the image to obtain the reconstructed image for the corresponding scene.

[0027] According to a third aspect of the present invention, an imaging training apparatus is provided, comprising:

[0028] The training dataset acquisition module is used to acquire multiple sets of training data and their corresponding labels. Each set of training data includes frame images of the same scene under low light intensity captured by a frame camera and a set of events synchronously captured by an event camera. The corresponding label is the frame image of the same scene under high light intensity. The low light intensity and the high light intensity are relative terms.

[0029] The backpropagation control module is used to input each set of training data into the ultra-low light imaging system as described in the second aspect and obtain a reconstructed image. Based on the loss between the reconstructed image and the corresponding label, the network parameters of the first image encoder, the second image encoder, and the feature decoder in the ultra-low light imaging system are adjusted in reverse, and the training is repeated until the loss converges to a preset level.

[0030] Optionally, the training dataset acquisition module includes a first beam splitter, a second beam splitter, a first filter, a second filter, a first frame camera, a second frame camera, and an event camera;

[0031] The first beam splitter is used to split external light into two parts and input them to the first frame camera and the second beam splitter respectively. The second beam splitter splits the light it receives into two parts again. One part is input to the first frame camera after passing through the first filter, and the other part is input to the event camera after passing through the second filter. The frame images and event sets synchronously acquired by the second frame camera and the event camera are spatially aligned and used as a set of training data. The frame images synchronously acquired by the first frame camera are used as the corresponding labels.

[0032] According to a fourth aspect of the invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, wherein the processor, when executing the computer program, implements the steps of the method as described in any of the first aspects.

[0033] According to a fifth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0034] According to a sixth aspect of the invention, a computer program product is provided, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method as described in any of the first aspects.

[0035] In summary, compared with the prior art, the technical solutions conceived in this invention have the following main advantages:

[0036] In this invention, a frame camera and an event camera are combined to perform fusion processing on synchronously acquired frame images and events. Specifically, an image encoder is used to extract image features from the frame images, and the illumination prior of the frame images is also extracted. A spatiotemporal filtering method guided by the illumination prior is used, applying more constrained denoising parameters to areas with stronger illumination. The denoised event signal is then converted into voxel form before extracting event features. Through an illumination-weighted adaptive fusion method, the frame image features and event voxels are fused and decoded to obtain a high-precision reconstructed image. In this scheme, due to the use of dual-mode fusion, the high dynamic range event signal and the image signal, which provides absolute brightness, complement each other. The event signal provides clear texture edges in dark areas, while the image signal provides absolute light intensity to guide event denoising and reconstruction. Therefore, high-precision imaging can ultimately be achieved in extremely low-light environments (such as illumination conditions below 1 lux). Attached Figure Description

[0037] Figure 1 This is a flowchart of the steps of an ultra-low light imaging method according to an embodiment of the present invention.

[0038] Figure 2 This is a signal flow diagram of an ultra-low illumination imaging method according to an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of a spatiotemporal filtering method according to an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the structure of an ultra-low light imaging system according to an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of the training dataset acquisition module in one embodiment of the present invention.

[0042] Figure 6 This is a calibration diagram of three cameras in one embodiment of the present invention.

[0043] Figure 7 This refers to data acquired under low light conditions in one embodiment of the present invention, as well as the reconstruction results obtained by the ultra-low light imaging method of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0045] In a first aspect, the present invention provides an ultra-low illumination imaging method based on event-image fusion.

[0046] like Figure 1 The diagram shown is a flowchart of the ultra-low illumination imaging method according to an embodiment of the present invention. Figure 2 The diagram shown is a signal flow graph of an ultra-low illumination imaging method according to an embodiment of the present invention. The relevant steps are described in detail below.

[0047] S1. Obtain frame images captured by the frame camera and event sets synchronously captured by the event camera in the same scene.

[0048] Understandably, the frame camera and the event camera need to acquire data synchronously. The frame camera outputs a single frame image captured during its exposure period, while the event camera continuously outputs an event stream. The event stream from the event camera during the exposure period of the frame camera needs to be acquired as a set of events that are time-synchronized with the frame image obtained by the corresponding exposure of the frame camera. Subsequently, by comprehensively processing the time-synchronized frame image and events, a clear image at the corresponding moment can be obtained.

[0049] Specifically, the frame image I under the current low-light environment and the event set during the corresponding exposure period are obtained, and the i-th event in the event set is denoted as E. i (x i ,y i ,t i ,p i ), where i represents the event number, E i Represents an event, x i ,y i Indicates the pixel position where the event was triggered, t i This represents the timestamp of the event trigger, i=1,2,3,……,G, where G is the number of events captured by the event camera during the corresponding exposure period of the frame camera, p i p represents the polarity of the event. i When the light intensity at pixel ∈{1,-1} changes from dark to bright, p i =1, otherwise p i =-1.

[0050] Understandably, after spatial alignment calibration, the data synchronously acquired by the frame camera and event camera after calibration meets the spatial alignment requirements.

[0051] Specifically, firstly, checkerboard calibration board data is collected, and frame image data is processed by adjusting the aperture and using algorithm enhancement. Event signals are reconstructed according to timestamps to obtain corresponding calibration data. Subsequently, the calibration data of the two cameras are projected onto the same plane and calibrated to determine the intrinsic and extrinsic parameters of the cameras, thus achieving spatial alignment of the camera data.

[0052] S2. Extract the illumination prior that reflects the relative intensity of illumination in different pixel regions of the frame image, and extract the image features F of the frame image through the first image encoder. I .

[0053] This step processes frame image I by inputting it into two different branches. The first branch obtains the illumination prior of frame image I, and the second branch extracts the image features F of frame image I through the first image encoder. I .

[0054] In the first branch, the illumination prior reflects the relative intensity of illumination in different pixel regions of the frame image. Specifically, the illumination prior can be extracted using Retinex theory.

[0055] In this embodiment, the operation of extracting illumination priors using Retinex theory includes:

[0056] Perform Gaussian convolution on the frame image, and then take the maximum value of each channel in the convolved image to obtain the illumination prior.

[0057] For example, perform a 3×3 Gaussian convolution on frame image I, and take the maximum value of each channel in the convolutioned W×H×3 image to obtain the illumination prior M, which has a size of W×H.

[0058] In the second branch, the first image encoder is a trainable network. Through prior training, optimal network parameters can be determined, thereby enabling feature extraction.

[0059] S3. Denoise the event set using a spatiotemporal filtering method guided by illumination prior. The spatiotemporal filtering method includes: traversing each event in the event set and treating it as an event to be processed. For each event to be processed, take all events within the previous ε seconds as a comparison event set. If the number of events in the comparison event set with a Manhattan distance less than σ from the corresponding event to be processed is less than ξ, then the corresponding event to be processed is judged as noise and filtered out; otherwise, it is a signal and retained. ε, σ, and ξ are all preset values, and events in pixel regions with different light intensities correspond to different σ and ξ. σ is inversely correlated with light intensity, and ξ is positively correlated with light intensity.

[0060] The study found that the noise of the event camera in extremely low light conditions is mainly photon shot noise, which follows a Poisson distribution with light intensity. Therefore, a spatiotemporal filtering noise reduction method guided by prior illumination was designed.

[0061] like Figure 3The diagram illustrates a spatiotemporal filtering method according to an embodiment of the present invention. Specifically, for each event to be processed, events within the preceding ε seconds are selected. For example, event N is the event to be processed, and events 1 to 12 are its comparison event set. Taking σ=2 as an example, the region whose Manhattan distance to event N is less than or equal to σ=2 is considered a neighboring region. Events in events 1 to 12 that are in the neighboring region of event N are considered neighboring events; otherwise, they are considered non-neighboring events. As shown in the diagram, events 1, 4, 8, 9, and 10 are all neighboring events, while events 2, 3, 5, 6, 7, 11, and 12 are all non-neighboring events. If the number of neighboring events is less than a preset threshold ξ, the event to be processed is determined to be noise; otherwise, it is considered a signal. ε, σ, and ξ are all preset values. For brighter regions in the illumination prior, stronger spatiotemporal filtering parameters are required, i.e., smaller σ and larger ξ. Based on this relative relationship, the σ and ξ corresponding to different pixel regions are determined experimentally.

[0062] In this step, if the number of events in the event set whose Manhattan distance to the corresponding event to be processed is less than σ is less than ξ, it indicates that the signal triggering frequency within a small neighborhood of the event is low, which does not conform to the characteristic of effective signal local aggregation, but is closer to the characteristic of event noise discrete sparseness. Therefore, the corresponding event to be processed can be judged as noise. Furthermore, events in pixel regions with different light intensities correspond to different σ and ξ. This invention makes σ inversely correlated with light intensity and ξ positively correlated with light intensity. Therefore, for events to be processed in brighter areas, the range of the neighboring area is smaller, and the number of neighboring events judged as signals is required to be greater, which conforms to the characteristic that event noise is positively correlated with light intensity under extremely low illumination. By denoising the event set using the above spatiotemporal filtering method, more noise can be removed while retaining the effective signal, thereby obtaining an event set with a high signal-to-noise ratio and improving the accuracy of the reconstruction results.

[0063] S4. Convert the event voxels in the denoised event set into three-dimensional tensors, and extract the event features F from the three-dimensional tensors using the second image encoder. e .

[0064] Specifically, voxelization of event point clouds can be achieved using conventional methods to obtain a three-dimensional tensor of size W×H×B. For example, the event set is {E}. i (x i ,y i ,t i ,p i )} i∈[1,N] The formula for voxelizing it can be expressed as:

[0065] ;

[0066] ;

[0067] In the formula, B represents the number of discrete units divided along the time dimension, i.e., the time axis is uniformly divided into B intervals, and k b (a)=max(0,1-|a|) is the interpolation function.

[0068] After voxelization, a second image encoder is used to extract features from the three-dimensional tensor to obtain the event features F of W×H. e The second image encoder is a trainable network. Through prior training, optimal network parameters can be determined, thereby enabling feature extraction.

[0069] S5, Image features F based on illumination priors I and event characteristics F e Weighted fusion is performed, where the fusion focuses more on image features in pixel regions with higher light intensity and more on event features in pixel regions with lower light intensity. The fused feature F is obtained after fusion. m .

[0070] In this invention, considering that frame cameras can provide relatively accurate absolute brightness, but suffer severe loss of texture details in underexposed areas, while event cameras have the advantage of high dynamic range, which allows them to retain rich texture details, the two are weighted and fused based on illumination priors. This makes the fusion process focus more on event modalities when the illumination is weak, and more on frame image modalities when the illumination is strong. In this way, the high dynamic event information of dark areas can be effectively utilized to perform high-precision reconstruction based on absolute light intensity.

[0071] In one embodiment, the illumination prior M is a normalized W×H feature, and the formula for weighted fusion is:

[0072] ;

[0073] In the formula, The operator represents the Hadamard product, and M represents the normalized illumination prior.

[0074] S6, Fusing features F m Input the feature decoder to obtain the reconstructed image for the corresponding scene.

[0075] The feature decoder is a trainable network. Through prior training, optimal network parameters can be determined, thereby enabling feature decoding and outputting image segmentation results.

[0076] In one specific embodiment, the feature decoder can be a convolutional neural network, specifically based on a U-Net neural network.

[0077] In this invention, a frame camera and an event camera are combined. An image encoder is used to extract image features from the frame image. An illumination prior is obtained based on the frame image. A spatiotemporal filtering method guided by the illumination prior is used to apply more constrained denoising parameters to areas with stronger illumination. The denoised event signal is then converted into voxel form and event features are extracted. The frame image features and event voxels are fused and decoded through an illumination-weighted adaptive fusion method to achieve high-precision imaging in extremely low-light environments.

[0078] Secondly, the present invention provides an ultra-low illumination imaging system based on event-image fusion.

[0079] like Figure 4 The diagram shown is a schematic of the structure of an ultra-low illumination imaging system according to an embodiment of the present invention, which includes a data acquisition module, an illumination prior module, a first image encoder, a spatiotemporal filtering module, a voxelization module, a second image encoder, a feature fusion module, and a feature decoder.

[0080] The data acquisition module is used to acquire frame images captured by the frame camera and event sets synchronously captured by the event camera in the same scene.

[0081] The illumination prior module is used to extract illumination priors that reflect the relative intensity of illumination in different pixel regions of the frame image;

[0082] The first image encoder is used to extract image features F from the frame image. I ;

[0083] The spatiotemporal filtering module is used to denoise the event set using a spatiotemporal filtering method guided by illumination prior. The spatiotemporal filtering method includes: traversing each event in the event set and treating it as an event to be processed; for each event to be processed, taking all events within the previous ε seconds as a comparison event set; if the number of events in the comparison event set with a Manhattan distance less than σ from the corresponding event to be processed is less than ξ, then the corresponding event to be processed is judged as noise and filtered out; otherwise, it is a signal and retained. ε, σ, and ξ are all preset values, and events in pixel regions with different light intensities correspond to different σ and ξ. σ is inversely correlated with light intensity, and ξ is positively correlated with light intensity.

[0084] The voxelization module is used to voxelize the events in the denoised event set into three-dimensional tensors;

[0085] The second image encoder is used to extract event features F from the three-dimensional tensor. e ;

[0086] The feature fusion module is used to perform image feature fusion based on illumination priors. I and event characteristics F eWeighted fusion is performed, where the fusion focuses more on image features in pixel regions with higher light intensity and more on event features in pixel regions with lower light intensity. The fused feature F is obtained after fusion. m ;

[0087] Feature decoder, used for fused features F m Decode the image to obtain the reconstructed image for the corresponding scene.

[0088] In this ultra-low light imaging system, the first image encoder, the second image encoder, and the feature decoder are all trainable networks. In the early stages, the optimal network parameters can be determined through training, thereby outputting high-quality images.

[0089] Specifically, the above modules are used to implement the corresponding steps of the ultra-low illumination imaging method introduced above. For details, please refer to the above introduction, which will not be repeated here.

[0090] Thirdly, the present invention provides an imaging training device.

[0091] This imaging training device is used to train the trainable network parameters in the extremely low-light imaging system described in the second aspect. The imaging training device includes a training dataset acquisition module and a backpropagation control module.

[0092] The training dataset acquisition module is used to acquire multiple sets of training data and their corresponding labels. Each set of training data includes frame images of the same scene under low light intensity captured by the frame camera and a set of events synchronously captured by the event camera. The corresponding label is the frame image of the same scene under high light intensity. Low light intensity and high light intensity are relative terms.

[0093] The backpropagation control module is used to input each set of training data into the ultra-low light imaging system and obtain the reconstructed image. Based on the loss between the reconstructed image and the corresponding label, the network parameters of the first image encoder, the second image encoder and the feature decoder in the ultra-low light imaging system are adjusted in reverse, and the training is repeated until the loss converges to a preset level.

[0094] The training data consists of data acquired under low light intensity, which has lower resolution, while the labels consist of data acquired under high light intensity, which has higher resolution. The purpose of training the imaging training device on the ultra-low light imaging system is to enable the neural network in the ultra-low light imaging system to learn the internal relationship between low-light intensity data and high-light intensity data. This allows the ultra-low light imaging system to reconstruct images based on low-light intensity data and generate clear images under high light intensity.

[0095] In one embodiment, in order to quickly obtain multiple sets of training data and their corresponding labels, the training dataset acquisition module is designed as follows.

[0096] like Figure 5 The diagram shows a schematic representation of the training dataset acquisition module in one embodiment of the present invention. In this embodiment, the training dataset acquisition module includes a first beam splitter, a second beam splitter, a first filter, a second filter, a first frame camera, a second frame camera, and an event camera.

[0097] The first beam splitter divides external light into two parts, which are then input to the first frame camera and the second beam splitter, respectively. The second beam splitter further divides the received light into two parts; one part passes through a first filter and is input to the first frame camera, while the other part passes through a second filter and is input to the event camera. The frame images and event sets synchronously acquired by the second frame camera and the event camera are spatially aligned to form a set of training data, with the frame images synchronously acquired by the first frame camera serving as the corresponding labels. Furthermore, the training dataset acquisition module also includes an external trigger. This external trigger controls the synchronous acquisition of the three cameras by outputting a stable level signal. Specifically, a stable square wave can be output from a high-frequency crystal oscillator to trigger the internal crystal oscillators of the event camera and the two frame cameras, thus achieving time synchronization between the cameras. The frequency and high-level duration of the square wave signal from the frame camera are determined by the frame rate and exposure time parameters provided by the host computer.

[0098] Understandably, the three cameras mentioned above are also calibrated cameras. For example... Figure 6 The diagram shown is a calibration schematic of three cameras in one embodiment of the present invention. The three cameras respectively collect data from the checkerboard calibration board. According to Zhang Zhengyou's calibration method, the camera data is projected onto the same plane. First, the two frame cameras are calibrated once, and then the frame cameras and the event cameras are calibrated again. This achieves spatial alignment of the data from the three cameras. Since the beam splitter will flip the image horizontally once, the event signal needs to be flipped once before calibration.

[0099] In the above embodiments, two beam splitters divide external light into three beams, which are then incident on the frame camera and event camera from the same angle. This ensures that all cameras acquire data under the same scene. A large amount of external light is filtered to simulate extremely low light conditions. The two frame cameras are used to generate bright image data and simulated image data under extremely low light conditions, respectively. The event camera is used to generate event data simulating extremely low light conditions. This allows for the rapid acquisition of data for the same scene under low light and strong light conditions. Based on time-synchronized and spatially aligned data, training data and its corresponding labels can be constructed. Using the resulting low-light images and low-light events as input, and normal light images as supervised learning labels, the event-image dual-mode fusion imaging quality under extremely low light conditions can be effectively improved.

[0100] like Figure 7The image shows data acquired under low light conditions in one embodiment of the present invention and the reconstruction result (fusion result) obtained by the ultra-low light imaging method of the present invention. It can be seen that the reconstruction image obtained by the present invention has better visualization effect and can significantly improve the performance of downstream tasks such as target recognition and semantic segmentation.

[0101] Fourthly, the present invention provides an electronic device.

[0102] The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0103] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0104] Fifthly, the present invention provides a computer-readable storage medium.

[0105] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described above.

[0106] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0107] In a sixth aspect, the present invention provides a computer program product or a computer program.

[0108] The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of the present invention.

[0109] In summary, this invention fully utilizes the high temporal resolution and high dynamic range characteristics of event streams and the dense light intensity information of image data. It uses illumination as a priori to guide the event signal for adaptive denoising and further achieves the fusion of effective information between modalities through weighted fusion. It has physical interpretability and provides a new solution for high-quality imaging under extremely low illumination conditions (such as illumination conditions below 1 lux).

[0110] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "again" are intended to illustrate the present invention and are not intended to limit the present invention.

[0111] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. An ultra-low illumination imaging method based on event-image fusion, characterized in that, include: Acquire frame images captured by a frame camera and a set of events synchronously captured by an event camera in the same scene; Extract the illumination prior that reflects the relative intensity of illumination in different pixel regions from the frame image, and extract the image features F of the frame image using the first image encoder. I ; The event set is denoised using a spatiotemporal filtering method guided by the aforementioned illumination prior. The spatiotemporal filtering method includes: traversing each event in the event set and treating it as an event to be processed; for each event to be processed, taking all events within the previous ε seconds as a comparison event set; if the number of events in the comparison event set with a Manhattan distance less than σ to the corresponding event to be processed is less than ξ, then the corresponding event to be processed is determined to be noise and filtered out; otherwise, it is a signal and retained. ε, σ, and ξ are all preset values, and events in pixel regions with different light intensities correspond to different σ and ξ. σ is inversely correlated with light intensity, and ξ is positively correlated with light intensity. The events in the denoised event set are voxels converted into three-dimensional tensors, and the event features F of the three-dimensional tensors are extracted by a second image encoder. e ; Based on the illumination prior, the image features F I and the event feature F e A weighted fusion is performed, whereby the fusion focuses more on image features in pixel regions with higher light intensity and more on event features in pixel regions with lower light intensity. The fusion process yields a fused feature F. m ; The fusion feature F m Input the feature decoder to obtain the reconstructed image for the corresponding scene.

2. The ultra-low illumination imaging method as described in claim 1, characterized in that, The illumination prior of the frame image is obtained using Retinex theory; The operation of extracting illumination prior using Retinex theory includes: performing Gaussian convolution on the frame image, taking the maximum value of each channel in the convolved image, and obtaining the corresponding illumination prior.

3. The ultra-low illumination imaging method as described in claim 1, characterized in that, The formula for weighted fusion is: ； In the formula, The operator represents the Hadamard product, and M represents the normalized illumination prior.

4. The ultra-low illumination imaging method as described in claim 1, characterized in that, The feature decoder is based on the U-Net neural network to perform feature decoding.

5. An ultra-low illumination imaging system based on event-image fusion, characterized in that, include: The data acquisition module is used to acquire frame images captured by the frame camera and event sets synchronously captured by the event camera in the same scene. The illumination prior module is used to extract the illumination prior of the frame image, which reflects the relative intensity of illumination in different pixel regions. A first image encoder is used to extract image features F of the frame image. I ; A spatiotemporal filtering module is used to denoise the event set using a spatiotemporal filtering method guided by the illumination prior. The spatiotemporal filtering method includes: traversing each event in the event set and treating it as an event to be processed; for each event to be processed, taking all events within the previous ε seconds as a comparison event set; if the number of events in the comparison event set with a Manhattan distance less than σ to the corresponding event to be processed is less than ξ, then the corresponding event to be processed is determined to be noise and filtered out; otherwise, it is a signal and retained. ε, σ, and ξ are all preset values, and events in pixel regions with different light intensities correspond to different σ and ξ. σ is inversely correlated with light intensity, and ξ is positively correlated with light intensity. The voxelization module is used to voxelize the events in the denoised event set into three-dimensional tensors; A second image encoder is used to extract the event features F of the three-dimensional tensor. e ; The feature fusion module is used to perform image feature F based on the illumination prior. I and the event feature F e A weighted fusion is performed, whereby the fusion focuses more on image features in pixel regions with higher light intensity and more on event features in pixel regions with lower light intensity. The fusion process yields a fused feature F. m ; Feature decoder, used for the fused feature F m Decode the image to obtain the reconstructed image for the corresponding scene.

6. An imaging training device, characterized in that, include; The training dataset acquisition module is used to acquire multiple sets of training data and their corresponding labels. Each set of training data includes frame images of the same scene under low light intensity captured by a frame camera and a set of events synchronously captured by an event camera. The corresponding label is the frame image of the same scene under high light intensity. The low light intensity and the high light intensity are relative terms. The backpropagation control module is used to input each set of training data into the ultra-low light imaging system as described in claim 5 and obtain a reconstructed image. Based on the loss between the reconstructed image and the corresponding label, the network parameters of the first image encoder, the second image encoder, and the feature decoder in the ultra-low light imaging system are adjusted in reverse, and the training is repeated until the loss converges to a preset level.

7. The imaging training apparatus as described in claim 6, characterized in that, The training dataset acquisition module includes a first beam splitter, a second beam splitter, a first filter, a second filter, a first frame camera, a second frame camera, and an event camera; The first beam splitter is used to split external light into two parts and input them to the first frame camera and the second beam splitter respectively. The second beam splitter splits the light it receives into two parts again. One part is input to the first frame camera after passing through the first filter, and the other part is input to the event camera after passing through the second filter. The frame images and event sets synchronously acquired by the second frame camera and the event camera are spatially aligned and used as a set of training data. The frame images synchronously acquired by the first frame camera are used as the corresponding labels.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 4.

Images