Reconstruction of hdr with blocking

By combining data from traditional cameras and event cameras, extracting and spatially aligning event features and image features, the problem of limited artifacts and dynamic range in HDR images is solved, and high-quality and high-temporal resolution HDR image generation is achieved.

CN118891653APending Publication Date: 2024-11-01HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280089456.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-01-26
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The prior art is difficult to overcome the problems of motion-related artifacts and limited dynamic range in dynamic scenes when capturing details in high contrast scenarios, especially in image-based HDR methods where the temporal resolution is reduced and the artifacts are severe.

Method used

Improved HDR images are generated by combining traditional frame-based cameras and event-based cameras, using a neural network to process input bracketed images and event data streams, extracting and spatially aligning event features and image features.

Benefits of technology

It significantly reduces motion-related artifacts, improves the quality and temporal resolution of HDR images, can handle a larger range of motion scenes, and relies on the bracketing method to prevent the failure of the event HDR method when there is no motion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118891653A_ABST
    Figure CN118891653A_ABST
Patent Text Reader

Abstract

An imaging apparatus for acquiring two or more low dynamic range (LDR) images (605), the two or more LDR images (605) comprising an LDR image represented as a reference LDR image (605b), and an image processing apparatus (606) for processing the two or more LDR images (605), the two or more LDR images (605) comprising a reference LDR image (605b), the reference LDR image (605b), the reference LDR image (605b), and the reference LDR image (605b). Acquiring two or more segments (602) of an event stream, the two or more segments (602) of the event stream corresponding to the two or more LDR images (605); extracting one or more event features from a segment (602) of the event stream; converting the one or more event features into image features to generate event-based image features; the event-based image features are spatially aligned to one or more image features of the reference LDR image (605b) to generate spatially aligned event-based image features. In this way, the spatially aligned event-based image features may be generated from image features from the LDR image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to high dynamic range (HDR) images, for example for generating spatially aligned event-based image features that are compatible with image-based features for generating HDR images. Background Art

[0002] Dynamic range in an image is defined as the ratio between the maximum and minimum measurable light intensity, white light and black light. The human visual system has a very wide dynamic range. In other words, humans can perceive details in very bright scenes and very dark scenes. However, it can be difficult for a camera to capture such rich details due to the nature of its photoreceptors. A camera's sensor is made up of photosites. These photosites can provide a "light cavity" that generates an electrical signal when a photon "falls" into the cavity.

[0003] Due to the inherent noise level in the signal, there is a minimum number of photons that can be detected by each cavity, so the "minimum" light that can be measured by the camera can be set. Figure 1A The minimum measurable light due to the sensor noise level 100 is schematically shown. The minimum number of photons 101 is shown therein.

[0004] At the other end of the brightness scale, when too many electrons fall into a photosite, it can overflow and saturate. This effectively sets the maximum measurable light, or white light. Figure 1B The maximum measurable light due to overflow of the photosite 102 is schematically shown. The maximum number of photons 103 is shown therein.

[0005] Any brightness present in the scene outside this range will be underexposed by a single exposure value or overexposed by a single exposure value. Brightness may be lost in these parts of the scene.

[0006] If the brightness of a scene is at different exposure values ​​in different frames, the details of the scene can be revealed. Figure 2A An example input image 200 with underexposed areas is shown. Figure 2B An example input image 201 with overexposed areas is shown. Figure 2A and Figure 2B Other areas in the image are well exposed.

[0007] Figure 2A and 2B Shows photos of the same scene at different exposure values. Figure 2A In , bright details around the doors and windows are shown, but the back of the room is underexposed. Figure 2B In this image, the areas around the doors and windows are blown out, but details in the back of the room are well exposed. Figure 2Aand Figure 2B The frames in the image can show the details of the scene in the frame.

[0008] It is known that the current dynamic range of the camera cannot capture all the details present in high contrast scenes. To overcome this, for a given static camera and scene, several low dynamic range (LDR) frames can be captured with different exposure values ​​and merged by weighted fusion. This is called automatic exposure bracketing (AEB) and enables the resulting image to retain the details present in all the input images, effectively increasing the dynamic range. The final image is called a high dynamic range (HDR) image.

[0009] Figure 3 Titled "Fusion Exposure", presented by Mertens et al. at the 2007 Pacific Rim Conference on Computer Graphics and Applications, three low dynamic range (LDR) frames 301a, 301b, 301c are shown for generating an HDR image 303. 302a, 302b, 303c show weight maps corresponding to the LDR frames 301a, 301b, 301c.

[0010] By changing the capture parameters that regulate the amount of light captured by the camera, ie the lens aperture (also denoted f-stop), exposure time, sensor gain or ISO, capturing several frames with different exposure values ​​can be achieved.

[0011] A key challenge of AEB is in its implementation for dynamic scenes. If there is motion between the captured exposure sequence (camera motion or scene motion), the images must be spatially aligned with a reference frame (usually an intermediate frame is chosen). If there is misalignment between the frames, motion-related artifacts may appear in the final HDR reconstructed image.

[0012] Figure 4 An example of misalignment between frames is shown in "Attention-Guided Networks for High Dynamic Range Imaging without Ghosting" by Q. Yan et al., 2019. Motion-related artifacts (ghosting) caused by the motion between the bracketed exposures can be seen in the frames of the first image 401a, 401b, 401c and the frames of the second image 402a, 402b, 402c. State-of-the-art networks adapted for AEB HDR attempt to address this issue but can still suffer from artifacts in very challenging dynamic scenes.

[0013] The main limitations of AEB include: (i) limited dynamic range per exposure, which may make alignment of LDRs difficult, (ii) presence of motion-related artifacts (ghosting / double edges / edge bleeding), and (iii) loss of temporal resolution, i.e., the temporal resolution decreases by a factor of 3 (for 3 input exposures).

[0014] An event-based camera (EBC) is a bio-inspired imaging sensor that may help address issues with traditional image-based HDR approaches such as AEB.

[0015] EBC only captures the motion in the scene. The sensor can contain independent asynchronous pixels, which have the following advantages: (i) high temporal resolution and low latency (both in the order of microseconds), (ii) very high dynamic range (up to 140dB, compared to 60dB for standard cameras), and (iii) low power consumption.

[0016] In contrast to a standard camera, which acquires a complete image at a rate specified by an external clock (e.g., 30fps), an event camera responds asynchronously and independently to brightness changes in the scene for each pixel. The output of an event camera can therefore be a variable data rate sequence of binary "events" or "spikes," each event representing a change in brightness (log intensity) of a predefined magnitude at a pixel at a specific time.

[0017] Figure 5 "EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames" by Gehrig et al., International Journal of Computer Vision, 2019, illustrates the difference between a standard frame-based camera and an event-based camera (EBC). The standard camera 502 captures every frame of a spinning disk 501, while the EBC 503 only captures pixels that are moving.

[0018] An individual pixel of an event camera responds to changes in its logarithmic photocurrent L = log(I) (“brightness”). When the brightness increment since the last event at that pixel (given by equation 1) reaches a threshold (given by equation 2), an event ei = (x, y, t, p) is triggered at pixel location (x, y) and timestamp t with polarity p.

[0019] ΔL(x,y,t)=L(x,y,t)-L(x,y,Δt) (1)

[0020] ΔL(x,y,t)=pC (2)

[0021] C is called the contrast threshold and p∈{-1,+1} is the polarity of the event, indicating whether the brightness has increased or decreased since the last event. Not all event cameras can capture event polarity, some EBCs may only detect whether an event occurred, while others capture the actual pixel intensity of the event. Detection depends on the specific type of EBC used.

[0022] The prior art for this system can be divided into two categories: i) image-based HDR and ii) event-based HDR.

[0023] Prior art in image-based HDR approaches attempt to reconstruct an HDR image from a single low dynamic range (LDR) image or from multiple LDR images captured with different exposures (bracketed exposures), typically captured with staggered equally spaced exposure values.

[0024] Multi-frame exposure methods are generally preferred over single-image methods because a single LDR image contains a limited dynamic range (about 60 dB). Therefore, information in overexposed or underexposed areas may need to be hallucinated, for example, using a neural network.

[0025] Exposure bracketing methods can perform better by leveraging information from a wider range of input exposures. However, since these exposures are captured sequentially, they can suffer from motion-related artifacts, especially in scenes containing fast motion. In addition, since each LDR has a limited dynamic range, saturated image regions can make alignment of bracketed exposures challenging.

[0026] State-of-the-art methods (e.g. AHDRNet, ADNet, etc.) use an attention module to suppress misaligned underexposed or overexposed areas of the input LDR. If ADNet is used, the LDR can be spatially aligned to the reference frame using a pyramidal deformable convolution (PCD) alignment module in the feature domain. The features can then be fused through a residual network backbone and an HDR image reconstructed based on the features. While these architectures may do a good job of suppressing ghosting artifacts, they may still not perform well for images containing saturated areas and large displacements between bracketed exposures.

[0027] The existing state-of-the-art in event-based HDR methods usually try to reconstruct the intensity image from the raw event data stream alone, i.e., it is a task to recover the intensity image from the sequence of binary events E = (x, y, timestamp, polarity). However, the quality of state-of-the-art schemes can be poor (PNSR is about 30dB). They are usually limited to grayscale reconstruction (because most event cameras do not capture color information) and have low resolution (the resolution of the Celepixel event camera is only 720×800 pixels).

[0028] In "High-Speed ​​and High Dynamic Range Video Using Event Cameras" by Rebecq et al., TPAMI, 2020, they attempt to reconstruct grayscale video from an event data stream using a recurrent neural network. However, the quality of the reconstruction can be lower than that obtained from image-based methods, with severe temporal artifacts, lack of detail, and complete failure to reconstruct when there is no motion. The method may also not explicitly predict linear HDR image frames, but only 8-bit tone-mapped images. The method can achieve high dynamic range by using only event-based cameras as input.

[0029] In "Neuromorphic Camera-Guided High Dynamic Range Imaging" by Han et al., published in CVPR 2020, they attempted to combine a single LDR image with event data to accomplish the HDR image reconstruction task. However, the method only uses a single exposure LDR image from a traditional frame-based camera and inputs the events in the form of intensity maps, either output by the event-based camera itself (some event cameras can capture image intensity frames and events simultaneously), or reconstructed using a neural network such as E2Vid. The results of the method are still of poor quality. The method does not seem to exploit the temporal nature of events by using only a single intensity map as input. In addition, the method also does not seem to be effective in scenes without motion, because when there is no event signal, the network may reconstruct with only the help of a single LDR input. This may perform worse than the bracketing exposure method.

[0030] Image-based HDR offers the following advantages: high pixel count, high-quality optics, accurate color representation, and fine detail reconstruction.

[0031] Image-based HDR has the following disadvantages: limited dynamic range per exposure (about 60dB), motion-related artifacts (ghosting / double edges), and loss of temporal resolution (effective temporal resolution drops by a factor of 3).

[0032] Event-based HDR has the following advantages: higher temporal resolution (about 10μs), higher dynamic range (140dB vs. 60dB), ultra-low power consumption (1mW vs. 1W), and no motion blur.

[0033] Event-based HDR has the following disadvantages: the quality of SOTA results is usually poor (as shown in "Neuromorphic Camera-Guided High Dynamic Range Imaging" published in CVPR 2020 by Han et al., the result is only about 30dB PSNR), grayscale output, low resolution (e.g. the DAVIS 346 event camera is only 346×260 pixels), information is only in the moving areas (such as temporal artifacts), so there is no signal without motion, and it is difficult to reconstruct fine details.

[0034] In view of this, we hope to develop a device and method to overcome the above problems. Summary of the invention

[0035] According to a first aspect, an imaging device is provided, the device comprising one or more processors and a memory storing data defining program code executable by the one or more processors in a non-transitory form, the program code, when executed by the one or more processors, causing the device to: acquire two or more low dynamic range (LDR) images, the two or more LDR images including an LDR image represented as a reference LDR image; acquire two or more segments of an event stream, the two or more segments of the event stream corresponding to the two or more LDR images; extract one or more event features from the segments of the event stream; convert the one or more event features into image features to generate event-based image features; spatially align the event-based image features to one or more image features of the reference LDR image to generate spatially aligned event-based image features. By spatially aligning the event-based image features to one or more image features of the reference LDR image to generate spatially aligned event-based image features, the spatially aligned event-based image features can be generated based on image features from the LDR image. This can provide image-feature-based event features for generating HDR images.

[0036] In some implementations, the apparatus may be used to generate an HDR image based on spatially aligned event-based image features. This may enable the HDR image to include a combination of event features and image features, and the event features are aligned with the image features. This may provide improved HDR image estimation.

[0037] In some implementations, the apparatus may be used to extract one or more image features from an LDR image. This may provide image features for event feature conversion.

[0038] In some implementations, the apparatus may be configured to extract one or more event features from a segment of an event stream based on one or more image features from an LDR image. This allows for the extraction of specific portions of the event stream based on the image features.

[0039] In some implementations, the apparatus may be used to convert the one or more event features into image features to generate event-based image features based on the one or more image features from the LDR image. In some implementations, the apparatus may be used to convert the one or more event features into image features to generate event-based image features based on self-supervision of the one or more image features from the LDR image. This enables teaching the LDR image features how to convert.

[0040] In some implementations, the apparatus may be used to extract one or more event features from a segment of an event stream via a neural network. The neural network may provide a means for training event feature extraction.

[0041] In some implementations, the apparatus may be configured to temporally subsample segments of the event stream. This allows the segments of the event stream to be decomposed for evaluation by the mixed modality module and comparison with the LDR image features.

[0042] In some implementations, the apparatus may be configured to extract the one or more event features from a subsample of segments of the event stream that are in the same feature space as the one or more image features extracted from the LDR image. This enables subsampling of portions of the LDR image where image features are present.

[0043] In some implementations, the apparatus may be used to extract one or more additional event features from a segment of the event stream, independent of one or more image features from the LDR image, and spatially align the additional event features with one or more image features of a reference LDR image. This enables the generation of more event features that are independent of the LDR image features. In this way, an HDR image may be estimated based on a wider range of event features.

[0044] In some implementations, the apparatus can be used to obtain the event stream, and the apparatus can be used to generate two or more segments of the event stream by converting the event stream into two or more voxel grids. This can convert the event stream into a format that the event pipeline neural network can understand.

[0045] In some implementations, the apparatus may be used to extract one or more image features from an LDR image via a neural network. The neural network may provide a means for training image feature extraction.

[0046] In some implementations, the apparatus may be configured to spatially align any misaligned image features or saturated image features from the LDR image to one or more image features of the reference LDR image by suppressing the misaligned image features or saturated image features. This may provide more image features for HDR image estimation.

[0047] In some implementations, the apparatus may be used to convert two or more LDR images into a linear domain by a gamma correction operation. When encoding the LDR image 605, the gamma correction may optimize the use of bits.

[0048] In some implementations, the apparatus may be used to extract one or more gamma-corrected image features from a gamma-corrected LDR image via a neural network. The neural network may provide a means for training gamma correction of image features.

[0049] In some implementations, the apparatus may be used to spatially align gamma-corrected image features with one or more image features of a reference LDR image. This may provide more image features for HDR image estimation.

[0050] In some implementations, the apparatus may be configured to combine spatially aligned event-based image features with spatially aligned more event features, spatially aligned misaligned image features or saturated image features, and spatially aligned gamma-corrected image features. This may provide more image features for HDR image estimation.

[0051] In some implementations, the apparatus may be configured to generate an HDR image based on a combination of spatially aligned event-based image features and spatially aligned more event features, spatially aligned misaligned image features or saturated image features, and spatially aligned gamma-corrected image features. This may provide more image features for HDR image estimation.

[0052] In some implementations, the apparatus may be configured to capture the two or more LDR images using different bracketing exposure levels. This may provide a range of exposure levels in the LDR images to combine and generate an HDR image.

[0053] According to a second aspect, there is provided a computing device comprising the device as claimed in any one of the preceding claims.

[0054] In some implementations, the computing device may include a frame-based camera and an event-based camera. Combining a frame-based camera and an event-based camera in the same computing device may provide a means for combining image-based features and event-based features in the same computing device.

[0055] In some implementations, the computing device may include a cloud server. This enables the device to be run remotely from the camera device.

[0056] According to a third aspect, there is provided a computer-implemented method for imaging, the method comprising: acquiring two or more low dynamic range (LDR) images, the two or more LDR images including a reference LDR image; acquiring two or more segments of an event stream, the two or more segments of the event stream corresponding to the two or more LDR images; extracting one or more event features from the segments of the event stream; converting the one or more event features into image features to generate event-based image features; spatially aligning the event-based image features with one or more image features of the reference LDR image to generate spatially aligned event-based image features. The spatially aligned event-based image features can be generated based on image features from the LDR images by spatially aligning the event-based image features to one or more image features of the reference LDR image to generate the spatially aligned event-based image features. This can provide image-feature-based event features for generating HDR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention is described below by way of examples in conjunction with the accompanying drawings. In the accompanying drawings:

[0058] Figure 1A The minimum measurable light due to the sensor noise level is schematically shown. Figure 1B The maximum measurable light due to overflow of the photosite is schematically shown.

[0059] Figure 2A An example input image with underexposed areas is shown. Figure 2B An example input image with overexposed areas is shown.

[0060] Figure 3 Three example LDR input image frames, three corresponding weight maps and output HDR images are shown.

[0061] Figure 4 Two HDR images generated by three different AEB methods are shown.

[0062] Figure 5 The output of motion images from a standard camera and an event camera is schematically shown.

[0063] Figure 6 The pipeline of a first exemplary imaging device is schematically shown.

[0064] Figure 7 The ducting of a second exemplary imaging apparatus is schematically shown.

[0065] Figure 8 The network design of an exemplary event feature extractor is schematically shown.

[0066] Fig. 9 The network design of an exemplary event alignment module is schematically shown.

[0067] Fig.10 The network design of an exemplary mixed-modality processor is schematically shown.

[0068] Fig.11A The network design of the first exemplary feature fusion module is schematically shown. Fig. 11B The network design of the second exemplary feature fusion module is schematically shown.

[0069] Fig.12 The network design of the HDR enhancement module is schematically shown.

[0070] Fig.13 An example of an apparatus for performing the methods described herein is shown.

[0071] Fig.14 An example method for generating spatially aligned event-based image features is shown.

[0072] Fig.15 Quantitative results of the HdM HDR dataset of our system compared with the prior art are shown.

[0073] Fig.16 Qualitative results on the HdM HDR dataset of our system compared with the prior art and ground truth are shown. DETAILED DESCRIPTION

[0074] The apparatus and method described herein relate to generating HDR images. In particular, the apparatus and method relate to generating spatially aligned event-based image features (for generating HDR images).

[0075] Embodiments of the present system can solve one or more of the above-mentioned problems by spatially aligning event-based image features to one or more image features of a reference LDR image to generate spatially aligned event-based image features. In this way, spatially aligned event-based image features can be generated based on image features from the LDR image. This can provide event features based on image features for generating HDR images.

[0076] The present system can solve the problem of capturing and estimating HDR images, and in particular, the difficulties caused by dynamic scenes, by utilizing the bracketing exposure of traditional frame-based cameras and the events of event-based cameras using an artificial intelligence (AI) multimodal camera system.

[0077] The system can do this by processing input events and input bracketed exposure images using a neural network pipeline. The pipeline can process input events and images using separate image and event processing modules or jointly with a mixed modality module. The output features of each processing module can be fused together and passed through an HDR enhancement module.

[0078] Specifically, the input to the present system may include a sequence of interlaced LDR images with bracketed exposures (e.g., short: -3ev, medium: 0ev, long: +3ev) and corresponding timestamps (t1, t2, t3), and a stream of simultaneously captured events captured from time t0 to t3, where t0 is a period of time before the first LDR capture t1. In other words, the input includes (i) a sequence of interlaced LDR images captured with bracketed exposure values, and (ii) a stream of event data captured by an event-based camera. The output of the present system may include one or more HDR images.

[0079] Figure 6 The pipeline of a first exemplary imaging device is schematically shown.

[0080] In summary, the pipeline can be composed of many separate modules. Input images and events can be processed by separate modules or jointly by a mixed modality module 610. The surrounding input image 605 can be processed by an LDR image processor 611. The input LDR image 605 can be converted to a linear image domain by gamma correction and processed using a linear image processor 612. Input events can be processed by an event processor 609. Subsamples of events can be processed by a mixed modality processor 610, which can convert them to a linear image feature domain.

[0081] Pipeline 600 may include different input streams. Figure 6 The exemplary pipeline 600 shown includes 4 input streams. In the first input stream, the event-based camera 601 can be fed into the event processor 609. In the second input stream, the event sampler 607 fed by the event-based camera 601 can be fed into the mixed modality processor 610. In the third input stream, the image-based camera 604 can be fed into the LDR image processor 611. In the fourth input stream, the image-based camera can be fed into the linear image processor 612 through the gamma correction function.

[0082] The different input streams 600 of the pipeline may be combined into a single output stream. The output stream may include a feature fusion module 613 which feeds into an HDR enhancement module 614. The HDR enhancement module 614 may output an HDR image 607.

[0083] Figure 7 The conduit 700 of the second exemplary imaging device is schematically shown. The conduit 700 may include a more detailed illustration of the conduit 600 of the first exemplary imaging device.

[0084] Specifically, the first input stream may further include an event-to-voxel module 701 implemented between the event-based camera 601 and the event processor 609. The event processor 609 may further include an event feature extractor 702, which feeds an event alignment module 703. The mixed modality processor 610 may further include an event feature extractor 704, which feeds a mixed modality alignment module 705. The LDR image processor 611 may further include an LDR feature extractor 706, which feeds an LDR spatial attention module 707. The linear image processor 612 may further include a linear feature extractor 708, which feeds a linear alignment module 709.

[0085] The apparatus as described herein may be implemented on any of pipelines 600, 700. The steps performed by the apparatus may be implemented on different processors and modules described herein. Alternatively, the apparatus described herein may be implemented on other suitable pipelines. Each module and processor may be implemented as a neural network.

[0086] The device can be used to obtain two or more LDR images 605. The LDR images 605 can be captured by the device. For example, the image can be captured by a frame-based camera. The device can be implemented on such a frame-based camera. The frame-based camera can include a digital single-lens reflex (DSLR) camera, a mirrorless camera and / or a smartphone camera. Alternatively, the device can be used to receive two or more LDR images from a separate device (e.g., a separate image-based camera as described herein).

[0087] The LDR images 605 can be captured using different bracketed exposure levels. Each of the LDR images 605 can be captured using a different bracketed exposure level. In other words, Figure 6 and 7As shown in the exemplary embodiment in FIG. 5 , the device can acquire three 5a, LDR images 605a, 605b, 605c. Preferably, at least three LDR images 605a, 605b, 605c are captured. However, two or more LDR images 605 can be used to generate a suitable HDR image 607. Each of the three LDR images 605a, 605b, 605c can be captured using a different bracketing exposure level. The middle or average bracketing exposure level can be designated as the reference LDR image 605b. This motion of the event in the reference LDR image 605b can be used to generate the final HDR image 607.

[0088] The LDR images 605a, 605b, 605c may be sequential. The exposure brackets of the LDR images 605a, 605b, 605c may be represented as ev-, ev0, ev+. The corresponding sequential timestamps of the LDR images 605a, 605b, 605c may be represented as t1, t2, t3. If more LDR images 605 are used in addition to the exemplary three LDR images 605a, 605b, 605c, more exposure brackets and timestamps may be used.

[0089] The device can be used to obtain an event stream. The device can be used to receive an event stream. The event stream can be a continuous stream captured while capturing the LDR image 605. The event stream can be captured by the device. For example, the event stream can be captured by an event-based camera. The device can be implemented on such an event-based camera. The event-based camera can include a CeleX camera, a DAVIS camera, and / or a DVS camera. Alternatively, the device can be used to receive an event stream from a separate device, such as a separate event-based camera described herein.

[0090] The apparatus may be used to obtain two or more segments 602 of an event stream. Figure 6 and 7 Three segments 602a, 602b, 602c of an event stream are shown. Preferably, the segments 602a, 602b, 602c may correspond to the LDR images 605a, 605b, 605c. In other words, the number of segments 602a, 602b, 602c may correspond to the number of LDR images 605a, 605b, 605c. Furthermore, the timing of the segments 602a, 602b, 602c may correspond to the timing of the LDR images 605a, 605b, 605c. In this way, the events captured in the segments 602a, 602b, 602c may correspond to the events captured in the LDR images 605a, 605b, 605c.

[0091] The apparatus may be used to generate two or more segments 602 of an event stream. The apparatus may be used to generate two or more segments 602 of an event stream by converting the event stream into two or more voxel grids. The voxel grid may provide a suitable structure for the apparatus, specifically, it may provide a three-dimensional structure including an x-coordinate, a y-coordinate, and a time coordinate. The conversion of the event stream to a voxel grid or segment 602 may be performed by an event to voxel module 701. Alternatively, the apparatus may be used to receive segments 602 of an event stream from a separate apparatus, such as a separate event-based camera as described herein.

[0092] The apparatus may be used to extract one or more image features from the LDR image 605. The image feature may include a portion, region, or feature of the LDR image 605, which describes good exposure information present in each of the differently exposed LDR images, and extracts temporal and spatial cues that are most suitable for use in the HDR reconstruction stage 614. Additionally or alternatively, the image feature may include a portion or region of the LDR image 605, the feature has low or high saturation, the feature moves between the LDR images 605a, 605b, 605c, or the feature includes other deformations.

[0093] The apparatus may be used to extract one or more image features from the LDR image 605 by means of a neural network.

[0094] refer to Figure 6 and 7 The LDR feature extractor 706 of the LDR image processor 611 may be used to receive the LDR image 605. The LDR feature extractor 706 may be used to extract one or more image features from the LDR image 605. The LDR feature extractor 706 may output one or more image features from the LDR image 605.

[0095] The apparatus may be used to spatially align any misaligned image features or saturated image features from the LDR image 605 to one or more image features of the reference LDR image 605b by suppressing the misaligned image features or saturated image features. By spatially aligning any misaligned image features or saturated image features from the LDR image 605, the apparatus may be used to generate spatially aligned misaligned image features or saturated image features 1101. In other words, the apparatus may be used to spatially align previously misaligned image features or saturated image features.

[0096] refer to Figure 6 and 7, the LDR spatial attention module 707 of the LDR image processor 611 may be used to receive features from the LDR image 605. The LDR spatial attention module 707 may be used to receive features from the LDR image 605 from the LDR feature extractor 706. The LDR spatial attention module 707 may be used to spatially align any misaligned image features or saturated image features from the LDR image 605 with one or more image features of the reference LDR image 605b by suppressing the misaligned image features or saturated image features. The LDR spatial attention module 707 may output spatially aligned misaligned image features or saturated image features 1101.

[0097] The apparatus may also be used to convert two or more LDR images 605 into a linear domain 606 by a gamma correction operation. Each of the two or more LDR images 605a, 605b, 605c may be converted into a linear domain 606 by a gamma correction operation to generate a gamma-corrected LDR image 606a, 606b, 606c, such as Figure 6 and 7 When encoding the LDR image 605, the gamma correction can optimize the use of bits.

[0098] The gamma correction module may be used to receive one or more LDR images 605. The gamma correction module may be used to generate gamma-corrected LDR images 606a, 606b, 606c. The gamma correction module may be used to output gamma-corrected LDR images 606a, 606b, 606c.

[0099] The apparatus may be used to extract one or more gamma-corrected image features from the gamma-corrected LDR image 606. The image feature may include a portion, region, or feature of the gamma-corrected LDR image 606 that describes good exposure information present in each of the differently exposed LDR images and extracts temporal and spatial cues that are most suitable for use in the HDR reconstruction stage 614. Additionally or alternatively, the image feature may include a portion or region of the gamma-corrected LDR image 606, the feature having low or high saturation, the feature moving between the gamma-corrected images 606a, 606b, 605c, or the feature including other deformations.

[0100] The apparatus may be configured to extract one or more gamma-corrected image features from the gamma-corrected LDR image 606 by means of a neural network.

[0101] refer to Figure 6 and 7, the LDR feature extractor 708 of the linear image processor 612 may be used to receive the gamma-corrected LDR image 606. The LDR feature extractor 708 may be used to extract one or more image features from the gamma-corrected LDR image 606. The LDR feature extractor 708 may output one or more image features from the gamma-corrected LDR image 606.

[0102] The apparatus may be used to spatially align one or more image features from the gamma-corrected LDR image 606 with one or more image features of the reference LDR image 605 b. By spatially aligning one or more image features from the gamma-corrected LDR image 606, the apparatus may be used to generate spatially aligned gamma-corrected image features 1003.

[0103] refer to Figure 6 and 7 , the linear alignment module 709 of the linear image processor 612 may be used to receive features from the gamma-corrected LDR image 606. The linear alignment module 709 may be used to receive features from the gamma-corrected LDR image 606 from the LDR feature extractor 708. The linear alignment module 709 may be used to spatially align one or more image features from the gamma-corrected LDR image 606 to one or more image features of the reference LDR image 605b. The linear alignment module 709 may output spatially aligned gamma-corrected image features 1003.

[0104] refer to Figure 6 and 7 , the event processor 609 can divide the simultaneously captured input event stream into chunks corresponding to events acquired during the timestamps of the bracketed exposure (e.g., t0 to t1, t1 to t2, t2 to t3), where t0 is the start of capture. The event processor 609 can extract features from the quantized input events using a neural network. The event feature space can then be aligned to the event feature at the reference timestamp (t2) using the event-feature alignment module 703.

[0105] Figure 8 Schematically illustrated is a network design of exemplary event feature extractors 702, 704. The event feature extractors 702, 704 may form part of, or be used for, the event processor 609 and / or the mixed modality processor 610. In other words, both the event feature extractor 702 and the event feature extractor 704 may have the same structure, as described herein.

[0106] In summary, the event feature extractors 702, 704 can use strided convolutions to extract feature pyramids 805, 806, 807 at multiple scales. The residual block 804 is used to improve training stability. In addition, alternative feature extractors can be used. In other words, different neural network architectures can be used. In some implementations, any method for extracting features from input events can be used. Figure 8 In the network design of the exemplary event feature extractors 702, 704 shown, a series of simple convolutions, residual blocks and strided convolutions 801, 802, 803 are used to extract pyramids of feature maps 805, 806, 807 at different image scales.

[0107] The event feature extractors 702, 704 may receive an input stream of raw events from the event-based camera 601 and output a set of spatially aligned event features aligned with the event features at a reference timestamp. Event-feature alignment enables further exploitation of information from events that are not directly close to the reference time instance, which may enrich the feature representation.

[0108] Before event features can be aligned, a first portion of the event processor 609 computes features from the quantized events using the event feature extractors 702, 704.

[0109] Specifically, the apparatus may be used to extract one or more event features from the event stream segment 602. The event feature includes a portion or region of the event stream segment 602, and the feature moves between different segments 602a, 602b, 602c.

[0110] The apparatus may be used to extract one or more event features from the segment 602 of the event stream based on one or more image features from the LDR image 605. In other words, the position and / or characteristics of the image features from the LDR image 605 may be used to extract one or more event features from the segment 602.

[0111] The apparatus may be used to extract one or more event features from a segment 602 of an event stream by a neural network. The neural network may be trained by features of the LDR image 605 to extract event features from the segment 602. Specifically, the event feature extractor 704 in the mixed modality processor 610 may be trained to extract using features of the LDR image 605, while the event feature extractor 702 in the event processor 609 may not be trained to extract using features of the LDR image 605.

[0112] The device can also be used to extract one or more additional event features from the fragment 602 of the event stream, without relying on one or more image features from the LDR image 605. In other words, there is no need to be taught by the image features from the LDR image 605 how to extract more event features from the fragment 602 of the event stream. The device can be used to extract one or more additional event features from the fragment 602 of the event stream by a neural network that is not trained by the LDR image 605. In this way, the device can generate more event features that have different characteristics and / or attributes from other event features. This can provide a wider range of event features for generating HDR image 607.

[0113] refer to Figure 6 and 7 , the event feature extractor 704 of the mixed modality processor 610 may be used to receive the segment 602 of the event stream. The event feature extractor 704 may be used to receive the segment 602 of the event stream from the event sampler 607 or directly from the event-based camera 601. The event feature extractor 704 may be used to extract one or more event features from the segment 602 of the event stream. The event feature extractor 704 may output event features from the segment 602 of the event stream.

[0114] refer to Figure 6 and Figure 7 , the event feature extractor 702 of the event processor 609 may be used to receive the segment 602 of the event stream. The event feature extractor 702 may be used to receive the segment 602 of the event stream from the event-based camera 601. The event feature extractor 702 may be used to extract one or more additional event features from the segment 602 of the event stream. The event feature extractor 702 may output more event features from the segment 602 of the event stream.

[0115] Fig. 9 The network design of an exemplary event alignment module 703 is schematically shown. The event alignment module 703 may form a part of the event processor 609. In summary, the event alignment module 703 may spatially align the extracted event features to the event features at a reference timestamp.

[0116] The event alignment module 703 may use deformable convolution to perform spatial alignment at the feature level, where the input is a pyramid of multi-scale features. Other suitable implementations may use different feature alignment methods, such as optical flow.

[0117] Specifically, Fig. 9 As shown, the event alignment module 703 can use pyramidal deformable convolution (PCD) to spatially align features 901, 902, 903 at different scales. The input is the ith group of event features Pyramids of 901, 902, 903 and reference event features to be aligned 904, 905, 906. Alternatively, to perform the alignment, a different architecture may be used, for example, a lightweight alignment based on optical flow may be used instead. The aligned features 907 may be output by the alignment module 703.

[0118] The apparatus may be used to spatially align more event features 901, 902, 903 to one or more image features of a reference LDR image 605. Specifically, the apparatus may be used to spatially align more event features 901, 902, 903 extracted from a segment of an event stream (independent of one or more image features extracted from an LDR image) to one or more image features of a reference LDR image.

[0119] refer to Figure 6 and Figure 7 , the event alignment module 703 of the event processor 609 may be used to receive more event features 901, 902, 903. The event alignment module 703 may be used to receive more event features 901, 902, 903 from the event feature extractor 702. The event alignment module 703 may be used to spatially align more event features 901, 902, 903 to one or more image features of the reference LDR image 605. The event alignment module 703 may output more spatially aligned event features.

[0120] Fig.10 The network design of an exemplary mixed modality processor 610 is schematically shown. The mixed modality processor 610 can be used to work in different HDR image devices. In other words, the mixed modality processor 610 can be used as an independent pipeline.

[0121] In summary, the input event stream may be sampled by the event sampler 607 using sliding windows 1001a, 1001b, 1001c corresponding to different timestamps along the event stream. An event feature extractor 704, such as a neural network, may be used to extract features 1002a, 1002b, 1002c from each subsample of the event, thereby generating intermediate features 1004 between the LDR timestamps (t-1, t, t+1). A self-supervised L2 loss 1005 may be placed between the extracted event features 1002a, 1002b, 1002c and the linear image features at the corresponding timestamps, such as Fig.10 This can encourage the network to transform event features 1002a, 1002b, 1002c into a linear image feature domain 1003a, 1003b, 1003c.

[0122] The goal of the mixed modality processor is to convert a sampled subset of event features into the linear image feature domain. By converting a subset of events into the image feature domain, the feature representation of the RGB image can be enriched in time and space.

[0123] The mixed modality processor 610 can subsample the input event stream. This can be achieved using a simple sliding window 1001a, 1001b, 1001c on the input event stream, such as Fig.10 As shown. Additionally, alternative methods can be used to extract subsets of events. The goal can be to extract more features 1004 between LDR timestamps, which can increase the temporal resolution of the input features, thereby generating better spatial alignment and more input feature information.

[0124] The apparatus may be used to perform sub-sampling on the fragment 602 of the event stream. Specifically, the apparatus may be used to perform temporal sub-sampling on the fragment 602 of the event stream. The sub-sampling may be performed using a sliding window 1001a method, such as Fig.10 shown.

[0125] refer to Figure 6 and Figure 7 , the event sampler 607 may be used to receive the segment 602 of the event stream. The event sampler 607 may be used to receive the segment 602 of the event stream from the event-based camera 601. The event sampler 607 may be used to sub-sample the segment 602 of the event stream. The event sampler 607 may output a sub-sample of the segment 602 of the event stream.

[0126] The apparatus may be used to extract one or more event features from a subsample 602 of segments of the event stream. Specifically, the apparatus may be used to extract one or more event features from a subsample 602 of segments of the event stream (in the same feature space as one or more image features extracted from the LDR image). In other words, the extraction of event features may be performed such that the event features obtained are compatible with the image features. The extraction of one or more event features may be performed by a feature extractor 704, such as Fig.10 shown.

[0127] The device can be used to convert the one or more event features into image features to generate event-based image features. In other words, the device can be used to obtain event features and convert them into image features.

[0128] The device can also be used to convert the event features that have been sub-sampled so that the sub-sampled event features 1001a, 1001b, 1001c are used to generate event-based image features. In other words, the output of the event sampler 607, the sub-sampled event features 1001a, 1001b, 1001c, can be used by the mixed modality processor 610 to generate event-based image features.

[0129] The device can be used to convert one or more event features into image features to generate event-based image features based on one or more image features from the LDR image 605. In other words, the LDR image 605 and the corresponding image features of the LDR image 605 can be used to generate event-based image features. Specifically, the device can be used to convert one or more event features into image features to generate event-based image features based on self-supervision of one or more image features from the LDR image 605. One or more image features from the LDR image 605 can be used to teach the mixed modality module 610 how to convert one or more event features into image features. The L2 loss 1005 can be used to compare one or more image features from the LDR image 605 and one or more event features, and then the L2 loss 1005 can be used to convert the one or more event features into image features.

[0130] The apparatus may be used to spatially align event-based image features with one or more image features of a reference LDR image 605b to generate spatially aligned event-based image features 1002a, 1002b, 1002c.

[0131] refer to Figure 6 and Figure 7 , the hybrid modality alignment module 705 of the hybrid modality module 610 may be used to receive event-based features. The hybrid modality alignment module 705 may be used to receive event-based features from the event feature extractor 704. The hybrid modality alignment module 705 may be used to spatially align event-based image features to one or more image features of the reference LDR image to generate spatially aligned event-based image features 1002a, 1002b, 1002c. The hybrid modality alignment module 705 may output spatially aligned event-based image features 1002a, 1002b, 1002c.

[0132] Fig.11A The network design of the first exemplary feature fusion module is schematically shown. Fig. 11B The network design of the second exemplary feature fusion module is schematically shown.

[0133] In summary, the feature fusion module 613 can combine all features from the input branch and perform feature fusion before passing to the HDR enhancement module 614.

[0134] Specifically, the apparatus may be used to combine the spatially aligned event-based image features 1002 with one or more of the spatially aligned more event features 707, the spatially aligned misaligned image features or saturated image features 1101, and the spatially aligned gamma-corrected image features 1003. Fig.11A and 11B In the exemplary fusion module 613 shown, the spatially aligned event-based image features 1002 are combined with all spatially aligned more event features 707, spatially aligned unaligned image features or saturated image features 1101, and spatially aligned gamma corrected image features 1003. By combining all spatially aligned features, a wider range of inputs can be provided for the HDR image 607 to be generated therefrom.

[0135] refer to Figure 6 and Figure 7 The feature fusion module 613 may be used to receive one or more of the spatially aligned event-based image features 1002 and the spatially aligned more event features 707, the spatially aligned unaligned image features or saturated image features 1101, and the spatially aligned gamma-corrected image features 1003. The feature fusion module 613 may be used to receive one or more of the spatially aligned event-based image features 1002 and the spatially aligned more event features 707, the spatially aligned unaligned image features or saturated image features 1101, and the spatially aligned gamma-corrected image features 1003 from the mixed modality processor 610, the event processor 609, the LDR image processor 611, and the linear image processor 612, respectively. The feature fusion module 613 may be used to combine the spatially aligned event-based image features 1002 with one or more of the spatially aligned more event features 707, the spatially aligned unaligned image features or saturated image features 1101, and the spatially aligned gamma-corrected image features 1003. The feature fusion module 613 may output the fused features 1103.

[0136] The first exemplary feature fusion module 613 can cascade all features and pass them through convolution 1102 and activation functions, such as Fig.11A shown.

[0137] The second exemplary feature fusion module 613 may use a more complex attention averaging module to weight the contribution of each input feature, such as Fig. 11BAs shown. Compared with the first exemplary feature fusion module 613, the second exemplary feature fusion module 613 achieves an improvement. The second exemplary feature fusion module 613 can use the attention average module 1104 to weight the input features differently. These features will be cascaded and passed through the attention network 1105 and the Softmax function 1106 to generate a weight map for each input. These attention weight maps are multiplied with the input and added to perform a weighted average of the features 1103.

[0138] Fig.12 The network design of the HDR enhancement module is schematically shown.

[0139] In summary, the fused features 1103 from the feature fusion module are passed through a series of dense residual blocks (DRDB) 1201. The linear image features corresponding to the reference image timestamp 1003b are added 1202 to the output of the DRDB 1201 before being passed through the final series of convolutions 1203 to predict the output HDR image 607.

[0140] The apparatus may be used to generate an HDR image 607 based on spatially aligned event-based image features 1002. The apparatus may be used to generate an HDR image 607 based on fused features 1103. Specifically, the apparatus may be used to generate an HDR image 607 based on a combination of spatially aligned event-based image features 1002 and one or more of spatially aligned more event features 707, spatially aligned unaligned image features or saturated image features 1101, and spatially aligned gamma corrected image features 1003. In other words, different types of spatially aligned features may be combined or fused, and the fused features 1103 may be used to generate an HDR image 607. This enables the HDR enhancement module 614 to reconstruct the LDR image 605 into an HDR image 607.

[0141] In some implementations, such as Figure 7 and 12 As shown, the apparatus may be used to generate an HDR image 607 based on the fused features 1103 and the linear reference features 1003b. This enables the HDR reconstruction or estimation to be calibrated to the linear reference features 1003b.

[0142] refer to Figure 6 and Figure 7, the HDR enhancement module 614 may be used to receive the fused features 1103. The HDR enhancement module 614 may be used to receive the fused features 1103 from the feature fusion module 613. The HDR enhancement module 614 may be used to generate an HDR image 607 based on a combination of one or more of the spatially aligned event-based image features 1002 and the spatially aligned more event features 707, the spatially aligned unaligned image features or saturated image features 1101, and the spatially aligned gamma-corrected image features 1003. The HDR enhancement module 614 may output an HDR image.

[0143] The combination of modules in pipeline 600 can provide excellent HDR quality (ie, higher PSNR value).

[0144] The advantage of this system is that it takes advantage of two separate input modalities, exposure bracketing and event. Event data can provide: high temporal resolution (about 10μs), high dynamic range (about 140dB), and low power consumption. Exposure bracketing can provide: high spatial resolution (1080p / 4K and higher), RGB color, high-quality optics, and fine details.

[0145] The present system can provide many advantages over existing methods: the system can utilize information from two complementary modalities (bracketing and events), the system can utilize content from events in areas that may be underexposed or overexposed in the input LDR frame, the system can handle large degrees of motion between bracketing, the system can significantly reduce ghosting / edge artifacts by utilizing the higher temporal resolution of event-based cameras, and the system can prevent event HDR methods from failing when there is no event signal (no motion) by using bracketing HDR, which still performs well.

[0146] To train the pipeline 600, the entire HDR reconstruction network can be trained end-to-end. The training can include the event processor 609 and the mixed modality processor 610. During the backward pass of the first-order iterative optimization algorithm, the gradients can be calculated relative to the learnable weights of all modules. In addition, the mixed modality module 610 can be trained by self-supervision on the extracted features, as described in this paper Fig.10 However, the gradient from the self-supervised loss in the mixed modality module 610 may be back-propagated only through the event branch, thereby not updating the weights of the image feature extractors 706, 708.

[0147] The training can minimize the tone mapping prediction image by Equation 3 and tone mapping ground truth The pixel-level reconstruction loss (L1 loss) between:

[0148]

[0149] The tone mapping operator is defined in Equation 4:

[0150]

[0151] Alternatively, you can use a self-supervised L2 loss on mixed-modal features, such as Fig.10 shown.

[0152] Fig.13 An example of an apparatus 1300 for implementing the method described herein is schematically shown.

[0153] The device 1300 may be implemented on an electronic device such as a laptop, a tablet computer, a smart phone, or a television. In other words, a computing device may include the imaging device 1300 .

[0154] The computing device may include a frame-based camera and an event-based camera. In this way, the computing device may be used to obtain an LDR image 605 and an event stream 601. The frame-based camera may capture the LDR image 605. The event-based camera may capture the event stream 601.

[0155] The computing device may include a cloud server. In other words, the imaging device 1300 may be configured to run remotely on a server.

[0156] The apparatus 1300 includes a processor 1301 for processing a data set in the manner described herein. For example, the processor 1301 may be implemented as a computer program running on a programmable device such as a central processing unit (CPU). The apparatus 1300 includes a memory 1302 for communicating with the processor 1301. The memory 1302 may be a non-volatile memory. The processor 1301 may also include a cache ( Fig.13 1302). The apparatus 1300 may include more than one processor 1301 and more than one memory 1302. The memory 1302 may store data executable by the processor 1301. The processor 1301 may be configured to operate according to a computer program stored in a machine-readable storage medium in a non-transitory form. The computer program may store instructions that cause the processor 1301 to perform its method in the manner described herein.

[0157] Fig.14An example of a method 1400 for generating spatially aligned event-based image features is summarized. In step 1401, the method 1400 includes acquiring two or more low dynamic range (LDR) images, the two or more LDR images including a reference LDR image. In step 1402, the method 1400 includes acquiring two or more segments of an event stream, the two or more segments of the event stream corresponding to the two or more LDR images. In step 1403, the method 1400 includes extracting one or more event features from the segments of the event stream. In step 1404, the method 1400 includes converting the one or more event features into image features to generate event-based image features. In step 1405, the method 1400 includes spatially aligning the event-based image features to one or more image features of the reference LDR image to generate spatially aligned event-based image features.

[0158] Fig.15 Quantitative results of the HdM HDR dataset of our system compared with the prior art are shown.

[0159] exist Fig.15 In , the results of the present system 1501 using bracketed exposure images and events are quantitatively compared with the state-of-the-art bracketed exposure-only HDR method (ADNet) 1502. Fig.15 The table in also compares the results of the system using only the event processor and using the entire pipeline (event processor + mixed modal processor). The PSNR results are in both the linear (L-PSNR) domain and the tone mapped (mu-PSNR) domain. It can be seen that the system (including the full pipeline) achieves improvements over ADNet, with L-PSNR increasing by more than +3dB and mu-PSNR increasing by almost +1dB.

[0160] Fig.16 Qualitative results of the HdM HDR dataset of the present system compared with the prior art and ground truth are shown. A first image 1601a and a second image 1601b are used for comparison.

[0161] exist Fig.16 In the figure, the output HDR images of the present system 1603a, 1603b using bracketed exposure images and events are qualitatively compared with the state-of-the-art bracketed exposure only HDR method (ADNet) images 1602a, 1602b and the reference truth images 1601a, 1601b. It can be seen that the output HDR images of the present system 1603a, 1603b are closer to the reference truth images 1601a, 1602b than the state-of-the-art bracketed exposure only HDR method (ADNet) images 1602a, 1602b.

[0162] The applicant hereby discloses separately each individual feature described herein and any combination of two or more such features. With the common knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, without considering whether such features or combinations of features can solve any problems disclosed herein; and without limiting the scope of the claims. The applicant points out that aspects of the present invention may include any such individual features or combinations of features. In view of the above description, it will be obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. An imaging device (1300), It is characterized in that The apparatus (1300) comprises one or more processors (1301) and a memory (1302), the memory (1302) storing data defining program code executable by the one or more processors (1301) in a non-transitory form, the program code, when executed by the one or more processors (1301), causing the apparatus (1300) to: acquiring two or more low dynamic range (LDR) images (605), the two or more LDR images (605) including an LDR image denoted as a reference LDR image (605b); Obtaining two or more segments (602) of an event stream, the two or more segments (602) of the event stream corresponding to the two or more LDR images (605); extracting one or more event features from a segment (602) of the event stream; converting the one or more event features into image features to generate event-based image features; The event-based image features are spatially aligned to one or more image features of the reference LDR image (605b) to generate spatially aligned event-based image features (1002).

2. The device (1300) according to claim 1, It is characterized in that The device (1440) is used to generate an HDR image (607) based on the spatially aligned event-based image features (1002).

3. The device (1300) according to claim 1 or 2, It is characterized in that The apparatus (1300) is used to extract the one or more image features from the LDR image (605).

4. The device (1300) according to claim 3, It is characterized in that The device (1300) is used to extract the one or more event features from the segment (602) of the event stream based on the one or more image features in the LDR image (605).

5. The device (1300) according to any one of the preceding claims, It is characterized in that The apparatus (1300) is used to convert the one or more event features into image features to generate event-based image features based on the one or more image features from the LDR image (605).

6. The device (1300) according to claim 5, It is characterized in that The apparatus (1300) is used to convert the one or more event features into image features to generate event-based image features based on self-supervision of the one or more image features from the LDR image (605).

7. The device (1300) according to any one of the preceding claims, It is characterized in that The device (1300) is used to extract the one or more event features from the fragment (602) of the event stream through a neural network.

8. The device (1300) according to any one of the preceding claims, It is characterized in that The apparatus (1300) is used to temporally subsample segments (602) of the event stream.

9. The device (1300) according to claim 8, when dependent on claim 3, It is characterized in that The apparatus (1300) is used to extract the one or more event features from the subsample of the segment (602) of the event stream (in the same feature space as the one or more image features extracted from the LDR image (605)).

10. The device (1300) according to any one of the preceding claims, It is characterized in that The device (1300) is used to extract one or more additional event features from a segment (602) of the event stream (independent of the one or more image features in the LDR image (605)) and to spatially align the additional event features with one or more image features of the reference LDR image (605).

11. The device (1300) according to any one of the preceding claims, It is characterized in that The apparatus (1300) is used to obtain the event stream, and the apparatus is used to generate two or more segments (602) of the event stream by converting the event stream into two or more voxel grids.

12. The device (1300) according to any one of claims 3 to 11, It is characterized in that The device (1300) is used to extract the one or more image features from the LDR image (605) by means of a neural network.

13. The device (1300) according to any one of claims 3 to 12, It is characterized in that The apparatus (1300) is used to spatially align any misaligned image features or saturated image features from the LDR image (605) to one or more image features of the reference LDR image (605b) by suppressing the misaligned image features or saturated image features.

14. The device (1300) according to any one of the preceding claims, It is characterized in that The apparatus (1300) is used to convert the two or more LDR images (605) into the linear domain (606) by a gamma correction operation.

15. The device (1300) according to claim 14, It is characterized in that The apparatus (1300) is used to extract one or more gamma-corrected image features from the gamma-corrected LDR image (606) by means of a neural network.

16. The device (1300) according to claim 15, It is characterized in that The apparatus (1300) is used to spatially align the gamma-corrected image features with one or more image features of the reference LDR image (605b).

17. The device (1300) according to any one of claims 10 to 16, It is characterized in that The device (1300) is used to combine the spatially aligned event-based image features (1002) with one or more of the spatially aligned more event features (707), the spatially aligned unaligned image features or saturated image features (1101) and the spatially aligned gamma-corrected image features (1003).

18. The device (1300) according to claim 17, when dependent on claim 2, It is characterized in that The device (1300) is used to generate the HDR image (607) based on a combination of the spatially aligned event-based image features (1002) and one or more of the spatially aligned more event features (707), the spatially aligned unaligned image features or saturated image features (1101) and the spatially aligned gamma-corrected image features (1003).

19. The device (1300) according to any one of the preceding claims, It is characterized in that The two or more LDR images (605) are each captured using a different bracketing exposure level.

20. A computing device, It is characterized in that Comprising a device (1300) as claimed in any one of the preceding claims.

21. The computing device according to claim 20, It is characterized in that The computing device includes a frame-based camera (604) and an event-based camera (601).

22. The computing device according to claim 21, It is characterized in that The computing device includes a cloud server.

23. A computer-implemented method (1400) for imaging, It is characterized in that The method (1400) comprises: Acquiring two or more low dynamic range (LDR) images, wherein the two or more LDR images include a reference LDR image (1401); Obtaining two or more segments of an event stream, the two or more segments of the event stream corresponding to the two or more LDR images (1402); extracting one or more event features from a segment of the event stream (1403); converting the one or more event features into image features to generate event-based image features (1404); The event-based image features are spatially aligned to one or more image features of the reference LDR image (1405) to generate spatially aligned event-based image features.