Event domain fast continuous bounding exposure

Abstract

An apparatus and method for event-based image surround exposure are provided. The device comprises an image camera and an event camera. The image camera captures one or more images with a fixed camera arrangement. The setting of the image camera changes rapidly, and the event camera records an event. The acquired one or more images and a portion of the surrounded exposure event are then fused to generate a composite image. The composite image may be an image having a desired camera setting, or an image having attributes that the image camera cannot capture at any setting, such as a full focus image. The distance of the sharpest focus setting may also be determined from the surrounded exposure event. The focus of the image camera is then set to this determined distance.

Description

TECHNICAL FIELD

[0001] Aspects of the disclosed embodiments generally relate to video cameras, and more particularly, to image bracketing. BACKGROUND

[0002] Image bracketing is a video camera technique in which multiple images of the same scene are captured with different camera settings. Different settings can include, but are not limited to, exposure, focus, aperture, and zoom. Many modern video cameras have built-in image bracketing functionality, making it easier to take multiple images and perform image bracketing.

[0003] There are many ways to implement image bracketing. Typical approaches generally support capturing a small number of images (e.g., 3 images) with different camera settings. However, these approaches tend to be time consuming and resource intensive. Moreover, during the image acquisition process, traditional image bracketing can be affected by blurring and misalignment caused by motion.

[0004] Accordingly, there is a need for an improved apparatus and method for image bracketing. Accordingly, it is desirable to provide an apparatus and method that at least partially address the above issues. SUMMARY

[0005] Aspects of the disclosed embodiments are directed to an apparatus and method for image bracketing. Aspects of the disclosed embodiments combine the properties of event cameras and image cameras to overcome the limitations of traditional bracketing. One or several well-exposed anchor images are acquired with fixed camera settings. Then, bracketing events are captured while the camera settings are changed in a swift manner. The camera settings are changed in a manner that is generally referred to herein as "swiftly." In one embodiment, specialized actuators or devices are used to change the camera settings in an automated manner. This speed efficiency is possible because the camera settings can be changed without exposing an image. Because events have an extremely high temporal resolution, the lens elements of the camera can be adjusted quickly, or as fast as the corresponding hardware actuators or controls allow. The camera settings can include, for example, but are not limited to, focus distance, aperture opening, or lens magnification. In this manner, aspects of the disclosed embodiments produce a composite image with superior properties or characteristics. For example, in the case of focus bracketing, a full-focus image can be generated in which all parts of the image appear sharp.

[0006] According to a first aspect, the above mentioned and further aims are achieved, in one embodiment, by an apparatus comprising an image camera and an event camera. A processor can be communicatively coupled with the image camera and the event camera. A memory can be communicatively coupled with the processor. The memory typically comprises program instructions that, when executed by the processor, cause the apparatus to capture one or more images of a region of interest with the image camera and record encircled exposure events of the region of interest with the event camera while changing one or more settings of the camera. Then, the captured one or more images and at least a portion of the recorded encircled exposure events are combined or fused to produce an image with the desired settings of the camera. Performing image encircled exposure based on events is more memory and energy efficient, enables faster acquisition, and the resulting one or more composite images are not degraded by object motion during the acquisition process.

[0007] In one possible implementation, the apparatus is configured to register a continuous change of camera settings to the event camera. Aspects of the disclosed embodiments enable continuous sampling of camera settings and probing all possible settings of the camera.

[0008] In one possible implementation, the apparatus is configured to change the settings of the camera quickly. Aspects of the disclosed embodiments enable the settings of the camera to be changed as fast as the camera hardware or actuators allow. This enables fast acquisition because only one image is captured with fixed settings of the camera and events are recorded while the settings are changed quickly.

[0009] In one possible implementation, the apparatus is configured to synchronize the image camera and the event camera in time. Aspects of the disclosed embodiments enable capturing or recording events while the camera settings are changed.

[0010] In one possible implementation, the apparatus is further configured to record encircled exposure events while the focus setting of the camera is changed. Then, images can be fused to acquire images with superior quality, e.g. all-in-focus images, to extend the depth of field. Aspects of the disclosed embodiments enable fast acquisition because one image is captured with fixed settings and events are recorded while the settings are changed quickly.

[0011] In one possible implementation, the apparatus is further configured to determine a focus distance and a focus frame from the recorded events. Aspects of the disclosed embodiments enable fast determination of the focus distance, e.g. 5 milliseconds, while being robust to motion blur, strong light, and weak light.

[0012] In one possible implementation, the determined focus distance is used to focus a frame-based camera, such as but not limited to a red-green-blue (RGB) camera. Aspects of the disclosed embodiments are able to quickly determine a focus distance while being robust to motion blur, strong light, and weak light.

[0013] In one possible implementation, the focus setting of the camera is changed from a closest distance to a farthest distance or from a farthest distance to a closest distance. The focus setting adjustment can start from any arbitrary setting and span the entire range of settings, such as closest to farthest or farthest to closest, or any suitable range in between.

[0014] In one possible implementation, the change in focus setting of the camera is arbitrary. The focus setting adjustment can start from any setting and span the entire range of settings. Aspects of the disclosed embodiments provide for continuous rather than discrete surround exposure.

[0015] In one possible implementation, the focus distance is determined by maximizing the contrast of the event or event image of the region of interest. Faster focusing is provided compared to contrast-based focusing.

[0016] In one possible implementation, the one or more images of the region of interest are captured by the camera using different focus settings of the camera, and the surround exposure event is recorded while changing the focus distance of the camera. Aspects of the disclosed embodiments are able to capture events while changing or adjusting the settings of the camera.

[0017] In one possible implementation, the apparatus is further configured to combine the one or more images with the recorded surround exposure event, the recorded surround exposure event being a focus surround exposure event. This enables capturing images without adjusting the focus.

[0018] In one possible implementation, the fusion of the one or more captured images with at least a portion of the surround exposure event is used to control the settings of the camera. Aspects of the disclosed embodiments allow for controlling the camera to generate higher quality images.

[0019] In one possible implementation, the image camera and the event camera comprise a hybrid sensor. The use of a hybrid sensor eliminates the need for separate devices and does not require geometric alignment of events and images.

[0020] According to a second aspect, the above described and other implementations and advantages are achieved by a method. In one embodiment, the method includes capturing one or more images with a camera; changing one or more settings of the camera; recording one or more wrapped exposure events with an event camera during the changing of the one or more settings; and combining the captured one or more images with the recorded one or more wrapped exposure events to generate an image with desired settings of the camera. Performing image wrapped exposure based on events is more memory and energy efficient, enables faster capture process, and does not degrade during the capture process due to object motion.

[0021] In one possible implementation, the method further includes recording a wrapped exposure event when a focus setting of the camera is changed. Aspects of the disclosed embodiments eliminate the need to spend time focusing a frame camera in a snapshot photography.

[0022] In one possible implementation, the method further includes determining a focus distance and a focus frame from the recorded wrapped exposure event.

[0023] In one possible implementation, the method further includes changing the focus setting of the camera from a closest distance to a farthest distance.

[0024] In one possible implementation, the method further includes capturing one or more images of a region of interest with the camera using different focus settings of the camera and recording a wrapped exposure event when the focus setting of the camera is changed. Aspects of the disclosed embodiments provide fast continuous image wrapped exposure in event domain and apply to auto focus, refocus, and all-in-focus imaging.

[0025] In one possible implementation, the method further includes synchronizing the image camera and the event camera in time. Aspects of the disclosed embodiments reduce problems due to object motion during image capture.

[0026] In one possible implementation, the method further includes aligning the captured image and the recorded event in geometry. Aspects of the disclosed embodiments reduce problems due to object motion during image capture.

[0027] In one possible implementation, the method further includes scanning all possible settings of the camera. Aspects of the disclosed embodiments enable camera setting changes to start from any arbitrary setting and span the entire camera setting range.

[0028] In one possible implementation, the method further includes capturing events with the event camera while scanning all possible settings of the camera. Aspects of the disclosed embodiments enable camera setting changes to start from any arbitrary setting and span the entire camera setting range.

[0029] In one possible implementation, the method further includes generating a refocused image based on a combination of the captured image and the recorded surround exposure event. Aspects of the disclosed embodiments generate a synthetic image from an image and a surround exposure event with attributes such as "all focus" that cannot be achieved with a conventional image surround exposure.

[0030] In one possible implementation, the method further includes generating an all focus image based on a combination of the captured image and the recorded surround exposure event. Aspects of the disclosed embodiments are able to compute an all focus image when it is not possible to directly capture an image due to the limited depth of field of the camera.

[0031] According to a third aspect, the above described and further implementations and advantages are obtained by a computer program product comprising non-transitory computer program instructions for causing a processor to perform the method according to any of the possible implementations when the non-transitory computer program instructions are executed by the processor.

[0032] According to a fourth aspect, the above described and further implementations and advantages are obtained by an apparatus. In one embodiment, the apparatus comprises an image camera and an event camera. A processor can be communicatively coupled with the image camera and the event camera. A memory can be communicatively coupled with the processor. The memory typically includes program instructions that, when executed by the processor, cause the apparatus to record a surround exposure event of a region of interest with the event camera while changing one or more settings of the camera. A distance to a sharpest focus point is computed from the surround exposure event, and a focus of the image camera is set to the distance. Performing an image surround exposure based on the event is more memory and energy efficient, enables faster capture, and the generated synthetic image or images are not degraded by object motion during the capture process.

[0033] These and other aspects, implementations, and advantages of the exemplary embodiments will appear from the following description and the accompanying drawings. It is to be understood, however, that the description and drawings are designed solely for purposes of illustration and not as a definition of the limits of the disclosed embodiments; any limitations of the disclosed embodiments are set forth only in the claims. Additional aspects and advantages of the present application will be set forth in the description to follow, and in part will become apparent to those having ordinary skill in the art upon examination of the following or can be learned from practice of the application. The aspects and advantages of the application will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0034] In the following detailed portion of the specification, the application will be explained with reference to the example embodiments illustrated in the drawings, in which like elements are identified with the same reference numerals, and:

[0035] Figure 1 A block diagram of an exemplary apparatus incorporating aspects of the disclosed embodiments is shown.

[0036] Figure 2 A block diagram of an exemplary apparatus incorporating aspects of the disclosed embodiments is shown.

[0037] Figure 3 A flow diagram depicting an exemplary method for event image surround exposure incorporating aspects of the disclosed embodiments is shown.

[0038] Figure 4 A flow diagram depicting an exemplary process flow for event image surround exposure incorporating aspects of the disclosed embodiments is shown.

[0039] Figure 5 An exemplary process flow for event image surround exposure incorporating aspects of the disclosed embodiments is shown.

[0040] Figure 6 An exemplary process flow for event image surround exposure incorporating aspects of the disclosed embodiments is shown.

[0041] Figure 7 An exemplary process flow for event image surround exposure incorporating aspects of the disclosed embodiments is shown.

[0042] Figure 8 An exemplary process flow for event image surround exposure incorporating aspects of the disclosed embodiments is shown. DETAILED DESCRIPTION

[0043] Figure 1 A block diagram of an exemplary apparatus 100 for event domain image surround exposure incorporating aspects of the disclosed embodiments is shown. As shown in the example of FIG. 1, in one embodiment, exemplary apparatus 100 generally includes an image camera 102 and an event camera 106. Image camera 102 can also be referred to or understood as a "picture" camera or a "frame" based camera. Figure 1

[0044] Image camera 102 is generally used to capture one or more images of a scene of interest 108 with a fixed camera setup. In one embodiment, image camera 102 is a red-green-blue (RGB) type camera. In alternative embodiments, any suitable camera or sensor can be used for image camera 102, including for example but not limited to a greyscale pattern camera or other Bayer pattern camera.

[0045] ​The event camera 106 is used to record the enclosed exposure event when adjusting or changing the settings of the image camera 102. The event camera 106 generally includes an imaging sensor that is responsive to local changes in scene brightness. The event camera 106 can also be referred to or include a neuromorphic camera, a silicon retina sensor, or a dynamic vision sensor. Generally, aspects of the disclosed embodiments are not intended to be limited to a particular type of event camera.

[0046] Generally, each pixel of the event camera 106 operates independently and asynchronously. When a change in brightness occurs, an event is generated. The event camera 106 is used to asynchronously measure the change in brightness per pixel and output an event stream that encodes the time, location, and sign of the change in brightness. In one embodiment, the event camera 106 of the disclosed embodiments can include a high temporal resolution camera. Using a high temporal resolution event camera 106 allows for recording a continuous image space with all intermediate camera settings in the form of enclosed exposure events over very short time intervals. The event representation is highly compressed, requiring less memory than storing images. In alternative embodiments, the event camera 106 can include any suitable type of event camera.

[0047] The event camera 106 of the disclosed embodiments generally has lower power requirements, enabling aspects of the disclosed embodiments to achieve very high energy efficiency. Additionally, because the acquisition time of the event camera 106 is short, events are not affected by motion blur. Thus, problems due to motion during image acquisition are largely avoided.

[0048] In one embodiment, the apparatus 100 is used to combine or "fuse" the acquired image or images and at least a portion of the recorded enclosed exposure events. The fusion is generally used to generate an image with the desired settings of the camera 102. Alternatively, an image is produced with properties that the image camera 102 is not capable of capturing at any setting, such as a full focus image. Additionally, the fusion process can be used to further control the image camera 102, for example, by changing the focus of the image camera 102.

[0049] Aspects of the disclosed embodiments are also capable of controlling the settings of the image camera 102, such as the camera focus. For example, in one embodiment, aspects of the disclosed embodiments are capable of calculating the distance to the sharpest focus of the enclosed exposure event. The focus of the image camera 102 can then be set to this calculated distance, either manually or automatically.

[0050] In one embodiment, the apparatus 100 includes a processor 110 and a memory 112. The processor 110 is generally communicatively coupled with the memory 112. The memory 112 generally includes, among other things, program instructions that, when executed by the processor 110, cause the apparatus 100 to acquire one or more images of a region of interest with the image camera 102. The acquired images can also be referred to as "anchor images," which are generally well-exposed or good-quality images. While generally one or more images are referred to herein, aspects of the disclosed embodiments allow for an image bracketing exposure function with a single image captured.

[0051] Generally, the one or more images are captured with the image camera 102 settings fixed. The optics of the image camera 102 and the event camera 106 can be set to the same settings when the images are captured. For example, for a focus bracketing exposure, the same focus setting and depth of field setting are used.

[0052] Aspects of the disclosed embodiments allow for events to be captured by the event camera 106 before or after the image camera 102 acquires images. The camera settings, such as but not limited to focus, focal length, or aperture, are rapidly changed when the event camera 106 is capturing events. Depending on the type of lens or optics, the speed at which the camera settings are changed can be on the order of milliseconds, such as but not limited to 5 milliseconds. The speed at which the settings of the camera 102 are changed or adjusted is generally limited only by the capabilities of the actuator hardware of the camera 102 used to control and adjust the camera settings.

[0053] In one embodiment, using the event camera 106, the apparatus 100 is used to record events of the region of interest 110 as the one or more settings of the image camera 102 are changed, also referred to herein as "bracketed exposure events." Aspects of the disclosed embodiments enable continuous sampling of the camera settings as events are recorded with the event camera 106. Aspects of the disclosed embodiments are capable of probing all possible settings of the camera, such as starting from an arbitrary setting, and across the entire or other suitable range of settings of the camera.

[0054] The image camera 102 and the event camera 106 are generally positioned in a manner that enables images and events, respectively, to be captured from the scene of interest 108. In alternative embodiments, the image camera 102 and the event camera 106 can be positioned in any suitable manner, including, for example, but not limited to, side-by-side, a stereo configuration, a configuration with a beam splitter, or implemented as a single hybrid sensor, as will be further described below.

[0055] Still referring to Figure 1 In one embodiment, the image camera 102 and the event camera 106 can be temporally and geometrically synchronized and aligned. In Figure 1In the example of FIG. 1, the synchronization block or module 104 can be used to temporally synchronize the image camera 102 and the event camera 106. In alternative embodiments, the image camera 102 and the event camera 106 can be temporally and geometrically synchronized in any suitable manner.

[0056] Although Figure 1 The example of FIG. 1 shows a separate synchronization block or module 104, aspects of the disclosed embodiments are not limited thereto. In alternative embodiments, the synchronization and / or geometric alignment of the image camera 102 and the event camera 106 can be implemented in any suitable manner. For example, one or more of the image camera 102 or the event camera 106 can include or incorporate the synchronization module 104.

[0057] The image camera 102 and the event camera 106 can also include "frame-based" sensors and "event-based" sensors. Such sensors will have similar fields of view and be temporally synchronized. In some implementations, the image camera 102 and the event camera 106 will have similar optics.

[0058] In one embodiment, a liquid lens is implemented in one or more of the image camera 102 and the event camera 106. Using an adjustable liquid lens can enable fast focus adjustment, for example, on the order of 5 milliseconds.

[0059] Figure 2 A block diagram of an apparatus 200 incorporating aspects of the disclosed embodiments is shown. In this example, one or more of the image camera 102 and the event camera 106 are replaced or implemented as a hybrid sensor 210. A hybrid sensor typically implements event pixels and normal RGB pixels on one silicon device. This eliminates the need for separate devices for the image camera 102 and the event camera 106. One of the main benefits of using a hybrid sensor in conjunction with aspects of the disclosed embodiments is that there is no need for geometric alignment of events and images.

[0060] Figure 3 An example of a process 300 incorporating aspects of the disclosed embodiments is shown. In one embodiment, one or more images are captured or taken 302. The captured images, also referred to herein as "anchor images," are typically well-exposed or good quality images. In one embodiment, the one or more images are taken by the image camera 102 of FIG. 1. Figure 1 The one or more images taken by the image camera 102 of FIG. 1 are taken with fixed camera settings. Camera settings typically include, but are not limited to, focus distance, aperture opening, or lens magnification.

[0061] The process 300 then includes utilizing the one or more anchor images to determine 304 a set of camera settings for the image camera 102 and the event camera 106. In one embodiment, the set of camera settings is determined by the synchronization block or module 104 of FIG. 1. Figure 1The event camera 106 records events 304, also referred to herein as "surrounding exposure events." Changing (304) the settings of the image camera 102 can include sweeping the settings of the image camera 102. In one embodiment, changing (304) the settings can involve sweeping the camera settings without any intermediate stops. Aspects of the disclosed embodiments are capable of probing all possible settings of the image camera 102 in an extremely short time interval. The time interval can be as small as possible with the support of a lens actuator. Typically, the captured images and the recorded events are temporally synchronized and geometrically aligned.

[0062] In one embodiment, one or more of the captured images and one or more surrounding exposure events are combined or fused (306). In one embodiment, the method 300 includes fusing (306) at least a portion of the captured images and surrounding exposure events that describe a change from the captured image to the latent image with the desired camera settings. For example, in one embodiment, from all recorded events, only the events that were triggered when the lens changed focus from the settings corresponding to the captured image to the settings corresponding to the latent image are captured. In alternative embodiments, all surrounding exposure events are used.

[0063] In one embodiment, the fusing (306) can generate (308) an image with the desired camera settings. The one or more desired camera settings can include, for example, but are not limited to, focus, aperture opening, or zoom. The generated image 308 can also be referred to as a synthetic image. In alternative embodiments, the fusing (306) can generate an image with properties that the image camera 102 is not capable of capturing, such as a full focus image. In another embodiment, the fusing 306 can be used to control the settings of the image camera 102, such as the camera focus.

[0064] These synthetic images typically have properties that are superior to the original camera settings. For example, in the case of focus surrounding exposure, a full focus image is generated in which all parts of the image appear sharp.

[0065] Auto focus using focus bracketed exposure events

[0066] Aspects of the disclosed embodiments are capable of finding the best focus distance using the surrounding exposure event stack. The best focus distance can then be used to set the focus of the image camera 102, such as an RGB camera accordingly. When using a liquid lens on the image camera 102, the best focus distance can be determined as fast as possible in about five (5) milliseconds. Although a time frame of about five (5) milliseconds is generally referred to herein, aspects of the disclosed embodiments are not limited thereto. In alternative embodiments, the time frame can be any suitable time frame other than including five (5) milliseconds.

[0067] Typical autofocus methods generally include contrast detection autofocus (CDAF) and phase detection autofocus (PDAF). These methods capture the exposed image surrounded by the focus point and then find the image with the best contrast in the region of interest. However, each image must be properly exposed and blur-free. Typical CDAF is usually slow and cannot handle fast-moving objects. Phase detection autofocus (PDAF) tends to perform poorly in challenging lighting conditions, such as strong or low light.

[0068] By focusing the camera 102 based on contrast detection in a focus-enclosed exposure event, aspects of the disclosed embodiments can provide autofocus for either event-enclosed exposure or focus-enclosed exposure events. Reference Figure 4 This illustrates one embodiment of an autofocus process flow 400 using a focus-enclosed exposure event. Aspects of the disclosed embodiment are capable of determining the distance to the sharpest focus from the enclosed exposure event, and then setting or physically refocusing the image camera 102 to that distance.

[0069] like Figure 4 As shown, when one or more focus settings of image camera 102 change, event camera 106 captures one or more bounded exposure events 402, referred to herein as "focus bounded exposure events," as described above. As an illustrative, non-limiting example, a focus bounded exposure event may be captured by event camera 106 when image camera 102 is refocused from closest to farthest distance, or from farthest to closest distance, or any distance in between. In an alternative embodiment, the focus setting change may begin from any desired or arbitrary setting and span the entire setting range, rather than including closest to farthest or farthest to closest.

[0070] The distance 404 to the sharpest point is then determined or calculated from the surrounded exposure events. For example, aspects of the disclosed embodiments may discretize the events into event images and find the event image with the highest contrast. The focus frame of the image camera 102 or the RGB camera may be set or adjusted (406) to this distance to provide a refocused image.

[0071] In one embodiment, the optimal focus distance is determined or calculated by maximizing the contrast of an event slice in the region of interest 108. Compared to contrast-based focusing, the method of the disclosed embodiments provides faster focusing. The method of the disclosed embodiments is also robust to object motion and performs well in challenging lighting conditions such as strong and low light due to the high temporal resolution and high dynamic range of the event camera 106.

[0072] Refocus of focus bracketed exposure events

[0073] Existing refocusing methods based on depth and image segmentation typically depend on the quality of the depth sensor. The refocusing results often do not allow for the recovery of blurred areas.

[0074] refer to Figure 5 The disclosed embodiments enable refocusing of a focus-enclosed exposure event using a single captured image and the focus-enclosed exposure event. The disclosed embodiments achieve "digital" refocusing by fusing the event and image with another focus point.

[0075] like Figure 5 As shown in the exemplary process flow, the acquired image 502 is the input to the fusion module 506. The acquired image 502 includes images composed of... Figure 1 Images captured by image camera 102. In one embodiment, the acquired images 502 include images captured by image camera 102 at different focal points.

[0076] In this example, the enclosed exposure event 504 occurs when the focus distance of the image camera 102 changes rapidly. Figure 1 Events recorded or captured by the event camera 106. For example, this could include events captured as the focus changes from nearest to farthest, from farthest to nearest, or any range in between (traversing all possible settings). The enclosed exposure event 504 in this example describes the transition from the acquired image to a potential image with the desired focus. In one embodiment, event input 504 includes only those events triggered when the lens of the image camera 102 changes focus from a setting corresponding to the acquired image to a setting corresponding to the potential image.

[0077] In this example, the fusion module or algorithm 506 processes one or more acquired images 502 and captured event inputs 504 to generate a composite image 508 with the desired focus, also referred to as a "refocused image". In one embodiment, motion during event-enclosed exposure acquisition is handled by performing basic event-based image interpolation.

[0078] In one embodiment, the fusion module 506 can include a neural network architecture that can be used to generate the refocused image 508. One example of an exemplary neural network architecture is a“UNET.” In alternative embodiments, any suitable neural network architecture can be used.

[0079] Unlike traditional refocusing methods, aspects of the disclosed embodiments produce a real refocus without the need for depth and image segmentation and allow recovery of blurred regions on the anchor image. The fast continuous event encircled exposure of the disclosed embodiments provides advantages of less memory usage, energy efficiency, and high speed. Aspects of the disclosed embodiments can capture images without focusing, which is especially important in time-sensitive snapshot photography.

[0080] Full focus of focus bracketed exposure events

[0081] Existing focal stack methods require multiple images with different focal points. These methods tend to be slow, energy and memory inefficient, and suffer from motion blur and misalignment due to motion during image capture.

[0082] Reference Figure 6 Aspects of the disclosed embodiments achieve full focus of the focal encircled exposure event from a single image input 602 captured or acquired and all focal encircled exposure events 604. In one embodiment, the process flow 600 fuses one or more acquired image inputs 602 and encircled exposure event inputs 604 using a fusion algorithm 606. The result 608 is a full focus image.

[0083] In Figure 6 the example, the encircled exposure event inputs 604 include all focal encircled exposure events recorded by the event camera 106. This is different from Figure 5 the example in Figure 5 where the recorded events are only those detected when the focal point changes from the focal point of the input image to the desired focal point.

[0084] The method of the disclosed embodiments allows the computation of a full focus image in cases where the image cannot be directly acquired due to the limited depth of field of the camera 102. This is particularly applicable in cases of macro photography or landscape photography, among others.

[0085] Event and image geometric fusion

[0086] Due to the limited contrast sensitivity of event sensors, direct fusion of event-bound exposure and anchor images for image refocusing and all-focus imaging does not provide good results when event data is sparse. To address this issue, aspects of the disclosed embodiments can use geometric fusion for refocusing and all-focus imaging. Instead of directly predicting pixel intensity, aspects of the disclosed embodiments apply geometric methods and predict a per-pixel blur / deblur kernel, and apply the predicted per-pixel kernel to the original image.

[0087] refer to Figure 7 In one embodiment, process flow 700 takes one or more acquired images 702 as input and input 704 of all events captured when the focus setting is changed, and estimates a per-pixel blur or pseudo-deblur kernel 708. This kernel 708 is applied to the input image 702 to generate a full-focus image 710. Process 700 leverages the fact that the blur / deblur kernel can be densely estimated from sparse events.

[0088] Figure 8 and Figure 7 Similarly, the difference lies in the event input 804. In this example, it's similar to... Figure 4 The recorded event input 804 includes only those events recorded when the focus of the imaging camera 102 changes from the focus of the input image to the desired focus. For example, event input 804 may include all events captured when the focus of the imaging camera 102 changes from closest to furthest (traversing all possible settings). A per-pixel blur or pseudo-deblur kernel is estimated (808) and applied to the input image 802 to generate a refocused image 810.

[0089] The disclosed embodiments acquire or capture one or more images using a fixed camera setup. Then, a bracketed exposure event is recorded by an event camera while the camera setup changes rapidly without intermediate stops. The event representation is highly compressed, requiring less memory than the images. By fusing one or more acquired images with the bracketed exposure event, the event imaging bracketed exposure of the disclosed embodiments offers advantages such as better memory and energy efficiency, a fast image acquisition process, continuous rather than discrete bracketed exposures, and fewer problems arising from object movement during the acquisition process.

[0090] Using focus-enclosed exposure event autofocus does not require capturing both the image and the event and then fusing them together. It simply calculates the distance to the sharpest point from the enclosed exposure event and then sets the camera's focus at that distance based on the frame.

[0091] Thus, although there have been described herein examples of the present application in accordance with the embodiments thereof, various modifications, alterations and improvements can be suggested to one skilled in the art and are within the spirit and scope of the application. Further, it is expressly intended that all combinations of the components and / or elements, variations and / or modifications as described herein, can be achieved, save that such combinations, variations and / or modifications are not expressly disclosed in the claims.

Claims

1. An apparatus (100), characterized by An apparatus (100) comprising an image camera (102), an event camera (106), and a processor (110) communicatively coupled with a memory (112), wherein the memory (112) comprises program instructions that, when executed by the processor (110), cause the apparatus (100) to: capture one or more images of a region of interest with the image camera (102); record, with the event camera (106), an enclosed exposure event of the region of interest while changing one or more settings of the image camera (102); combine the captured one or more images and at least a portion of the recorded enclosed exposure event to generate an image with desired settings of the image camera (102).

2. The apparatus (100) according to claim 1, characterized in that The apparatus (100) is configured to register a continuous change in camera settings to the event camera (106).

3. The apparatus (100) according to claim 1 or 2, characterized in that The apparatus (100) is configured to synchronize the image camera (102) and the event camera (106) in time.

4. The apparatus (100) according to any one of the preceding claims, characterized in that The apparatus (100) is further configured to record the enclosed exposure event while changing a focus setting of the image camera (102).

5. The apparatus (100) according to any one of the preceding claims, characterized in that The apparatus (100) is further configured to determine one or more of a focus distance and a focus frame of the image camera from the recorded enclosed exposure event.

6. The apparatus (100) according to any one of the preceding claims, characterized in that The focus setting of the image camera (102) is changed from a closest distance to a farthest distance or from a farthest distance to a closest distance.

7. The apparatus (100) according to any one of the preceding claims, characterized in that The change in the focus setting of the image camera (102) is arbitrary.

8. The apparatus (100) according to any one of the preceding claims, characterized in that The focus distance is determined by maximizing a contrast of the region of interest.

9. The apparatus (100) according to claim 1, characterized in that The one or more images of the region of interest are captured by the image camera (102) using different focus settings of the image camera (102), and the enclosed exposure event is recorded while changing a focus distance of the image camera (102).

10. The apparatus (100) according to any one of the preceding claims, characterized in that The apparatus is further configured to combine the one or more images with the recorded enclosed exposure event, the recorded enclosed exposure event being a focus enclosed exposure event.

11. A method (300) characterized by, comprising: capturing (302) one or more images with an image camera; recording (304) one or more enclosed exposure events with an event camera while changing one or more settings of the image camera; combining (208) the captured one or more images and the recorded one or more enclosed exposure events to generate an image with desired settings of the image camera.

12. The method (300) according to claim 9, characterized by Further comprising recording an enclosed exposure event while changing a focus setting of the image camera.

13. The method (300) according to claim 11 or 12, characterized by The focus setting of the image camera is changed from a closest distance to a farthest distance or from a farthest distance to a closest distance.

14. The method (300) according to claim 11, characterized by The one or more images of the region of interest are captured by the image camera using different focus settings of the image camera, and the enclosed exposure event is recorded while changing a focus distance of the image camera.

15. A computer program product, characterised in that, comprising non-transitory computer program instructions, which, when executed by a processor (110), are configured to cause the processor (110) to perform the method according to any one of claims 11 to 14.

Images