Event-based motion estimation for multi-frame stacking

Abstract

An image enhancement method using multi-frame stacking comprises the steps of: storing a plurality of frames of a target scene captured by a frame-based camera (10) during an acquisition interval (Tacq); storing a series of events generated by the event-based camera (12) viewing the target scene during the acquisition interval; synchronizing the timestamps of the events and the timestamps of the frames; selecting an anchor frame from the plurality of frames; aligning the current frame with respect to the anchor frame based on a stored event (16) having a timestamp corresponding to an interval between the anchor frame and the current frame; and merging the aligned frame and the anchor frame.

Description

Technical Field

[0001] The present invention relates to still frame image processing and more particularly to so-called multi-frame stacking techniques for producing an enhanced single image from multiple frames captured of the same scene. Background Art

[0002] Multi-frame stacking is commonly used in cameras for noise reduction, high dynamic range (HDR) imaging, super-resolution imaging, focus stacking, etc. In multi-frame pipeline processing, multiple consecutive frames are captured for the same scene at a given time interval. Among these frames, the so-called anchor frame is selected as the reference frame. The remaining frames are aligned with the reference frame and then merged.

[0003] More specifically, the multi-frame planning step decides the number of frames to use, the exposure of each frame, and which frame to use as the anchor.

[0004] After the planned multi-frame acquisition, the alignment step estimates the motion between each frame and the anchor frame. The resulting motion information is used to create a displacement map or optical flow for each frame with reference to the anchor frame. The displacement map includes the displacement vector for each pixel of the frame. Each frame is then aligned or registered with the anchor using its corresponding displacement map.

[0005] In the merging step, the aligned frames including the anchors are combined according to any of several possible techniques to produce an enhanced image.

[0006] For example, aligned frames can be averaged pixel by pixel for noise reduction. See, e.g., [Godard C, Matzen K, Uyttendaele M, 2018, "Deep burst denoising", European Conference on Computer Vision, pp. 560–577].

[0007] For high dynamic range images, frames are captured with different exposures. For each pixel of the enhanced image, matching pixels in the aligned frames are evaluated to select the pixel with the best exposure. See, for example, [Hasinoff SW, Sharlet D, Geiss R, Adams A, Barron JT et al., 2016, "Burst photography for high dynamic range and low-light imaging on mobile cameras", "ACM Transactions on Graphics", 35:192].

[0008] The generation of super-resolution images relies on some motion to create sub-pixels between the pixels of the frames. The motion may be generated internally by the camera's optical image stabilization device, which shifts the sensor or optics by half a pixel between frames, or the motion may be the inevitable motion of a handheld camera. Each frame is then aligned with a set of sub-pixels of an anchor frame involving minimal displacement, and the merging step involves creating sub-pixels in the enhanced image from the corresponding aligned frames. See, for example [Farsiu S, Elad M, Milanfar P, 2006, "Multiframe demosaicing and super-resolution of color images", "IEEE Trans. Image Processing", 15: 141–159].

[0009] Accurate motion estimation is important for good multi-frame stacking results. Wrong motion estimation may lead to wrong alignment between frames, which will affect the quality of the merging step, producing artifacts such as ghosting, double edges, texture loss, etc. Summary of the invention

[0010] An image enhancement method using multi-frame stacking generally includes the following steps: storing multiple frames of a target scene captured by a frame-based camera during an acquisition interval; storing a series of events generated by an event-based camera viewing the target scene during the acquisition interval; synchronizing timestamps of the events and timestamps of the frames; selecting an anchor frame from the multiple frames; aligning a current frame relative to the anchor frame based on stored events having timestamps corresponding to the interval between the anchor frame and the current frame; and merging the aligned frame and the anchor frame.

[0011] The aligning step may include the steps of: generating an optical flow of the current frame relative to the anchor frame based on the stored events; and creating an aligned frame by applying the optical flow to the current frame.

[0012] The step of selecting an anchor frame may include the steps of: for each frame, counting the number of stored events having a timestamp corresponding to an exposure interval of the frame; and selecting an anchor frame based on the event count.

[0013] The method may comprise the further step of subjecting the events to an ambient flicker filtering operation prior to the counting step.

[0014] The interval between the anchor frame and the current frame may be defined between the middle of the respective exposure intervals of the anchor frame and the current frame.

[0015] The method may comprise the further step of subjecting each frame to a motion deblurring operation based on a stored event having a timestamp corresponding to an exposure interval of the frame.

[0016] The step of generating the optical flow may include the step of applying a convolutional neural network to stored events having a timestamp corresponding to an interval between an anchor frame and a current frame.

[0017] A camera is also provided, which generally includes: a frame-based camera module, which is configured to store multiple frames captured from a target scene during an acquisition interval; an event-based camera module, which is configured to store a series of events generated by viewing the target scene during the acquisition interval, the events including timestamps synchronized with the timestamps of the frames; a control circuit, which operates on the stored frames and events, and the control circuit is configured to: select an anchor frame from multiple frames; align the current frame relative to the anchor frame based on events occurring in the interval between the anchor frame and the current frame; and merge the aligned frame and the anchor frame.

[0018] The camera may include an event counter configured to count, for each frame, a number of events that occur during an exposure interval of the frame; and a control circuit programmed to select the frame that produces the lowest event count as the anchor frame.

[0019] The camera may include a flicker filter configured to remove ambient flicker events from the events provided to the event counter.

[0020] The interval between the anchor frame and the current frame may be defined between the middle of the respective exposure intervals of the anchor frame and the current frame.

[0021] The camera may include a motion deblur filter configured to operate on each frame based on events occurring during an exposure interval of the frame.

[0022] The camera may include a convolutional neural network configured to generate an optical flow for a current frame based on events occurring in an interval between an anchor frame and the current frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments will be presented in the following description provided for exemplary purposes only, in conjunction with the accompanying drawings, in which:

[0024] Figure 1 is a schematic diagram of a multi-frame stacking pipeline according to an embodiment; and

[0025] Figure 2 is a timing diagram showing the operational details of the pipeline. DETAILED DESCRIPTION

[0026] As mentioned earlier, the accuracy of motion estimation implemented in the multi-frame stacking technique is important for obtaining good quality image enhancement. Frame-based motion estimation relies on the presence of sharp edges, i.e., high contrast transitions in the frames. When the acquired frames have motion blur, the edges become smooth and have lower contrast, which adversely affects the accuracy of motion estimation based on these frames.

[0027] Motion blur is mainly caused by excessive exposure time, which is generally selected by the camera to achieve a compromise between motion blur and noise, while generally favoring noise reduction. Shorter exposure times can be used with higher amplification of the signal (i.e., setting a higher ISO number), which reduces motion blur but also reduces the signal-to-noise ratio SNR.

[0028] In order to overcome the effect of motion blur on motion estimation, this paper provides an event-based camera module that views the same scene as a frame-based camera module and performs motion estimation on events occurring between consecutive frames. Event-based cameras are asynchronous and react much faster (microseconds) at the scale of exposure times (tens of milliseconds) involved in frame-based imaging, and do not suffer from motion blur.

[0029] Figure 1 is a schematic diagram of a multi-frame stacking pipeline using an event-based sensor according to an embodiment. The pipeline can be implemented in a camera or a smart phone. It includes a frame-based camera module 10 and an event-based camera module 12 arranged to view the same scene. A control circuit or microcontroller 13 can be programmed to perform or manage various operations in the pipeline, some of which, such as motion estimation, can be delegated to a hardware accelerator dedicated to image processing.

[0030] Preferably, the two camera modules are implemented as hybrid cameras, i.e., using a single sensor array comprising both frame-based pixels and event-based pixels. In such hybrid cameras, event and brightness signals are spatially registered. If spatially separated and off-axis cameras are used, additional conventional steps may be implemented to register events with corresponding pixels of the frame-based camera modules, such as disclosed in patent application EP4068207.

[0031] The frame-based camera module and the event-based camera module are connected to synchronize their timestamps. The resolution of the timestamps is at least the resolution of the events.

[0032] When the user presses the shutter button, the camera programs a multi-frame sequence, typically determining the number of frames, the corresponding exposure time interval, and the inter-frame interval or frame rate. A typical program may simply apply default settings for the number of frames and frame rate stored by the user, while automatically setting the exposure based on the conditions measured when the shutter button was pressed. Subsequently, the frame-based module 10 captures bursts of frames in the storage device 14 during the acquisition interval, while the event-based module 12 records all events that occurred during the acquisition interval in the storage device 16.

[0033] With the first frame used as an anchor to avoid "shutter lag", once the second or subsequent frames of the burst are stored, motion estimation 18 can be initiated based on the events recorded between the current frame i and the anchor frame, which is the first frame here. Any of several known motion estimation techniques that operate on events can be selected, such as a convolutional neural network (CNN) that generates an optical flow for the current frame, which is stored at 20. The optical flow typically includes, for each pixel of the current frame, a vector that displaces the pixel to a matching pixel in the anchor frame. An exemplary CNN is disclosed in [Zhu A., Yuan L., Chaney K., Danielidis K., "EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras", "Robotics: Science and Systems (RSS)", 2018].

[0034] Once the optical flow 20 of the current frame i is obtained, it can be used at 22 to align the current frame i with the anchor frame and store the corresponding aligned frame at 24. The alignment operation typically involves shifting each pixel of the current frame by the corresponding vector provided in the current optical flow i, whereby the shifted pixel occupies another pixel position in the aligned frame. This operation may result in the loss of some pixels at the frame edge when the vector crosses the frame edge.

[0035] There are alternative solutions for directly producing aligned frames without using optical flow or vectors, for example by using a time-shift operator disclosed in patent application EP4168977, or by using machine learning methods as described in [TULYAKOV Stepan, GEHRIG Daniel, GEORGOULIS Stamatios, et al., "Time lens: Event-based video frame interpolation", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16155-16164].

[0036] The aligned frames 24 and the anchor frames are merged at 26 to produce the final enhanced image. The merging technique used depends on the purpose of the multi-frame stacking (noise reduction, HDR, super-resolution, etc.). The aligned frames are similar to the frames produced by conventional multi-frame stacking techniques, so the merging technique can be any known technique. However, the aligned frames are the result of event-based rather than frame-based motion estimation, so they are significantly more accurate, especially when the frames contain motion blur or when the frames have different exposure times.

[0037] When the selected anchor frame is not the first frame of the burst, motion estimation may start as early as when the anchor frame is captured. In some applications, it is preferred to select the best anchor frame according to certain criteria, in which case the anchor frame is not necessarily the first frame. Such anchor selection methods are further disclosed below.

[0038] Figure 2 A timing diagram illustrating the pipeline operation is shown and certain aspects of its operation are highlighted.

[0039] The top or "frame" diagram illustrates the capture phase of a burst of five frames F1 to F5 of the frame-based module 10 starting at time t0, which is planned after the user presses the shutter button to initiate a multi-frame stacking operation. The acquisition time interval Tacq starts at t0 and ends at the end of the last frame F5. Each frame is exposed during an exposure time interval Texp and consecutive frames are separated by a time interval Tf. For example, when generating HDR images, the exposure time Texp of the frames can be different. The time intervals Tf are generally equal and as short as the capabilities of the frame-based module allow, while respecting the exposure time interval Texp.

[0040] Each frame may be stored with timing information, such as timestamps of the start and end of its exposure interval Texp.

[0041] The middle or "event" diagram illustrates exemplary events generated and stored by the event-based module 12 during the acquisition interval Tacq. As shown, events can occur at a significantly faster rate than frames. This example corresponds to a motion in the scene that accelerates until the end of frame F2 and then decelerates to accelerate again. Each event is typically stored with a timestamp, pixel coordinates, and polarity. As previously described, the timestamp is synchronized with the timing information stored with frames F1 to F5.

[0042] In this example, the anchor frame is F4*, which is selected as further explained below. Therefore, motion estimation can start when frame F4* is captured. The horizontal arrows below the event diagram overlay exemplary time intervals over which motion is estimated for each frame. The direction of the arrow is towards the anchor frame, indicating that the displacement vector is calculated to displace the pixels of the frame towards the matching pixels of the anchor frame. In addition, the arrow starts and ends in the middle of the exposure interval Texp, which means that the motion is estimated between the middle of the exposure interval, thereby representing the frame state at intermediate displacements during the corresponding exposure interval. This choice of motion estimation interval is particularly suitable when the frames have different exposure intervals such as for generating HDR images.

[0043] If the frames are for example stored with time stamps corresponding to the start and end of an exposure interval Texp, a median time stamp Tm corresponding to each exposure interval may be calculated as the average of the time stamps of the start and end of the exposure interval.

[0044] Then, more specifically, the optical flow of frames F1, F2, F3 and F5 is calculated using the stored events with timestamps between the paired intermediate timestamps (Tm1, Tm4), (Tm2, Tm4), (Tm3, Tm4) and (Tm4, Tm5), respectively. It should be noted that the optical flow calculated for each frame follows in principle the increasing time, resulting in a vector representing the displacement of each pixel of the earlier frame towards the pixel of the later frame. Therefore, for the frames before anchor F4*, the direction of the displacement vector is correct according to the standard calculation, as shown by the forward arrow. However, for frame F5, the anchor point F4* is an earlier frame, which means that the vector generated by the calculation point is in the opposite direction and should be inverted, as shown by the corresponding reverse arrow.

[0045] Since the motion of each frame of the burst is calculated relative to a common anchor frame, this anchor frame preferably has the best quality, and in particular is the sharpest in the burst. Detecting a sharp frame with an event-based camera happens to be particularly simple: it is the frame in which the fewest events occurred during its exposure time. In fact, the number of events occurring during a given time interval is a direct indication of the amount of motion, provided that spurious causes of event triggering, such as ambient flicker, are removed. Before counting the events during each exposure interval, ambient flicker can be compensated for using known techniques such as disclosed in patent application EP3744084.

[0046] Figure 2 The bottom figure of illustrates a method for selecting the sharpest anchor frame in terms of motion blur. The method is implemented during the initial multi-frame planning phase. The previously reset event counter is started at the beginning of each exposure interval Texp and stopped at the end of this exposure interval. The reached counts are stored within the acquisition interval Tacq. These counts are then compared and the frame with the lowest count is selected as the anchor frame. In the example shown, consecutive counts 7, 17, 8, 5 and 8 are obtained. The lowest count 5 corresponds to the frame F4 where the header of the anchor frame is obtained. Motion estimation can then start from the stored frames and events.

[0047] In some photography applications, it may be desirable to maximize sharpness in an area while leaving motion blur in other areas. This can be achieved by counting events in separate areas of the image plane, computing scores based on these areas and their corresponding counts, and selecting frames based on the best scores. An exemplary application is photography of a fast moving object (i.e., a cyclist), where the cyclist should be sharp (low event counts in the center area) while the background should be motion blurred (high event counts in the surrounding areas).

[0048] The optical flow has been calculated above independently for each frame, so that some calculations reuse events that have been used in previous calculations (as illustrated by the overlapping parts of the arrows below the event graph). As an alternative, more computationally efficient solution, the continuous-time optical flow can be predicted by feeding all events that occurred during the acquisition interval Tacq to a convolutional neural network, for example see [Gehrig M., Muglikar M., Scaramuzza D., "Dense Continuous-Time Optical Flow from Events and Frames", arXiv, 2022]. Such an operation creates a prediction function B(t,x,y) that produces a pixel displacement vector for any triplet of parameters (t,x,y), where x and y are the coordinates of the pixel to be mapped, and t is the timestamp of the frame to which the pixel is mapped. The anchor frame is then assumed to be the first frame of the burst.

[0049] The new coordinates of the displaced pixel X at time t are thus expressed as:

[0050] X'(t)=X+B(t,X), where X=(x,y) is the pixel to be mapped at time t=0.

[0051] If at time t aIf the anchor frame is selected at time t, then the displacement of the reference anchor frame at time t is the displacement from the anchor frame to the first frame (at t=0) plus the displacement from the first frame to the frame at time t, so:

[0052] B -1 (t a ,X a )+B(t,B -1 (t a ,X a )),

[0053] Among them B -1 is the inverse function of B, X a =(x a ,y a ) is to be at time t a The pixel mapped in the anchor frame, and B -1 (t a ,X a ) is the mapped pixel X a Displacement vector to the first frame at time t=0.

[0054] The new coordinates of the displaced pixel X' at time t are then expressed by:

[0055] X'(t)=X a +B -1 (t a ,X a )+B(t,B -1 (t a ,X a )).

[0056] Although motion blur in a frame does not affect the accuracy of event-based motion estimation, merging over-blurred frames may still have an adverse effect. Events can be effectively used to remove motion blur, such as disclosed in patent application EP4168977. Therefore, after capturing a current frame with a corresponding event during an exposure interval, motion blur can be compensated by using such a deblurring technique using events before the merging operation 26.

[0057] The resolution of the frame-based module and the event-based module may be different. The event-based sensor array may have a lower resolution than the frame-based sensor. Although the motion estimation calculation may not be affected, the accuracy of the displacement vector may be at most the same as the resolution of the event-based sensor. It is common practice in frame-based algorithms to work at a much lower resolution than the initial image, see [Hasinoff SW, Sharlet D, Geiss R, Adams A, Barron JT et al., 2016, "Burst photography for high dynamic range and low-light imaging on mobile cameras", "ACM Transactions on Graphics" 35:192].

Claims

1. An image enhancement method using multi-frame stacking, the method comprising the following steps: storing a plurality of frames of a target scene captured by a frame-based camera (10) during an acquisition interval (Tacq); storing a series of events generated by an event-based camera (12) viewing the target scene during the acquisition interval; synchronizing a timestamp of the event with a timestamp of the frame; selecting an anchor frame from the plurality of frames; aligning the current frame relative to the anchor frame based on a stored event (16) having a timestamp corresponding to an interval between the anchor frame and the current frame; as well as The aligned frames and the anchor frame are merged.

2. The method of claim 1, wherein the aligning step comprises: Based on the stored events (16), generating an optical flow (20) of the current frame relative to the anchor frame; as well as An aligned frame is created by applying the optical flow to the current frame (24).

3. The method according to claim 1, wherein the step of selecting the anchor frame comprises: For each frame, counting the number of stored events having a timestamp corresponding to an exposure interval (Texp) of the frame; as well as The anchor frame is selected based on the event count.

4. A method according to claim 3, comprising a further step of subjecting the events to an ambient flicker filtering operation before the counting step.

5. The method of claim 1, wherein the interval between the anchor frame and the current frame is defined between the middle (Tm) of the respective exposure intervals of the anchor frame and the current frame.

6. The method according to claim 1, comprising the further step of subjecting each frame to a motion deblurring operation based on a stored event having a timestamp corresponding to an exposure interval of the frame.

7. The method of claim 1, wherein the step of generating the optical flow comprises applying a convolutional neural network to stored events having a timestamp corresponding to an interval between the anchor frame and the current frame.

8. A camera, comprising: a frame-based camera module (10) configured to store a plurality of frames captured from a target scene during an acquisition interval (Tacq); an event-based camera module (12) configured to store a series of events generated by viewing the target scene during the acquisition interval, the events including time stamps synchronized with time stamps of the frames; A control circuit (13) operates on the stored frames and events and is configured to: selecting an anchor frame from the plurality of frames; aligning the current frame relative to the anchor frame based on an event (16) occurring in an interval between the anchor frame and the current frame; as well as The aligned frames and the anchor frame are merged.

9. The camera of claim 8, comprising: an event counter configured to count, for each frame, a number of events that occur during an exposure interval (Texp) of the frame; and A control circuit is programmed to select the frame that produces the lowest event count as the anchor frame.

10. The camera of claim 9, comprising a flicker filter configured to remove ambient flicker events from the events provided to the event counter.

11. The camera of claim 8, wherein the interval between the anchor frame and the current frame is defined between the middle (Tm) of the respective exposure intervals of the anchor frame and the current frame.

12. The camera of claim 8, comprising a motion deblur filter configured to operate on each frame based on events occurring during an exposure interval of the frame.

13. The camera of claim 8, comprising a convolutional neural network configured to generate the optical flow of the current frame based on events occurring in an interval between the anchor frame and the current frame.

Images