Event-based motion estimation for multi-frame stacking

Abstract

To improve the precision of motion estimation by reducing an influence of a motion blur of multi-frame stacking.SOLUTION: An image emphasis method using multi-frame stacking includes the steps of: storing a plurality of frames of a target scene captured by a frame-based camera (10) during acquisition intervals (Tacq); storing a series of events generated during the acquisition intervals by an event-based camera (12) viewing the target scene; synchronizing time stamps of the events with time stamps of the frames; selecting an anchor frame out of the plurality of frames; arraying a current frame for the anchor frame based upon stored events (16) having time stamps corresponding to the interval between the anchor frame and current frame; and merging the arrayed frame and anchor frame.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to still frame image processing, and more particularly to the so-called multi-frame stacking technique, which generates a single enhanced image from multiple frames captured of the same scene. [Background technology]

[0002] Multi-frame stacking is typically used in cameras for noise reduction, high dynamic range (HDR) imaging, super-resolution imaging, focus stacking, etc. In a multi-frame pipeline process, multiple consecutive frames are captured of the same scene in a given time interval. Among these frames, a so-called anchor frame is selected as a reference frame. The remaining frames are aligned with the reference frame and then merged.

[0003] More specifically, the multi-frame planning step determines how many frames to use, the exposure of each frame, and which frames to use as anchors.

[0004] After planned multi-frame acquisition, an alignment step estimates the motion between each frame and the anchor frame. The resulting motion information is used to create a displacement map or optical flow for each frame referenced to the anchor frame. The displacement map includes a displacement vector for each pixel of the frame. Each frame is then aligned or registered with the anchor using its respective displacement map.

[0005] In the merging step, the aligned frames containing the anchors are combined to produce an image that is enhanced according to any of several possible techniques.

[0006] For example, for noise reduction, the aligned frames can be averaged pixel by pixel, see, for example, [Godard C, Matzen K, Uyttendaele M., 2018, “Deep burst denoising”, European Conference on Computer Vision, pp. 560-577].

[0007] For high dynamic range images, frames are captured at different exposures. For each pixel in the enhanced image, the corresponding pixels in the aligned frames are evaluated and the pixel with the best exposure is selected. See, for example, [Hasinoff SW, Sharlet D, Geiss R, Adams A, Barron JT et al., 2016, "Burst photography for high dynamic range and low-light imaging on mobile cameras", ACM Transactions on Graphics 35:192].

[0008] The generation of super-resolution images relies on some motion to create sub-pixels between pixels of a frame. This motion can be created internally by the camera's optical image stabilization device, which shifts the sensor or optics by half a pixel between frames, or it can be the inevitable motion of a handheld camera. Each frame is then aligned with the set of sub-pixels of the anchor frame with the smallest displacement, and the merging step involves creating sub-pixels in the enhanced image from the correspondingly aligned frames. See, for example, [Farsiu S, Elad M, Milanfar P., 2006, "Multiframe demosaicing and super-resolution of color images", IEEE Trans. Image Processing 15:141-159].

[0009] Accurate motion estimation is important for good multi-frame stacking results: incorrect motion estimation can lead to incorrect alignment between frames, which impairs the quality of the merging step and causes artifacts such as ghosting, double edges, and texture loss. Summary of the Invention

[0010] A method for image enhancement using multi-frame stacking generally includes the steps of storing multiple frames of a target scene captured by a frame-based camera during an acquisition interval, storing a series of events generated during the acquisition interval by an event-based camera viewing the target scene, synchronizing timestamps of the events and timestamps of the frames, selecting an anchor frame from among the multiple frames, aligning the current frame with respect to the anchor frame based on a stored event having a timestamp corresponding to the interval between the anchor frame and the current frame, and merging the aligned frames and the anchor frame.

[0011] The aligning step may include generating an optical flow of the current frame relative to the anchor frame based on the stored events, and creating the aligned frame by applying the optical flow to the current frame.

[0012] The step of selecting an anchor frame may include counting, for each frame, the number of stored events having a timestamp corresponding to the exposure interval of the frame, and selecting the anchor frame based on the event count.

[0013] The method may include the further step of subjecting the events to an ambient flicker filtering operation prior to the counting step.

[0014] The interval between the anchor frame and the current frame may be defined midway between the exposure intervals of the anchor frame and the current frame.

[0015] The method may include the further step of subjecting each frame to a motion deblurring operation based on stored events having a timestamp corresponding to the exposure interval of the frame.

[0016] The step of generating the optical flow may include applying a convolutional neural network to stored events having timestamps corresponding to the interval between the anchor frame and the current frame.

[0017] Generally, a camera is provided that includes a frame-based camera module configured to store a plurality of frames captured from a target scene during an acquisition interval; an event-based camera module configured to store a series of events triggered by viewing the target scene during the acquisition interval, the events including timestamps synchronized with the timestamps of the frames; and a control circuit that operates on the stored frames and events, the control circuit being configured to: select an anchor frame from among the plurality of frames, align the current frame to the anchor frame based on events occurring in an interval between the anchor frame and the current frame, and merge the aligned frame and the anchor frame.

[0018] The camera may include an event counter configured to count, for each frame, the number of events that occur during the exposure interval of the frame, and control circuitry programmed to select the frame producing the lowest event count as the anchor frame.

[0019] The camera may be equipped with a flicker filter configured to remove ambient flicker events from the events provided to the event counter.

[0020] The interval between the anchor frame and the current frame may be defined midway between the exposure intervals of the anchor frame and the current frame.

[0021] The camera may include a motion deblur filter configured to operate on each frame based on events that occur during the exposure interval of the frame.

[0022] The system may include a convolutional neural network configured to generate optical flow for the current frame from events occurring in the interval between the anchor frame and the current frame. [Brief description of the drawings]

[0023] Embodiments are set forth in the following description, given by way of example only, in connection with the accompanying drawings, in which: [Figure 1] FIG. 2 is a schematic diagram of a multi-frame stacking pipeline, according to one embodiment. [Diagram 2] FIG. 2 is a timing diagram detailing the operation of the pipeline. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] As mentioned above, the accuracy of the motion estimation performed in the multi-frame stacking technique is important to obtain good quality image enhancement. Frame-based motion estimation relies on the presence of sharp edges, i.e., high-contrast transitions within a frame. When the captured frames have motion blur, the edges become smooth and have lower contrast, which adversely affects the accuracy of the motion estimation based on these frames.

[0025] Motion blur is mainly caused by excessive exposure times, which are usually chosen by the camera to achieve a compromise between motion blur and noise, with noise reduction usually taking precedence. Shorter exposure times may be used together with higher amplification of the signal (i.e. setting a higher ISO number), which reduces not only motion blur but also the signal-to-noise ratio SNR.

[0026] To overcome the impact of motion blur on motion estimation, an event-based camera module is used herein that views the same scene as the frame-based camera module, and performs motion estimation on events that occur between successive frames. Event-based cameras are asynchronous, react much faster (on the order of microseconds) on the scale of exposure times involved in frame-based imaging (tens of milliseconds), and are not affected by motion blur.

[0027] 1 is a schematic diagram of a multi-frame stacking pipeline using an event-based sensor according to one embodiment. The pipeline may be implemented in a camera or a smartphone. The pipeline includes a frame-based camera module 10 and an event-based camera module 12 configured to view the same scene. A control circuit or microcontroller 13 may be programmed to perform or manage various operations in the pipeline, and some of the operations, such as motion estimation, may be allocated to a hardware accelerator dedicated to image processing.

[0028] Preferably, both camera modules are implemented as hybrid cameras, i.e. using a single sensor array that includes both frame-based and event-based pixels. In such hybrid cameras, the event and luminance signals are spatially aligned. If a spatially separated off-axis camera is used, an additional conventional step may be performed to align the events to the corresponding pixels of the frame-based camera module, as disclosed in European Patent Application No. 4068207.

[0029] The frame-based camera module and the event-based camera module are connected to synchronize their timestamps, the resolution of the timestamps being at least the resolution of the events.

[0030] When the user presses the shutter button, the camera plans a multi-frame sequence, generally determining the number of frames, the corresponding exposure time interval, and the interframe interval or frame rate. A typical plan might simply apply default settings stored by the user for the number of frames and frame rate, while automatically setting the exposure based on the conditions measured at the time the shutter button was pressed. The frame-based module 10 then captures a burst of frames in storage device 14 during the acquisition time interval, and the event-based module 12 records in storage device 16 all events that occur during the acquisition interval.

[0031] If the first frame is used as an anchor to avoid "shutter lag", as soon as the second or subsequent frame of the burst is stored, motion estimation 18 can be started based on events recorded between the current frame i and the anchor frame, here the first frame. Any of several known motion estimation techniques operating on events can be chosen, such as a convolutional neural network (CNN), which generates an optical flow for the current frame that is stored at 20. The optical flow typically includes, for each pixel of the current frame, a vector that displaces that pixel to a matching pixel in the anchor frame. An exemplary CNN is disclosed in [Zhu, A., Yuan, L., Chaney, K., Daniilidis, K., "EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras", Robotics: Science and Systems (RSS), 2018].

[0032] Once the optical flow 20 for the current frame i is prepared, it can be used at 22 to align the current frame i with the anchor frame and store the corresponding aligned frame at 24. This alignment operation typically involves displacing each pixel of the current frame by the corresponding vector provided in the current optical flow i, so that the displaced pixel occupies a different pixel location in the aligned frame. This operation may cause the loss of some pixels at the edges of the frame when the vector crosses the frame edge.

[0033] Alternative solutions exist that generate aligned frames directly without using optical flow or vectors, for example by using a time shift operator as disclosed in European Patent Application No. 4168977, or by using machine learning methods as described in [TULYAKOV, Stepan, GEHRIG, Daniel, GEORGOULIS, Stamatios et al., "Time lens: Event-based video frame interpolation", Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2021, pp. 16155-16164].

[0034] The aligned frames 24 and the anchor frame are merged at 26 to generate a final enhanced image. The merging technique used depends on the purpose of the multi-frame stacking (noise reduction, HDR, super-resolution, etc.). The aligned frames are similar to frames generated using conventional multi-frame stacking techniques, and therefore the merging technique can be any known technique. However, the aligned frames are the result of motion estimation that is based on events rather than frames, and therefore they are significantly more accurate, especially when the frames contain motion blur or have different exposure times.

[0035] If the selected anchor frame is not the first one in the burst, motion estimation may start at the earliest point in time at which the anchor frame is captured. In some applications, it is preferable to select the optimal anchor frame according to specific criteria, in which case the anchor frame is not necessarily the first. Such anchor selection methods are disclosed further below.

[0036] FIG. 2 shows a timing diagram of the operation of the pipeline, highlighting specific aspects of its operation.

[0037] The top or "frame" diagram shows the capture phase of a burst of five frames F1-F5 by the frame-based module 10 from time t0, planned after the user presses the shutter button to start the multi-frame stacking operation. The acquisition time interval Tacq starts at t0 and ends at the end of the last frame F5. Each frame is exposed during the exposure time interval Texp, with successive frames separated by a time interval Tf. The exposure times Texp of the frames can be different, for example when generating HDR images. The time interval Tf is usually equal to and as short as possible, while taking into account the exposure time interval Texp.

[0038] Each frame may be stored along with timing information, for example timestamps for the start and end of its exposure interval Texp.

[0039] The center or "event" diagram shows exemplary events that are generated and stored by the event-based module 12 during the acquisition interval Tacq. As shown, events can occur at a rate significantly faster than the frames. This example corresponds to motion in a scene that accelerates until the end of frame F2, then decelerates and accelerates again. Each event is typically stored with a timestamp, pixel coordinates, and polarity. As previously mentioned, the timestamps are synchronized with the timing information stored with frames F1-F5.

[0040] In this example, the anchor frame is F4 * and is selected as explained further below. Therefore, motion estimation is performed for frame F4 * is captured. The horizontal arrows below this event diagram span an exemplary time interval during which motion is estimated for each frame. The direction of the arrow is toward the anchor frame, indicating that a displacement vector is calculated to displace a pixel of the frame toward a corresponding pixel of the anchor frame. Furthermore, the arrows start and end at the center of the exposure interval Texp, meaning that motion is estimated between the centers of the exposure intervals, and represent a frame state at the center displacement during the corresponding exposure interval. Such a selection of motion estimation intervals is particularly well suited when frames have different exposure intervals, such as for generating HDR images.

[0041] If the frames are stored, for example, with timestamps corresponding to the start and end of the exposure interval Texp, the timestamp Tm corresponding to the middle of each exposure interval can be calculated as the average of the timestamps of the start and end of the exposure interval.

[0042] More specifically, optical flows for frames F1, F2, F3, and F5 are then calculated using stored events with timestamps between the intermediate timestamp pairs (Tm1, Tm4), (Tm2, Tm4), (Tm3, Tm4), and (Tm4, Tm5), respectively. Note that the optical flows calculated for each frame essentially produce vectors representing the displacement of each pixel of the earlier frame towards a pixel of the later frame according to increasing time. Thus, the anchor F4 * For frames preceding anchor F4, the direction of the displacement vector is correct from standard calculations, as indicated by the forward arrow. However, for frame F5, anchor F4 *is an earlier frame, which means that the vector produced by the calculation points in the opposite direction and should be reversed, as indicated by the corresponding backward arrow.

[0043] Since the motion for each frame of a burst is calculated with respect to a common anchor frame, that anchor frame is preferably of the best quality, and in particular the sharpest within the burst. Detecting a sharp frame with an event-based camera happens to be particularly simple: it is the frame with the fewest events occurring during its exposure time. In fact, the number of events occurring during a given time interval is a direct indicator of the amount of motion, if spurious causes of event triggering, such as ambient flicker, are eliminated. Ambient flicker can be compensated for using known techniques, such as those disclosed in European Patent Application No. 3744084, before counting the events during each exposure interval.

[0044] The bottom graph of Fig. 2 shows the method of selecting the sharpest anchor frame in terms of motion blur. This method is implemented during the initial multiframe planning phase. A previously reset event counter is started at the beginning of each exposure interval Texp and stopped at the end of the exposure interval. The counts reached are stored over the acquisition interval Tacq. The counts are then compared and the frame with the lowest count is selected as the anchor frame. In the illustrated example, successive counts of 7, 17, 8, 5 and 8 are obtained. The lowest count of 5 corresponds to frame F4, which acquires the role of anchor frame. Motion estimation can then start from the stored frames and events.

[0045] In some photography applications, one may want to maximize sharpness in some areas while leaving motion blur in others. This can be achieved by counting events in separate regions of the image plane, computing a score based on the regions and their respective counts, and selecting the frame based on the best score. An example application is the photography of a fast moving object (i.e., a cyclist), where the cyclist should be sharp (low event count in the central region) and the background should be motion blurred (high event count in the surrounding regions).

[0046] The optical flow has been calculated above independently for each frame, making some calculations reuse events already used in previous calculations (as indicated by the overlapping arrows under the event graph). As an alternative, more computationally efficient solution, the continuous-time optical flow can be predicted by feeding all events occurring during the acquisition interval Tacq to a convolutional neural network, see for example [Gehrig, M., Muglikar, M., Scaramuzza, D., "Dense Continuous-Time Optical Flow from Events and Frames", arXiv, 2022]. Such an operation creates a prediction function B(t, x, y) that generates a pixel displacement vector for any triplet of parameters (t, x, y), where x and y are the coordinates of the pixel to map and t is the timestamp of the frame to which the pixel is mapped. The anchor frame is then assumed to be the beginning of the burst.

[0047] Therefore, the new coordinates of the displaced pixel X' at time t are expressed as: X'(t)=X+B(t,X), where X=(x,y) is the pixel to be mapped at time t=0.

[0048] The anchor frame is at time t aIf the first frame is selected at t=0, then the displacement at time t relative to the anchor frame is the displacement from the anchor frame to the first frame (at t=0) plus the displacement from the first frame to the frame at time t, thus: B -1 (t a ,X a )+B(t,B -1 (t a ,X a )) Here, B -1 is the inverse function of B, and X a =(x a y a ) at time t a is the pixel to be mapped into the anchor frame in B -1 (t a ,X a ) is pixel X a is the displacement vector that maps

[0049] Therefore, the new coordinates of the displaced pixel X' at time t are expressed as: X'(t)=X a +B -1 (t a ,X a )+B(t,B -1 (t a ,X a ))

[0050] Although motion blur in a frame does not affect the accuracy of event-based motion estimation, merging excessively blurred frames may still produce undesired effects. As disclosed in European Patent Application No. 4168977, there are cases where events can be effectively used to remove motion blur. Thus, after a current frame has been captured with a corresponding event during an exposure interval, the motion blur can be compensated for before the merging operation 26 by using such deblurring techniques with the event.

[0051] The resolution of the frame-based and event-based modules may be different. The event-based sensor array may have a lower resolution than the frame-based sensor. The motion estimation calculation may not be affected, but the accuracy of the displacement vectors may be at most the same as the resolution of the event-based sensor. It is common practice for frame-based algorithms to work at a significantly lower resolution than the original image, see [Hasinoff SW, Sharlet D, Geiss R, Adams A, Barron JT et al., 2016, "Burst photography for high dynamic range and low-light imaging on mobile cameras", ACM Transactions on Graphics 35:192].

Claims

1. 1. A method for image enhancement using multi-frame stacking, the method comprising: storing a number of frames of a target scene captured by a frame-based camera (10) during an acquisition interval (Tacq); storing a sequence of events generated during the acquisition interval by an event-based camera (12) viewing the target scene; synchronising the event timestamps and the frame timestamps; selecting an anchor frame from the plurality of frames; aligning the current frame with respect to the anchor frame based on stored events (16) having a timestamp corresponding to an interval between the anchor frame and the current frame; and b. merging the aligned frame and the anchor frame.

2. The aligning step further comprises: generating an optical flow (20) of a current frame relative to the anchor frame based on the stored events (16); and creating the aligned frame (24) by applying the optical flow to the current frame.

3. the step of selecting the anchor frame comprises: For each frame, counting the number of stored events having a timestamp corresponding to the exposure interval (Texp) of said frame; and selecting the anchor frame based on the count of events.

4. The method of claim 3 , comprising the further step of subjecting said events to an ambient flicker filtering operation prior to said counting step.

5. The method of claim 1 , wherein the interval between the anchor frame and the current frame is defined during a midpoint (Tm) of an exposure interval of each of the anchor frame and the current frame.

6. 2. The method of claim 1, comprising the further step of subjecting each frame to a motion deblurring operation based on stored events having a timestamp corresponding to an exposure interval of the frame.

7. 2. The method of claim 1 , wherein the generating the optical flow step comprises applying a convolutional neural network to stored events having timestamps corresponding to intervals between the anchor frame and the current frame.

8. A camera, a frame-based camera module (10) configured to store a plurality of frames captured from a target scene during an acquisition interval (Tacq); an event-based camera module (12) configured to store a series of events triggered by viewing the target scene during the acquisition interval, the events including timestamps synchronized with timestamps of the frames; a control circuit (13) operating on said stored frames and events, selecting an anchor frame from the plurality of frames; aligning the current frame with respect to the anchor frame based on events (16) occurring in an interval between the anchor frame and the current frame; and a control circuit (13) configured to merge the aligned frame and the anchor frame.

9. an event counter configured to count, for each frame, the number of events occurring during an exposure interval (Texp) of said frame; The camera of claim 8 , wherein the control circuitry is programmed to select as the anchor frame the frame that produces the lowest event count.

10. The camera of claim 9 , comprising a flicker filter configured to remove ambient flicker events from the events provided to the event counter.

11. The camera of claim 8 , wherein the interval between the anchor frame and the current frame is defined between the midpoint (Tm) of an exposure interval of each of the anchor frame and the current frame.

12. 10. The camera of claim 8, comprising a motion deblur filter configured to operate on each frame based on events occurring during an exposure interval of the frame.

13. 9. The camera of claim 8, comprising a convolutional neural network configured to generate the optical flow for the current frame from events occurring in an interval between the anchor frame and the current frame.

Images