Event-based motion estimation for auto exposure determination

Abstract

The overall image acquisition method includes the following steps: providing a frame-based camera (10) with a field of view; storing images by an event-based camera (12) during the acquisition time interval (Δ). T, Texp A series of events generated during the acquisition interval, the event-based camera views the target scene within the field of view of the frame-based camera; motion in the target scene is estimated based on events stored during the acquisition interval (16); the exposure time is determined based on the estimated motion ( Texp ); and apply the exposure time to a frame-based camera (10) for capturing frames.

Description

Technical Field

[0001] This invention relates to still frame image processing, and more particularly to a technique for adjusting the exposure time of frames using motion estimation. Background Technology

[0002] Scientific paper [Liba O. et al., " Handheld mobile photography in very low light [Article 164, Vol. 38, No. 6, ACM Trans. Graph (November 2019)] discloses a low-light image processing technique based on capturing, aligning, and combining multiple frames. The system employs "motion metrology," which uses estimates of motion amplitude (whether due to hand shakiness or moving objects) to identify the frame number and exposure time per frame. Together, the frame number and exposure time minimize both noise and motion blur during the capture burst. These frames are combined using robust alignment and merging techniques specifically designed for high-noise images.

[0003] Accurate motion estimation is crucial for obtaining satisfactory results. Incorrect motion estimation can lead to incorrect calculation of exposure time and misalignment between frames, which will impair the quality of the combination step and produce artifacts such as ghosting, double-sided images, and texture loss. Summary of the Invention

[0004] A general image acquisition method is provided, comprising the following steps: providing a frame-based camera with a field of view; storing a series of events generated during an acquisition time interval by an event-based camera, the event-based camera viewing a target scene within the field of view of the frame-based camera; estimating motion in the target scene based on the events stored during the acquisition time interval; determining an exposure time based on the estimated motion; and applying the exposure time to the frame-based camera for frame capture.

[0005] The duration of the acquisition interval can be dynamically determined according to the following steps: starting the acquisition interval; continuously calculating the motion score based on the stored events; and terminating the acquisition interval when the motion score reaches a threshold standard.

[0006] The method may include the following steps: a) initiating exposure of a frame-based camera when an acquisition interval is started; b) configuring motion estimation to generate the maximum motion amplitude in the scene as a motion score; and c) stopping exposure when the acquisition interval ends.

[0007] The motion score can be the number of events.

[0008] The method may include the following steps: setting a threshold to a value that causes motion blur that can be compensated by a motion deblurring operation for the corresponding maximum motion amplitude; and performing a motion deblurring operation on the frame based on events stored during the exposure time interval of the frame.

[0009] The method may include the following steps: repeating steps a), b) and c) multiple times to capture multiple frames; synchronizing the timestamps of events and frames; selecting an anchor frame from the multiple frames; aligning the current frame relative to the anchor frame based on a stored event corresponding to the interval between the anchor frame and the current frame; and merging the aligned frames and the anchor frame.

[0010] The alignment step may include the following steps: generating an optical flow of the current frame relative to the anchor frame based on stored events; and creating an aligned frame by applying the optical flow to the current frame.

[0011] Anchor frames can be selected as frames that achieve the minimum to maximum motion amplitude.

[0012] The steps of generating optical flow may include applying a convolutional neural network to stored events corresponding to the interval between the anchor frame and the current frame.

[0013] A camera is also provided, comprising: a frame-based camera module; an event-based camera module storing a series of events generated from a target scene in the field of view of the frame-based camera module during an acquisition interval; a motion estimator configured to estimate motion in the target scene based on the events stored during the acquisition interval; and control circuitry operating on the stored events, the control circuitry being configured to: determine an exposure time based on the estimated motion; and apply the exposure time to the frame-based camera module for capturing frames.

[0014] The control circuit can be configured to: initiate a dynamic acquisition interval; configure a motion estimator to continuously calculate motion scores based on stored events; and terminate the acquisition interval when the motion score reaches a threshold criterion.

[0015] The control circuit can be further configured to: initiate exposure of the frame-based camera module when the acquisition interval is started; configure the motion estimator to generate the maximum motion amplitude in the scene as the motion score; and stop exposure when the acquisition interval ends.

[0016] Frame-based camera modules and event-based camera modules can be a single hybrid camera module comprising a single sensor array that includes both frame-based pixels and event-based pixels. Attached Figure Description

[0017] The embodiments will be illustrated in the following description, provided for illustrative purposes only, with reference to the accompanying drawings, in which: Figure 1 [ Figure 1 This is a schematic diagram illustrating the pipeline determination based on the frame exposure time of an event-based sensor in the implementation scheme. Figure 2 [ Figure 2 This is a sequence diagram illustrating an implementation using dynamically adjusted event acquisition timing. Figure 3 [ Figure 3 [This is a timing diagram illustrating an implementation using dynamically adjusted exposure time; and] Figure 4 [ Figure 4 ] is used Figure 3 A schematic diagram of an implementation scheme for a multi-frame stacking pipeline with dynamically adjusted exposure times. Detailed Implementation

[0018] Frame-based motion estimation relies on the presence of sharp edges in consecutive frames, i.e., high-contrast transitions within frames. When the acquired frames have motion blur, the edges become smooth and have lower contrast due to the long exposure time, which adversely affects the accuracy of motion estimation based on these frames.

[0019] To reduce motion blur, shorter exposure times can be used, but this requires higher signal gain (or ISO number) to obtain correctly exposed frames, thus reducing the signal-to-noise ratio (SNR). In noise reduction applications, using frames with lower SNR requires more frames to achieve satisfactory results, thus increasing latency and computational resources.

[0020] To overcome the impact of motion blur on motion estimation, this paper presents a method using an event-based camera module to view the same scene as a frame-based camera module and perform motion estimation on events generated by that scene. The event-based camera is asynchronous and reacts much faster (microseconds) at the exposure time scale (tens of milliseconds) involved in frame-based imaging, and is not subject to motion blur.

[0021] Regarding motion estimation for determining exposure time, the asynchronous and real-time nature of event data generated by event-based cameras can be used online to dynamically adjust parameters, such as adjusting frame exposure time when capturing frames.

[0022] [ Figure 1This is a schematic diagram illustrating a pipeline that uses frame exposure time determined by an event-based sensor according to an implementation scheme. This pipeline can be implemented in a camera or smartphone. It includes a frame-based camera module 10 and an event-based camera module 12 arranged to view the same scene. Control circuitry or a microcontroller 13 can be programmed to perform or manage various operations in the pipeline, where some operations (such as motion estimation) can be delegated to a hardware accelerator dedicated to image processing.

[0023] Events generated by the event-based module 12 are stored in buffer 14, where they can be analyzed by the motion estimator 16. For example, during the acquisition time interval Δ... T Motion is analyzed within the frame, and the acquisition interval can be shorter than the typically achieved exposure time or at least shorter than the frame rate period achievable by the camera, and long enough to capture sufficient event data for accurate motion amplitude estimation. Since motion estimation is used to determine the exposure time, the motion amplitude is sufficient to replace optical flow (i.e., a vector), which simplifies the calculations involved. The motion amplitude for each pixel is stored as shown in Figure 18. Figure 18 is analyzed at point 20 to determine the exposure time to be used by the frame-based camera module 10. The camera module 10 can apply this exposure time to capture the current frame or a burst of multiple frames 22. Multiple frames can then be used for noise reduction techniques or other multi-frame stacking techniques.

[0024] Exposure time can be set to less than or equal to: Texp = Δ T•B / M max , Where B is the maximum expected blur value in pixels, and M max The maximum motion amplitude, also measured in pixels, is found in Figure 18.

[0025] To improve robustness, amplitudes can be filtered during online determination to remove noise and outliers. The resolution of the amplitude map and the event-based camera can be lower than that of the frame-based camera, since global motion estimation within the scene is sufficient.

[0026] [ Figure 2 ] is an example of using dynamically adjusted event acquisition time Δ T The timing diagram of the implementation scheme. When the user triggers frame capture, the camera starts the current event acquisition time interval. ΔT It then begins recording events from event-based module 12. As events are recorded in buffer 14, event statistics are collected, and motion estimation 16 is performed in parallel for the same events.

[0027] The statistics used are customized to generate motion scores that represent the amount of motion, such as the number of events counted since the start of the acquisition interval.

[0028] When the motion score reaches the threshold standard, the data collection interval is stopped. ΔT Motion estimation 16 performs operations on the events stored up to that point, thereby generating, for example, a corresponding amplitude diagram 18, from which the exposure time is determined. Texp The frame-based camera module 10 is configured with a defined exposure time. The camera module 10 can then capture frames while simultaneously setting the corresponding aperture (if it is adjustable) and the corresponding gain (ISO number) for proper exposure.

[0029] If the score does not reach the threshold, it means there is almost no movement in the scene; when the maximum sampling interval is reached... ΔT max At that time, the data collection will be forcibly stopped.

[0030] A typical motion score can be the number of events in the thousands, while ΔT max It can be tens of milliseconds.

[0031] By using this process, the acquisition time interval can be dynamically adjusted to a minimum, thereby providing sufficient event data for satisfactory motion estimation and reducing the waiting time between the time the user triggers the process and the time the first frame is captured.

[0032] However, the exposure time is therefore determined as an estimate based on a pre-analysis of motion in the scene and is applied only after a certain waiting time to capture frames or frame bursts. During the waiting time, although the process tends to reduce the waiting time, conditions may change, especially the introduction of more motion or less light, thus the exposure time may no longer be applicable.

[0033] [ Figure 3 This is a timing diagram illustrating an implementation scheme that provides real-time dynamic adjustment of exposure for the current frame. When the user triggers frame capture, the camera initiates an exposure time interval applied to the frame-based module 10. Texp That is, the frame-based module 10 is controlled to operate within a time interval. Texp The current frame is exposed during this period. Simultaneously, event-based module 12 is controlled to record events. While events are recorded in buffer 14, motion estimation 16 is performed online, incrementally updating the amplitude map 18.

[0034] When the amplitude in the graph reaches the threshold corresponding to the maximum desired motion blur B, the running exposure time interval is adjusted. TexpStop, thus halting the exposure of the current frame. Then store the current frame, which can be post-processed.

[0035] If the amplitude does not reach the threshold, it means there is almost no movement in the scene; when the maximum exposure interval is reached... Texp max When this happens, the exposure is forcibly stopped to avoid overexposure.

[0036] value Texp max A frame-based camera module can automatically set the exposure time based on the lighting conditions at the time the process is triggered, along with the aperture (if it is adjustable) and the gain (ISO number) for proper exposure.

[0037] Therefore, in motion blur applications, the optimal exposure time is determined and applied in real time. Under low light and high motion conditions, the exposure time may typically be shorter than [previous value]. Texp max This compensates for the "missing" exposure time by increasing the signal gain after the exposure interval. This, in turn, reduces the SNR. However, the SNR is reduced to just enough to satisfy the motion blur target.

[0038] As a less accurate alternative, motion estimation stage 16 can be used [ Figure 2 Instead, a statistical score is calculated, and thus, when the score reaches a threshold, such as when the event count reaches a threshold number, the exposure interval is adjusted. Texp Termination. The advantage of this alternative is its lower computational density compared to maintaining an incremental motion amplitude map, but it is more prone to error because a large number of events does not necessarily represent large-amplitude motion. For example, objects in a scene may oscillate at high frequencies with low amplitudes; in such cases, the amplitude threshold will not be reached when using auto-estimation.

[0039] To relax motion blur constraints and thus improve SNR, the maximum motion blur value B can be increased to a level that can be satisfactorily compensated for by motion deblurring techniques. In practice, events can be effectively used to remove motion blur, as disclosed in patent application EP4168977. In this case, the incremental motion estimation performed on the event is configured to generate optical flow, not just an amplitude map. Motion blur in the current frame can then be compensated based on the optical flow generated during the exposure time interval.

[0040] [ Figure 3The dynamic and real-time process is particularly well-suited for handling bursts of multiple consecutive frames, such as multi-frame stacking techniques commonly used for noise reduction, high dynamic range (HDR) imaging, super-resolution imaging, focus stacking, etc. In practice, this technique can be applied to each frame in a burst to dynamically set its exposure time and simultaneously perform event-based motion estimation, which is then used in the frame alignment process.

[0041] The alignment step typically estimates the motion between each frame and the anchor frame. The resulting motion information is used to create a displacement map or optical flow for each frame, referencing the anchor frame. The displacement map includes the displacement vector for each pixel of the frame. Each frame is then aligned or registered with the anchor using the corresponding displacement map for each frame.

[0042] [ Figure 4 [This is a schematic diagram of a multi-frame stacking pipeline using event-based motion estimation, which can be based on [...] Figure 3 Dynamic exposure time determination is applied to each frame. Figure 1 The pipeline has been supplemented with additional steps involved in multi-frame stacking.

[0043] Frame-based camera module 10 and event-based camera module 12 are connected to synchronize their timestamps. The resolution of the timestamps is at least the resolution of the events.

[0044] Preferably, the two camera modules are implemented as a hybrid camera, i.e., using a single sensor array comprising both frame-based pixels and event-based pixels. In such a hybrid camera, the event and luminance signals are spatially registered. If spatially separated and off-axis cameras are used, additional conventional steps can be implemented to register the events with corresponding pixels of the frame-based camera module, as disclosed in patent application EP4068207.

[0045] When the user presses the shutter button, the camera plans a default multi-frame sequence, including multiple frames to be stored in buffer 22. The frame-based module 10 is controlled to... Figure 3 The process captures each frame, wherein once the current frame is stored after its dynamic exposure time, the next frame is captured with its own dynamic exposure time, and so on. The event-based module 12 records all events that occur during the consecutive exposure intervals in buffer 14.

[0046] The first frame can be used as the anchor frame for subsequent frame alignment. Once the burst of the second or subsequent frames is stored, it can be used as a basis for the current frame. iEvents recorded between the anchor frame and the first frame are used to initiate motion estimation 18. Any of several known motion estimation techniques that operate on the events can be selected, such as a convolutional neural network (CNN) that generates optical flow for the current frame. Such optical flow can typically be configured to provide a vector for each pixel of the current frame that shifts that pixel to a matching pixel in the anchor frame. An exemplary CNN is disclosed in [Zhu, Alex et al., " EV-FlowNet: Self- Supervised Optical Flow Estimation for Event-based Cameras [Robotics: Science and Systems (RSS), 2018]

[0047] Once the current frame i Following optical flow 18, it can be used at position 23 to transfer the current frame. i Align with the anchor frame and store the corresponding aligned frame at position 24. The alignment operation typically involves shifting each pixel of the current frame by the current optical flow. i The corresponding vector provided in the image is used to shift the pixel position to another pixel in the aligned frame. When the vector crosses a frame edge, this operation may cause some pixels at the frame edge to be lost.

[0048] There are alternative solutions that directly generate aligned frames without using optical flow or vectors, for example by using the time-shift operator disclosed in patent application EP4168977, or by using, as in [Tulyakov, Stepan et al., " Time lens: Event-based video frame interpolation The machine learning methods described in "Proceedings of the IEEE / CVFconference on computer vision and pattern recognition, 2021, pp. 16155-16164"

[0049] The aligned frame 24 and the anchor frame are merged at 26 to produce the final enhanced image. The merging technique used depends on the purpose of multi-frame stacking (noise reduction, HDR, super-resolution, etc.). The aligned frames are similar to those produced using conventional multi-frame stacking techniques, so the merging technique can be any known technique. However, the aligned frames are the result of event-based rather than frame-based motion estimation, and therefore they are significantly more accurate, especially when the frames contain motion blur or when the frames have different exposure times.

[0050] Since the motion of each frame in a burst is calculated relative to a common anchor frame, that anchor frame preferably has the best quality, especially the sharpest one in the burst. The sharpest frame can be determined as the frame with the smallest maximum amplitude in the motion amplitude map.

[0051] In such as [ Figure 4In the multi-frame stacking process illustrated, there are actually two separate uses for the events captured and stored across multiple frames: i) based solely on the dynamic exposure time interval within the frame. Texp The events stored during the period are used to generate a motion amplitude map for each frame, and ii) optical flow is generated for each frame based on the events stored between the current frame and the anchor frame. Separate processes can be implemented for each purpose, sharing the event data stored in buffer 14.

[0052] A more computationally efficient solution is to use the same motion estimation process for both uses, such as constructing continuous-time optical flow by feeding all events into a convolutional neural network, see, for example, [Gehrig, M. et al., " Dense Continuous-Time Optical Flow from Events and Frames (arXiv, 2022). Such operations create prediction functions. B(t, x, y) This prediction function produces results for any triplet parameters. (t, x, y) , where x and y These are the coordinates of the pixels to be mapped, and t It is the timestamp of the frame to which the pixel is mapped.

[0053] Then, targeting t = 0 Assuming the anchor frame is the first frame of the burst, for alignment purposes, the time can be selected at a given position (such as in the middle) within the exposure time interval of each frame. t Regarding the determination of dynamic exposure time, time t It can be the current time.

[0054] Displaced pixels X’ In time t The new coordinates at that point are therefore represented as: X'(t) = X + B(t, X) ,in X = (x, y) It is necessary to be in time t = The pixel mapped at 0.

[0055] If in time t a ≠ 0 If an anchor frame is selected, then in time... t Displacement of the reference anchor frame D(t, t a ) It is the displacement from the anchor frame to the first frame (in) t = at 0) plus from the first frame to time t The frame shift at that point, therefore: D(t, t a ) = B -1 (ta , X a ) + B(t, B -1 (t a , X a )) (1) in B -1 yes B inverse function, X a = (x a , y a ) It is necessary to be in time t a The pixels mapped in the anchor frame, and B -1 (t a , X a ) It is in time t = 0 will be the pixel X a The displacement vector mapped to the first frame.

[0056] Therefore, in time t relative to time t a The pixel displacement of the anchor frame at that location X' The new coordinates are represented as: X'(t) = X a + D(t, t a ) .

[0057] For dynamic exposure time adjustment, the displacement within the exposure time interval needs to be considered. Texp The time interval between the start and the current time can be determined in equation (1) by influencing the start to time of the exposure time interval. t a And it affects the current time to the end time. t To obtain. Then, from relation (1) D(t, t a ) Provides the start time of the interval. t a Each pixel at position ) to the current time t The displacement vectors of the pixels at that location. Therefore, the magnitudes of these displacement vectors form an amplitude map used to determine the exposure time.

[0058] The resolutions of frame-based and event-based modules can differ. Event-based sensor arrays can have lower resolutions than frame-based sensors. While motion estimation computations may not be affected, the accuracy of the displacement vectors can at most be the same as that of event-based sensors. However, it is common practice in frame-based algorithms to work at much lower resolutions than the original image while still obtaining satisfactory results; see [Hasinoff SW et al., 2016, " Burst photography for high dynamic range and low-light imaging on mobile cameras ”, ACM Transactions on Graphics 35:192].

Claims

1. An image acquisition method, the image acquisition method comprising the following steps: Provide a frame-based camera with a field of view (10); The event-based camera (12) stores the data during the acquisition time interval (Δ). T, Texp A series of events generated during the event-based camera view the target scene within the field of view of the frame-based camera; Motion in the target scene is estimated based on the events stored during the acquisition interval (16). The exposure time is determined based on the estimated motion. Texp );as well as The exposure time is applied to the frame-based camera (10) to capture frames; Wherein, the acquisition time interval (Δ) T, Texp The duration of ) is dynamically determined according to the following steps: Start the aforementioned data collection time interval; The motion score is continuously calculated based on the stored events (14); and The data collection time interval is terminated when the motion score reaches the threshold standard.

2. The method according to claim 1, wherein the method comprises the following steps: a) When the acquisition interval is initiated, the exposure of the frame-based camera is initiated. Texp ); b) Configure the motion estimation (16) to generate the maximum motion amplitude in the scene as the motion score; and c) Stop the exposure when the acquisition time interval ends.

3. The method according to claim 1, wherein, The motion score is the number of events.

4. The method according to claim 2, wherein the method comprises the following steps: Set the threshold to a value that causes motion blur that can be compensated for by motion deblurring operations, based on the corresponding maximum motion amplitude. as well as Based on the exposure time interval in the frame ( Texp The events stored during the period are used to perform motion deblurring on the frame.

5. The method according to claim 2, wherein the method comprises the following steps: Repeat steps a), b), and c) multiple times to capture multiple frames; Synchronize the timestamp of the event with the timestamp of the frame; Select an anchor frame from the plurality of frames; Based on the stored event (14) corresponding to the interval between the anchor frame and the current frame, align (23) the current frame relative to the anchor frame; and Merge (26) the frame that is aligned with the anchor frame.

6. The method according to claim 5, wherein, The alignment step includes: Based on the stored events (14), generate the optical flow (18) of the current frame relative to the anchor frame; and The aligned frame is created by applying the optical flow to the current frame (23).

7. The method according to claim 5, wherein, The anchor frame is selected as the frame that achieves the minimum maximum motion amplitude.

8. The method according to claim 5, wherein, The step of generating the optical flow includes applying a convolutional neural network to stored events corresponding to the interval between the anchor frame and the current frame.

9. The method according to claim 1, wherein, The frame-based camera (10) and the event-based camera (12) are a single hybrid camera, which includes a single sensor array comprising both frame-based pixels and event-based pixels.

10. A camera, said camera comprising: Frame-based camera module (10). An event-based camera module (12) stores data during the acquisition time interval (Δ). T, Texp During this period, a series of events generated from the target scene in the field of view of the frame-based camera module; Motion estimator (16), the motion estimator being configured to estimate motion in the target scene based on the events stored during the acquisition interval; and Control circuit (13), which operates on the stored events, is configured to: The exposure time is determined based on the estimated motion. Texp ); The exposure time is applied to the frame-based camera module to capture frames; Start dynamic data acquisition interval; Configure the motion estimator (16) to continuously calculate motion scores based on stored events; and The acquisition interval is terminated when the motion score reaches the threshold standard.

11. The camera according to claim 10, wherein, The control circuit is configured as follows: When the acquisition interval is activated, the exposure of the frame-based camera module is activated; Configure the motion estimator (16) to generate the maximum motion amplitude in the scene as the motion score; and The exposure is stopped when the acquisition interval ends.

12. The camera according to claim 10, wherein, The frame-based camera module (10) and the event-based camera module (12) are a single hybrid camera module, which includes a single sensor array comprising both frame-based pixels and event-based pixels.

Images