Image enhancement methods, devices, and systems

The combination of frame-based and event-based sensors in image enhancement methods addresses motion artifacts and HDR challenges, enabling high-quality image capture in low-light conditions with improved dynamic range and reduced computational complexity.

KR103046157B1Active Publication Date: 2026-11-10PROPHESEE SOLUTIONS PVT LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
KR1020227044520
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-06-23
Filing Date
2021-06-23
Publication Date
2026-11-10
Estimated Expiration
2041-06-23

AI Technical Summary

Technical Problem

Conventional image and video cameras face limitations such as motion artifacts due to rolling shutter readouts, exposure/ISO trade-offs leading to motion-blurred images, and computational complexity in generating High Dynamic Range (HDR) images, especially in low-light conditions with large dynamic ranges.

Method used

An image enhancement method combining frame-based and event-based sensors to synchronize and process image frames and event streams, utilizing modules for image time shifting, deblurring, reblurring, resolution and color recovery, and non-linear compensation to correct artifacts and enhance image quality.

Benefits of technology

Enables the capture of images and videos with motion and large dynamic range in low-light conditions with limited computational complexity, effectively removing artifacts and improving image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112022136506739-PCT00008_ABST
    Figure 112022136506739-PCT00008_ABST
Patent Text Reader

Abstract

An image enhancement method is disclosed, comprising: capturing a reference image frame of a scene by means of a frame-based sensor; acquiring an event stream for at least an exposure time by means of an event-based sensor synchronized with the frame-based sensor; and deriving an artifact-free corrected image frame from the reference image frame using the event stream, wherein the reference image frame contains artifacts and has an exposure time, and the event encodes a brightness change of the scene corresponding to the reference image.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to image processing technology, particularly image enhancement. Background Technology

[0002] Conventional image and video cameras and systems are commonly limited by several factors. In particular, at the sensor design level, image sensors often use rolling shutter readouts to make them smaller with lower cost and higher sensitivity. While this readout has advantages, it can lead to artifacts due to motion occurring while the frame is being captured.

[0003] Furthermore, when video or images are captured by conventional cameras, camera settings offer different trade-offs for acquisition, such as the typical exposure / ISO trade-offs of changing exposure duration; while more or less light is incorporated, there is still a risk of producing motion-blurred images. Maintaining a short exposure duration can reduce motion blur but will produce images that are too dark in low-light situations. In the latter case, it is common practice to increase the sensor's ISO gain to compensate for the smaller input signal. However, this also amplifies sensor noise.

[0004] Furthermore, some scenes contain both bright and dark areas simultaneously. In such cases, one often has to choose between an exposure that exposes the bright areas well and an exposure that exposes the dark areas well. A common solution to this problem is to combine multiple images and use some non-linear function to compress the measured light intensity to generate an HDR image. However, this solution typically requires the alignment of 2 to 8 multiple images and is therefore computationally too complex for high frame rates used for conventional image and video acquisition.

[0005] In this field, there are already several solutions designed to compensate for the aforementioned limitations of conventional imaging and video acquisition. For example, WO2014042894A1, US10194101B1, and US9554048B2 disclose how to remove specific rolling shutter effects, WO2016107961A1, EP2454876B1, and US8687913B2 disclose how to remove blur without rolling shutter effects, and some other literature discusses computing High Dynamic Range (HDR) images and videos without removing rolling shutter and blur. However, these are generally tailored to specific artifacts and / or entail significant computational complexity. The problem to be solved

[0006] The present invention aims to improve these disadvantages. means of solving the problem

[0007] In this regard, according to one aspect of the present invention, an image enhancement method comprising the following is provided.

[0008] A step of acquiring a reference image frame of a scene containing artifacts and having an exposure time by means of a frame-based sensor;

[0009] A step of acquiring an event stream for at least the exposure time of a reference image frame by an event-based sensor synchronized with a frame-based sensor (the event encodes a brightness change of the scene corresponding to the reference image); and

[0010] A step of deriving an artifact-free corrected image frame from a reference image frame using an event stream.

[0011] Accordingly, according to this arrangement according to the present invention, the frame image and event data are combined to remove image artifacts such as wobble (also known as jello effect), skew, spatial aliasing, and temporal aliasing in the frame image, particularly in the case of scenes with movement at low light and a large dynamic range.

[0012] In one embodiment according to the present invention, a reference image frame is acquired at a reference time, and the step of deriving a corrected image frame includes an image time shifting step in which the reference image frame is shifted to a corrected image frame at a target time within or outside the exposure time by establishing a relationship between an event and a reference image frame and accumulating a brightness change encoded by an event acquired during an interval [t_r,t] between a reference time and a target time, thereby shifting the image as if it were acquired at a different time, i.e., at the target time.

[0013] Furthermore, according to the present invention, a reference image frame is a blurry frame, and an image deblurring step for restoring the blurry frame to a sharp frame is included, for example, to acquire a sharp reference image when the exposure time is too long.

[0014] Alternatively, the method according to the present invention includes the step of reblurring a corrected image frame to simulate a larger exposure time effect that makes the image more realistic, particularly in a scene of fast motion from a sharp image.

[0015] Alternatively, the method according to the present invention includes a non-linear compensation step for compensating for non-linear processing in both image frames and events.

[0016] Furthermore, in another embodiment of the present invention, a reference image includes a first resolution and / or a first color space, and an event stream includes a second resolution and / or a second color space different from the first resolution and / or the first color space, and the method further includes a resolution and color recovery step for compensating for the difference in resolution and / or color space between the reference image and the event stream.

[0017] In another embodiment of the present invention, an image frame is generated by a frame-based camera comprising a rolling shutter sensor that includes a row of sensor elements (pixels). Furthermore, for example, the step of deriving a corrected image frame is performed sequentially for each row of the rolling shutter sensor.

[0018] In another embodiment of the present invention, the method includes the step of generating a plurality of corrected image frames distributed at different times of exposure time.

[0019] Alternatively, the method according to the present invention includes the step of aligning a plurality of corrected image frames derived at different exposure times.

[0020] According to another aspect of the present invention, an image enhancement device comprising the following is further provided.

[0021] Frame-based camera including a frame-based sensor;

[0022] An event-based camera comprising an event-based sensor, wherein a frame-based camera and an event-based camera observe the same scene and generate image frames and event streams corresponding to the scene, respectively, and the frame-based camera and the event-based camera are synchronized event-based cameras; and

[0023] A processor for executing the above-described method.

[0024] Alternatively, according to another aspect of the present invention, another image enhancement device comprising the following is further provided.

[0025] A camera configured to generate image frames and event streams corresponding to a scene observed by the camera;

[0026] A processor for executing the above-described method.

[0027] Furthermore, according to another aspect of the present invention, an image enhancement system comprising the following is further provided.

[0028] A capture module configured to acquire a reference image that may contain artifacts and has an exposure time, and to acquire an event stream corresponding to a scene during the exposure time of the reference image frame;

[0029] An image enhancement module configured to derive an artifact-free corrected image frame from a reference image by means of an event stream.

[0030] In one embodiment of the system according to the present invention, the image enhancement module includes an image time shifting module for shifting a reference image frame to a corrected image frame at any time (t) inside or outside the exposure time by accumulating a brightness change encoded by an event.

[0031] In another embodiment of the system according to the present invention, the image enhancement module includes a non-linear compensation module for compensating for non-linear processing in image frames and events.

[0032] In another embodiment of the system according to the present invention, the reference image includes a first resolution and / or a first color space, the event stream includes a second resolution and / or a second color space different from the first resolution and / or the first color space, and the image enhancement module further includes a resolution and color recovery module for compensating for the difference in resolution and / or color space between the reference image and the event stream. Effects of the invention

[0033] The present invention extends the capabilities of a typical frame-based image sensor by pairing it with, for example, an event stream from a DVS sensor, in order to expand the conventional trade-offs of image sensors and enable the capture of video and images that better match scene characteristics. In this regard, the present invention enables the capture of images and videos of scenes with motion and large dynamic range in low light conditions with limited computational complexity requirements. Brief explanation of the drawing

[0034] Other features and advantages of the present invention will be shown in the following description with reference to the accompanying drawings. FIG. 1 illustrates an exposure window for an image captured by a frame-based sensor and an image corrected by a method according to the present invention. FIG. 2 illustrates a functional block diagram of a first exemplary method according to the present invention. FIG. 3 illustrates a functional block diagram of a second exemplary method according to the present invention. FIG. 4 illustrates a functional block diagram of a third exemplary method according to the present invention. FIG. 5 illustrates a functional block diagram of a fourth exemplary method according to the present invention. FIG. 6 illustrates an exposure window for two images captured by a frame-based sensor and a plurality of images corrected by a method according to the present invention. Specific details for implementing the invention

[0035] The context of the present invention includes an exemplary image capture architecture comprising a conventional image frame-based camera and an event-based camera, wherein the frame-based camera and the event-based camera observe the same scene in which an object or the entire scene moves with respect to the frame-based camera and the event-based camera, and the frame-based camera captures and generates image frames of the scene (including the moving object), and the event-based camera generates a stream of events corresponding to the same scene, including the moving object, captured during the acquisition of the image frames (frame duration) and possibly before and after.

[0036] In particular, frame-based cameras have a rolling shutter sensor that has rows of sensors, such as a CMOS sensor, and the rows of sensors are scanned rapidly row by row across the scene. Thus, an image is captured or generated when scanning from the first row of the sensor to the last row of the sensor during a period called the frame period.

[0037] An event-based camera includes an asynchronous vision sensor directed toward the same scene observed by a frame-based camera and receives light from the same scene. The asynchronous vision sensor includes an array of sensing elements (pixels), each containing a photosensitive element. Each sensing element (pixel) generates an event corresponding to a change in light in the scene. The asynchronous vision sensor may be a Contrast Detection (CD) sensor or a Dynamic Vision Sensor (DVS).

[0038] Alternatively, the present invention may use other possible exemplary image capture architectures comprising a single sensor, such as a DAVIS sensor or an ATIS (asynchronous time-based image sensor), that records both frames and events.

[0039] In the present invention, it is necessary to synchronize frame data with event data.

[0040] In particular, in the case of a combination of an image frame-based sensor and a DVS or CD sensor, it is possible to synchronize the time bases of the two sensors by exchanging synchronization signals between the sensors.

[0041] For example, one possibility is to rely on a flash-out signal from a frame base, which is typically used to trigger the flash during the frame acquisition exposure time, and to use this signal to initiate the exposure time of the CD sensor. The CD sensor can then timestamp the received signal on its time base, allowing the processing element to associate the event timestamp with the exposure time of the frame-based sensor.

[0042] This synchronization is important because the present invention is based on the relationship between frames and events, and that relationship is discussed below.

[0043] Events of an event-based camera (DVS or CD) relate to the temporal evolution of the scene's brightness. The event-based sensor records "events" that encode changes in brightness relative to a reference state. This reference state is referred to as the reference image L(t_r) at reference time t_r. The event-based camera [at] (pixel) position x at time t

[0044] log(L(x,t) / L(x,t_r))>c+

[0045] In this case, generate a positive event and

[0046] log(L(x,t) / L(x,t_r)) <c-

[0047] In the case of, a negative event is generated. c+ and c- are the positive and negative thresholds, respectively. It should be noted that the use of the logarithmic function here is merely an example, and this relationship can be applied to monotonic or continuous functions. Also, e p Defines the polarity associated with a specific event e. e p It can be written as follows.

[0048] ep =τ(log(L(x,t) / L(x,t_r)),c+,c-)

[0049] Here, if x >= c+, τ(x, c+, c-) = 1, if x ∈ (- c-, c+), 0, and if x ≤ - c-, -1.

[0050] If the time interval T=[t_r,t_n], receive the train of events E[t_r,t_n]=e1,…, ek at position x, and

[0051] ∑ e∈E[t_r,t_n] e p =τ(log(L(x,t1) / L(x,t_r)),c+,c-)+...+τ(log(L(x,t_n) / L(x,t_(n-1))),c+,c-) log(L(x,t_n) / L(x,t_r)),c+,c-)

[0052] has.

[0053] h(x,E[t_r,t_n],c+,c-)=∑ e+∈E[t_r,t_n],ep>0 c+ + ∑ e-∈E[t_r,t_n],ep<0 By defining c-

[0054]

[0055] has.

[0056] Mathematical formula 1 is applied independently to each pixel.

[0057] In this case, Equation 1 establishes a relationship between two images of different times and an event received between these times. Next, how to manipulate and enhance images according to the present invention using this relationship is discussed.

[0058] In the context of the present invention, a processor or computing unit adapted to execute an image enhancement method based on the following module is also included.

[0059] ● Image Time Shifting Module

[0060] In Equation 1, by accumulating the brightness change encoded by the event E[t_r,t] and updating the brightness of L(x,t_r) as follows, t_r <t인 샤프 기준 프레임 L(x,t_r)로부터 임의의 시간 t의 샤프 이미지 L(x,t)를 도출하는 것이 가능하다.

[0061]

[0062] If t_r>t, then h(x,E[t_r,t],c+,c-)=-(∑ e+∈E[t_r,t],ep>0 c+ + ∑ e-∈E[t_r,t],ep<0 c-) is.

[0063] Equation 2 enables the estimation of an image at any time t given a reference image at time t_r and an event recorded at the time interval [t_r,t]. This equation forms an image time shifting module, which is one of the basic modules of the method according to the present invention, and enables the image to be "shifted" as it is acquired at different times.

[0064] ● Image deblurring module

[0065] Equation 2 assumes that a reference image is available at time t_r. This may be true if the exposure time is sufficiently short (i.e., if the image is sharp). However, if the exposure time is too long, it is impossible to have a sharp reference image available in the case of scene movement or relative movement between the scene and the camera(s).

[0066] Therefore, it is necessary to ensure that the sharp image of t_r is shifted. Mathematically, a blurred frame can be defined as follows.

[0067] B=1 / T∫ f+T / 2 f-T / 2 L(t)dt

[0068] Here, B is a blurred image, f is a given time within the exposure time and is used as the reference time, T is the exposure time and L(t) is the sharp image at time t.

[0069] Mathematical Equation 2 can be substituted into the position of L(t):

[0070]

[0071] Equation 3 defines an image deblurring module that enables the recovery of a sharp image from a blurred image at any time within the exposure time. With the image deblurring module and the image time shifting module, a sharp image can be reconstructed at any given time t.

[0072] ● Image Reblurring Module

[0073] Sometimes, photographers or videographers want to retain or add some blur to an image or video. In this regard, an image can be simulated at an arbitrary exposure time T_n by simply averaging all possible different images that occur during the exposure. For example, if a sharpened image is reconstructed at time t_r and the image is adjusted to the exposure T_n at time t_r, it can be calculated as follows.

[0074]

[0075] Mathematical Equation 4 defines an image reblurring module, where a larger exposure time is simulated in sharp images.

[0076] ● Resolution and Color Restoration Module

[0077] The module defined above acts independently on each pixel. Additionally, the module assumes that there is a one-to-one correspondence between event-based sensor pixels, such as DVS sensors, and that c+ and c- are known for each pixel. However, this may not be the case even for combined sensors. For example, WO 2020 / 002562 A1, titled "Image Sensor with a Plurality of Super Pixels," has event pixels and frame pixels in the same sensor, but there is no one-to-one correspondence.

[0078] Therefore, in a more general manner, the system can combine two separate sensors with different spatial resolutions. Furthermore, typical conventional sensors use Bayer filters to generate color images. Similarly, it is possible to place Bayer filters in event-based sensors. In this case, the aforementioned different operations (deblurring, time shifting, reblurring) need to be applied per color channel. However, most event-based sensors do not have Bayer filters and only record brightness changes. Therefore, the aforementioned different modules can be applied only to grayscale images.

[0079] Accordingly, in order to enable the use of frame and event-based sensors having different numbers of pixels (i.e., sensor resolution) and to account for differences in color space between the frame and event sensors, the present invention proposes a resolution and color restoration module.

[0080] These modules can independently or serially reference the mapping from the recorded event buffer to the event buffer ready for use by the deblurring, image shifting, and reblurring modules. This variation may also be performed at different stages of the method or at the output.

[0081] Furthermore, while the sensor is exposed to noise with thresholds that are not ideal and, more specifically, vary depending on different factors, the output of the resolution and color restoration module is still expected to ideally follow the mathematical definition of a "color-reference" event. Therefore, this resolution and color restoration module also functions as a noise and artifact removal algorithm.

[0082] Generally, this module can be implemented in several ways for each of the resolution, color restoration, and idealization problems as follows.

[0083] ● By using approaches implemented serially or independently, such as corrected distortion, eigen and extrinsic parameters of two sensors, and reprojection algorithms that may be based on machine learning, or by combining them with machine learning approaches for improvement that connect the corrected distortion, eigen and extrinsic parameters to a colorization algorithm that may be machine learning-based or model-based, and also to a noise removal module that may be machine learning-based.

[0084] ● By using a machine learning approach on two of these modules and pairing the acquired model with the remaining module.

[0085] ● By using an end-to-end machine learning approach to model the entire mapping.

[0086] These approaches can be arranged in series in any order.

[0087] Next, some examples of different submodules implementing resolution and color restoration modules are given.

[0088] - Event-based resolution and color restoration submodule

[0089] This submodule modifies the event buffer before applying the aforementioned module.

[0090] In fact, the frame and event sensors are exposed to the same scene. Therefore, the original event buffer e=(e_i)_i=1..N, acquired during the exposure time of the blurred frame, can be mapped to buffer e'=(e'_i)_i=1..N' so that the coordinates of event e'_i are in the frame sensor's coordinate system. It should be noted that set e' does not need to have the same number of events as e.

[0091] In addition, applying image shifting modules, image deblurring modules, and image reblurring modules to a color image can be performed by preprocessing the event stream to determine which change in the color channel triggered the change in luminosity.

[0092] This preprocessing can be thought of as selecting a 3D color space and then finding a mapping between the frame-based coordinates e'=(e'_i)i=1..N' and the color reference events e''=(e''_i)_i=1..N'' of the already aligned event buffer.

[0093] e''_i exists at pixel x when one of the following is true.

[0094] ● log(Ch(x,t) / Ch(x,t_r))>c + _ch and assigned polarity 1 and channel Ch.

[0095] ● log(Ch(x,t) / Ch(x,t_r)) <c-_ch이고 부여된 극성 0 및 채널 Ch.

[0096] Ch is any of the three color channels and c + _ch and c-_ch are color thresholds according to the e' to e'' mapping algorithm. Therefore, events in e'' have additional color channel information. Also, in this case, for each event in e', depending on the mapping function used, some events are generated in e'' or not generated.

[0097] To go from set e to set e'', the model typically learns a function f(e) -> [e''] which, when an event is given as input, generates a list of events with different color references as output.

[0098] By knowing the eigenparameters of each camera (camera matrix K and lens distortion parameter D), the rotation R and translation T between the two cameras, and the depth map of the scene, a map M can be calculated that gives the event camera coordinates for each pixel of the frame camera. This map can be used to transform the coordinates of each event to obtain the frame camera coordinates. Furthermore, an event can be represented as a vector v=[x,y,t,p] where (x,y) is the event location, t is the time, and p is the polarity, or as a binary vector b=[0...0 1 0...] or a floating-point vector v^f=[f_1,f_2,f_3,f_4] encoding v in different spaces. And f(e) can be a linear or non-linear function of v, b, or v^f that outputs a matrix where each column is a vector [x,y,t,p,ch] where ch is the color channel. The matrix may have an additional row indicating whether a column is a valid event. This function is a combination of linear operators, which is a warping function obtained from M, or can be learned by providing sufficient examples of input / output pairs.

[0099] - Medium Resolution and Color Restoration Submodule

[0100] Another possible example is acting on the intermediate map used by the aforementioned module. First, the time shifting, deblurring, and reblurring modules are all assumed to act on the quantity h(E[t_i,t_j],c-,c+) which can be discretized in the time bin for computational efficiency.

[0101] The input to the time-shifting, deblurring, and reblurring modulo can be a collection of binned event sums. For example, using binning with a fixed time bin width ΔT, a positive (h(E[t_0+i*ΔT,t_0+(I+1)*ΔT],c-,c+)_I=1..B can be used as input. Additionally, when Δk is the bin width in terms of the number of events, a positive (∑ e+∈i*Δk..(i+1)*Δk,ep>0 c+ + ∑ e-∈ i*Δk..(i+1)*Δk,ep<0 c -By using )_i=1..B, discretization can be performed with a fixed number of events.

[0102] Since this intermediate resolution and color restoration submodule can be implemented with this in mind, it has the inputs and outputs of the configuration module, regardless of whether it is executed individually or abstracted together through a machine learning pipeline. In practice, the quantity h(E[t_I,t_j],c-,c+) is the matrix H \in R where m_eb x n_eb is the resolution of the event-based sensor and B is the number of time bins with a fixed time width or a fixed number of events. m_eb x n_eb x B It can be expressed as. Therefore, this module is the function f: R m_eb x n_eb x B -> R m_fb x n_fb x B' x ch It can be defined as a transformation, and thus the matrix of the equation has frames of the same dimension and a selected number of empty B'. Also, in this case, using camera calibration, f may include a warping operator obtained from camera calibration, may be a guided filter or a joint bidirectional filter, or may be learned using input / output pairs.

[0103] - Post-resolution and color restoration submodule

[0104] Finally, the aforementioned module converts frame F to the resolution of the event, removes color information, and F d Deblurring this version of the frame denoted as, and D d The deblurred F denoted as d It can be applied by converting it to the original frame resolution and reapplying color information.

[0105] In this case, the geometric mapping M mapping the two camera pixels is F d It is applied to deblurring. It is also possible to maintain the original resolution of the frame camera and apply some simple interpolation to both h(E[t_I,t_j],c-,c+). D dOnce obtained, a function f is needed to convert it into a color version (and one that has the correct details at the frame camera's resolution). This function can be a guided filter or a joint bidirectional filter, or it can be learned using input / output pairs.

[0106] ● Non-linear compensation module

[0107] The aforementioned module can be applied to images acquired from a frame-based sensor before or after the non-linear processing module by the ISP. In the latter case, it is necessary to compensate for non-linear processing even in events. Therefore, the non-linear compensation module is defined as follows.

[0108] Assuming the camera response function is an identity function, mathematical equations 2, 3, and 4 describing the image time shifting, deblurring, and reblurring modules can be written as follows.

[0109] When F_1 and F_2 are frames, (e_i)_i=1..N is an event stream, and f is a transformation function, F_1=F_2*f( (e_i)_i=1..N )

[0110] Therefore, assuming the frame of interest is F_2, it can be searched using simple division.

[0111] If the camera response function is some function g, the above relationship becomes as follows.

[0112] g(F_1) = g( F_2 * f( (e_i)_i=1..N) )

[0113] Non-linear processing often involves gamma correction. In this setup, it has the following:

[0114] (F_1)^γ=(F_2)^γ*f((e_I)_I=1..N)^γ

[0115] The search for the frame of interest is obtained by (F_2)^γ = (F_1)^γ / f((e_I)_I=1..N)^γ).

[0116] A nonlinear compensation module applied to a function f in the event buffer (e_i)_i=1..N is defined to generate a value f((e_i)_i=1..N)^γ in order to compensate for the nonlinearity of the output function of a frame-based sensor and to enable the application of the aforementioned frame processing module. Therefore, in general, it can be treated as any function g that can be factored into the aforementioned gamma correction in this manner. However, this approach cannot be used when applying a reblurring module or if the function g cannot be factored. In this case, if g is invertible, g -1 When (F_1)=F_2*f( (e_i)_i=1..N) can be calculated and F_2 is obtained, g(F_2) is applied to the obtained F_2, which has the same response function as F_1.

[0117] To further explain the invention, some embodiments of an exemplary method according to the invention are described below, which improve the image quality of an imaging system using an event stream to modify a captured image as if it were captured in a different ideal setting using the module described above.

[0118] FIG. 1 illustrates how a row of a rolling shutter sensor is exposed during one frame. Each black element represents the exposure window of a given row, and the exposure window for each row is shifted due to the rolling shutter because it is read sequentially by the sensor circuit. It should be noted that while the exposure windows all have the same exposure period or exposure time, they start and end at different times for each sensor row. The light gray vertical bar represents the exposure window of the global shutter sensor at the target time to be recovered by the present invention, and the exposure windows of all rows are aligned. In this embodiment of FIG. 1, the rolling shutter sensor scans from the top (first) row to the bottom (last) row, and a temporally distorted image (an image with artifacts) is captured.

[0119] As shown in FIG. 1, for each row, the time t_s^r within the exposure window is indicated as a reference. Alternatively, for a global shutter sensor, the exposure window starts and ends at the same time for all rows. Furthermore, the target time t_ref is also defined as the new target exposure time (indicated by the gray bar shape).

[0120] In the following paragraph, it will be shown how an image frame captured by the exposure window of a rolling shutter sensor, represented by a black element, is converted / corrected into something else represented by a gray bar shape by using the method according to the present invention as illustrated in FIG. 1, where each row r of the rolling shutter sensor has a start exposure time se_r, an end exposure time ee^r, and a reference time t_s^r \in [se^r, ee^r]. A target exposure time T_ref is also defined together with a target time t_ref.

[0121] FIG. 2 illustrates a first exemplary method according to the present invention comprising the following steps: image frame B is captured by a row of sensor elements of a rolling shutter sensor of a frame-based camera, the exposure time of each row is illustrated as a black element in FIG. 1, an event stream E=[e_1, e_n] corresponding to a scene depicted in image frame B is generated by an event-based camera, and the frame-based camera and the event-based camera have different resolutions and color spaces.

[0122] Step S1: B'=g -1 The response function is inverted by applying (B).

[0123] Step S2: For each row of B':

[0124] a. Apply the image deblurring module. This corresponds to the following:

[0125] i. Step S2.1: Estimate the corresponding row of event stream r^e, and from the event stream, row d^r^e(t_s^r)=1 / T∫ se^ree^r e h(E^r^e[t_s^r,t],c+,c-) Calculate dt.

[0126] ii. Step S2.2: If the event stream is at different resolutions and colors of the image, apply the intermediate resolution and color recovery submodule to obtain row d^r(t_s^r) corresponding to row r of B.

[0127] iii. Step S2.3: Finally, L^r(t_s^r) at t_s^r Calculate the sharp row using B'^r / d^r(t_s^r).

[0128] b. Apply the image time shifting module for each time t_d \in [t_ref-T_ref / 2,t_ref+T_ref / 2]. This corresponds to the following:

[0129] i. Step S2.4: Estimate the corresponding row of event stream r^e, and from the event stream, row e^r^e(t_d)=e h(x,E[t_s^r,t_d],c+,c-) Calculate.

[0130] ii. Step S2.5: If the event stream is at a different resolution and color of the image, apply the intermediate resolution and color recovery submodule to obtain row e^r(t_d) corresponding to row r of B.

[0131] c. Step S2.6: Apply the image reblurring module and time t_ref L^r(t_ref)=L^r(t_s^r) / T∫ t_ref+T_ref / 2 t_ref-T_ref / 2 Estimate row r of the image in e^r(t)dt.

[0132] Steps S2.1 through S2.6 of this Step 2 will be repeated until the last row of B is processed, and then move to the next Step 3.

[0133] Step S3: g is applied to L to obtain the final corrected image L', which is similar to the one captured by the global shutter sensor with the target exposure time T_ref at the target time t_ref.

[0134] Through these steps, it is possible to correct images using event streams by correcting image frames acquired by a frame-based camera.

[0135] In the embodiments described above, the term "row of sensor(s)" is used only for clarity. It should be understood that the mathematical formula is applied pixel by pixel in each row. Similarly, due to different lens distortions, different sensor sizes and orientations, and distances to objects in the scene, a row of a frame-based sensor does not necessarily correspond to a single row of an event-based sensor, but corresponds to one or more discontinuous curves of an event-based sensor. The term "estimate the corresponding row of the event stream" means to involve an action on each pixel of the event-based sensor that is on one or more curves corresponding to a row of the frame-based sensor. Consequently, each processing of event data generated by a pixel of a row of the event-based sensor may use a different t_s^r depending on the relevant row of the frame-based sensor.

[0136] Alternatively, the exposure time T_ref can be sufficiently short so that the image reblurring module does not need to be applied. In that case, step c (step S2.6) is omitted. Having a large target exposure time T_ref only has the effect of adding some artistic blur to the resulting image.

[0137] FIG. 3 illustrates a second exemplary method according to the present invention, which differs from the first exemplary method in that a reblurring module is not used and the camera response function g is factorable, and thus the non-linear response function can be corrected without g inversion. This second exemplary method comprises the following steps:

[0138] Step S2: For each row of B:

[0139] Step S2.1: Estimate the corresponding row of event stream r^e, and from the event stream, row d^r^e(t_s^r)=1 / T∫ se^ree^r e h(E^r^e[t_s^r,t],c+,c-) Calculate dt.

[0140] Step S2.2: If the event stream is at different resolutions and colors of the image, apply the intermediate resolution and color recovery submodule to obtain row d^r(t_s^r) corresponding to row r of B.

[0141] Step S2.2b: Correct d^r(t_s^r) with a non-linear correction module.

[0142] Step S2.3: And finally, L^r(t_s^r) at t_s^r Calculate the sharp row using B'^r / d^r(t_s^r).

[0143] Steps S2.1 through S2.6 of Step 2 of the second exemplary method will be repeated until the last row of B is processed.

[0144] In the exemplary method above, an intermediate resolution and color recovery submodule is used. On the other hand, in order to use an event-based resolution and color correction submodule compared to the exemplary method shown in FIG. 2, in the third exemplary method shown in FIG. 4, the intermediate resolution and color recovery submodule of steps S2.2 and S2.5 is removed in step S2, and the event-based resolution and color correction submodule of step S1b is applied before step S2.1 for calculating e^r^e(t_s^r). In particular, as shown in FIG. 4, the event-based resolution and color correction submodule of step S1b is applied before step S2 to obtain the aforementioned E', i.e., e', in the "event-based resolution and color recovery submodule" item, and in step S2.1, E' is used to estimate the corresponding row of the event stream s^e and to calculate row d^r^e(t_s^r) in the event stream.

[0145] In another variation, FIG. 5 illustrates a fourth exemplary method according to the present invention, in which a post-resolution and color recovery submodule is used. In this case, compared to the exemplary embodiment illustrated in FIG. 2, the intermediate resolution and color recovery submodule of steps S2.2 and S2.5 is removed, image B is downscaled to the resolution of the event, the intermediate resolution and color recovery submodule is replaced by simple interpolation in steps S2.2' and S2.5', and step S2b is added to apply a post-resolution and color recovery correction module. In particular, as illustrated in FIG. 5, the post-resolution and color recovery submodule step S2b is applied after step 2 to convert grayscale L into color L'.

[0146] Furthermore, the present invention also proposes some embodiments of frame enhancement using a fully data-driven approach. In particular, the present invention proposes deblurring, generating, or enhancing the dynamic range of multiple frames from a single blurred frame using an end-to-end machine learning model. By utilizing additional information provided by event-based sensors, the model size and thus the computational load can be reduced.

[0147] That is, the present invention proposes the use of machine learning to model the following:

[0148] ● Conversion of blurry frame B within or outside the exposure time and event stream (e_i)_i=1..N into a sharp image S at a given time within or outside the exposure time

[0149] ● Conversion into a set of N regularly time-separated sharp frames (S_0, ..., S_N) through the exposure time of the blurred frame B and the event stream (e_i)_i=1..N during the exposure time

[0150] ● Conversion to Frame F2 with enhanced dynamic range of Frame F1 with Event(e_i)_i=1..N during exposure time or any preprocessing of this event stream

[0151] The present invention also proposes a modification of the above mapping in which the event buffer (e_i)_i=1..N is replaced with any appropriate preprocessor.

[0152] Furthermore, according to the present invention, with the module, submodule, and approach described above, in some additional embodiments, it is possible to obtain slow-motion video and video having an enhanced dynamic range, which will be described below.

[0153] For slow motion video, the exemplary method described above can generate an image without artifacts at a specific time t_ref. However, it is also possible to generate one or more corrected images of output for each image of input. Accordingly, an embodiment according to the present invention can generate multiple frames of output and increase the frame rate of the generated video, as shown in FIG. 6 below.

[0154] Figure 6 illustrates an exposure window for two consecutive images captured by a frame-based camera. Similar to that of Figure 1, it consists of slanted black elements representing the exposure window for each row of the rolling shutter, which are converted into multiple corrected exposure windows, namely light gray vertical bars representing 13 frames at 13 different times, so that the frame rate is greatly increased.

[0155] Therefore, it is possible to generate high frame rate video without imposing the exposure time constraints that occur when shooting high frame rate video with only a frame camera (exposure time must be less than 1 / (frames per second)). By using the present invention, for example, a video of 60 frames per second (and thus with a maximum exposure of 16ms) can be shot, and a video of a different (generally higher) number of frames per second can be generated, and the maximum number of frames per second is limited only by the temporal resolution of the event camera.

[0156] For video having an enhanced dynamic range, in addition to the above-described embodiment intended to enhance frame generation using only different frames of the input and event streams, multiple images (or frames) sequentially captured by the sensor may also be aligned and merged. Using multiple images corrected / enhanced by the present invention enables the improvement of image or video quality and the expansion of the dynamic range.

[0157] This improves upon methods in the field where multiple frames are acquired and spatially aligned before being averaged to reduce image noise. Alignment is difficult to achieve while the scene changes during frame acquisition due to moving objects in the scene or relative movement between the scene and the camera. Methods in this field typically align frames using computationally intensive motion estimation and motion compensation. In the present invention, it is possible to "align" multiple frames to a unique t_ref in a more efficient manner without explicit motion compensation by using an event stream and an image time shifting module.

[0158] The present invention is particularly advantageous when exposure times change between frame acquisitions. Changing exposure times is often used to better capture the dynamic range of a scene: when the highest and lowest brightness levels of a scene exceed the highest and lowest light intensities achievable in a single exposure due to the physical limitations of a (frame-based) sensor, N frames are commonly captured with different exposure times E_i, i \in [0, N), merged into a "composite" frame, and thus the highest and lowest light intensities are combined into a single frame. The varying exposure of N frames makes alignment much more difficult for motion compensation techniques in this field.

[0159] In this case, the module and method described above can be used to capture multiple images of a scene and align them to a unique target time t_ref to obtain a high-dynamic-range (HDR) result.

[0160] Accordingly, the present invention generates image / video frames and event streams using a combination of a conventional frame-based image sensor and an event-based sensor, such as a DVS sensor, and combines the image / video frames with event data to more efficiently remove artifacts and improve dynamic range.

[0161] Furthermore, the aforementioned module / method according to the present invention may be implemented in various ways, such as, for example, program instructions for execution by a processor, software modules, microcode, computer program products on a computer-readable medium, logic circuits, application-specific semiconductors, firmware, etc. Embodiments of the present invention may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments including both hardware and software elements. In a preferred embodiment, the present invention may be implemented in software, including but not limited to firmware, resident software, microcode, etc.

[0162] Furthermore, embodiments of the present invention may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in conjunction with a computer, processing device, or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any device capable of containing, storing, communicating, or transmitting a program for use by or in conjunction with an instruction execution system, device, or device. The medium may be an electronic, magnetic, optical, or semiconductor system (or device or apparatus). Computer-readable media include, but are not limited to, semiconductors or solid-state memory, magnetic tape, removable computer diskettes, RAM, ROM (read-only memory), ferromagnetic disks, optical disks, etc. Current examples of optical disks include CD-ROM (compact disk-read-only memory), CD-R / W (compact disk-read / write), and DVD.

[0163] The embodiments described above are examples of the present invention. Various modifications may be made without departing from the scope of the present invention as derived from the appended claims.

Claims

Claim 1 An image enhancement method comprising: a step of capturing a reference image frame of a scene at a reference time (t_r) by means of a frame-based sensor, wherein the reference image frame contains artifacts and has an exposure time; a step of acquiring an event stream for at least an exposure time by means of an event-based sensor synchronized with the frame-based sensor, wherein the event encodes a brightness change of the scene corresponding to the reference image; and a step of deriving an artifact-free corrected image frame from the reference image frame using the event stream, wherein the step of deriving the corrected image frame includes an image time shifting step, and wherein the reference image frame is shifted to a corrected image frame at a target time (t) within or outside the exposure time by establishing a relationship between the event and the reference image frame and accumulating the brightness change encoded by the event acquired during the interval [t_r, t] between the reference time and the target time. Claim 2 An image enhancement method according to claim 1, wherein the reference image frame is a blurred frame and the method includes an image deblurring step for converting the blurred frame into a sharp frame. Claim 3 An image enhancement method according to claim 1, comprising the step of reblurring a corrected image frame. Claim 4 An image enhancement method according to claim 1, comprising a non-linear compensation step for compensating for non-linear processing in an image frame and an event. Claim 5 An image enhancement method according to any one of claims 1 to 4, wherein the reference image frame includes a first resolution and / or a first color space, the event stream includes a second resolution and / or a second color space different from the first resolution and / or the first color space, and further comprises a resolution and color recovery step for compensating for a difference in resolution and / or color space between the reference image frame and the event stream. Claim 6 The image enhancement method of claim 1, wherein the frame-based sensor is a rolling shutter sensor comprising a row of sensor elements, and the step of deriving a corrected image frame is performed sequentially for each row of the rolling shutter sensor. Claim 7 An image enhancement method according to claim 1, comprising the step of generating a plurality of corrected image frames distributed at different times within or outside the exposure time. Claim 8 An image enhancement method according to claim 1, comprising the step of aligning a plurality of corrected image frames derived at different exposure times. Claim 9 A frame-based camera comprising a frame-based sensor; an event-based camera comprising an event-based sensor; and an image enhancement device comprising a processor for executing the method according to claim 1, wherein the frame-based camera and the event-based camera observe at least a portion of the same scene and generate an image frame and an event stream corresponding to the scene, respectively, and the frame-based camera and the event-based camera are synchronized. Claim 10 An image enhancement device comprising: a camera configured to generate image frames and event streams corresponding to a scene observed by the camera; and a processor for executing the method according to claim 1. Claim 11 An image enhancement system comprising: a capture module configured to acquire a reference image containing artifacts at a reference time and having an exposure time, and further configured to acquire an event stream encoding brightness changes of a scene corresponding to the reference image during at least the exposure time; and an image enhancement module configured to derive an artifact-free corrected image frame from the reference image using the event stream, wherein the image enhancement module includes an image time shifting module, and by establishing a relationship between an event and a reference image frame and accumulating brightness changes encoded by the event acquired during a time interval between the reference time and a target time, the reference image frame is shifted into a corrected image frame at a target time within or outside the exposure time. Claim 12 The image enhancement system of claim 11, wherein the image enhancement module comprises a non-linear compensation module for compensating for non-linear processing in image frames and events. Claim 13 An image enhancement system according to claim 11, wherein the reference image includes a first resolution and / or a first color space, the event stream includes a second resolution and / or a second color space different from the first resolution and / or the first color space, and the image enhancement module further includes a resolution and color recovery module for compensating for a difference in resolution and / or color space between the reference image and the event stream. Claim 14 delete Claim 15 delete

Citation Information

Patent Citations

  • Apparatus and method for digital auto-focus

    KR1020090054301A

  • Method and Device of Image Deblurring

    KR1020170087814A

  • Method of flicker reduction

    KR1020190064522A