Image processing method and its implementation device

Abstract

This disclosure relates to an image processing method and an apparatus configured to implement the image processing method. The method includes: obtaining an optical input from a hybrid imaging device, where the obtained optical input includes a first component and a second component that corresponds in time to the first component; where the first component of the obtained optical input corresponds to a first temporal resolution, and the second component of the obtained optical input corresponds to a second temporal resolution that is higher than the temporal resolution of the first component; performing an image restoration operation on a first subset of the first component of the obtained optical input based on data from the second component of the obtained optical input; and performing an image fusion operation to generate fused image data from the output of the image restoration operation and a second subset of the first component of the obtained optical input.

Description

Technical Field

[0001] The present disclosure generally relates to image processing methods and apparatuses for implementing the same, and more particularly, to an image processing implementation of a hybrid imaging device that utilizes dual sensing elements with different characteristics to achieve enhanced high dynamic range (HDR) processing. Background Art

[0002] High dynamic range (HDR) imaging in a conventional pipeline typically involves capturing optical images at different exposure sensitivities at different times. However, a conventional HDR pipeline typically only involves a single type of intensity-frame based imaging component.

[0003] When it comes to motion-induced image distortion caused by relative motion between an image capture device and an object to be optically captured, the limitations of such an arrangement can become apparent.

[0004] Although the above disadvantages can be addressed by several processing techniques in an intensity-based HDR pipeline, it is still desirable to achieve a higher resistance to image distortion at a lower computational cost. Summary of the Invention

[0005] In one embodiment of the present disclosure, a method of image processing includes: obtaining an optical input from a hybrid imaging device, wherein the obtained optical input includes a first component and a second component having motion-related information corresponding to the first component, wherein the first component of the obtained optical input corresponds to a first temporal resolution, and the second component of the obtained optical input corresponds to a second temporal resolution higher than the first component; performing an image restoration operation on a first subset of the first component of the obtained optical input according to data from the second component of the obtained optical input, wherein the first subset of the first component of the obtained optical input includes a first exposure frame having a first exposure duration; and performing an image fusion operation to generate fused image data from an output of the image restoration operation and a second subset of the first component of the obtained optical input, wherein the second subset of the first component of the obtained optical input includes a second exposure frame having a second exposure duration shorter than the first exposure duration, and the first subset and the second subset of the first component of the obtained optical input are temporally offset from each other.

[0006] In another embodiment of the present disclosure, an image processing system includes: a hybrid imaging device configured to obtain an optical input, wherein the obtained optical input includes a first component and a second component having motion-related information corresponding to the first component, wherein the second component corresponds to a temporal resolution higher than that of the first component; and a processing device communicatively coupled to the hybrid imaging device. The processing device includes: an image restoration circuit configured to process a first subset of the first component of the obtained optical input based on data of the second component from the obtained optical input, wherein the first subset of the first component of the obtained optical input includes a first exposure frame having a first exposure duration; and an image fusion circuit configured to generate fused image data based on the output of the image restoration circuit and a second subset of the first component of the obtained optical input, wherein the second subset of the first component of the obtained optical input includes a second exposure frame having a second exposure duration shorter than the first exposure duration, and wherein the first subset and the second subset of the first component of the obtained optical input are temporally offset from each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To understand the above features of the present disclosure in detail, the present disclosure briefly summarized above may be described in more detail with reference to the embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate only typical embodiments of the present disclosure and should not be considered as limiting the scope of the present disclosure, as the present disclosure may admit the existence of other equally effective embodiments.

[0008] Figure 1 Illustrates exemplary image processing implementations in accordance with some embodiments of the present disclosure.

[0009] Figure 2 Illustrates a schematic image processing operation in accordance with some embodiments of the present disclosure.

[0010] Figure 3 Illustrates a schematic image processing operation in accordance with some embodiments of the present disclosure.

[0011] Figures 4A - 4C Illustrates a pair of temporally correlated intensity-frame based image data and event-based volume data and some exemplary image restoration techniques in accordance with some embodiments of the present disclosure.

[0012] Figures 5A - 5C Illustrates an exemplary image processing pipeline in accordance with some embodiments of the present disclosure.

[0013] Figure 6Shows a schematic component configuration of an image processing system according to some embodiments of the present disclosure.

[0014] Figure 7 Shows a schematic diagram of a hybrid imaging device according to some embodiments of the present disclosure. Detailed Description

[0015] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. Like reference numerals refer to like elements throughout.

[0016] The terms used herein are for the purpose of describing particular exemplary embodiments only and are not intended to limit the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular forms "a" and "the" are also intended to include the plural forms. It should be further understood that when used herein, the terms "comprises / comprising", "includes / including" or "has / having" specify the presence of the stated features, regions, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, regions, integers, steps, operations, elements, components and / or groups thereof.

[0017] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It should be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0018] The present disclosure describes methods and apparatuses configured to perform the following operations: integrating both an intensity-frame based image sensor (e.g., complementary metal oxide semiconductor image sensor / CIS or charge coupled device / CCD) and a dynamic event-based sensor (dynamic / event vision sensor (DVS / EVS), a very novel type of neuromorphic sensor) imager to improve the accuracy of inter-frame synchronization, thereby achieving enhanced de-blurring, de-ghosting, and rolling shutter distortion correction (RSDC) performance in a multi-frame HDR imaging pipeline.

[0019] Currently available are a variety of multiple-exposure HDR capture techniques, which may include sequential HDR, rolling HDR, interleaved HDR, downsampled HDR, and dual-merged diode HDR (to name just a few).

[0020] The rolling HDR scheme works by asserting an intermediate voltage pulse on the transfer gate during photon charge integration after a first duration (i.e., a long exposure time T1), the falling edge of the intermediate voltage pulse defining the start of a second exposure time (i.e., a short exposure time T2). The ratio between T1 and T2 (i.e., (T1+T2) / T2) defines the dynamic range (DR) expansion. The interleaved multi-frame HDR scheme works by combining long integration time captures and short integration time captures. The downsampled multi-frame HDR scheme compromises pixel resolution for increased DR by combining adjacent pixels with different integration times. The dual-merged diode HDR scheme was developed for automotive applications, where each sensor pixel is provided with a large photodiode and a small photodiode. The pair of photodiodes is configured to be simultaneously exposed, which makes this type of HDR scheme inherently resistant to ghosting artifacts caused by scene motion.

[0021] However, current HDR pipelines mainly involve a single type of intensity-frame-based imaging component (e.g., CIS or CCD). For example, current HDR solutions in commercially available products are mainly based on conventional image data and typically only incorporate 1) camera shake correction using feature matching or inertial measurement unit (IMU) data, and 2) de-ghosting using optical flow, block matching, optimization techniques, or deep learning.

[0022] When it comes to motion-induced image distortion caused by relative motion between an image capture device and an object to be optically captured (e.g., attributable to camera shake or object motion, or both), the limitations of such arrangements can be obvious. Observations of such undesirable image distortion can include motion blur and ghosting artifacts (e.g., as a result of accuracy limitations in temporal registration).

[0023] Although the above-mentioned drawbacks can be addressed by several processing techniques / algorithms in an intensity-based HDR pipeline (e.g., depending on the specific capture technique of the application), the pursuit of higher resistance to image distortion under a conventional HDR pipeline usually translates into higher-cost hardware specifications, greater power consumption, and / or longer processing times.

[0024] On the one hand, in a conventional HDR pipeline, deblurring operations for long-exposure frames in multi-exposure techniques are typically not considered, although deblurring may be necessary to correct image distortions in fast-moving objects and / or camera motion to obtain a sharp image in the final result. On the other hand, it is difficult to perform temporal registration between multi-exposure frames using only sparse data because conventional intensity-based sensors have limited temporal resolution (e.g., frame rate). The limited temporal resolution particularly imposes limitations on the execution of existing algorithms that attempt to estimate motion from sparse data. Additionally, rolling shutter distortion is typically not considered in existing algorithms, but rolling shutter distortion can be a critical consideration, especially when the image capture mechanism / technique involves a combination of CMOS-based sensors and neuromorphic sensor data. Furthermore, current end-to-end deep learning methods for HDR are computationally expensive, especially in applications where integration between neuromorphic sensor data and conventional sensor data is required.

[0025] Figure 1 FIG. shows an image processing implementation according to some embodiments of the present disclosure. By way of example, an exemplary image processing implementation may be integrated in a multi-frame high dynamic range (HDR) imaging pipeline to provide enhanced resilience to motion-induced image distortion problems (e.g., motion blur, ghosting artifacts, and rolling shutter distortion effects).

[0026] The exemplary image processing operation 10 begins at an initial stage of obtaining an optical input using a processing device (e.g., the device 60 as Figure 6 shown). The imaging device according to the present disclosure is provided with a dual-sensing component having different characteristics, the dual-sensing component being configured to capture two different components of the optical input in parallel. For example, a hybrid imaging device (e.g., the device 70 as Figure 7 shown) includes a first type of sensor component (e.g., components C11, C12, C13) integrated together and designed to capture a first optical component S1 of the optical input, and a second type of sensor (e.g., component C2) designed to capture a second optical component S2 of the optical input.

[0027] In a typical embodiment, one of the two sensing components may include a camera based on standard intensity frames (e.g., having a CCD or CIS component) capable of obtaining frame-based image data with a high pixel resolution (e.g., 1920×1080 or higher). In the context of the present disclosure, frame-based image data generally refers to the captured optical input component depicted in absolute intensity measurements. The frame-based image component of the optical input may be referred to as an active pixel sensor (APS) frame, which is characterized by its relatively low frame rate (e.g., typically greater than 5 ms latency). Thus, the frame-based sensor component is characterized by the ability to obtain image data with a high spatial (pixel) resolution at a lower temporal resolution (frame rate per unit time). This makes the frame-based camera itself prone to motion blur when recording high-dynamic scenes. On the one hand, since standard cameras produce frames with relatively low temporal information, it is challenging for them to handle motion-induced image distortion problems (e.g., blur, ghost artifacts, and rolling shutter distortion). Although the intensity frame rate can be increased to reduce the motion-based distortion effects, embodiments of higher frame rate performance for intensity-based sensors typically involve additional costs in terms of hardware specifications, power consumption, and / or processing time.

[0028] While the spatial resolution capabilities of standard frame-based cameras are fundamental for producing high-quality images, embodiments of the present disclosure incorporate a second type of optical sensor that itself operates with distinct characteristics as a complementary metric for accurate and effective motion compensation. In some embodiments, other types of sensors may include event-based cameras (e.g., EVS sensors), which are characterized by their significantly higher sensitivity to a second component S2 of the optical input O in on a standard intensity-frame-based sensor.

[0029] The working principle of event-based cameras (DVS / EVS) is very different from that of traditional frame-based cameras. Event cameras employ independent pixels that generate only information called "events" at the exact moment of such phenomena in the presence of brightness changes in the scene. Thus, the output of an event sensor is not an intensity image but an asynchronous event stream recorded with a high-definition temporal resolution (e.g., microseconds), where each event includes the recording time and the location (or address) of the corresponding pixel where the brightness change was detected, as well as the polarity of the brightness change (indicating the binary intensity change with a positive or negative sign).

[0030] Since event generation is induced by luminance changes over time, an event camera itself responds to edge detection in a scene with relative motion. Thus, an event camera offers certain characteristics superior to those of a standard frame-based camera, particularly in terms of, for example, significantly lower latency (in microseconds), low power consumption, and significantly higher dynamic range (e.g., 130 dB as compared to 60 dB of a standard frame-based camera). More importantly, since the pixels of an event sensor are non-dependent, such a sensor itself is not affected by the motion blur problem.

[0031] In some embodiments, the process of obtaining an optical input involves acquiring the optical input (e.g., as shown in Figure 6 Device 610 as shown) by a hybrid imaging device (e.g., as shown in Figure 6 O as illustrated in in ), the hybrid imaging device including a first type of sensor component (e.g., sensor component 612) and a second type of sensor component (e.g., sensor component 614) integrated together. The first type of sensor component is configured to capture a first component S1 of the optical input. Correspondingly, the second type of sensor component is configured to record a second component S2 of the optical input having characteristics different from those of the first component S1. In some embodiments, the first type of sensor component is provided with a first pixel resolution. In some embodiments, the second type of sensor component has a second pixel resolution less than the first pixel resolution. In some embodiments, the first type of sensor component includes a frame-based image sensor (e.g., a CIS component). In some embodiments, the second type of sensor component includes an event-based vision sensor (e.g., an EVS).

[0032] For example, in the illustrated embodiment, the first type of sensor (corresponding to the frame-based component S1 of the optical input O in ) is provided with a first pixel resolution and a first temporal resolution. On the other hand, the second type of sensor (corresponding to the event-based component S2 of the optical input O in ) is provided with a second pixel resolution lower than the first pixel resolution and a second temporal resolution greater than the first temporal resolution. It should be noted that depending on the operating conditions and application requirements, other types of sensor components may be integrated in the hybrid imaging device to augment the frame-based sensor elements. On the one hand, a hybrid imaging device integrating different types of sensor components with different characteristics can achieve enhanced imaging processing capabilities suitable for specific surrounding conditions or application types. In some embodiments, the hybrid imaging device may include a combination of different types of sensor components, which may include, for example, a standard frame-based camera component, a high-speed camera component, a spiking sensor, a structured light sensor, an event sensor component, an ultrasonic imaging component, an infrared imaging component, a laser imaging component, etc.

[0033] The combination of two types of sensor elements from a hybrid imaging device enables the recording of two distinct components (e.g., S1, S2) of the acquired optical input. Among them, the optical component S1 corresponds to the data stream of high pixel resolution frames at a relatively low update frequency (e.g., 30 fps).

[0034] In an ideal (but rare) scenario where there is no relative motion between the captured object and the edge of the image frame (e.g., under the condition that the camera operation is stable when recording a stationary object), the optical component S1 from a frame-based standard sensing component (e.g., Figure 6 the sensor 612 shown) can be presented by a high redundancy image frame stream. On the other hand, the component S2 corresponds to the event-based volume data from a complementary event-based sensing component (e.g., Figure 6 the sensor 612 shown), which only reflects the relative movement path of moving objects at a significantly higher update rate. Therefore, in the exemplary image processing operation 10, the acquired optical input includes a first component S1 (which may include subsets S1-1 and S1-2) and a second component S2 that is temporally corresponding to the first component S1. In some embodiments, the first component S1 of the acquired optical input corresponds to a first temporal resolution, while the second component S2 of the acquired optical input corresponds to a second temporal resolution that is higher than the first component S1's temporal resolution.

[0035] As Figure 1 illustrated in the embodiments of, after the synchronous capture of high temporal event data (e.g., the second component S2) and its corresponding standard intensity frames (e.g., the first component S1: S1-1, S1-2), the image processing proceeds to the image restoration operation 122, where an image restoration operation is performed on the first subset S1-1 of the first optical component S1 according to the data from the second component S2 of the acquired optical input. Compared with image enhancement, during the image restoration process, image distortion / degradation is modeled. Therefore, through the comprehensive information collected by two distinct types of image sensor components, the unwanted image degradation effects can be largely removed.

[0036] In the context of a multi-frame high dynamic range (HDR) imaging application, a first subset S1-1 of a first optical component S1 of an acquired optical input may be a long exposure frame (LEF), typically reflecting an intensity frame with a higher exposure value (EV). On the other hand, a second subset S1-2 of the first local component S1 of the acquired optical input may be a short exposure frame (SEF) with a lower EV. In the illustrated embodiment, incorporating high temporal resolution EVS data (i.e., a second component S2 from, e.g., a neuromorphic image sensor such as an EVS) helps to accurately synchronize the capture times between the intensity-based data of the LEF and SEF (i.e., the first subset S1-1 and the second subset S1-2 of the first component S1) during a subsequent image fusion process, in order to improve resilience to adverse effects (such as motion blur and ghosting artifacts) commonly observed in conventional HDR schemes.

[0037] After the image restoration operation 122, the exemplary image processing implementation 10 proceeds to perform an image fusion operation 126, where the output of the image restoration operation 122 is fused with intensity frame data corresponding to the second subset S1-2 of the acquired optical input to produce an enhanced HDR image output. In some embodiments involving multi-frame HDR applications, the first subset S1-1 and the second subset S1-2 of the first component of the acquired optical input are temporally offset from each other (e.g., sequential sampling data taken in a successive HDR scheme). In some embodiments, the first subset S1-1 of the first component of the acquired optical input has a longer time interval compared to the second subset S1-2. In the illustrated embodiment, the first subset S1-1 of the first optical component corresponds to a first exposure frame with a first exposure duration (e.g., LEF), while the second subset S1-2 of the first optical component corresponds to a second exposure frame with a shorter second exposure duration (e.g., SEF). However, depending on the specific implementation arrangement, in some embodiments, the LEF (e.g., S1-1) and the SEF (e.g., S1-2) may overlap in time. In some embodiments, the sampling of the LEF and the SEF may be initiated simultaneously.

[0038] In some embodiments, the image restoration operation 122 includes a selective combination of performing a deblurring operation or a rolling shutter distortion correction (RSDC) operation. In the illustrated embodiment, the image restoration operation 122 includes a first restoration process 122a that specifically assigns processing to a first subset S1-1 (e.g., LEF) of a first optical component (e.g., an intensity-frame based component) because the longer exposure duration of the first subset S1-1 makes the first sampled frame itself susceptible to motion-based image distortion. In applications where a CMOS image sensor without a global shutter arrangement is used, the selective RSDC operation in the first restoration process 122a will play a crucial role in maintaining image restoration accuracy during image processing implementation 10. In some embodiments, the first image restoration process 122a is performed according to a first portion S2-1 of a second component S2, the first portion being temporally associated with the first subset S1-1 of the first component of the acquired optical input. For example, event-based data having a high temporal resolution is used as a reference basis for the restoration of the intensity-frame based CIS data of the subset S1-1.

[0039] In some embodiments where additional restoration accuracy is desired, the image restoration operation 122 may additionally include performing an additional image restoration operation (e.g., a second restoration process 122b) on a second subset S1-2 of the first component S1 of the acquired optical input. In such embodiments, the additional image restoration operation 122b may be performed according to a second portion S2-2 of the second component S2 of the optical input. In some embodiments, the additional image restoration operation (e.g., the second restoration process 122b) may include RSDC processing on the second subset S1-2 of the first component S1 of the acquired optical input. For example, in embodiments where a CMOS-based sensor assembly with a rolling shutter arrangement is employed, the settings of the additional RSDC module may be known.

[0040] In the illustrated embodiment, the image processing implementation 10 further includes performing a time registration operation 124 on a first subset S1-1 of the first component S1 of the acquired optical input. In some embodiments, the time registration operation 124 is performed according to data from a second component S2 (e.g., S2-1). Due to the higher time resolution of the second component S2 (e.g., EVS data), the time registration accuracy between the LEF (e.g., S1-1) and the SEF (e.g., S1-2) can be increased. Specifically, in some embodiments, the time registration operation 124 can be performed according to a first portion S2-1 of the second component S2, and the first portion corresponds in time to the first subset S1-1 of the first component S1 of the acquired optical input. In addition, in some embodiments, the time registration operation 124 is further performed according to a second portion S2-2 of the second component S2, and the second portion corresponds in time to a second subset S1-2 of the first component S1 of the acquired optical input. For example, in the illustrated embodiment, the second portion S2-2 of the second optical component S2 is used to generate an SEF timestamp TS1-2 to further assist in improving the accuracy of image data synchronization.

[0041] Figure 2 Schematic image processing operations according to some embodiments of the present disclosure are shown. As Figure 2 shown in the illustration, the exemplary image processing operations are applied to an optical input including a restaurant image, where bright outdoor scenery is presented in the far background inside the restaurant through a series of floor-to-ceiling windows in the middle view.

[0042] The first component (e.g., CIS data 212) of the acquired optical input (e.g., intensity-frame-based image data of the scenery of the restaurant) is shown along the top of the illustration. The first component 212 of the acquired optical data can be captured, for example, by the CIS component of the hybrid imaging device (e.g., the component 612 as Figure 6 illustrated). In the illustrated embodiment, the CIS component 212 of the acquired optical input includes a pair of sequentially recorded subsets, namely, the previous long exposure frame (LEF) data S1-1 and the later short exposure frame (SEF) data S2-1 that does not overlap in time. However, it should be noted that the order and the number of subsets among the CIS data shown in the current embodiment are provided for illustrative purposes only. Depending on the specific application or operational requirements, the number of CIS data subsets (i.e., the number of sampled frames) and the order and duration of each of the sampled frames can be different from those shown in the current embodiment.

[0043] As can be observed from the illustration, the recorded image in LEF S1-1 provides a clear view of the interior details of the restaurant, but it is difficult to properly reflect the outdoor details beyond the floor-to-ceiling windows due to overexposure of the bright outdoor scene. Conversely, the recorded image in SEF S1-2 can show a clear view of the outdoor scene beyond the series of floor-to-ceiling windows, but it is significantly lacking in showing the details of the interior foreground in the restaurant.

[0044] The second component (e.g., EVS data 214) of the acquired optical input (e.g., event-based volumetric data of the restaurant scene) is shown along the bottom of the illustration. The second component 214 of the acquired optical data can be recorded, for example, by the EVS component of the hybrid imaging device (e.g., the component 614 as illustrated in Figure 6 ). Compared with conventional intensity-based image data, EVS data 214 is not an intensity image but an asynchronous event stream at microsecond resolution (e.g., as indicated by a series of upward-pointing and downward-pointing arrows along the time axis). In addition, each resulting event data includes associated spatio-temporal coordinates and the sign of the corresponding brightness change (positive or negative polarity, as illustrated by the upward-pointing and downward-pointing directions).

[0045] The high temporal resolution of EVS data 214 enables the generation of precision timestamps corresponding to CIS data 212, which in turn provides crucial information for further refined image restoration and for subsequent frame synchronization operations. It should also be noted that in the current embodiment, although subsets of CIS data 212 (e.g., LEF S1-1 and SEF S1-2) are sequentially acquired in a temporally non-overlapping manner, the corresponding portions of EVS data 214 for image restoration processing (e.g., S2-1, S2-2) do not completely shift from each other (see, for example, the temporally overlapping portion being fed to the time registration operation module 224 between S2-1 and S2-2).

[0046] After the synchronous capture of the high-temporal EVS data 214 and the corresponding intensity-based CIS data 212, the two components of the acquired optical input are forwarded to the image restoration operation module. For example, a first image restoration operation 222a is performed on the LEF portion S1-1 of CIS data 212 according to the corresponding portion S2-1 of EVS data 214. In the current embodiment, the first image restoration operation 222a performs both deblurring and RSDC processes on the first subset S1-1 of CIS data 212 in order to address the inherent susceptibility of LEF to motion-based image distortion.

[0047] For the handling of relatively static scenes with relatively significant luminance contrast, the current embodiment additionally incorporates a second image restoration operation 222b of the corresponding portion S2-2 of the EVS data 214 on the SEF portion S1-2 of the CIS data 212. Specifically, the second image restoration operation 222b includes performing additional RSDC processing on the SEF data S1-2 based on the corresponding portion of the EVS data 214. The additional RSDC processing of the operation 222b can provide enhanced robustness for applications using a CMOS image sensor without a global shutter arrangement.

[0048] After the image restoration operation 222a, the processed CIS data subset (e.g., LEF S1-1) and the corresponding EVS data 214 are forwarded to the time registration module 224 for further processing. Thus, the time registration of the operation module 224 can be performed according to the data from the EVS data 214 (e.g., parts of S2-1 and S2-2). Due to the higher time resolution of the EVS data 214, the accuracy of the time registration between the LEF S1-1 and the SEF S1-2 can be increased.

[0049] Next, the output of the time registration operation 224 on the LEF data subset S1-1 and the additional processed SEF data subset S1-2 are forwarded to the HDR fusion operation module 226 for HDR image fusion processing. In the case of incorporating high-resolution time data (e.g., data 214) from the EVS sensor assembly in a multi-frame HDR imaging pipeline, the execution of image restoration can be further refined and frame synchronization can be more accurate, thereby providing higher resistance to unwanted image distortion effects (such as motion blur, ghosting artifacts, and rolling shutter distortion). The fused image data from the HDR fusion operation 226 is then forwarded to the tone mapping operation module 228 for further processing, after which the output 240 can be generated. In some embodiments, a conventional tone mapping scheme can be applied to the post-HDR fusion processing. As can be observed from the output 240, due to the image processing operations according to the current embodiment, the resulting image retains clear internal details in the restaurant foreground and a vivid outdoor background outside the series of floor-to-ceiling windows.

[0050] Figure 3 Illustrates a schematic image processing operation according to some embodiments of the present disclosure. As Figure 3 shown in the illustration, the exemplary image processing operation is applied to a more dynamic scene, in which a hand hits a Tetra Pak milk carton causing the milk carton to fall in a free-fall manner.

[0051] The first subset of data (e.g., SEF) of the CIS component S1 of the obtained optical input (e.g., intensity-frame-based image data of a falling milk carton scene) is shown in the upper left of the illustration. In the illustrated embodiment, the first subset SEF corresponds to a short-exposure frame of the object scene (observable through relative darkness). On the other hand, the middle left of the illustration shows the second subset of data (e.g., LEF) of the CIS component S1, which in this case corresponds to a long-exposure frame of the object scene (distinguishable by its relative brightness). The lower left of the illustration shows the EVS component S2 of the obtained optical input (e.g., event-based volume data of the scene). The EVS data S2 includes associated spatio-temporal coordinates and corresponding brightness changes reflected in positive or negative polarities, as illustrated by the upward-pointing and downward-pointing arrows. As discussed previously, the specific order, duration, and number of subsets among the CIS data should not be limited to those provided in the illustration of the exemplary embodiment. Depending on the specific application or operational requirements, the number of CIS data subsets (i.e., the number of sampled frames) and the order and duration of each of the sampled frames may differ from those shown in the current embodiment.

[0052] In the current embodiment, an image restoration operation 322 is performed on the LEF data of the CIS component S1 in the presence of the corresponding EVS data S2. Due to the dynamic nature of the falling object, the image restoration operation 322 of the current embodiment incorporates both a deblurring and an RSDC module to compensate for its inherent susceptibility to longer exposure durations, thereby enhancing the resilience against the unwanted effects of motion blur and ghosting artifacts. Due to the image restoration operation 322, enhanced image data of LEF' is generated.

[0053] In contrast, due to the brevity of the SEF data, the current embodiment chooses to omit additional image restoration operations (e.g., additional RSDC processing) on the SEF data. On the one hand, in embodiments where a sufficiently short exposure time is applied, or when the motion of the target object is sufficiently slow, the deblurring and RSDC processing can be selectively omitted to favor increased computational efficiency.

[0054] After the image restoration operation 322, the processed CIS data subset (e.g., LEF') and the corresponding EVS data S2 are forwarded to a temporal registration module 324 for further processing. Due to the higher temporal resolution from the EVS data S2, the accuracy of subsequent registration between the LEF and SEF can be increased.

[0055] Next, the output of the temporal registration operation 324 (e.g., LEF') and the SEF data subset are forwarded to the HDR fusion operation module 326 for HDR image fusion processing. In the case where high-resolution temporal data from the EVS sensor assembly is incorporated into the multi-frame HDR imaging pipeline, the execution of image restoration can be further refined, and frame synchronization can be more accurate, thereby providing higher immunity to unwanted image distortion effects (e.g., motion blur, ghosting artifacts, and rolling shutter distortion).

[0056] The fused image data from the HDR fusion operation 326 is then forwarded to the tone mapping operation module 328 for further processing, after which the output 340 can be generated. In some embodiments, conventional tone mapping schemes can be applied to the post-HDR fusion processing.

[0057] Figures 4A - 4C Illustrate a pair of temporally correlated frame-based intensity image data and event-based voxel data according to some embodiments of the present disclosure and some exemplary image restoration algorithms. Specifically, Figures 4A - 4C each of which illustrates a pair of corresponding CIS data (e.g., an upper coordinate system showing the logarithm of pixel illuminance with respect to time) and EVS data (e.g., a lower coordinate system showing the polarity of the event stream with respect to time). Among the labels along the axis, the upward and downward arrows in the EVS data indicate the corresponding polarities of the event voxels, while the threshold c in the CIS data corresponds to a predetermined threshold (e.g., sensitivity increment) of the event sensor assembly.

[0058] A variety of known techniques can be employed for the processing of CIS and EVS data. By way of example, respectively, Figure 4A illustrate techniques that make it possible to apply to temporal registration by utilizing the corresponding CIS data and EVS data of the pair, Figure 4B illustrate an exemplary algorithm for de-RSDC processing, while Figure 4C illustrate a computational scheme applicable to deblurring operations. The CIS data and the EVS data can cooperate to achieve the restoration / reconstruction of the initially captured CIS data (e.g., LEF data) that is prone to motion-induced image distortion. Since the actual image restoration techniques applied can depend on the actual requirements of a particular implementation (and thus are not the main focus of the present disclosure), the above examples will only be provided herein for reference purposes; thus, further discussion of specific computational techniques will be omitted in the interest of brevity of the disclosure.

[0059] The main applicable field of the present disclosure remains within an imaging pipeline incorporating HDR capabilities. This can be, for example, part of a pipeline of an Image Signal Processor (ISP). By way of example, CIS data can be fed to the currently disclosed image processing pipeline after some conventional functional blocks, which can include but are not limited to Black Level Correction (BLC), Defective Pixel Correction (DPC), denoising, demosaicking, etc. Similarly, the results of the currently disclosed algorithms can be further processed by other functional blocks such as denoising, demosaicking, color correction, sharpening, etc. By way of non-exhaustive examples, Figures 5A - 5C Illustrate some exemplary image processing pipelines according to some embodiments of the present disclosure.

[0060] In addition, the foregoing embodiments according to the present disclosure can be implemented in hardware, firmware, or via software or computer code that can be stored on a recording medium (such as a CD ROM, RAM, floppy disk, hard disk drive, magneto-optical disk), stored on a remote recording medium, or stored in a non-transitory machine-readable medium and downloaded computer code on a data network storing data on a local recording medium. In some embodiments, such software stored on a recording medium can be used to implement the image processing methods described herein using a general-purpose computer, a dedicated processor, or in programmable or dedicated hardware (such as an ASIC or FPGA). As will be understood in the art, a computer, a processor, a microprocessor controller, or programmable hardware can include memory components such as RAM, ROM, flash, etc., which can be used to store or receive software or computer code such that when accessed and executed by a computing device, the processor or hardware is caused to implement to perform the processing methods described herein. In addition, it will be recognized that when a general-purpose computer accesses code for implementing the processing shown herein, execution of the code transforms the general-purpose computer into a dedicated computer for performing the processing described herein.

[0061] Figure 6 Illustrate a schematic component configuration of an image processing system according to some embodiments of the present disclosure. By way of example, an exemplary image processing system can be used to perform an image processing implementation according to the present disclosure, as depicted in the previous embodiments.

[0062] The exemplary image processing system 60 includes a camera 600 incorporating a hybrid imaging device 610, the hybrid imaging device including a first type sensor component 612 and a second type sensor component 614. Among them, the first type sensor 612 is configured to be sensitive to a first component of the obtained optical input O in (e.g., S1 as shown in Figure 1 ), while the second type sensor 614 is configured to be sensitive to a second component of the captured input (e.g., as shown in Figure 1in response to S2) shown in. The second component S2 corresponds in time to the first component S1. In some embodiments, the second component S2 corresponds to a time resolution that is higher than the time resolution of the first component S1.

[0063] For example, in some embodiments, the first type of sensor 612 includes an intensity-based optical sensor, and the first type of component of the optical input includes intensity-frame-based image data (e.g., S1-1, S1-2). In some embodiments, the second type of sensor 614 includes an event-based optical sensor, and the second component of the optical input includes event-based voxel data (e.g., component S2). Thus, the second type of sensor 614 has a spatial resolution that is lower than the spatial resolution of the first type of sensor 612, while its time resolution is increased to be significantly higher than the time resolution of the first type of sensor 612.

[0064] The camera 600 further includes a processing device 620 arranged to communicate with the hybrid imaging device 610 in signal, and the processing device is configured to receive and process the outputs from both the first type of sensor 612 and the second type of sensor 614. In some embodiments, the processing device 620 may include a processor incorporated as part of an integrated circuit, and the integrated circuit includes various circuits each for performing specific functions as depicted in the previous embodiments. For example, depending on the application, the processing device 620 may be implemented as a multi-functional computing hardware or a dedicated hardware. For example, applicable types of processing devices may include a central processing unit (CPU), a digital signal processor (DSP), an image signal processor (ISP), etc. In some embodiments, the processor may include a multi-core processor, and the multi-core processor contains multiple processing cores in the computing device. In some embodiments, the respective elements associated with the processing device 620 may be shared by other devices.

[0065] In the illustrated embodiment, the exemplary processing device 620 is provided with an image restoration module 622, and the image restoration module may contain dedicated or shared hardware circuitry, software, or firmware components to perform an image restoration operation based on the first component S1 of the obtained optical input. By way of example, the image restoration module 622 is configured to process the first component S1 of the optical input from the first type of sensor 612 and the second component S2 of the optical input from the second type of sensor 614, respectively. For example, the image restoration unit 622 may be configured to process the first subset S1-1 of the first optical component S1 at least partially based on the second component data S2 from the second type of sensor 614.

[0066] The processing device 620 is further provided with an image fusion module 624 arranged downstream of the image restoration module 622. The image fusion module may incorporate the necessary hardware circuitry, software, and firmware components to perform image fusion operations based on the output from the image restoration module 622. In the current embodiment, the image fusion module 626 is configured to generate image data based on the output from the image restoration module 622 and based on the second subset data S1-2 of the first component S1 of the obtained optical input O in In some embodiments, the first subset S1-1 and the second subset S1-2 of the first component S1 may be arranged with a time offset from each other. In some embodiments, the first subset S1-1 of the first optical component S1 may be configured with a longer time interval compared to the time interval of the second subset S1-2. For example, in the context of a multi-frame high dynamic range imaging application, the first subset data S1-1 of the first optical component S1 may correspond to the first exposure frame of a first duration. For example, the first subset data S1-1 may correspond to a long exposure frame (LEF) that typically has a higher exposure value (EV).

[0067] The image restoration operations performed by the image restoration module 622 may include algorithms having a distortion / degradation model designed to mitigate motion blur, ghosting artifacts, and rolling shutter distortion effects. In some embodiments, the image restoration module 622 is configured to perform a selective combination of deblurring operations and rolling shutter distortion correction (RSDC) operations on the first subset S1-1 of the first optical component S1. For example, since the longer exposure duration of the LEF (e.g., subset data S1-1) makes the first sampled frame itself prone to motion-based image distortion, in some embodiments, the image restoration module 622 is provided with a first restoration unit (not explicitly shown but functionally corresponding to Figure 1 block 122a) specifically designated to perform deblurring / deghosting operations on the first subset S1-1 of the first optical component S1 (e.g., LEF). On the other hand, in applications where a CMOS image sensor without a global shutter arrangement is used, the first restoration unit may be further configured to incorporate RSDC processing functionality.

[0068] In some embodiments, the image restoration module 622 is configured to operate based on the first part of the second optical component S2 (e.g., event data S2-1), which temporally corresponds to the first subset S1-1 of the first optical component S1. For example, in the current embodiment, the event-based data S2-1 with high temporal resolution is used as a reference basis for the intensity-frame-based CIS data of the restoration subset S1-1.

[0069] In some embodiments, the image restoration module 622 is additionally provided with an additional image restoration unit (not explicitly illustrated, but functionally corresponding to Figure 1 block 122b), and the additional image restoration unit is configured to process the second subset data S1-2 of the first optical component S1. In some embodiments, the additional image restoration unit may be configured to perform an image restoration operation on the second subset S1-2 data at least partially based on the second part of the second optical component S2. In some embodiments, for example, in an application that employs a CMOS-based sensor assembly with a rolling shutter arrangement, the additional image restoration unit may be configured to perform RSDC processing on the second subset S1-2 of the first optical component S1.

[0070] In the illustrated embodiment, the time registration module 624 is arranged in signal communication between the image restoration module 622 and the image fusion module 626. The time registration module 624 may be configured to perform a time registration operation on the first subset data S1-1 of the first optical component S1 at least partially based on the data from the second optical component S2. In some embodiments (for example, where the time interval of the second subset data S1-2 is long and comparable to the time interval of the first subset data S1-1), the time registration module 624 may be further configured to perform a time registration operation on the second subset S1-2 of the first component of the obtained optical input.

[0071] It should be noted that each of the various functional modules / circuits 622, 624, and 626 of the processing device 620 may be formed by common or different circuitry within the processing unit and is configured to execute program instructions read from the memory 630 coupled thereto. For example, the memory 630 is available for intermediate storage during calculations by one or more of the illustrated circuits of the processing device 620 and for storing the data of the calculations. The memory 630 may also store program instructions that are read and executed by the processing device 620 to perform its operations.

[0072] The display 640 may be a stationary or mobile device and is coupled to the processing device 620 in signal communication for displaying image or video output.

[0073] Figure 7 A schematic diagram of a hybrid imaging device according to some embodiments of the present disclosure is shown. By way of example, in some embodiments, the hybrid imaging device includes a fused image sensor chip incorporating both CMOS elements and a DVS sensor assembly on a common sensing interface.

[0074] For example, Figure 7Shows a schematic sensor assembly layout for a hybrid imaging device 70, which includes a plurality of first type sensor units C11, C12, C13 and a plurality of second type sensor units C2 arranged in a common interleaved matrix pattern. In the illustrated embodiment, the first type sensor units C11, C12, and C13 correspond to the red R, green G, and blue B sub-pixels of a CMOS image sensor, respectively, while the second type sensor units C2 correspond to an array of dynamic vision sensor (DVS) units inserted in an interleaved manner among the second type sensor units. However, it should be noted that without departing from the context of the present disclosure, the actual hardware implementation of the hybrid imaging device may be different from that shown in the current figure. For example, the specific relative positions of the sensor units, the actual unit density distribution, the relative sizes, or the spectral responses of each sensor unit may vary. In some embodiments, different types of sensor units may be implemented in an overlapping manner at different layers in an integrated sensor chip.

[0075] For such embodiments, since for event-based DVS units (e.g., unit C2), motion-related information is only generated when there is a change in luminance, the exemplary hybrid imaging device 70 can remove the inherent redundancy of frame-based standard sensor units (e.g., C11, C12, C13), thus requiring a significantly lower data rate.

[0076] For example, the output of an event-based sensor unit (e.g., Figure 1 S2 illustrated in) is not an intensity image, but an asynchronous event stream at microsecond resolution. Therefore, each resulting event data includes associated spatio-temporal coordinates and the sign of the corresponding luminance change (positive or negative, no intensity information).

[0077] In addition, due to the spontaneous sensitivity of event-based sensors to event triggering, the resulting event frames can represent events captured in less than a millisecond. Therefore, the event voxels can form a sparsely populated type of edge map, in which only the regions that provide information for the image restoration process are processed, and other regions can be discarded without any calculation. Therefore, event-based sensors allow the system processing to be triggered only when sufficient event data has been accumulated (e.g., more events can be triggered when there is severe camera jitter). When there is less movement in the camera device, fewer event frames will be generated (and thus less calculation will be performed), thereby saving power and processing resources.

[0078] However, as previously discussed, depending on the specific application environment and operating requirements, without departing from the spirit of the present disclosure, other types of sensor components with different response characteristics to optical inputs may be used in the hybrid imaging device.

[0079] Accordingly, one aspect of the present disclosure provides a method for image processing, the method comprising: obtaining an optical input from a hybrid imaging device, wherein the obtained optical input includes a first component and a second component that corresponds to the first component in time; wherein the first component of the obtained optical input corresponds to a first temporal resolution, and the second component of the obtained optical input corresponds to a second temporal resolution that is higher than the temporal resolution of the first component; performing an image restoration operation on a first subset of the first component of the obtained optical input according to data from the second component of the obtained optical input; and performing an image fusion operation to generate fused image data from the output of the image restoration operation and a second subset of the first component of the obtained optical input.

[0080] Accordingly, another aspect of the present disclosure provides an image processing system, which comprises: a hybrid imaging device configured to obtain an optical input, wherein the obtained optical input includes a first component and a second component that corresponds to the first component in time, wherein the first component corresponds to a temporal resolution that is higher than the temporal resolution of the second component; and a processing device communicatively coupled to the hybrid imaging device by a signal, the processing device comprising: an image restoration circuit configured to process a first subset of the first component of the obtained optical input according to data from the second component of the obtained optical input; and an image fusion circuit configured to generate fused image data based on the output of the image restoration circuit and a second subset of the first component of the obtained optical input.

[0081] Those skilled in the art will readily find that many modifications and changes can be made to the apparatus and method while maintaining the teachings of the present disclosure. Therefore, the above disclosure should be understood to be limited only by the boundaries and limitations of the appended claims.

[0082] Translation of the attached drawings

[0083] Figure 1

[0084] 112 Complementary Metal Oxide Semiconductor Image Sensor Data (Complementary Metal Oxide Semiconductor / Charge-Coupled Device) 114 Event Vision Sensor Data (Event)

[0085] RSDC Rolling Shutter Distortion Correction

[0086] Deblurring

[0087] 126 High Dynamic Range Fusion

[0088] 124 Temporal Registration

[0089] 128 Tone Mapping

[0090] 140 Output

[0091] Figure 2

[0092] Long Exposure Long Exposure

[0093] Short Exposure Short Exposure

[0094] LEF Long Exposure Frame

[0095] SEF Short Exposure Frame

[0096] 222a Deblurring + Rolling Shutter Distortion Correction

[0097] 222b Rolling Shutter Distortion Correction

[0098] 224 Time Registration

[0099] 226 High Dynamic Range Fusion

[0100] 228 Tone Mapping

[0101] Figure 3

[0102] S1: SEF S1: Short Exposure Frame

[0103] S1: LEF S1: Long Exposure Frame

[0104] 326 High Dynamic Range Fusion

[0105] 328 Tone Mapping

[0106] 322 Deblurring + Rolling Shutter Distortion Correction

[0107] 324 Time Registration

[0108] EVS Data Event Vision Sensor Data

[0109] Figure 4A

[0110] Log pixel illuminance Log Pixel Illuminance

[0111] LEF Restoration / Reconstruction Long Exposure Frame Restoration / Reconstruction

[0112] c(threshold) c(Threshold)

[0113] time Time

[0114] Event flow (e(t))

[0115] Temporal Alignment

[0116] Polarity

[0117] Impulse

[0118] fun ction Pulse function

[0119] Event integral

[0120] Reference time

[0121] Temporally aligned image Temporally aligned image

[0122] Reference image

[0123] Detection threshold

[0124] Figure 4B

[0125] Log pixel illuminance

[0126] LEF Restoration / Reconstruction Long Exposure Frame Restoration / Reconstruction

[0127] c(threshold) c(threshold)

[0128] time

[0129] Event flow (e(t))

[0130] n-row De-blurring+ROlling Shutter Distortion Correction n-row Deblurring+Rolling Shutter Distortion Correction

[0131] Figure 4C

[0132] Log pixel illuminance

[0133] LEF Restoration / Reconstruction Long Exposure Frame Restoration / Reconstruction

[0134] c(threshold) c(threshold)

[0135] time time

[0136] Event flow(e(t)) Event flow (e(t))

[0137] Blurry i mag e Blurry image

[0138] Deblurred imag e Deblurred image

[0139] De-blurring De-blurring

[0140] Blurry image Blurry image

[0141] Eposure time Exposure time

[0142] Deblurred image Deblurred image

[0143] Figure 5A

[0144] 514 Event vision sensor data

[0145] 512 Complementary metal oxide semiconductor image sensor data

[0146] 540 Output

[0147] BLC Black level correction

[0148] DPC Defective pixel correction

[0149] 522 De-blurring + Rolling shutter distortion correction

[0150] 524 Time registration

[0151] De-noising De-noising

[0152] 526 High dynamic range fusion

[0153] 528 Tone mapping

[0154] Demosaicing Demosaicing

[0155] Figure 5B

[0156] 514′ Event vision sensor data

[0157] 512′ Complementary metal oxide semiconductor image sensor data

[0158] 540 Output

[0159] Denoising

[0160] 522' Deblurring + Rolling Shutter Distortion Correction

[0161] 524' Temporal Registration

[0162] Denoising

[0163] 526' High Dynamic Range Fusion

[0164] 528' Tone Mapping

[0165] Demosaicing

[0166] De - noising

[0167] Figure 5C

[0168] 514" Event Vision Sensor Data

[0169] 512" Complementary Metal Oxide Semiconductor Image Sensor Data

[0170] 522" Demosaicing

[0171] 524" Deblurring + Rolling Shutter Distortion Correction

[0172] Temporal

[0173] Alignment Temporal Registration

[0174] 526" High Dynamic Range Fusion

[0175] 528" Tone Mapping

[0176] 540 Output

[0177] Figure 6

[0178] 612 Sensor 1

[0179] 614 Sensor 2

[0180] 630 Memory

[0181] 622 Image Restoration Circuit

[0182] 624 Temporal Registration Circuit

[0183] 626 Image Fusion Circuit

Claims

1. A method for image processing, comprising: Obtaining an optical input from a hybrid imaging device, wherein the obtained optical input includes a first component and a second component having motion-related information corresponding to the first component; wherein the first component of the obtained optical input corresponds to a first time resolution, and the second component of the obtained optical input corresponds to a second time resolution higher than the time resolution of the first component; Performing an image restoration operation on a first subset of the first component of the obtained optical input according to data from the second component of the obtained optical input, wherein the first subset of the first component of the obtained optical input includes first exposure frames having a first exposure duration; and Performing an image fusion operation to generate fused image data from the output of the image restoration operation and a second subset of the first component of the obtained optical input, wherein the second subset of the first component of the obtained optical input includes second exposure frames having a second exposure duration shorter than the first exposure duration, and the first subset and the second subset of the first component of the obtained optical input are temporally offset from each other.

2. The method according to claim 1, wherein the first subset of the first component of the obtained optical input has a longer time interval compared to its second subset.

3. The method according to claim 1, wherein obtaining the optical input through the hybrid imaging device includes acquiring the optical input through a hybrid imaging device including a first type of sensor component and a second type of sensor component integrated together; wherein the first component of the optical input corresponds to the first type of sensor component; wherein the second component of the optical input corresponds to the second type of sensor component.

4. The method according to claim 3, wherein the first type of sensor component has a first pixel resolution; wherein the second type of sensor component has a second pixel resolution smaller than the first pixel resolution.

5. The method according to claim 3, wherein the first type of sensor component includes a frame-based image sensor; wherein the second type of sensor component includes an event-based vision sensor.

6. The method according to claim 1, wherein performing the image restoration operation includes performing at least one of a deblurring operation or a rolling shutter distortion correction (RSDC) operation on the first subset of the first component.

7. The method according to claim 6, wherein the image restoration operation is performed according to a first part of the second component of the obtained optical input, and the first part corresponds in time to the first subset of the first component of the obtained optical input.

8. The method according to claim 7, which further includes: Performing an additional image restoration operation on the second subset of the first component of the obtained optical input according to a second part of the second component of the obtained optical input.

9. The method according to claim 8, Wherein the additional image restoration operation includes performing an RSDC operation on the second subset of the first component of the obtained optical input.

10. The method according to claim 1, further comprising: Performing a temporal registration operation on the first subset of the first component of the obtained optical input according to data from the second component of the obtained optical input.

11. The method according to claim 10, wherein the temporal registration operation is performed according to a first part of the second component, and the first part corresponds in time to the first subset of the first component of the obtained optical input.

12. The method according to claim 11, wherein the temporal registration operation is performed according to a second part of the second component, and the second part corresponds in time to the second subset of the first component of the obtained optical input.

13. An image processing system, comprising: A hybrid imaging device configured to obtain an optical input, Wherein the obtained optical input includes a first component and a second component having motion-related information corresponding to the first component, Wherein the second component corresponds to a time resolution higher than that of the first component; And A processing device communicatively coupled to the hybrid imaging device, the processing device comprising: An image restoration circuit configured to process a first subset of the first component of the obtained optical input according to data from the second component of the obtained optical input, wherein the first subset of the first component of the obtained optical input includes a first exposure frame having a first exposure duration; And An image fusion circuit configured to generate fused image data based on the output of the image restoration circuit and a second subset of the first component of the obtained optical input, wherein the second subset of the first component of the obtained optical input includes a second exposure frame having a second exposure duration shorter than the first exposure duration, and the first subset and the second subset of the first component of the obtained optical input are offset from each other in time.

14. The image processing system according to claim 13, Wherein the hybrid imaging device includes a first type of sensor assembly and a second type of sensor assembly; Wherein the first component of the optical input corresponds to the first type of sensor assembly; Wherein the second component of the optical input corresponds to the second type of sensor assembly.

15. The image processing system according to claim 14, Wherein the image restoration circuit is configured to process the first component of the optical input from the first type of sensor and the second component of the optical input from the second type of sensor separately.

16. The image processing system according to claim 15, Wherein the first type of sensor assembly has a first pixel resolution and a first time resolution; Wherein the second type of sensor component has a second pixel resolution that is less than the first pixel resolution and a second temporal resolution that is greater than the first temporal resolution.

17. The image processing system according to claim 16, wherein the first type of sensor component includes an array of frame-based image sensor units; and wherein the second type of sensor component includes an array of event-based vision sensor units interleaved among the first type of sensor units.

18. The image processing system according to claim 13, wherein the first subset of the first component of the obtained optical input has a longer time interval compared to the second subset thereof.

19. The image processing system according to claim 13, wherein the image restoration circuit is configured to perform at least one of a deblurring operation or a rolling shutter distortion correction (RSDC) operation on the first subset of the first component.

20. The image processing system according to claim 19, wherein the image restoration circuit is configured to operate based on a first portion of the second component of the obtained optical input, the first portion corresponding in time to the first subset of the first component of the obtained optical input.

21. The image processing system according to claim 20, further comprising: an additional image restoration circuit configured to process the second subset of the first component of the obtained optical input based on a second portion of the second component of the obtained optical input.

22. The image processing system according to claim 21, wherein the additional image restoration circuit is configured to perform an RSDC operation on the second subset of the first component of the obtained optical input.

23. The image processing system according to claim 13, further comprising: a time registration circuit communicatively arranged between the image restoration circuit and the image fusion circuit, the time registration circuit being configured to perform a time registration operation on the first subset of the first component of the obtained optical input based on data from the second component of the obtained optical input.

24. The image processing system according to claim 23, wherein the time registration circuit is further configured to perform a time registration operation on the second subset of the first component of the obtained optical input.

Images