Hybrid image sensor with video interpolation

Abstract

Hybrid image sensors (and associated systems, devices, and methods) with video interpolation are disclosed herein. In one embodiment, an imaging system includes one or more event vision sensor (EVS) pixels and a plurality of CMOS image sensor (CIS) pixels. Each EVS pixel can be configured to capture event data corresponding to contrast information of light incident on the EVS pixel. Each CIS pixel can be configured to capture CIS data corresponding to an intensity of light incident on the CIS pixel. The imaging system can further include a deblurring circuit configured to deblurring the CIS data captured by the plurality of CIS pixels using a first portion of the event data captured by the one or more EVS pixels; and a system processor configured to interpolate a video frame using the deblurred CIS data and all the event data, or a subset thereof.

Description

[0001] CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 597,638, filed on November 9, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to image sensors. For example, several embodiments of the present technology relate to hybrid image sensors with video interpolation. As a specific example, several embodiments of the present technology relate to hybrid image sensors that utilize event-based vision sensor (EVS) data to (a) interpolate video / image frames using complementary metal oxide semiconductor (CMOS) image sensor (CIS) data and / or (b) deblur the CIS data and / or correct rolling shutter distortion of the CIS data. Background Art

[0004] Image sensors have become ubiquitous and are now widely used in digital cameras, cell phones, video cameras, as well as in medical, automotive and other applications. As image sensors are integrated into a wider range of electronic devices, it is desirable to enhance their functionality, performance metrics, and the like in as many ways as possible (e.g., resolution, power consumption, dynamic range, etc.) both through device architecture design and image acquisition processing.

[0005] A typical image sensor operates in response to image light from an external scene being incident on the image sensor. The image sensor includes an array of pixels having photosensitive elements (e.g., photodiodes) that absorb a portion of the incident image light and generate image charges after absorbing the image light. The image charge generated by the pixel light can be measured as an analog output image signal on a column bit line, which varies depending on the incident image light. In other words, the amount of image charge generated is proportional to the intensity of the image light, and the image light is read out as an analog image signal from the column bit line and converted into a digital value to provide information representing the external scene. Summary of the invention

[0006] Embodiments of the present disclosure provide an imaging system comprising: an event-driven sensing array comprising one or more event vision sensor (EVS) pixels, wherein each of the one or more EVS pixels is configured to capture event data corresponding to contrast information of light incident on the EVS pixel; a pixel array comprising a plurality of CMOS image sensor (CIS) pixels arranged into one or more CIS pixel rows, wherein each of the plurality of CIS pixels is configured to capture CIS data corresponding to the intensity of light incident on the CIS pixel; a deblurring circuit configured to deblur the CIS data captured by the plurality of CIS pixels using a first portion of the event data captured by the one or more EVS pixels; and a system processor configured to interpolate a video frame using the deblurred CIS data and all or a subset of the event data.

[0007] Another embodiment of the present disclosure provides a method of operating an imaging system, the method comprising: accumulating first event data captured using one or more event vision sensor (EVS) pixels of the imaging system corresponding to the CIS pixels during an integration period of a CMOS image sensor (CIS) pixel of the imaging system; outputting at least a portion of the accumulated first event data to an EVS frame buffer; deblurring CIS data captured by the CIS pixels during the integration period using the accumulated first event data; and interpolating a video frame based at least in part on the deblurred CIS data and the portion of the accumulated first event data. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Non-limiting and non-exhaustive embodiments of the present technology are described below with reference to the following figures, wherein like or similar reference numerals are used to refer to all like or similar components unless otherwise specified.

[0009] Figure 1 is a partial schematic diagram of an EVS pixel configured according to various embodiments of the present technology.

[0010] Figure 2 are plots illustrating (a) exposure period, (b) sequence of event pulses formed into a time-continuous signal, (c) sum of the sequence of event pulses over time, and (d) integral of the sum of the sequence of event pulses over time.

[0011] Figure 3 is a partial schematic diagram illustrating an example of an imaging system with off-chip image deblurring.

[0012] Figure 4Ais a partial schematic diagram of a stacked hybrid complementary metal oxide semiconductor (CMOS) image sensor (CIS) and event-based vision sensor (EVS) system configured in accordance with various embodiments of the present technology.

[0013] Figure 4B yes Figure 4A A partial schematic diagram of a specific example of a system.

[0014] Figure 4C is a partial schematic diagram of a 4x4 pixel cluster configured in accordance with various embodiments of the present technology.

[0015] Figure 5 is a partial schematic block diagram of an image sensor configured in accordance with various embodiments of the present technology.

[0016] Figure 6 is a partial schematic diagram of a deblurring circuit configured in accordance with various embodiments of the present technology.

[0017] Figure 7 are diagrams illustrating (a) an exposure period of an example image frame and (b) four corresponding latent image frames that have been deblurred in accordance with various embodiments of the present technology.

[0018] Figure 8 are plots illustrating (a) an exposure period of an example image frame and (b) four corresponding latent image frames that have been deblurred and corrected for rolling shutter distortion in accordance with various other embodiments of the present technology.

[0019] Fig.9A is a partial schematic diagram of a deblurring and rolling shutter distortion correction circuit configured in accordance with various embodiments of the present technology.

[0020] Fig. 9B is a partial schematic diagram of another deblurring and rolling shutter distortion correction circuit configured in accordance with various embodiments of the present technology.

[0021] Fig.10 is a flow chart illustrating a method of operating an image sensor according to various embodiments of the present technology.

[0022] Fig.11 and 12 is a timing diagram corresponding to a method of operating an image sensor and / or a corresponding imaging system according to various embodiments of the present technology.

[0023] Fig.13A is a partial schematic diagram illustrating an event-driven sensing array and a row-scan readout scheme configured in accordance with various embodiments of the present technology.

[0024] Fig. 13B is the use of various embodiments according to the present technology Fig.13A The line scan readout scheme from Fig.13A Plotting of detected events from event-driven sensing array readout.

[0025] Fig.14 is a flow chart illustrating another method of operating an image sensor according to various embodiments of the present technology.

[0026] Fig.15A and 15B is a timing diagram corresponding to a method of operating an image sensor and / or a corresponding imaging system according to various embodiments of the present technology.

[0027] Fig.16 is a flow chart illustrating yet another method of operating an image sensor in accordance with various embodiments of the present technology.

[0028] Fig.17A and 17B is a timing diagram corresponding to a method of operating an image sensor and / or a corresponding imaging system according to various embodiments of the present technology.

[0029] Fig.18 is a partial schematic diagram illustrating an imaging system configured in accordance with various embodiments of the present technology.

[0030] Fig.19 is a partial schematic diagram illustrating another imaging system configured in accordance with various embodiments of the present technology.

[0031] Fig. 20 is a partial schematic diagram illustrating a video frame interpolation pipeline configured in accordance with various embodiments of the present technology.

[0032] Those skilled in the art will appreciate that the elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the figures may be exaggerated relative to other elements to aid in understanding various aspects of the present technology. In addition, common but well-known elements or methods that are useful or required in commercially feasible embodiments are generally not depicted in the figures or described in detail below so as not to unnecessarily obscure the description of various aspects of the present technology. DETAILED DESCRIPTION

[0033] The present disclosure generally relates to imaging systems with video interpolation. For example, several embodiments disclosed herein relate to imaging systems that utilize event-based vision sensor (EVS) data to interpolate video / image frames using complementary metal oxide semiconductor (CMOS) image sensor (CIS) data. In some embodiments, the imaging system also utilizes EVS data to deblur CIS data using accumulated event data to produce deblurred image / video data corrected for rolling shutter distortion. The imaging system may include an EVS sensor and a separate active (CIS) sensor. In other embodiments, the imaging system includes a hybrid image sensor that includes both EVS sensing and active sensing components. In at least some of these embodiments, deblurring CIS data and / or correcting rolling shutter distortion of CIS data using EVS data may be performed at least partially on-chip and then output to a downstream processor (e.g., an application processor) of the imaging system.

[0034] In the following description, specific details are set forth to provide a thorough understanding of aspects of the present technology. However, one skilled in the relevant art will recognize that the systems, devices, and techniques described herein may be practiced without one or more of the specific details set forth herein or with other methods, components, materials, etc.

[0035] Reference throughout this specification to an "example" or "embodiment" means that a particular feature, structure, or characteristic described in connection with the example or embodiment is included in at least one example or embodiment of the present technology. Thus, the use of the phrases "for example," "as an example," or "embodiment" herein does not necessarily all refer to the same example or embodiment and is not necessarily limited to the particular example or embodiment discussed. Furthermore, the features, structures, or characteristics of the present technology described herein may be combined in any suitable manner to provide additional examples or embodiments of the present technology.

[0036] For ease of description, spatially relative terms (e.g., "below," "below," "above," "below," "above," "up," "top," "bottom," "left," "right," "center," "middle," and the like) may be used herein to describe the relationship of one element or feature relative to one or more other elements or features, as illustrated in the figures. It should be understood that, in addition to the orientation depicted in the figures, spatially relative terms are intended to cover different orientations of the device or system in use or operation. For example, if the device or system illustrated in the figures is rotated, turned, or flipped around a horizontal axis, an element or feature described as being "below" or "below" or "below" one or more other elements or features may be oriented to be "above" one or more other elements or features. Therefore, the exemplary terms "below" or "below" are non-limiting and may cover both above and below orientations. The device or system may be additionally or alternatively oriented in a manner other than that illustrated in the figures (e.g., rotated 90 degrees around a vertical axis or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly. In addition, it will also be understood that when an element is referred to as being “between” two other elements, it can be the only element between the other two elements, or one or more intervening elements may also be present.

[0037] Throughout this specification, several terms of the art are used. These terms have their ordinary meaning in the art, unless specifically defined herein or the context of their use clearly indicates otherwise. It should be noted that element names and symbols (e.g., Si vs. silicon) are used interchangeably throughout this document; however, both have the same meaning.

[0038] A. Overview

[0039] The active pixel sensor employs a pixel array for capturing an intensity image / video of an external scene. More specifically, the pixels are used to obtain CIS information (e.g., intensity information) corresponding to light from the external scene incident on the pixels. The CIS information obtained during an integration period is read out at the end of the integration period and used to generate a corresponding intensity image of the external scene.

[0040] The pixels of an active pixel sensor typically have a globally defined integration time. Thus, the pixels in an array of an active pixel sensor typically have the same integration time, and each pixel in the array is typically converted into a digital signal regardless of its content (e.g., whether there have been changes in the external scene captured by the pixel since the pixel was last read out). Thus, relatively large amounts of memory and power are required to operate an active pixel sensor at a high frame rate. Therefore, due in part to memory and power constraints, it is difficult to use active pixel sensors alone to obtain intensity images / videos of an external scene at ultra-high frame rates.

[0041] Furthermore, when motion or other changes occur in the external scene during the integration period, motion artifacts may be observed as blur in the resulting intensity image of the external scene. Blurring may be particularly prominent in low light conditions where longer exposure times are used. Thus, active pixel image sensors are not inherently good at obtaining sharp intensity images / videos of highly dynamic scenes.

[0042] In contrast, an event vision sensor (e.g., an event-driven sensor or a dynamic vision sensor) employs EVS pixels that can be used to obtain non-CIS information (e.g., contrast information, intensity changes, event data) corresponding to light from an external scene incident on those EVS pixels. The event vision sensor reads out the EVS pixels and / or converts the corresponding pixel signals into digital signals only when the EVS pixels detect a change in the external scene (e.g., an event). In other words, the EVS pixels of the event vision sensor that do not detect a change in the external scene are not read out, and / or the pixel signals corresponding to such EVS pixels are not converted into digital signals (thereby saving power). Therefore, each EVS pixel of the event vision sensor can be independent of the other EVS pixels of the event vision sensor, and only the EVS pixels that detect a change in the external scene need to be read out and / or have their corresponding pixel signals converted into digital signals. Therefore, unlike active pixel sensors with synchronized integration times, event vision sensors are not affected by limited dynamic range and can accurately capture high-speed motion. Therefore, event vision sensors are generally more robust than active pixel sensors in weak lighting conditions and / or highly dynamic scenes because they are not affected by under / overexposure or motion blur associated with synchronized shutters. In other words, event vision sensors can be used to provide ultra-high frame rates and accurately capture high-speed motion.

[0043] A hybrid image sensor employs a pixel array comprising a combination of: (i) active (CIS) pixels, which can be used to obtain CIS information corresponding to light from an external scene; and (ii) EVS pixels, which can be used to obtain non-CIS information corresponding to light from an external scene. Such a hybrid image sensor is thus capable of simultaneously capturing (a) an intensity image / video of an external scene and (b) events occurring within the external scene. The event data captured by the EVS pixels can be used to address / mitigate (i) the low frame rate intensity image problem discussed above and (ii) the blurring effects inherent in intensity images captured using CIS pixels in the presence of motion. For example, using an event-based double integral (EDI) model, a high frame rate intensity image / video of an external scene can be reconstructed from a single (e.g., blurred) intensity image and its event sequence.

[0044] For clarity and understanding, a description of the EDI model is provided here. The instantaneous intensity (or irradiance / flux) at a pixel (x, y) at any time t is related to the rate at which photons arrive at that pixel (x, y) and is called the latent image L xy (t). Latent image L xy (t) is not directly output from the hybrid image sensor corresponding to the pixel (x, y). Instead, the hybrid image sensor outputs: (i) an intensity image (e.g., a blurred image) representing multiple latent images (including the latent image L) captured by one or more CIS pixels during the exposure period xy (t)); and (ii) a sequence of events that record intensity changes between latent images captured during the exposure period. A description of how events can be detected is provided below, followed by how the detected events can be used to obtain a latent image L of a pixel. xy Because each pixel of the image sensor can be processed individually, the subscripts x and y are omitted from the following equations and variables for readability. However, it should be understood that the latent image L of a pixel in the pixel array at a given time xy (t) represents the latent image L of the entire pixel array at the given time. F (t). Therefore, the latent image L of the entire pixel array at any time F (t) can be determined by determining the latent image L of each pixel in the array at the given time xy (t) to obtain.

[0045] Figure 1 1 is a partial schematic diagram of an EVS pixel 100 configured in accordance with various embodiments of the present technology. The EVS pixel 100 is also referred to herein as an “event sensing front end circuit.” As shown, the EVS pixel 100 includes a photosensor 101, a logarithmic amplifier 102, a buffer 103, a difference detector 104, and an up / down comparator 105.

[0046] The photosensor 101 is configured to photogenerate an image charge (photocurrent) in response to incident light. The photocurrent photogenerated by the photosensor 101 at time t is proportional to the latent image L(t) at time t (eg, the irradiance of the incident light), as indicated by the following equation 1:

[0047] Equation 1: I photo (t)∝L(t)

[0048] As discussed above, the latent image L(t) represents the instantaneous intensity at the EVS pixel 100 at time t related to the rate at which photons arrive at the EVS pixel 100 .

[0049] The photocurrent generated by the photosensor 101 is fed to the logarithmic amplifier 102. The logarithmic amplifier 102 in turn converts (a) the photocurrent linearly proportional to the latent image L(t) into (b) a voltage V logarithmically dependent on the latent image L(t). FE , as indicated by the following equation 2:

[0050] Equation 2: V FE ∝ln[I photo (t)])∝ln[L(t)]

[0051] Temporal contrast (also referred to herein as “linear contrast”) is defined as the light contrast at the EVS pixel 100 (eg, at the photosensor 101) relative to a reference time t 0 The change of is given by the following equation 3:

[0052] Equation 3:

[0053] The difference detector 104 of the EVS pixel 100 is used to monitor the temporal contrast of light incident on the photosensor 101. More specifically, when reset, the difference detector 104 samples the reference time t 0 The voltage V FE And then generate the output V shown by the following equation 4 O :

[0054] Equation 4:

[0055] The output V of the difference detector 104 O Tracking voltage V FE Over time relative to a reference time t 0 The voltage V FE As shown in the following equation 5, as the voltage V FE As time goes by, the output V of the difference detector 104 O The corresponding change in is proportional to the logarithm of the temporal contrast:

[0056] Equation 5:

[0057] The output V of the difference detector 104 O is fed to the up / down comparator 105. The up / down comparator 105 in turn compares the output V O The corresponding threshold voltage V is given by Equation 6 below +TH and V -TH , where C log-TH is a contrast threshold parameter that determines whether an event should be recorded. As shown by the following equation 7, when the output V of the difference detector 104 O Exceeding the threshold voltage V+TH When the output ΔV of the difference detector 104 is ΔV, the upward comparator detects an event. As shown by the following equation 8, when the output ΔV of the difference detector 104 is ΔV, the upward comparator detects an event. O Less than the threshold voltage V -TH , the down comparator detects an event.

[0058] Equation 6: V ±TH = ±β·C log-TH

[0059] Equation 7:

[0060] Equation 8:

[0061] Using the log ratio rule, Equations 7 and 8 provide Equations 9 and 10, respectively, that specify when the EVS pixel 100 detects an event:

[0062] Equation 9: ln[L(t i )]≥ln[L(t i-1 )]+C log-TH

[0063] Equation 10: ln[L(t i )]≤ln[L(t i-1 )]-C log-TH

[0064] In the above equation, time t i corresponds to the time when each event is detected, and time t i-1 Indicates the timestamp of the previous event. When an event is triggered in an EVS pixel, L(t i-1 ) is updated to a new intensity level (e.g., by resetting the difference detector 104 so that the difference detector 104 newly samples the voltage V FE ). Therefore, at time t i The detected event indicates that the i-1 The previous event detected occurred when the contrast threshold parameter C log-TH In other words, each detected event indicates a latent image (current latent image L(t i ) and the previous latent image L(t i-1 )) between the intensity changes. Therefore, the above equations 9 and 10 provide where c is equal to C log-TH And p i is the following equation 11 for the event polarity:

[0065] Equation 11: ln[L(t i )]=ln[L(t i-1 )]+c·p i

[0066] Every time t i The event polarity p i Given by the following equation 12. The polarity p of +1 i The irradiance of light incident on the photosensor 101 increases, and the polarity p of -1 i This indicates that the irradiance of light incident on the photosensor 101 is reduced.

[0067] Equation 12:

[0068] At every time t i The detected events can be calculated by multiplying the unit pulse (Dirac function δ) by the corresponding polarity p i The detected events can be defined as continuous time functions. For example, Figure 2 Various plots 210, 212, 214, 216, and 218 are illustrated. Plot 210 illustrates the exposure period of a CIS pixel row in a pixel array for one image frame. A rolling shutter is used such that the exposure period of a pixel row in the pixel array is staggered with the readout. The exposure period of the top two rows of the CIS pixel array extends from time t 0 With t 0+T and the exposure period of the bottom two rows of the CIS pixel array extends to time t s With t s+T Graph 212 illustrates the sequence of events detected by the EVS pixels corresponding to the bottom two rows of the CIS pixel array. More specifically, at time t 0 With time t s+T The sequence of events detected between is plotted as a time-continuous signal e(t) comprising a sequence of unit pulses. The time-continuous signal e(t) is modeled by the following equation 13:

[0069] Equation 13: e(t) = p i ·δ(tt i )

[0070] Because each event detected by an EVS pixel indicates a change between latent images captured at different times, the proportional change in intensity at the pixels of the bottom two rows of the CIS pixel array over the exposure period of those rows of pixels can be represented by the intensity of the corresponding EVS pixel at time t s The sum (or combination) of events detected between t and time t provides, as shown in Equation 14 below:

[0071] Equation 14: Figure 2 The plot 214 illustrates a time-continuous signal E(t) representing the exposure period corresponding to the bottom two rows of the CIS pixel array at time t sWith time t s+T Between Figure 2 The sum of events in the time-continuous signal e(t) of the plot 212 .

[0072] When ln[L(t i-1 )] is known, using the above equations 11 and 12, ln[L(t i )]. Thus, given a sequence of events specified by a time-continuous signal e(t) and assuming that c in Equation 11 above remains constant, it is possible (in the linear domain) to s The latent image L(t) of the bottom two rows of CIS pixels of the array at any given time t is determined by incrementing the starting latent image L(s) at time t over all events up to time t, as shown by the following equation 15:

[0073] Equation 15:

[0074] As discussed above with reference to Equation 1, the photocurrent generated by photosensor 101 is proportional to the latent image L(t) (or irradiance) at time t. Therefore, the latent image L(t) is proportional to the irradiance at time t. s With time t s+T The integral over the exposure period between t corresponds to the “charge” and can be related to the frame information captured by the CIS pixels of the frame-based image sensor. Furthermore, as discussed above, the blurred intensity image captured by the frame-based image sensor can be viewed as a latent image sequence extending over time t s With time t s+T The integral of the exposure period between (during which events are accumulated) or the integral of the combination of multiple latent images captured by the frame-based image sensor within the exposure period. Therefore, using the above equation 15, the blurred frame B captured by the frame-based image sensor can be expressed by the following equation 16:

[0075] Equation 16: Figure 2 Description of the drawing 216 Figure 2 The time continuous signal E(t) is plotted 214 from time t s To time t s+T The integral within the time.

[0076] The above equation 16 is called the EDI model and provides the blur frame B captured by the frame-based image sensor and the time t s (corresponding to the start of the frame / exposure period of the CIS pixel). This relationship can be rearranged to find the latent image L(s), as shown by the following Equation 17:

[0077] Equation 17:

[0078] The latent image L(s) in Equation 17 above is interpreted as a deblurred frame based on (a) a blurry frame B captured by a frame-based image sensor and (b) events detected by the EVS pixels across corresponding exposure periods. In other words, because the events detected by the EVS pixels during the exposure period indicate changes between the latent images captured by one or more active (CIS) pixels during the exposure period, the detected events can be used to perform event-directed deblurring.

[0079] As discussed above, a rolling shutter can be used to capture and read out CIS information. For example, Figure 2 The exposure period of the top two rows of the CIS pixel array shown in the plot 210 is at time t 0 Start, time t 0 The exposure period for the bottom two rows of the pixel array begins at time t s Before. Therefore, at least some of the CIS information captured by the top two rows of CIS pixels is captured at a different time than at least some of the CIS information captured by the bottom two rows of CIS pixels. Thus, when (a) the CIS information captured by the top two rows and the bottom two rows of CIS pixels of the CIS pixel array are read out and (b) the latent image L(0) of the top two rows of CIS pixels and the latent image L(s) of the bottom two rows of CIS pixels are determined using the above equation 17, the latent image L obtained from the CIS information captured by the CIS pixels of the entire array is F (s) may include rolling shutter distortion (e.g., due to the presence of 0 With time t s movement between them).

[0080] Events detected by the EVS pixels corresponding to the bottom two rows of the pixel array can be used to correct this distortion. More specifically, at time t 0 With time t s Events detected by EVS pixels between Figure 2 218) can be used to provide rolling shutter correction for the CIS information captured by the bottom two rows of CIS pixels. Specifically, at time t 0 (corresponding to the start of the exposure period for the top two pixel rows of the CIS pixel array) and time t s The proportional change in intensity during the period between t and t (corresponding to the start of the exposure period for the bottom two rows of the CIS pixel array) can be determined by the time t 0 With time t s The sum (or combination) of events detected by corresponding EVS pixels between is provided as shown in Equation 18 below:

[0081] Equation 18:

[0082] Figure 2 The plot 218 illustrates the time continuous signal E'(t), which represents the time at time t 0 With time t s Between Figure 2 The sum of events in the time-continuous signal e(t) of the plot 212 .

[0083] As discussed above, when ln[L(t i-1 )] is known, equations 11 and 12 can be used to determine ln[L(t i )]. Therefore, given that at time t 0 With time t s The sequence of events specified by the time continuous signal e(t) between the two can be (in the linear domain) calculated by (i) 0 To time t s The CIS pixel at time t is determined by integrating / incrementing over all events detected by the corresponding EVS pixel and (ii) multiplying the accumulated events by the starting latent image L(0). s The latent image L(s) of is shown in the following equation 19:

[0084] Equation 19:

[0085] Therefore, in view of the above equations 17 and 19, the latent image L(0) of the CIS pixel can be solved and obtained using the following equation 20:

[0086] Equation 20:

[0087] L(0) in Equation 20 above corresponds to the deblurred rolling shutter distortion correction latent frame. Therefore, the latent frame L of the entire pixel array is F (0) can be obtained using the latent frame L(0) of the individual CIS pixels in the array.

[0088] As discussed above, Equation 15 can be used to calculate the time from time t s The latent image L(t) of these CIS pixels at any given time t is determined by incrementing the starting latent image L(s) at time t to all events corresponding to the CIS pixels at time t. Using the above equations 15 and 19, the latent image L(t) at any given time t can be obtained by 0 (corresponding to Figure 2 ), as shown by the following Equation 21:

[0089] Equation 21: L(t) = L(0) exp(c∑i∈[0,s] p i )·exp(c∑ i∈[s,t] p i )

[0090] As discussed in more detail below, Equation 15 and / or Equation 21 may be used for video interpolation. Video interpolation (VFI) is a technique that involves generating additional (e.g., otherwise non-existent) frames of video / image data between consecutive video / image frames. For example, referring again to Figure 2 The plot 210 corresponds to time t 1 , time t 2 and time t s+T Three additional frames of video / image data can be generated using the starting latent image L(s) and Equation 15 above. 1 , time t 2 and time t s+T exist Figure 2 The start time of the image frame (time t s ) and before the next image frame (not shown) begins. Therefore, the latent image L(t 1 )、L(t 2 ) and L(t s+T ) can be used as Figure 2 The three additional video / image frames that occur between the image frame in and the next image frame.

[0091] Video interpolation is commonly used to make video playback smoother and more fluid, for example by making the video appear to have a higher refresh rate. Video interpolation is also commonly used in various video processing applications, such as video compression and video restoration. Additionally or alternatively, video interpolation can be used to implement slow motion video.

[0092] Several embodiments of the present technology described in detail below relate to imaging systems with event-directed video interpolation. For example, several embodiments of the present technology relate to imaging systems with hybrid image sensors that capture CIS data and corresponding EVS data. Continuing with this example, the hybrid image sensor can be configured to accumulate EVS data and output the accumulated EVS data to downstream components of the imaging system (e.g., an application processor). The accumulated EVS data can in turn be used to generate one or more interpolated video frames based on the CIS data.

[0093] In at least some of these embodiments, the imaging system may further utilize EVS data (e.g., raw EVS data or accumulated EVS data) to deblur the CIS data and / or correct rolling shutter distortion of the CIS data. This deblurring and / or rolling shutter distortion correction may be performed on-chip (e.g., on a hybrid image sensor) such that the hybrid image sensor is configured to output the deblurred and / or rolling shutter distortion corrected CIS data to a downstream application processor or image signal processor of the imaging system. On-chip deblurring and / or on-chip rolling shutter distortion correction may avoid many of the disadvantages discussed in detail below with reference to off-chip deblurring and / or off-chip rolling shutter distortion correction techniques. In other embodiments of the present technology, deblurring and / or rolling shutter distortion correction may be performed off-chip (e.g., outside the hybrid image sensor, e.g., after the hybrid image sensor outputs raw CIS data, raw EVS data, and / or accumulated EVS data).

[0094] As discussed above, many event-directed deblurring solutions use an active image sensor to capture CIS data and a separate event vision sensor to capture EVS data. However, such a dual sensor configuration has several disadvantages, such as parallax introduced due to the uncombined sensors, complexity of spatial and temporal synchronization of the CIS data with the EVS data, and increased cost (e.g., due to the need for two pairs of lenses, packaging, etc.). Although these disadvantages can be overcome by using a hybrid image sensor, all existing event-directed deblurring solutions known to the inventors of the present disclosure output the CIS data and EVS data to an application processor external to the hybrid image sensor so that the application processor performs off-chip event-directed deblurring and rolling shutter distortion. Such off-chip event-directed deblurring solutions and rolling shutter correction solutions have several additional disadvantages, many of which are discussed below with reference to FIG. Figure 3 Discussion.

[0095] Figure 3 3 is a partial schematic diagram illustrating an example of an imaging system 320 that performs off-chip event-directed deblurring. As shown, CIS data 321 and EVS data 322 are output from (a) a hybrid image sensor (not shown) of imaging system 320 or (b) an active pixel sensor (not shown) and a separate event vision sensor (not shown) of imaging system 320 to an application processor 323 of imaging system 320. Application processor 323 is configured to (a) perform event-directed deblurring of CIS data 321 using EVS data 322 and (b) output a deblurred image frame to image signal processor 352.

[0096] One disadvantage of an off-chip event-oriented deblurring solution is that a relatively large amount of memory is required. For example, to perform event-oriented deblurring, the application processor 323 uses a first buffer 324, a second buffer 325, a third buffer 326, and a fourth buffer 327. More specifically, the first buffer 324 is used to store two frames of CIS image data. One of the frames is used to synchronize CIS data 321 with EVS data 322 at the CIS / EVS synchronization block 328 of the application processor 323, and the other of the frames is deblurred by the application processor 323 using EVS data 322 at the deblurring block 329 of the application processor 323. In one example, for a 12-megapixel image sensor, the size of the first buffer 324 may be approximately 44MB. The second buffer 325 is used to store the EVS data 322 before decoding, and the third buffer 326 is used to store the EVS data 322 after decoding. In one example, the second buffer 325 may be configured to store the EVS data 322 for approximately 50ms and thus may be approximately 90MB in size. Thus, depending on the decoding, the size of the third buffer 326 may be approximately 100 to 400 MB. The fourth buffer 327 may be configured to store deblurred image frames and thus may be approximately 22 MB in size for a 12 megapixel image sensor. Thus, continuing with the above example, the application processor 323 may require approximately 550 MB of memory to perform event-directed deblurring of the CIS data 321 using the EVS data 322.

[0097] Other disadvantages of off-chip event-based deblurring solutions include: (i) relatively large latency and (ii) inability to support real-time video. Figure 3 As shown in FIG. 3 , CIS data 321 and EVS data 322 are output to an external application processor 323 (causing a 1-frame delay). In addition, the application processor 323 introduces additional delay when performing event-directed deblurring calculations. For example, it may take up to 1 second for the application processor 323 to process a 12-megapixel image. Figure 3 The described off-chip event-directed deblurring solution is only able to process still images, which means that this solution cannot support real-time video.

[0098] Other disadvantages of off-chip event-directed deblurring solutions include (i) requiring relatively high input / output (IO) bandwidth between the image sensor and the external application processor and (ii) consuming relatively large amounts of power. Figure 3, both CIS data 321 and EVS data 322 are output from the image sensor to an application processor 323, which requires a relatively high IO throughput (e.g., about 20 Gbps for a 12 megapixel, 30fps image sensor). This high IO throughput and long data processing time (e.g., up to 1 second, as discussed above) results in a relatively large amount of power being consumed.

[0099] Another disadvantage of off-chip event-directed deblurring solutions is the complexity of the interface between the image sensor and downstream components of the imaging system. For example, instead of outputting a deblurred image frame, Figure 3 The image sensor (not shown) of the imaging system 320 outputs raw CIS data 321 and raw EVS data 322, which requires a relatively complex application processor 323 and an interface corresponding to the image sensor.

[0100] To address at least some of these issues, several embodiments of the present technology described herein are generally directed to hybrid image sensors with on-chip image deblurring and rolling shutter distortion correction capabilities. For example, several embodiments of the present technology described in detail below are directed to image sensors with the following on-chip capabilities: (a) synchronization of CIS data captured using active (CIS) pixels and EVS data captured using EVS pixels; (b) image deblurring; and / or (c) rolling shutter distortion correction. In some embodiments, on-chip image deblurring may include on-chip event-directed deblurring. In these and other embodiments, on-chip rolling shutter distortion correction may include on-chip event-directed rolling shutter distortion correction.

[0101] Embodiments of the present technology that include on-chip image deblurring and / or on-chip rolling shutter distortion correction are expected to provide several advantages. For example, compared to the off-chip event-directed deblurring solutions discussed above, the present technology is expected to reduce or minimize: (a) the amount of memory required to perform image deblurring and rolling shutter distortion correction; (b) the latency associated with performing image deblurring and rolling shutter distortion correction; (c) the required IO bandwidth / throughput; and / or (d) the amount of power required to perform image deblurring and rolling shutter distortion correction. Thus, in addition to still image processing, the present technology is expected to support real-time video. In addition, because image deblurring and rolling shutter distortion correction are performed on-chip (e.g., completely or partially within the image sensor and / or without the need to first output or require the output of raw CIS data and / or raw EVS data from the image sensor), image sensors configured in accordance with various embodiments of the present technology are able to output deblurred, rolling shutter distortion corrected image frames (e.g., in addition to or in place of raw CIS data and / or raw EVS data), which means that the interface between such image sensors and downstream components of the corresponding imaging system can be simplified compared to the off-chip event-directed deblurring solution discussed above.

[0102] B. Selected embodiments of hybrid image sensors with video interpolation and associated systems, devices, and methods

[0103] Figure 4A 4 is a partial schematic diagram of a stacked complementary metal oxide semiconductor (CMOS) image sensor (CIS) having an event-based vision sensor (EVS) system 430 ("stacked system 430" or "image sensor 430") configured in accordance with various embodiments of the present technology. As shown, the stacked system 430 includes a first die 432, a second die 434, and a third die 436 stacked and coupled together in a stacked chip scheme. In some embodiments, the first die 432, the second die 434, and the third die 436 are semiconductor dies including suitable semiconductor materials, such as silicon. In the illustrated embodiment, the first die 432 (also referred to herein as the "top die") includes a pixel array 438. The third die 436 (also referred to herein as the "bottom die") includes an image readout circuitry 446 (also referred to herein as the "image readout mixed signal circuitry"). The image readout circuitry 446 can be coupled to the pixel array 438 of the first die 432 through a column level connection 440 for normal image readout. In some embodiments, column level connections 440 for normal image readout are implemented from column bit lines of pixel array 438 with through silicon vias (TSVs) extending between first die 432 and third die 436 and routed through second die 434 .

[0104] In some embodiments, the pixel array 438 is a two-dimensional (2D) array including a plurality of pixel units (also referred to as "pixels"), each of which includes at least one photosensor (e.g., at least one photodiode) exposed to incident light. As shown in the illustrated embodiment, the pixels are arranged in rows and columns. Some of the pixels may be configured as CMOS image sensor (CIS) pixels, which are configured to acquire image data of a person, location, object, etc., which may then be used to reproduce images and / or videos of the person, location, object, etc. For example, each CIS pixel is configured to photogenerate an image charge in response to incident light. After each CIS pixel acquires its image charge, the corresponding analog image charge data may be read out by the image readout circuitry 446 in the third die 436 through the column bit lines. In some embodiments, the image charge from each row of the pixel array 438 may be read out in parallel by the image readout circuitry 446 through the column bit lines. As discussed in more detail below, other of the pixels of the pixel array 438 may be configured as event vision sensor (EVS) pixels.

[0105] The image readout circuitry 446 in the third die 436 may include amplifiers for normal image readout and processing, analog / digital converter (ADC) circuitry, associated analog support circuitry, associated digital support circuitry, and the like. In some embodiments, the image readout circuitry 446 may also include event-driven readout circuitry, as will be described in more detail below. In operation, photogenerated analog image charge signals are read out from the pixel cells of the pixel array 438, amplified in the image readout circuitry 446 and converted into digital values. In some embodiments, the image readout circuitry 446 may read out a row of image data at a time. In other examples, the image readout circuitry 446 may read out the image data using various other techniques (not illustrated) such as serial readout or full parallel readout of all pixels simultaneously. The image data may be stored or even manipulated by applying post-image effects such as cropping, rotation, red-eye removal, adjusting brightness, adjusting contrast, and the like.

[0106] In the illustrated embodiment, the second die 434 (also referred to herein as the "middle die") includes an event-driven sensing array 442 coupled to at least some of the pixels (e.g., EVS pixels) of the pixel array 438 in the first die 432. In some embodiments, the event-driven sensing array 442 is coupled to the pixels of the pixel array 438 by hybrid bonding between the first die 432 and the second die 434. The event-driven sensing array 442 may include an array of event-driven circuits. In some embodiments, each of the event-driven circuits in the event-driven sensing array 442 is coupled to at least one of a plurality of pixels of the pixel array 438 by hybrid bonding between the first die 432 and the second die 434 to asynchronously detect an event occurring in light incident on the pixel array 438 in accordance with the teachings of the present disclosure.

[0107] In some embodiments, the corresponding event detection signal is generated by an event-driven circuit (e.g., similar to Figure 1 The event detection signal may be received and processed by the event-driven peripheral circuit system 444. In some embodiments, the event-driven peripheral circuit system 444 is arranged in the second die 434 around the periphery of the event-driven sensing array 442, such as Figure 4A Displayed in. Figure 4A The embodiment illustrated in FIG. 4 also illustrates column level connections 440 for normal image readout being routed through the second die 434 between the first die 432 and the third die 436 .

[0108] Figure 4B yes Figure 4A A partial schematic diagram of a specific example of a stacked system 430. Figure 4BAs shown in FIG. 4 , the stacked system 430 includes a pixel array 438 ( Figure 4B 438 is shown in the figure), the event-driven circuit 400 of the event-driven sensing array 442 on the second die 434, and the image readout circuit system 446 on the third die 436. The image readout circuit system 446 includes an analog-to-digital converter 451 ("ADC 451"), an image signal processor 452, a scanning readout circuit system 453, an event signal processor 454, a synchronous communication interface 455 (e.g., a mobile industry processor interface (MIPI) transmitter and / or receiver), and various auxiliary circuits 456. As discussed in more detail below, the image readout circuit system 446 may also include a deblurring circuit (e.g., for performing event-directed deblurring and / or rolling shutter distortion correction of CIS data).

[0109] Figure 4B The portion of the pixel array 438 shown in 438 corresponds to a 4x4 cluster of pixels in the pixel array 438. This cluster may be repeated across the pixel array 438. In the illustrated embodiment, fifteen (15) pixels in the cluster are configured as active (CIS) pixels 435 to capture CIS information (e.g., intensity information) corresponding to light incident on the photosensors of these pixels. Additionally, one pixel in the cluster is configured as an EVS pixel 437 to capture non-CIS information (e.g., contrast information, event data) corresponding to light incident on the photosensor of the EVS pixel 437. Figure 4C illustrate Figure 4B 4x4 pixel clusters, where CIS pixels 435 are arranged in a Bayer pattern to capture CIS (frame) information corresponding to light incident on the cluster, and EVS pixels 437 are arranged to detect EVS (asynchronous event) information corresponding to light incident on the cluster. Of course, in other embodiments of the present technology, CIS pixels 435 may be arranged in another pattern besides a Bayer pattern.

[0110] Reference again Figure 4B, independently read out the clustered CIS pixels 435 and the clustered EVS pixels 437. More specifically, the CIS information captured by the CIS pixels 435 is read out to the ADC 451 on the third die 436 through the second die 434 using the corresponding row / column control circuit system (not shown). The non-CIS information captured by the EVS pixels 437 is read out to the event-driven circuit 400 on the second die 434 using the corresponding row / column control circuit system (not shown), and the events detected by the event-driven circuit 400 are read out by the scanning readout circuit system 453 on the third die 436. The CIS information captured by the CIS pixels 435 of the pixel array 438 is frame-based and can be read out from the CIS pixels 435 row by row at the end of the exposure period. In contrast, the non-CIS information captured by the EVS pixels 437 is used by the event-driven circuit 400 to asynchronously detect / trigger events, and the events can be read out according to the row scanning readout scheme or the column scanning readout scheme. The row scan readout scheme is discussed in more detail below.

[0111] In some embodiments, the row / column control circuitry corresponding to the CIS pixel 435 may be allocated on the same or different die as the die on which the ADC 451 is allocated (e.g., the third die 436). In these and other embodiments, the row / column control circuitry corresponding to the EVS pixel 437 may be allocated on the same or different die as the die on which the scanning readout circuitry 453 is allocated (e.g., the third die 436). In these and other embodiments, the ADC 451 and / or the row / column control circuitry corresponding to the CIS pixel 435 may be allocated on the same or different die as the die on which the scanning readout circuitry 453 and / or the row / column control circuitry corresponding to the EVS pixel 437 is allocated (e.g., the third die 436).

[0112] In the illustrated embodiment, EVS pixels 437 are dedicated to capturing non-CIS (EVS) information, while CIS pixels 435 are dedicated to capturing CIS information. In other embodiments, one or more of EVS pixels 437 and / or CIS pixels 435 may be switched between being configured to capture CIS information and non-CIS information. This may enable stacked system 430 to operate in the following modes: a CIS-only mode, in which all pixels 435 and pixels 437 are used to capture CIS information; an EVS-only mode, in which all pixels 435 and pixels 437 are used to capture non-CIS (EVS) information; and / or a mixed CIS and EVS mode, in which a first subset of pixels 435, 437 are used to capture CIS information and a second subset of pixels 435, 437 are used to capture non-CIS (EVS) information.

[0113] In some embodiments, the event driven circuit 400 on the second die 434 has the same die size as the 4x4 pixel cluster on the first die 432. In other embodiments, the event driven circuit 400 may have a different die size than the 4x4 pixel cluster. Additionally or alternatively, although the ratio of CIS pixels to EVS pixels in the 4x4 pixel cluster is 15:1, other ratios of CIS pixels to EVS pixels (e.g., 14:2, 12:4, 8:8, 4:12, 2:14, 15:1) are possible and within the scope of the present technology. Furthermore, although Figure 4B The EVS pixels 457 correspond to 4x4 pixel clusters, but other arrangements (e.g., EVS pixels corresponding to 1X1 pixel clusters, 4X2 pixel clusters, etc.) are possible and within the scope of the present technology. Figure 4B In one embodiment, a row of EVS pixels (e.g., a row including EVS pixels 457) corresponds to four rows of CIS pixels, but other arrangements are possible and within the scope of the present technology. For example, each row of EVS pixels may correspond to (a) one row of CIS pixels, (b) two rows of CIS pixels, (c) three rows of CIS pixels, or (d) more than four rows of CIS pixels.

[0114] As discussed above, event data captured using EVS pixels may be used to perform event-directed deblurring and / or rolling shutter distortion correction of CIS (frame) information captured using CIS pixels. To this end, Figure 5 is a partially schematic block diagram of an image sensor 530 that may include on-chip image deblurring and / or rolling shutter correction capabilities and configured in accordance with various embodiments of the present technology. The image sensor 530 may be the Figure 4A and / or Figure 4B Examples of stacked system 430 or other image sensors configured in accordance with the present technology.

[0115] As shown, image sensor 530 includes a CIS pixel array 538 (e.g., similar to Figure 4A , Figure 4B and / or Figure 4C 438) and / or an event-driven sensing array 542 (e.g., similar to Figure 4A , Figure 4B and / or Figure 4CThe image sensor 530 further includes (a) a row / column control circuit system 561 and a column readout circuit system 563 for controlling the operation and readout of the CIS pixels included in the pixel array 538 and (b) a row / column control circuit system 562 for controlling the operation and readout of the EVS pixels included in the event-driven sensing array 542. The image sensor 530 may optionally include a pre-processing circuit 564 for performing various operations (e.g., denoising) on ​​the EVS data read out from the EVS pixels of the event-driven sensing array 542.

[0116] The image sensor 530 further includes a common control block 568 for synchronizing the operation of the pixel array 538 with the operation of the event-driven sensing array 542. More specifically, although the CIS pixels of the pixel array 538 and the EVS pixels of the event-driven sensing array 542 themselves include row / column control circuitry and are independently read by their own readout circuitry, the common control block synchronizes the operation (e.g., reset, exposure start time, exposure end time) of the CIS pixels, EVS pixels, row / column control circuitry, and / or readout circuitry. This synchronization is described below with reference to Figures 10 to 12 Describe in more detail.

[0117] In some embodiments, the image sensor 530 may include a first multiplexer 565, a second multiplexer 566, and / or a third multiplexer 567. As shown, the first multiplexer 565, the second multiplexer 566, and the third multiplexer 567 may be controlled using a deblur enable signal deblurEN. When the deblur enable signal deblurEN is not asserted (e.g., in a low or '0' state), the first multiplexer 565 and the third multiplexer 567 may stream CIS data (e.g., raw intensity image frames, blurred intensity image frames) to the image signal processor 552 of the image sensor 530, e.g., instead of streaming raw CIS data to the deblur and rolling shutter distortion correction circuit 570 ("deblur circuit 570" or "rolling shutter distortion correction circuit 570") of the image sensor 530. Image signal processor 552 may in turn provide the CIS data to a synchronous communication interface 555 a (eg, a MIPI interface / transmitter), for example, for output from image sensor 530 .

[0118] Additionally or alternatively, when the deblur enable signal deblurEN is not asserted (e.g., in a low or '0' state), the second multiplexer 566 may stream the EVS data to the column scan readout circuitry 553, e.g., instead of streaming the EVS data to the deblur circuit 570. The column scan readout circuitry 553 may, in turn, provide the EVS data to an event signal processor 554 of the image sensor 530, and the event signal processor 554 may provide the EVS data to a synchronous communication interface 555b (e.g., a MIPI interface / transmitter), e.g., for output from the image sensor 530.

[0119] In the illustrated embodiment, the synchronous communication interface 555a and the synchronous communication interface 555b may be independent physical interfaces. Alternatively, the synchronous communication interface 555a and the synchronous communication interface 555b may be merged. For example, CIS data and EVS data may be output from the image sensor 530 via a shared synchronous communication interface 555 (e.g., a shared MIPI interface, a shared virtual channel, an embedded line).

[0120] Referring again to the first multiplexer 565, the second multiplexer 566, and the third multiplexer 567, when the deblur enable signal deblurEN is asserted (e.g., in a high or '1' state), the first multiplexer 565 is enabled to stream CIS information read from the CIS pixels of the pixel array 538 into the deblur circuit 570, and the second multiplexer 566 is enabled to stream EVS information read from the EVS pixels of the event-driven sensing array 542 into the deblur circuit 570. For example, when the deblur enable signal deblurEN is asserted, the EVS information read from the EVS pixels of the event-driven sensing array 542 may be continuously streamed into the deblur circuit 570 via the second multiplexer 566 (e.g., when the CIS pixels of the pixel array 538 integrate photo-generated charge during an exposure period). Additionally or alternatively, when the CIS information is read out from the CIS pixels of the pixel array 538 after an exposure period, the digitized CIS information may be streamed into the de-blurring circuit 570 via the first multiplexer 565 .

[0121] The deblurring circuit 570 may (a) compute a fused image / video stream from the CIS data and the EVS data received via the first multiplexer 565 and the second multiplexer 566, respectively, and (b) may output the fused image / video stream into the third multiplexer 567 for streaming to the image signal processor 552. The fused image / video stream may then be provided from the image signal processor 552 to the synchronous communication interface 555a for output from the image sensor 530. The fusion calculations performed by the deblurring circuit 570 may target deblurring of CIS frame information captured by the CIS pixels of the pixel array 538, correcting rolling shutter artifacts, and / or creating interpolated video frames. On-chip deblurring of CIS frame information and rolling shutter correction of CIS frame information are described below with reference to Figures 6 to 20 Discuss in more detail.

[0122] In embodiments where image sensor 530 is a stacked system, components of deblurring circuit 570 may be located on one or more of the dies of the stacked system, such as the top die, the middle die, or the bottom die. As a specific example, image sensor 530 may be substantially similar to the above-described Figure 4A and 4B In one embodiment, the stacked system 430 may include a plurality of image sensors 530, and the deblurring circuit 570 of the image sensor 530 may be located on a third (or bottom) die of the image sensor 530. In other embodiments, at least a portion of the deblurring circuit 570 may be located off-chip (e.g., outside the image sensor 530), such as on a downstream application processor of an imaging system that includes the image sensor 530. In these embodiments, at least a portion of the deblurring and / or rolling shutter distortion correction may be performed off-chip, such as a portion of the video interpolation calculations.

[0123] In some embodiments, the Figure 5 565, the second multiplexer 566 and / or the third multiplexer 567 shown in FIG. In at least some of these embodiments, the image sensor 530 may be substantially similar to Figure 5 5B , the image sensor 530 described in FIG. 5B operates in the manner of how it would operate if the deblur enable signal deblurEN was permanently in an asserted state (e.g., by always streaming CIS information and EVS information into the deblur circuit 570 so that the deblur circuit 570 computes a fused image / video stream). Additionally or alternatively, the image sensor 530 may be configured to output raw CIS data and / or EVS data. For example, the image sensor 530 may output raw CIS data and / or EVS data in addition to or in lieu of outputting a fused image / video stream based on the raw CIS data and EVS data. Furthermore, although Figure 5The image signal processor 552 of the image sensor 530 illustrated in FIG. 5 is configured to process the fused image / video stream output by the deblurring circuit 570 when the deblurring enable signal deblurEN is asserted, but in other embodiments, the image signal processor 552 may be configured to process the CIS information read out from the CIS pixels of the pixel array 538 before the deblurring circuit 570 calculates the fused image / video stream. In such embodiments, after processing the CIS information read out from the CIS pixels of the pixel array 538, the image signal processor 552 may output the processed CIS information to the deblurring circuit 570 so that the deblurring circuit 570 calculates the fused image / video stream based on the processed CIS information (e.g., not based on the original CIS information).

[0124] Figure 6 is a partial schematic diagram of a deblurring circuit 670 configured in accordance with various embodiments of the present technology. The deblurring circuit 670 may be Figure 5 An example of a deblurring circuit 570 or other deblurring circuit configured according to the present technology. As discussed in more detail below, the deblurring circuit 670 may be, for example, on-chip (e.g., Figure 5 The device may perform event-directed deblurring of CIS data on an image sensor 530 of the image sensor 530 and thereafter output the deblurred image frame to a downstream image signal processor and / or a downstream application processor.

[0125] like Figure 6 , the deblurring circuit 670 includes an event-based double integral (EDI) calculation block 671 and a latent frame calculation block 672. The operation of at least a portion of the EDI calculation block 671 may be performed by a time-continuous signal e(t) that at least generally follows the above equation 13 (an example of which is shown in Figure 2 212 of the drawing. In other words, each time an event is triggered and read out from the EVS pixel, the operation of at least a portion of the EDI calculation block 671 can be enabled. The operation of the latent frame calculation block 672 can be timed by the control signal line_sync. The control signal line_sync can be controlled by a common control block (e.g., Figure 5 The control signal line_sync may be provided / controlled by a common control block 568 of the image sensor 530. In some embodiments, the control signal line_sync may correspond to the duration of the exposure period T, as shown in equations 16 and 17 above.

[0126] In the illustrated embodiment, the EDI calculation block 671 includes a plurality of EDI components. More specifically, the EDI calculation block 671 includes a counter 673, a first integral buffer 674, a product calculation block 675, an exponential calculation block 676, an integral calculation block 677, and a second integral buffer 678. The counter 673 may be an integer counter (e.g., an up / down counter) or another suitable type of counter for calculating a running sum of events detected by each EVS pixel during an exposure period, and the first integral buffer 674 may be configured to track / store a running sum over an exposure period.

[0127] The counter 673 and the first integral buffer 674 may be configured to calculate the first inner integral of the EDI model described above. For example, as event data is streamed into the deblurring circuit 670 during an exposure period, each time an event is detected, the counter 673 may be enabled via the control signal EVS_CLK, which may cause the counter 673 to increment or decrement (depending on the polarity of the detected event) the running sum maintained by the first integral buffer 674 for the EVS pixel where the event was detected. The running sum of each of the EVS pixels over time is equal to E(t) shown in Equation 14 above, an example of which is Figure 2 214. For this implementation, the buffer size of the first integral buffer 674 can be relatively small. As a specific example, assuming that time t s With time t s+T The duration of the exposure period between is 33 ms, and the first integration buffer 674 may include a buffer size of 10 bits / pixel. In some embodiments, a floating point calculator may be used instead of the counter 673. In such embodiments, the first integration buffer 674 may have a larger or smaller buffer size.

[0128] The product calculation block 675, the exponent calculation block 676, the integral calculation block 677 and the second integral buffer 678 may be configured to calculate the second outer integral of the above-mentioned EDI model. For example, the product calculation block 675 may multiply the running sum stored in the first integral buffer 674 by the contrast threshold parameter c, which is equal to C log-TH (described above with reference to equations 6-12, 15-17, and 19-21) and assumed to remain constant. Therefore, for each EVS pixel, the output of product calculation block 675 may be equal to c∑ i∈[s,t] p i Thereafter, for each EVS pixel, the exponent calculation block 676 may determine the exponent of the output of the product calculation block 675, thereby obtaining exp(c∑ i∈[s,t] p i ).

[0129] like Figure 66, the output of the exponent calculation block 676 (representing the exponent of the output of the product calculation block 675) can be provided to (i) an integral calculation block 677 and (ii) a downstream application processor, such as an off-chip application processor of an imaging system including a hybrid image sensor incorporating the deblurring circuit 670. As discussed in more detail below, the application processor can be configured to use the output of the exponent calculation block 676 to perform video interpolation. Because the exponent at the output of the exponent calculation block 676 includes accumulated EVS data, in some embodiments, the corresponding raw EVS data read out from the EVS pixels of the corresponding hybrid image sensor can be discarded without the corresponding raw EVS data being read out from the hybrid image sensor and / or to the application processor. Alternatively, for example, in addition to the accumulated EVS data and / or the output of the exponent calculation block 676, the corresponding raw EVS data can also be read out from the hybrid image sensor and / or to the application processor. In some embodiments, the accumulated EVS data stored to the first integration buffer 674 may be read out to a downstream application processor (e.g., for performing video interpolation), e.g., without first providing the accumulated EVS data to the product calculation block 675 and / or the exponent calculation block 676. In these embodiments, the downstream application processor may include one or more EDI components, such as a product calculation block and / or an exponent calculation block, for deblurring the corresponding CIS data and / or for interpolating additional video / image frames.

[0130] The integral calculation block 677 of the EDI calculation block 671 may continuously integrate the output of the index calculation block 676 over time for each EVS pixel. More specifically, the integral calculation block 677 may be configured to calculate the output of the index calculation block 676 from time t s (corresponding to the start of the current exposure period) to time t (at time t s+T The second integration buffer 678 can track / store the results of this time-continuous integration, each of which is equal to the end of the exposure period. The example result of the EVS pixel output by the integral calculation block 677 is Figure 2 This is shown in Figure 216.

[0131] Each of the calculations performed by the product calculation block 675, the exponent calculation block 676, the integral calculation block 677, and the second integral buffer 678 may be performed in a floating point representation, such as a 9-bit mantissa and a 4-bit exponent. In addition, although the operation of the counter 673 and / or the first integral buffer 674 may be clocked by the control signal EVS_CLK, the product calculation block 675, the exponent calculation block 676, the integral calculation block 677, and / or the second integral buffer 678 may be enabled to continuously perform their respective operations over time. As a specific example, in some embodiments, the operation of the integral calculation block 677 is neither clocked by the control signal EVS_CLK nor triggered by an event. Specifically, the integral calculation block 677 is configured to continuously integrate the output of the exponential calculation block 676 over time, at least at times t corresponding to the start and stop times of the corresponding exposure period / EVS accumulation period, respectively. s With time t s+T In these embodiments, the operations of the product calculation block 675 and / or the exponent calculation block 676 may be clocked or enabled by the control signal EVS_CLK to continuously perform their respective operations over time.

[0132] Furthermore, because the events detected at each EVS pixel are accumulated by the EDI calculation block 671, the raw EVS data input into the EDI calculation block 671 of the deblurring circuit 670 may be discarded once the events of the raw EVS data are accumulated by the EDI calculation block 671. Therefore, the second integral buffer 678 only needs to store / maintain the accumulated results of the integral calculation block 677, which means that the second integral buffer 678 may have a relatively small buffer size (e.g., about 9 MB, e.g., about 8.625 MB or about 13 bits / pixel) compared to the buffer used in the off-chip event-oriented deblurring solution. Additionally, because the raw EVS data may be discarded instead of being output from the image sensor corresponding to the deblurring circuit 670, IO throughput and power consumption may be reduced compared to the off-chip event-oriented deblurring solution in which the raw EVS data is output from the image sensor to an external application processor. In other embodiments of the present technology, after the events in the raw EVS data are accumulated, all of the raw EVS data or a subset thereof may be stored and / or output from the image sensor.

[0133] After the exposure period ends, the CIS data may be read out from the CIS pixels of the image sensor and streamed into the latent frame calculation block 672 of the deblurring circuit 670. At this point, the latent frame calculation block 672 may deblur the CIS data by combining / fusing the CIS data with the accumulated EVS data stored in the second integration buffer 678 of the EDI calculation block 671. More specifically, the latent frame calculation block 672 may calculate the time corresponding to time t by performing the operation specified in the above equation 17 for each EVS pixel. s(beginning of the exposure period). The final deblurred image data (e.g., latent image frame L(s)) can be output from the latent frame calculation block 672 to the application processor and / or the image signal processor of the corresponding imaging system. Because the CIS data can be read directly into the latent frame calculation block 672 after the exposure period and because the accumulated EVS data from the second integration buffer 678 is readily available and aligned at this time (as discussed in more detail below), there is no need for a CIS frame buffer to perform on-chip deblurring using the deblurring circuit 670. Therefore, in some embodiments, the deblurring circuit 670 and / or the corresponding image sensor may lack a CIS frame buffer. In other embodiments, the deblurring circuit 670 and / or the corresponding image sensor may include a CIS frame buffer, such as in embodiments in which raw CIS data plus fused image / video data can be output.

[0134] As discussed above, the output of the index calculation block 676 (or the first integration buffer 674) can be streamed to a downstream application processor to enable the application processor to perform video interpolation. In addition, when (i) the exposure period ends, (ii) CIS data is read out from the CIS pixels of the image sensor, and / or (iii) the latent frame calculation block 672 calculates the time corresponding to the time t s After the latent image frame L(s) is calculated by the latent frame calculation block 672, the latent image frame L(s) calculated by the latent frame calculation block 672 can be output to, for example, a CIS key frame buffer of a downstream application processor. Additionally or alternatively, raw CIS data can be read out from the CIS pixels of the image sensor and provided to the downstream application processor. The downstream application processor can, in turn, perform video interpolation (using the output of the index calculation block 676 or the first integration buffer 674, the latent image frame L(s) and / or the raw CIS data and the above equation 15) to calculate the time corresponding to time t s With time t s+T One or more latent image frames L(t) at one or more times between (end of exposure period). One or more latent image frames L(t) may represent one or more additional interpolated image frames, which may be used, for example, to increase the frame rate of the imaging system and / or to generate slow motion video.

[0135] Figure 7 is an illustration of an image frame and four corresponding latent image frames L that can be calculated using corresponding CIS data and EVS data according to various embodiments of the present technology. F (s), L F (t 1 )、L F (t 2 ) and L F (t s+T ) of the exposure period 710. More specifically, according to the above Figure 6 The discussion of defuzzification circuits (e.g. Figure 5 Deblurring circuit 570, Figure 6 The deblurring circuit 670, the chip deblurring circuit) can be used by using Figure 7 The corresponding EVS data accumulated during the exposure period described in t is used to deblur the CIS data to calculate the corresponding s (indicates the start time of the exposure period of the last few CIS pixel rows) F (s).

[0136] In addition, three additional latent frames L F (t 1 )、L F (t 2 ) and L F (t s+T ) can be generated using video interpolation. For example, the downstream application processor (a) can Figure 7 The accumulated EVS data is received from the deblurring circuit during the exposure period described in (b) and the latent image frame L calculated by the deblurring circuit can be obtained by using the accumulated EVS data. F (s) and / or raw CIS data captured during the exposure period to interpolate, for example, using equation 15 above Figure 7 The three additional latent frames L shown in F (t 1 )、L F (t 2 ) and L F (t s+T As part of this process, the downstream application processor may (a) combine the accumulated EVS data with the latent image frame L F (s) and / or corresponding CIS data synchronization, (b) calibrating (e.g., setting or adjusting) the contrast threshold c, (c) with respect to time t 1 , time t 2 and time t s+T Make the latent image frame L F (s) and / or corresponding CIS data deblurring, and / or (d) calculating the latent image frame L F In addition to (s), the latent frame L used as the interpolated video / image frame is also calculated F (t 1 )、L F (t 2 ) and L F (t s+T ).

[0137] like Figure 7 As shown in Figure 2, CIS data can be captured using a rolling shutter. Specifically, Figure 7 The exposure period of the first two rows of CIS pixels shown in FIG. 0 Start, and Figure 7 The exposure period of the last two rows of CIS pixels shown in FIG.s In other words, due to the use of a rolling shutter, there is a delay between the time when the exposure period of the first two CIS pixel rows begins and the time when the exposure period of each of the other CIS pixel rows (including the last two CIS pixel rows) begins. 0 The exposure time of the pixel row of the CIS pixel in the latent image frame L occurs after the start of the exposure time F (s) and / or latent image frame L(t 1 )、L(t 2 ) and L(t s+T ) may have rolling shutter distortion, especially when there are external scenes between time t 0 The motion that occurs between the start times of these exposure periods.

[0138] Thus, in embodiments where a rolling shutter is used, at least some deblurring circuits configured in accordance with various embodiments of the present technology may additionally include a rolling shutter distortion correction component that may be used to correct for distortions that may be present in the calculated latent image frame L F (s) and L F (t). Two such deblurring circuits are described below with reference to Fig.9A and 9B Describe in more detail.

[0139] Figure 8 is an illustration of an example image frame and four corresponding latent image frames L that have each been corrected for rolling shutter distortion according to various embodiments of the present technology. F (0), L F (t 1 )、L F (t 2 ) and L F (t 3 ) of the exposure period 810. Four latent image frames L F (0), L F (t 1 )、L F (t 2 ) and L F (t 3 ) can be calculated using the corresponding CIS data and EVS data. More specifically, according to the following Fig.9A and 9B The discussion of defuzzification circuits (e.g. Figure 5 Deblurring circuit 570, Fig.9A Deblurring circuit 970a, Fig. 9B The deblurring circuit 970b, the chip deblurring circuit) can be used by (i) Figure 8The corresponding EVS data accumulated during the exposure period described in Figure 8 The EVS data accumulated before the start of one or more of the exposure periods described in t is corrected for rolling shutter distortion of the CIS data to calculate the time corresponding to time t 0 (indicates the start time of the exposure period of the first few CIS rows) F (0).

[0140] In addition, three additional latent frames L F (t 1 )、L F (t 2 ) and L F (t 3 ) can be generated using video interpolation. For example, the downstream application processor (a) can Figure 8 The accumulated EVS data is received from the deblurring circuit before and during the exposure period described in (b) and the latent image frame L calculated by the deblurring circuit can be used to calculate the latent image frame L using the accumulated EVS data. F (0) and / or the raw CIS data captured during the exposure period to interpolate, for example, using Equation 21 above Figure 8 The three additional latent frames L shown in F (t 1 )、L F (t 2 ) and L F (t 3 As part of this process, the downstream application processor may: (a) combine the accumulated EVS data with the latent image frame L F (0) and / or corresponding CIS data synchronization; (b) calibrating contrast threshold c; (c) about time t 1 , time t 2 and time t 3 Make the latent image frame L F (s) and / or corresponding CIS data deblurring; (d) with respect to time t 1 , time t 2 and time t 3 Correction latent image frame L F (0) rolling shutter distortion; and / or (e) calculating the latent image frame L F (0) In addition, a latent frame L is calculated for interpolating video / image frames. F (t 1 )、L F (t 2 ) and L F (t 3 ).

[0141] Fig.9Ais a partial schematic diagram of a deblurring and rolling shutter distortion correction circuit 970a ("deblurring circuit 970a" or "rolling shutter distortion correction circuit 970a") configured in accordance with various embodiments of the present technology. Deblurring circuit 970a may be Figure 5 970a includes a rolling shutter distortion correction and event-based double integral (EDI) calculation block 971a ("calculation block 971a") and a latent frame calculation block 972. The operation of at least a portion of the calculation block 971a can be performed by at least generally following the time-continuous signal e(t) above in equation 13 (an example of which is shown in FIG. Figure 2 212 of the drawing. In other words, each time an event is triggered and a pixel is read out from the EVS, the operation of at least a portion of the calculation block 971a can be enabled. The operation of the latent frame calculation block 972 can be timed by the control signal line_sync. The control signal line_sync can be controlled by a common control block (e.g., Figure 5 The control signal line_sync may be provided / controlled by a common control block 568 of the image sensor 530. In some embodiments, the control signal line_sync may correspond to the duration of the exposure period T, as shown in equations 16 and 17 above. In these and other embodiments, the control signal line_sync may correspond to the duration extending between (i) the beginning of the exposure period of the first row of CIS pixels in the CIS pixel array and (ii) the end of the exposure period of the last row of CIS pixels in the CIS pixel array.

[0142] In the illustrated embodiment, computing block 971a includes an EDI component. The EDI component may be substantially similar to Figure 6 The EDI components of the calculation block 671 of the defuzzification circuit 670 are shown in FIG. 6. For example, the EDI components of the calculation block 971a include a counter 973, a first integral buffer 974, a product calculation block 975, an exponent calculation block 976, an integral calculation block 977, and a second integral buffer 978. The counter 973 is also referred to herein as a “first counter”, the product calculation block 975 is also referred to herein as a “first product calculation block”, and the exponent calculation block 976 is also referred to herein as a “first exponent calculation block”. The first integral buffer 974 and the second integral buffer 978 are also referred to herein as “EDI integral buffers”.

[0143] Counter 973 may be an integer counter (e.g., an up / down counter) or another suitable type of counter for calculating a running sum of events detected by each EVS pixel during an exposure period of the CIS pixel corresponding to the EVS pixel, and the first integral buffer 974 may be configured to track / store the running sum. More specifically, counter 973 and first integral buffer 974 may be configured to calculate a first inner integral of the EDI model described above. For example, as event data is detected by this EVS pixel during an exposure period of the CIS pixel corresponding to the EVS pixel and streamed into the deblurring circuit 970a, counter 973 may be enabled via control signal EVS_CLK, which may cause counter 973 to increment or decrement (depending on the polarity of the detected event) a running sum maintained by the first integral buffer 974 for this EVS pixel. For each EVS pixel, the running sum maintained over time by the first integral buffer 974 is equal to E(t) shown in Equation 14 above, an example of which is Figure 2 214. For this implementation, the buffer size of the first integration buffer 974 may be relatively small. As a specific example, assuming that the duration of the exposure period of the CIS pixel corresponding to the EVS pixel is 33ms, the first integration buffer 974 may include a buffer size of approximately 10 bits / pixel. In some embodiments, a floating point calculator may be used instead of the counter 973. In such embodiments, the first integration buffer 974 may have a larger or smaller buffer size.

[0144] The product calculation block 975, the exponent calculation block 976, the integral calculation block 977 and the second integral buffer 978 may be configured to calculate the second outer integral of the above-mentioned EDI model. For example, the product calculation block 975 may multiply the running sum stored in the first integral buffer 974 by the contrast threshold parameter c, which is equal to C log-TH (described above with reference to equations 6-12 and 15-17) and assumed to remain constant. Therefore, for each EVS pixel, the output of product calculation block 975 may be equal to c∑ i∈[s,t] p i Thereafter, for each EVS pixel, the exponent calculation block 976 may determine the exponent of the output of the product calculation block 975, thereby obtaining exp(c∑ i∈[s,t] p i ).

[0145] like Fig.9A, the output of the exponent calculation block 976 (representing the exponent of the output of the product calculation block 975) can be provided to (i) an integral calculation block 977 and (ii) a downstream application processor, such as an off-chip application processor of an imaging system including a hybrid image sensor incorporating the deblurring circuit 970a. As discussed in more detail below, the application processor can be configured to use the output of the exponent calculation block 976 to perform video interpolation. Because the exponent of the output of the exponent calculation block 976 includes accumulated EVS data, in some embodiments, the corresponding raw EVS data read out from the EVS pixels of the corresponding hybrid image sensor can be discarded without the corresponding raw EVS data being read out from the hybrid image sensor and / or read to the application processor. Alternatively, for example, in addition to the accumulated EVS data and / or the output of the exponent calculation block 976, the corresponding raw EVS data can also be read out from the hybrid image sensor and / or read to the application processor. In some embodiments, the accumulated EVS data stored to the first integration buffer 974 may be read out to a downstream application processor (e.g., for performing video interpolation), e.g., without first providing the accumulated EVS data to the product calculation block 975 and / or the exponent calculation block 976. In these embodiments, the downstream application processor may include one or more EDI components, such as a product calculation block and / or an exponent calculation block, for deblurring the corresponding CIS data and / or for interpolating additional video / image frames.

[0146] The integral calculation block 977 of the deblurring circuit 970a may continuously integrate the output of the index calculation block 976 over time for each EVS pixel. More specifically, the integral calculation block 977 may be configured to calculate the output of the index calculation block 976 from time t s (corresponding to the start of the exposure period of the CIS pixel corresponding to the corresponding EVS pixel) to time t (at time t s+T Each of the outputs of the index calculation block 976 is integrated at the end of the exposure period corresponding to the CIS pixel corresponding to the corresponding EVS pixel. The second integration buffer 978 can track / store the results of this time-continuous integration, which are each equal to the end of each corresponding exposure period. The example result of the EVS pixel output by the integral calculation block 977 is Figure 2 This is shown in Figure 216.

[0147] Each of the calculations performed by the product calculation block 975, the exponent calculation block 976, the integral calculation block 977, and the second integral buffer 978 may be performed in a floating point representation, such as a 9-bit mantissa and a 4-bit exponent. In addition, although the operation of the counter 973 and / or the first integral buffer 974 may be clocked by the control signal EVS_CLK, the product calculation block 975, the exponent calculation block 976, the integral calculation block 977, and / or the second integral buffer 978 may be enabled to continuously perform their respective operations over time. As a specific example, in some embodiments, the operation of the integral calculation block 977 is neither clocked by the control signal EVS_CLK nor triggered by an event. Specifically, the integral calculation block 977 is configured to continuously integrate the output of the exponential calculation block 976 over time, at least at times t corresponding to the start and stop times of the corresponding exposure period, respectively. s With time t s+T In these embodiments, the operations of the product calculation block 975 and / or the exponent calculation block 976 may be clocked or enabled by the control signal EVS_CLK to continuously perform their respective operations over time.

[0148] Because the events detected at each EVS pixel during the corresponding exposure period are accumulated by the EDI component of the calculation block 971a, the raw EVS data input into the calculation block 971a of the deblurring circuit 970a during the corresponding exposure period can be discarded once the events of the raw EVS data are accumulated by the EDI component of the calculation block 971a. Therefore, in some embodiments, the second integral buffer 978 only stores / maintains the accumulated results of the integral calculation block 977, which means that the second integral buffer 978 can have a relatively small buffer size (e.g., about 9MB, e.g., about 8.625MB or about 13 bits / pixel) compared to the buffer used in the off-chip event-oriented deblurring solution. In addition, because the raw EVS data can be discarded instead of being output from the image sensor corresponding to the deblurring circuit 970a, the IO throughput and power consumption can be reduced compared to the off-chip event-oriented deblurring solution in which the raw EVS data is output from the image sensor to the external application processor. In other embodiments of the present technology, after the events in the raw EVS data are accumulated, all of the raw EVS data or a subset thereof can be stored and / or output from the image sensor.

[0149] After the exposure period ends, the CIS data may be read out from the CIS pixels of the image sensor and streamed into the latent frame calculation block 972 of the deblurring circuit 970a. At this point, the latent frame calculation block 972 may deblur the CIS data by combining / fusing the CIS data with the accumulated EVS data stored in the second integral buffer 978 of the calculation block 971a. More specifically, for one or more of the CIS pixels, the latent frame calculation block 972 may calculate the respective corresponding time t by using the CIS data captured by the one or more CIS pixels and the corresponding EVS data accumulated in the second integral buffer 978 to perform the operations specified in the above equation 17. s (indicating the start of the corresponding exposure period). Because the CIS data can be read directly into the latent frame calculation block 972 at or after the end of the exposure period and because the EVS data accumulated in the second integration buffer 978 is readily available and aligned with the CIS data at this time (as discussed in more detail below), no CIS frame buffer is required to perform on-chip deblurring using the deblurring circuit 970a. Therefore, in some embodiments, the deblurring circuit 970a and / or the corresponding image sensor may lack a CIS frame buffer. In other embodiments, the deblurring circuit 970a and / or the corresponding image sensor may include a CIS frame buffer, such as in embodiments where raw CIS data plus fused image / video data can be output.

[0150] As mentioned above Figure 7 and 8 As discussed above, a rolling shutter may be used to capture and read out CIS data from a CIS pixel array. Therefore, due to the use of a rolling shutter, there is a delay between the time when the exposure period of the first few CIS pixel rows begins and the time when the exposure period of each of the other pixel rows (including the last few pixel rows) begins. Therefore, rolling shutter distortion may exist in the latent image frame L(s) calculated using the EDI component of calculation block 971a, especially when there is motion in the external scene between the start of a first exposure period of an image frame and the start time of another exposure period corresponding to the image frame. Thus, in embodiments in which a rolling shutter is used, Fig.9A The computation block 971a may additionally include a rolling shutter distortion correction component that may be used to correct rolling shutter distortion of the latent image frame L(s).

[0151] like Fig.9A, the rolling shutter distortion correction component of the calculation block 971a may include a counter 979 (also referred to herein as a “second counter”), an integral buffer 980 (also referred to herein as a “third integral buffer”), a product calculation block 981 (also referred to herein as a “second product calculation block”), and an exponent calculation block 982 (also referred to herein as a “second exponent calculation block”). The integral buffer 980 is also referred to herein as a rolling shutter distortion correction (RSDC) integral buffer.

[0152] The counter 979 may be an integer counter (eg, an up / down counter) or a counter for calculating the number of 0 (corresponding to the start of the first exposure period for a given image frame) and time t s Another suitable type of counter that can be used to store / maintain a running sum of events detected by each EVS pixel between time t and t (indicating the start of another exposure period for the CIS pixel corresponding to the EVS pixel for a given image frame). Additionally, the integration buffer 980 can be configured to store / maintain a running sum of events detected by each EVS pixel between time t 0 With time t s More specifically, counter 979 and integration buffer 980 may be configured to calculate, for each EVS pixel, the running sum at (i) time t 0 and (ii) the start of the exposure period of the CIS pixel corresponding to the EVS pixel (also referred to herein as time t s ) is the integral of events that occur between .

[0153] For example, consider Figure 8 The last two CIS pixel rows are shown in the plot 810 of FIG. Figure 8 and 9A , when the EVS pixels corresponding to the last two CIS pixel rows are at time t 0 With time t s When an event is detected between t and t, the corresponding event data may be read out from the EVS pixel into the computation block 971a of the deblurring circuit 970a. The counter 979 of the deblurring circuit 970a may in turn be enabled (e.g., via the control signal EVS_CLK or another control signal) to increment or decrement (depending on the polarity of the detected event) the running sum maintained by the integration buffer 980 for this EVS pixel. The running sum of the EVS pixel over time is equal to E'(t) shown in Equation 18 above, an example of which is Figure 2 In some embodiments, at time t s Starting with (corresponding to the start of the exposure period of the CIS pixels in the last two rows of CIS pixels), events detected by the EVS pixels may then be accumulated by the EDI component of computation block 971a (e.g., rather than the rolling shutter distortion correction component of computation block 971a).

[0154] Reference again Fig.9A , in some embodiments, the buffer size of the integration buffer 980 can be relatively small. As a specific example, assuming a readout speed of 30 frames / second, the maximum delay between the start of the first exposure period of a given frame and the start of the last exposure period of the given frame can be about 33ms. Continuing with this example, assuming (a) a 12-megapixel CIS pixel array is used, (b) a 3-megapixel EVS pixel array is used, and (c) an EVS readout rate of about 50 nanoseconds / row is used (meaning that the entire EVS pixel array can be read out once every 33ms), the counter 979 and the integration buffer 980 are expected to accumulate up to 400 events between the start of the first exposure period of a given frame and the start of the last exposure period of the given frame. Because the events have positive or negative polarity, the possible values ​​of the running sum calculated by the counter 979 and the integration buffer 980 for each EVS pixel can range from about -400 to about +400. Thus, in some embodiments, the integration buffer 980 can include a buffer size of about 10 bits / pixel. In these and other embodiments, a floating point calculator may be used in place of counter 979. In such embodiments, integration buffer 980 may have a buffer size that is larger or smaller than 10 bits / pixel.

[0155] The product calculation block 981 may multiply the running sum stored in the integration buffer 980 by a contrast threshold parameter c, which is equal to C log-TH (described above with reference to equations 6-12, 15-17, 19, and 20) and assumed to remain constant. Therefore, for each EVS pixel, the output of product calculation block 981 may be equal to c∑ i∈[0,t] p i , assuming that the contrast threshold parameter c remains constant. Thereafter, for each EVS pixel, the exponent calculation block 982 may determine the exponent of the output of the product calculation block 981, thereby obtaining exp(c∑ i∈[0,t] p i). The exponent of the output of the product calculation block 981 determined by the exponent calculation block 982 may be output to (i) a latent frame calculation block 972 of the deblurring circuit 970a and (ii) a downstream application processor, such as an off-chip application processor of an imaging system including a hybrid image sensor incorporating the deblurring circuit 970a. As discussed in more detail below, the application processor may be configured to perform video interpolation using the output of the exponent calculation block 982. Because the exponent at the output of the exponent calculation block 982 includes accumulated EVS data, in some embodiments, the corresponding raw EVS data read out from the EVS pixels of the corresponding hybrid image sensor may be discarded without the need to read out the corresponding raw EVS data from the hybrid image sensor and / or read to the application processor. Alternatively, for example, in addition to the accumulated EVS data and / or the output of the exponent calculation block 976, the corresponding raw EVS data may also be read out from the hybrid image sensor and / or read to the application processor.

[0156] In some embodiments, the calculations performed by the product calculation block 981 and / or the exponent calculation block 982 may be performed in a floating point representation, such as a 9-bit mantissa and a 4-bit exponent. Additionally or alternatively, although the operation of the counter 979 and / or the integral buffer 980 may be timed by the control signal EVS_CLK or another control signal, the product calculation block 981 and / or the exponent calculation block 982 may be enabled to continuously perform their respective operations over time. Alternatively, the operation of the product calculation block 981 and / or the exponent calculation block 982 may also be timed by the control signal EVS_CLK.

[0157] As discussed above, after the exposure period of the CIS pixel ends, the CIS data captured by the CIS pixel can be read out to the latent frame calculation block 972 of the deblurring circuit 970a. According to the above discussion, the latent frame calculation block 972 can deblur the CIS data by combining / fusing the CIS data with the accumulated EVS data stored in the second integration buffer 978 of the calculation block 971a. In addition, the latent frame calculation block 972 can use the corresponding output of the index calculation block 982 from the calculation block 971a to correct the rolling shutter distortion of the CIS data. More specifically, the latent frame calculation block 972 can calculate the corresponding time t by performing the operation specified in the above equation 17 for each EVS pixel. s (beginning of the exposure period) of the latent image frame L(s). In addition, the latent frame calculation block 972 may calculate the latent image frame L(s) corresponding to time t for each pixel using the following: 0Latent image frame L(0) (corresponding to the beginning of an image frame, e.g., corresponding to the beginning of a first exposure period of an image frame): (i) latent image frame L(s), (ii) corresponding output from index calculation block 982, and / or (iii) the operation specified in the above equation 20. Latent image frame L(s) may correspond to deblurred rolling shutter distortion corrected CIS data and may be output from latent frame calculation block 972 to an application processor and / or an image signal processor of a corresponding imaging system.

[0158] When the exposure periods of consecutive image frames do not overlap in time, Fig.9A In other words, a single (eg, only one) instance of counter 979 and integration buffer 980 may be used as long as the exposure periods of adjacent frames do not overlap in time. Fig.9A However, this relationship limits the maximum frame rate that can be used by the corresponding image sensor.

[0159] For example, the maximum frame rate that can be used by the corresponding image sensor can be increased by starting the first exposure period of the second frame before the end of the last exposure period of the first frame. However, this arrangement requires tracking both (a) events corresponding to the first image frame and (b) events corresponding to consecutive image frames. Implementing a ping-pong buffer into the rolling shutter distortion correction component of the deblurring circuit can achieve this functionality.

[0160] For example, Fig. 9B is a partial schematic diagram of a deblurring and rolling shutter distortion correction circuit 970b ("deblurring circuit 970b" or "rolling shutter distortion correction circuit 970b") configured in accordance with various embodiments of the present technology. The deblurring circuit 970b may be Figure 5 570 or other deblurring circuits configured in accordance with the present technology. As shown, the deblurring circuit 970b is generally similar to Fig.9A Thus, similar reference numbers span Fig.9A and 9B To represent identical or at least substantially similar components, and in view of the detailed description of deblurring circuit 970a provided above, a detailed description of deblurring circuit 970b is substantially omitted herein for the sake of brevity.

[0161] like Fig. 9B, the deblurring circuit 970b includes a rolling shutter distortion correction and event-based double integral (EDI) calculation block 971b ("calculation block 971b") and a latent frame calculation block 972. The calculation block 971b includes EDI components (e.g., a counter 973, a first integral buffer 974, a product calculation block 975, an exponent calculation block 976, an integral calculation block 977, and a second integral buffer 978. The counter 973 is also referred to herein as the "first counter", the product calculation block 975 is also referred to herein as the "first product calculation block", and the exponent calculation block 976 is also referred to herein as the "first exponent calculation block". The first integral buffer 974 and the second integral buffer 978 are also referred to herein as "EDI integral buffers".

[0162] The computation block 971b of the deblurring circuit 970b further includes a rolling shutter distortion correction component. Fig.9A The rolling shutter distortion correction component of the computation block 971a of the deblurring circuit 970a is compared to Fig. 9B The rolling shutter distortion correction component of the calculation block 971b of the deblurring circuit 970b includes a ping-pong buffer. More specifically, the rolling shutter distortion correction component includes a routing switch 983, a counter 979a (also referred to herein as a "second counter"), a counter 979b (also referred to herein as a "third counter"), an integral buffer 980a (also referred to herein as a "third integral buffer"), an integral buffer 980b (also referred to herein as a "fourth integral buffer"), and a multiplexer 984. The integral buffers 980a and 980b are also referred to herein as "rolling shutter distortion correction (RSDC) integral buffers".

[0163] As shown, the rolling shutter distortion component of the calculation block 971b also includes a product calculation block 981a (also referred to herein as a “second product calculation block”), a product calculation block 981b (also referred to herein as a “third product calculation block”), an exponent calculation block 982a (also referred to herein as a “second exponent calculation block”), and an exponent calculation block 982b (also referred to herein as a “third exponent calculation block”). In other embodiments, the rolling shutter distortion component may include a single (e.g., only one) instance of the product calculation block 981 and / or a single (e.g., only one) instance of the exponent calculation block 982. In such embodiments, the product calculation block 981 and the exponent calculation block 982 may be positioned downstream of the multiplexer 984, such as between the multiplexer 984 and the latent frame calculation block 972. Continuing with this example, product calculation block 981 and exponent calculation block 982 may be configured to perform operations on a running sum output from either integration buffer 980a or integration buffer 980b via multiplexer 984 .

[0164] Counters 979a and 979b of deobfuscation circuit 970b may be substantially similar to Fig.9A The counter 979 of the defuzzification circuit 970a, and the integral buffer 980a and the integral buffer 980b can be substantially similar to Fig.9A In addition, the product calculation block 981a and the product calculation block 981b can be substantially similar to Fig.9A The product calculation block 981 of the deblurring circuit 970a, and the exponent calculation block 982a and the exponent calculation block 982b may be substantially similar to Fig.9A The exponential calculation block 982 of the deblurring circuit 970a. Therefore, in view of the above reference Fig.9A A detailed description of each of these rolling shutter distortion correction components of deblurring circuit 970b is omitted herein for the sake of brevity, in contrast to the detailed description of similar components of deblurring circuit 970a provided herein.

[0165] In the illustrative embodiment, the counter 979a, the integral buffer 980a, the product calculation block 981a, and the exponent calculation block 982a (collectively referred to herein as the "first set of rolling shutter distortion correction (RSDC) components") may correspond to different frames from the counter 979b, the integral buffer 980b, the product calculation block 981b, and the exponent calculation block 982b (collectively referred to herein as the "second set of RSDC components"). For example, the first set of RSDC components may be used to accumulate events detected during a first frame, and the second set of RSDC components may be used to accumulate events detected during a second consecutive (or immediately adjacent) frame. Thereafter, the first set of RSDC components may be used to accumulate events detected during a third frame; and the second set of RSDC components may be used to accumulate events detected during a fourth frame; and so on. Thus, continuing the above example, at time t 0 The start time t of the exposure period corresponding to the first frame s Events detected by EVS pixels between t and t can be accumulated using the first set of RSDC components. 0 The start time t of the exposure period corresponding to the second frame s Events detected by EVS pixels can be accumulated using a second set of RSDC components.

[0166] Routing events detected by EVS pixels to the appropriate set of RSDC components may be handled via routing switch 983. More specifically, routing switch 983 is configured to receive a control signal ping_pong. In some embodiments, control signal ping_pong may be generated by a common control block (e.g., Figure 5 Alternatively, the control signal ping_pong may be provided and / or controlled by another control block (e.g., Figure 5 The image sensor 530 may be provided / controlled by the column control circuit system 562 and / or the column scanning readout circuit system 553).

[0167] The control signal ping_pong may be used to control to which counter (counter 979a or counter 979b) an event detected by an EVS pixel is routed via routing switch 983. For example, at time t 0 The time t of the first frame s During the time t, the control signal ping_pong may transition to or maintain the first state (eg, asserted state, high state, "1" state). 0 The start time t of the exposure period corresponding to the first frame s Events detected by the EVS pixels between t and t can be routed to the counter 979a via the routing switch 983. 0 The time t of the second frame s During the time t, the control signal ping_pong may transition to or maintain a second state (eg, a deasserted state, a low state, a "0" state). 0 The start time of the exposure period of the second frame is t s Events detected by an EVS pixel during the exposure period of a CIS pixel corresponding to an EVS pixel may be routed to counter 979b via routing switch 983. Events detected by this EVS pixel during the exposure period of a CIS pixel corresponding to an EVS pixel may be routed to counter 973 of the EDI component of computation block 971b, as described above with reference to Fig.9A The defuzzification circuit 970a is consistent with the description of the EDI component of the computation block 971a.

[0168] Referring now to the multiplexer 984, the control signal ping_pong may be used to control which input into the multiplexer 984 (e.g., which output of the exponent calculation block 982a and the exponent calculation block 982b) is output from the multiplexer 984 (e.g., to the latent frame calculation block 972 and / or a downstream application processor). In the illustrated embodiment, when the control signal ping_pong transitions or remains in a first state, the output of the exponent calculation block 982b may be routed to (i) the downstream application processor and (ii) the latent frame calculation block 972 via the multiplexer 984. On the other hand, when the control signal ping_pong transitions or remains in a second state, the output of the exponent calculation block 982a may be routed to (i) the downstream application processor and (ii) the latent frame calculation block 972 via the multiplexer 984. Thus, when the detected event is routed to the counter 979a via the routing switch 983, the output of the exponent calculation block 982b may be passed to the downstream application processor and the latent frame calculation block 972 via the multiplexer 984. Additionally, when the detected event is routed to the counter 979b via the routing switch 983, the output of the exponential calculation block 982a may be passed to the downstream application processor and the latent frame calculation block 972 via the multiplexer 984. In this manner, the ping-pong buffer implements rolling shutter distortion correction for CIS data corresponding to two different frames that at least partially overlap in time.

[0169] Fig.10 is a flow chart illustrating a method 1000 of operating an imaging system according to various embodiments of the present technology. For example, method 1000 may be a method of (i) performing (e.g., on-chip) deblurring of CIS data and / or (ii) video interpolation. Method 1000 is illustrated as a series of boxes 1001 to 1013 or steps. All boxes 1001 to 1013 or a subset of one or more thereof may be performed by a device or component of an imaging system configured according to various embodiments of the present technology. For example, all boxes 1001 to 1013 or a subset of one or more thereof may be performed by a hybrid image sensor, a CIS pixel of a pixel array, an EVS pixel of an event-driven sensing array, a common control block, a row / column control circuit system, a column readout circuit system, a column scan readout circuit system, a deblurring block or circuit, and / or an application processor. All boxes 1001 to 1013 of method 1000 or a subset of one or more thereof may be performed according to the above. Figures 1 to 9B In fact, some of the blocks 1001 to 1013 of method 1000 are referred to below. Figures 11 to 13B describe.

[0170] Method 1000 begins with block 1001, which aligns CIS pixel data with corresponding EVS pixel data. In some embodiments, aligning CIS pixel data with corresponding EVS pixel data may be performed at least in part using a common control block of the corresponding image sensor. For example, the common control block may synchronize the operation of the row / column control circuitry and / or the deblurring block of the image sensor, such as by using one or more control signals.

[0171] Aligning CIS pixel data with corresponding EVS pixel data at block 1001 may include aligning / synchronizing the timing of exposure periods of one or more rows of CIS pixels with event accumulation periods of one or more corresponding EVS pixels. In some embodiments, aligning exposure periods with event accumulation periods may include aligning exposure periods with each other and / or event accumulation periods such that the exposure periods and event accumulation periods have the same start time t s and / or the same end time t s+T . For example, before the exposure period and the event accumulation period begin, the CIS pixels of one or more CIS pixel rows may be reset at the same time as (a) each other and / or (b) one or more EVS pixels of one or more EVS pixel rows corresponding to the one or more CIS pixel rows. Thus, the exposure period of the CIS pixels and the event accumulation period of the EVS pixels may begin at the same time as each other. Additionally, assuming that the exposure period and the event accumulation period have the same duration, aligning the start times of the exposure period and the event accumulation period with each other may also align their stop times.

[0172] In addition, as mentioned above Figure 6 , 9A 9B, the deblurring circuit of the image sensor of the present technology may be configured to (a) integrate the events to calculate a running sum of the events detected by the EVS pixel, (b) store the running sum in a first integration buffer, (c) exponentially integrate the product of (i) the running sum and (ii) a contrast threshold parameter, and (d) store the result of the integration of the exponent in a second integration buffer. Therefore, in order to ensure that the running sum stored in the first integration buffer of the deblurring circuit and the result of the integration of the exponent stored in the second integration buffer correspond only to events detected by the EVS pixel during the corresponding event accumulation period (which, as discussed above, may be aligned with the exposure period of the corresponding CIS pixel), the first integration buffer and / or the second integration buffer may be reset before the aligned start time of the event accumulation period and the exposure period. In some embodiments, the first integration buffer and / or the second integration buffer may be reset simultaneously with the EVS pixel and / or the corresponding CIS pixel.

[0173] In addition, the accumulated EVS pixel data may be output to and stored in one or more EVS frame buffers (e.g., of a downstream application processor) during and / or after the integration period of the CIS pixel used to capture the CIS pixel data. For example, the running sum stored in the first integration buffer and / or the index output from the index calculation block may be provided to the one or more EVS frame buffers. Therefore, in order to ensure that the accumulated EVS pixel data stored in the EVS frame buffer corresponds only to events associated with a given image frame, the corresponding portion of the EVS frame buffer may be reset before the aligned start time of the event accumulation period and the exposure period. The corresponding portion of the CIS key frame buffer (e.g., of the downstream application processor) may also be reset before the event accumulation period is aligned with the start time of the exposure period. In some embodiments, the corresponding portion of the EVS frame buffer and / or the CIS key frame buffer may be reset simultaneously with the EVS pixel and / or the corresponding CIS pixel.

[0174] For clarity and understanding of the alignment performed at block 1001 of method 1000, consider Fig.11 and 12 , which respectively illustrate timing diagrams 1195 and 1290 according to various embodiments of the present technology. Fig.12 , timing diagram 1290 illustrates three EVS pixel rows (EVS pixel rows N, N+1, and N+2) and twelve CIS pixel rows (CIS pixel rows 4N-3 to 4N+8). In the illustrated embodiment, each of EVS pixel rows N, N+1, and N+2 corresponds to four of the twelve CIS pixel rows shown. For example, EVS pixel row N corresponds to CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3. In other words, the EVS data captured by the EVS pixels of EVS pixel row N can be used for event-directed deblurring of the CIS data captured by the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3.

[0175] The CIS data captured by the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2 and 4N-3 are read out synchronously on a frame basis after each corresponding exposure period ends. In many active pixel sensors, the CIS data is read out row by row. In such a configuration, different exposure period start and stop times are typically used for different rows. For example, in many active pixel sensors, the CIS pixels of CIS pixel row 4N will have a first exposure period that starts and stops at a different time than the second exposure period for the CIS pixels of CIS pixel row 4N-1. When CIS pixel row 4N and CIS pixel row 4N-1 correspond to the same EVS pixel row, this may be a problem for event-oriented deblurring because the misalignment between the first exposure period and the second exposure period means that the start and / or stop time of the event accumulation period for the corresponding EVS pixel row will be different from the start and / or stop time of the first exposure period and / or the second exposure period. Therefore, the EVS data captured by the EVS pixels of the EVS pixel row will be misaligned with the CIS data captured by the CIS pixels of the CIS pixel row 4N and / or CIS pixel row 4N- 1. This misalignment may affect the accuracy and / or efficacy of event-directed deblurring operations performed on the CIS data and / or require additional storage / processing after data capture and / or readout to align the CIS data with the EVS data.

[0176] To address this issue, at block 1001 of method 1000, the exposure periods of CIS pixel rows corresponding to the same EVS pixel row may be aligned with each other and the event accumulation period of the EVS pixel row. Fig.12 , CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 all correspond to EVS pixel row N. Therefore, at block 1001, the exposure periods 1197 of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 may be aligned with each other and the event accumulation period 1198 of EVS pixel row N. Therefore, the exposure period 1197 and the event accumulation period 1198 may each start at time t0. Additionally, because the exposure period 1197 and the event accumulation period 1198 are the same duration, aligning the start times at time t0 may align the exposure period 1197 with the end time of the event accumulation period 1198 at time t5. In this way, the EVS data captured by the EVS pixels of EVS pixel row N are aligned with the CIS data captured by the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3. The alignment between (i) the exposure / integration period 1197 of the CIS pixel row and (ii) the event / EVS accumulation period 1198 of the EVS row N is Fig.11 This is further shown in the timing diagram 1195.

[0177] Continuing with the above example, the alignment between the exposure period 1197 and the event accumulation period 1198 may be achieved by resetting the EVS pixels of EVS pixel row N and the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 simultaneously and / or before the start time t0. Fig.11 , the EVS pixels of the EVS pixel row N and the CIS pixels of the CIS pixel rows 4N, 4N-1, 4N-2 and 4N-3 may each be reset within the period 1193 before the start time t0 of the exposure period 1197 and the event accumulation period 1198.

[0178] In addition, in order to ensure that the deblurring calculation performed by the deblurring circuit of the corresponding image sensor corresponds only to the exposure period 1197 of the CIS pixel rows 4N, 4N-1, 4N-2 and 4N-3 and the event accumulation period 1198 of the EVS pixel row N, the first integration buffer and / or the second integration buffer of the deblurring circuit may be reset before the start time t0, for example (i) at Fig.11 In some embodiments, the portion of the EVS frame buffer (e.g., of a downstream application processor) corresponding to the EVS pixels of EVS pixel row N and / or the portion of the CIS key frame buffer (e.g., of a downstream application processor) corresponding to the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 may be additionally reset prior to the start time t0, e.g., (i) at the same time as the EVS pixels of EVS pixel row N and the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3. Fig.11 (i) during the time period 1193 of EVS pixel row N and / or (ii) simultaneously with the EVS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3. Other EVS pixel rows (e.g., EVS pixel rows N+1, N+2, N+3, N+4, etc.) and other CIS pixel rows (e.g., 4N+1, 4N+2, ..., 4N+8, etc.) may be operated in a similar manner.

[0179] Reference again Fig.10 , method 1000 may then, at block 1002, (a) capture CIS data using CIS pixels during corresponding exposure periods and (b) capture EVS data using EVS pixels (also referred to herein as “EVS pixel data” or “event data”). Referring again to Fig.11 For example, the CIS pixels of the CIS pixel row corresponding to the EVS pixel row N may integrate the photogenerated charge in the CIS pixels during an exposure period 1197 extending from the start time t0 to the stop time t5. At the same time, the EVS pixels of the EVS pixel row N may asynchronously detect events during an aligned EVS accumulation period 1198 also extending from the start time t0 to the stop time t5.

[0180] In some embodiments, the EVS pixels may be selectively enabled to capture EVS data during corresponding event accumulation periods (eg, selectively enabled to detect events). Fig.11 , the EVS pixels of EVS pixel row N may be enabled at start time t0 of (or shortly before) the event accumulation period 1198 such that the EVS pixels are configured to detect events occurring during the event accumulation period 1198 between time t0 and time t5.

[0181] At block 1003, method 1000 then reads out the events detected by the EVS pixels of the event-driven sense array. Block 1003 may be performed when block 1002 is performed. For example, when an EVS pixel detects an event, the event may be read out from the EVS pixel. When an event detected by an EVS pixel is read out from the EVS pixel, the EVS pixel may be reset and thereby enabled to detect subsequent events.

[0182] A single EVS pixel may detect hundreds of events within a single event accumulation period. In some embodiments, each of these events may be read out from the EVS pixel and / or provided to the deblurring circuitry of the corresponding image sensor for accumulation. Thus, for a single CIS frame, a relatively large amount of EVS data may be provided to the deblurring circuitry for accumulation and subsequent use in event-directed deblurring of CIS data corresponding to the CIS frame.

[0183] In some cases, this large amount of EVS data can complicate and / or slow down the deblurring calculations performed by the deblurring circuitry of the corresponding image sensor, which can be inappropriate or unacceptable for certain applications. Therefore, in some embodiments, the EVS data can be read out from the EVS pixels using a progressive scan readout. More specifically, the image sensor can progressively scan / step through the event-driven sensing array and take a uniform amount of time to read out each EVS pixel row. In this way, the scanning readout can limit the number of EVS readouts per CIS frame, which can simplify and / or speed up the deblurring calculations performed by the deblurring circuitry.

[0184] For the sake of clarity and example, consider Fig.13A and 13B , which illustrate (i) an example event-driven sensing array 1342 and (ii) a corresponding plot 1305 of detected events read out from the event-driven sensing array 1342 for a single CIS frame, respectively. Fig.13A , the event-driven sense array 1342 includes a plurality of EVS pixels 00 to 57 arranged in a plurality of rows Row_0 to Row_5 and a plurality of columns Col_0 to Col_7. EVS pixels 12, 13, 35, 42, 53, 54, and 55 in the sense array 1342 have detected an event.

[0185] To read out the EVS data captured by EVS pixels 00 to 57 of the event-driven sense array 1342, the image sensor may cycle through rows Row_0 to Row_5 of the event-driven sense array 1342, row by row, spending the same amount of time at each row to read out events from the EVS pixels of this row. For example, although none of the EVS pixels 00 to 07 of row Row_0 have detected an event, the image sensor may spend a fixed / preset amount of time (e.g., 50ns) at Row_0 before moving to row Row_1. Then, at row Row_1, the image sensor may spend the same fixed amount of time (e.g., 50ns) to read out events detected by EVS pixels 10 to 17. In the illustrated example, EVS pixels 12, 13, and 15 in row Row_1 have detected an event. Thus, during the fixed / preset amount of time allocated to row Row_1, the image sensor may (i) read out events detected by EVS pixels 12, 13, and 15 and / or (ii) reset EVS pixels 12, 13, and 15 so that they are enabled to detect subsequent events. At the end of the fixed / preset amount of time allocated to row Row_1, the image sensor may move to row Row_2 of the event-driven sense array 1342 to read out events detected by EVS pixels 20 to 27 of row Row_2, if any. Fig. 13B The drawing 1305 shows the Fig.13A The event drives the sensing array 1342 to perform a scan to read out the results.

[0186] The above-described row-by-row scan readout scheme thus limits the total number of EVS readouts per CIS frame and keeps the time required to scan each row in the event-driven sensing array constant within each scan cycle. For example, given that (i) the event-driven sensing array has 2,000 rows of EVS pixels and (ii) the preset amount of time for reading out each row of EVS pixels in the event-driven sensing array is 50 ns, the scanning readout requires 100 μs (2,000 rows of EVS pixels x 50 ns) to scan the entire event-driven sensing array once. Therefore, assuming that the duration of the CIS exposure period is 32 ms, the scanning readout can limit the number of EVS readouts per EVS pixel to 320 (320 ms divided by 100 μs) for each CIS readout. In other words, each CIS frame readout can correspond to a maximum of 320 EVS readouts per EVS pixel. This can limit the amount of EVS data fed to the deblurring circuit, which can simplify the calculations performed by the deblurring circuit and / or speed up the availability of the final results of such calculations.

[0187] In the example of row-by-row scan readout described above, the image sensor spends a fixed / preset amount of time reading out detected events (if any) at each EVS pixel row. As discussed above, this can keep the total time required to scan the entire event-driven sensing array once constant within each cycle of scan readout. In other embodiments, the image sensor can skip EVS pixel rows where no events are detected. For example, referring again to Fig.13A , none of the EVS pixels 20 to 27 of row Row_2 of the event-driven sense array 1342 have detected an event. Therefore, during scanning readout, the image sensor may spend a preset amount of time (e.g., 50 ns) reading out events detected by EVS pixels 12, 13, and 15 of row Row_1, skip row Row_2, and then spend the next preset amount of time (e.g., 50 ns) reading out events detected by EVS pixels 35 of row Row_3. In such embodiments, the total time taken for the scanning readout cycle to pass through the entire event-driven sense array 1342 may vary across cycles depending on which of rows Row_0 to Row_5 include EVS pixels that have detected an event.

[0188] Reference again Fig.10 , method 1000 may then, at block 1004, accumulate, for each EVS pixel of the event-driven sense array, EVS data read out from the EVS pixel during the corresponding event accumulation period. Block 1004 may be performed when blocks 1002 and / or 1003 are performed, as shown by an arrow returning from block 1004 to block 1002. Additionally or alternatively, block 1004 may be performed by an on-chip deblurring circuit of the image sensor.

[0189] As discussed above, accumulating EVS data may include, for each EVS pixel, calculating and maintaining a running sum of events, for example, using a counter (e.g., of a deblurring circuit of a hybrid image sensor) and a first integration buffer. Additionally, accumulating EVS data may include, for each EVS pixel, (a) calculating the product of the running sum and a contrast threshold parameter, (b) determining an exponent of the product, and (c) continuously integrating the exponent over a corresponding EVS accumulation period. The results of the integration for each EVS pixel may be stored in a second integration buffer (e.g., of a deblurring circuit of a hybrid image sensor).

[0190] In some embodiments, for example, at block 1004, block 1005, or block 1007 of method 1000, the running sum of events and / or the index of the product stored in the first integration buffer (e.g., the output of the index calculation block of the deblurring circuit of the hybrid image sensor) can be output to a downstream application processor, such as to the EVS frame buffer of the application processor and / or used in video interpolation calculation. For example, the running sum of events and / or the index of the product of each EVS pixel stored in the first integration buffer can be output to the downstream application processor and / or loaded into the EVS frame buffer at a timing corresponding to the time when the running sum of the event is stored in the first integration buffer and / or the time when the index of the product is calculated. As another example, the running sum of events and / or the index of the product stored in the first integration buffer can be output to the downstream application processor and / or loaded into the EVS frame buffer at a specified timing, such as at the end of the accumulation period, at the interpolation timing point, at the end of the exposure period, at the beginning of the exposure period, etc. The running sum of events and / or the exponent of the product of all EVS pixels stored in the first integration buffer may be output row by row to a downstream application processor and / or loaded into an EVS frame buffer. Additionally or alternatively, loading of (a) the running sum of events (e.g., from the first integration buffer) and / or (b) the exponent of the product (e.g., from the exponent calculation block) into an application processor and / or a corresponding EVS frame buffer may be gated or otherwise controlled via a flip-flop and corresponding switch, as discussed in more detail below.

[0191] As discussed above, for each EVS pixel, EVS data is accumulated over a corresponding event accumulation period. For example, for each EVS pixel, a corresponding portion of the first integration buffer and a corresponding portion of the second integration buffer may be reset before the start of the event accumulation period for the EVS pixel to reset (i) the running sum stored in the first integration buffer and (ii) the integration results stored in the corresponding portion of the second integration buffer. Event accumulation may then be enabled at the start of the event accumulation period and thereafter disabled at the end of the event accumulation period, such that the integration results stored in the corresponding portion of the second integration buffer at the end of the event accumulation period correspond only to events detected by the corresponding EVS pixel during the event accumulation period. In some embodiments, once event data has been accumulated, the corresponding raw event data may be discarded.

[0192] Reference again Fig.11For clarity and example, the first integration buffer and the portion of the second integration buffer corresponding to the EVS pixels of the EVS pixel row N may be reset during the period 1193 before the start t0 of the event accumulation period 1198. In some embodiments, the corresponding portion of the EVS frame buffer and / or the corresponding portion of the CIS key frame buffer (e.g., of the downstream application processor) may also be reset during the period 1193. At the start t0 of the event accumulation period 1198, event accumulation in the deblurring circuit may be enabled for the EVS pixels of the EVS pixel row N. When an event is detected by the EVS pixel during the event accumulation period 1198, the corresponding running count / sum maintained in the corresponding first integration buffer is updated. In addition, the running sum is multiplied by the contrast threshold, the exponent of the resulting product is calculated, and the exponent is integrated. The integration result of the EVS pixels corresponding to the EVS pixel row N is stored to the corresponding portion of the second integration buffer. In addition, in some embodiments, the running sum and / or the exponent may be output (e.g., streamed) to a downstream application processor, such as to the EVS frame buffer of the application processor.

[0193] At box 1005, method 1000 then reads out the CIS data at the end of the corresponding exposure period. In some embodiments, reading out the CIS data may include reading out the CIS data into a deblurring circuit, such as into a latent frame calculation block of the deblurring circuit. In these and other embodiments, reading out the CIS data may include reading out the CIS data from a CIS pixel in a row or a row group. For example, in an embodiment in which a plurality of CIS pixel rows correspond to the same EVS pixel row, the CIS data captured by the CIS pixels of the plurality of CIS pixel rows may be read out together / simultaneously at the end of or after the corresponding exposure period. In some embodiments, reading out the CIS data may include skipping the readout of one or more CIS pixel rows and / or columns (e.g., for reducing the resolution of the CIS data, matching the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array, and / or reducing the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array. The mismatch between). In these and other embodiments, reading out the CIS data may include merging one or more CIS pixel rows and / or columns (e.g., to reduce the resolution of the CIS data, match the resolution of EVS data captured by the EVS pixels of the event-driven sensing array, and / or reduce a mismatch between the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array).

[0194] Reference again Fig.11 and 12, for clarity and example, CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 all correspond to EVS pixel row N and have a common exposure period 1197 extending between time t0 and time t5 after being aligned with the event accumulation period 1198 of the EVS pixels of EVS pixel row N. Therefore, at or after the end time t5 of the exposure period 1197, the CIS data captured by the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 can be read out from the CIS pixels simultaneously. Additionally or alternatively, one or more of the CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 can be skipped or merged together during readout.

[0195] At block 1006, method 1000 then deblurs the CIS data (read out from the CIS pixel at block 1005) using the accumulated EVS data generated at block 1004 and stored in the corresponding portion of the second integration buffer. In some embodiments, deblurring the CIS data may include combining the CIS data with the accumulated EVS data to calculate one or more latent image frames, such as (a) a start time t corresponding to an exposure period of the CIS pixel; s In some embodiments, combining the CIS data with the accumulated EVS data may include interpolating the EVS data to generate additional EVS data corresponding to additional EVS pixel rows and / or columns (e.g., to increase the resolution of the EVS data, match the resolution of the CIS data captured by the CIS pixels of the CIS pixel array, and / or reduce the mismatch between the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array).

[0196] At block 1007, method 1000 then outputs the deblurred image data. Outputting the deblurred image data may include outputting the deblurred image data from the image sensor to a downstream application processor. As a specific example, outputting the deblurred image data may include outputting the deblurred image data to a CIS key frame buffer and / or a video interpolation calculation block of a downstream application processor. Additionally or alternatively, outputting the deblurred image data may include outputting the deblurred image data to an image signal processor, such as for previewing the deblurred image data. In these and other embodiments, outputting the deblurred image data may include outputting the deblurred image data (e.g., latent image frame L(s)) calculated at block 1006, such as in addition to or in lieu of outputting the raw CIS data read out from the CIS pixels at block 1005 and / or the raw EVS data generated and read out from the EVS pixels at block 1003.

[0197] Fig.11 The timing diagram 1195 provides Fig.101195 . For example, time period 1193 of timing diagram 1195 corresponds to a time period before exposure start time t0, wherein the EVS pixels of EVS pixel row N, the CIS pixels of the CIS pixel row corresponding to EVS pixel row N, the corresponding first integration buffer and / or the corresponding portion of the second integration buffer are reset. The corresponding portion of the EVS frame buffer and / or the corresponding portion of the CIS key frame buffer of the downstream application processor may also be reset at this timing. Due to the reset, the CIS pixel row has a common exposure period 1197 aligned with the EVS accumulation period 1198 of EVS pixel row N. Therefore, when the CIS pixels of the CIS pixel row capture CIS data during exposure period 1197, the EVS pixels of the EVS pixel row detect events aligned during the EVS accumulation period 1198. As events are detected and read into the deblurring circuitry of the corresponding image sensor (e.g., using a progressive scan readout scheme), the events are accumulated such that a running sum of the events integrated over the entire EVS accumulation period 1198 is available at the end of the exposure period 1197 and / or such that the accumulated event data is provided to a downstream application processor at one or more times over the entire exposure period. Fig.11 Arrow 1101 in 1190 shows that at the end of exposure period 1197, CIS data may be read out from the CIS pixels of the CIS pixel row corresponding to EVS pixel row N and combined with the accumulated event data to calculate final (deblurred) image data (e.g., one or more latent images, such as latent image L(s)) that may be output row by row from the image sensor to, for example, a CIS frame buffer of a downstream application processor. In some embodiments, loading the latent image L(s) into the CIS frame buffer of the downstream application processor may, for example, use a flip-flop and corresponding switch gating (discussed in more detail below).

[0198] Referring again to block 1005 of method 1000, after reading out the CIS pixel data at block 1005 at the end of the exposure period, method 1000 may additionally continue to blocks 1008-1012 to collect additional EVS pixel data outside the CIS exposure period. More specifically, at block 1008, method 1000 may then reset the EVS pixels and / or corresponding EVS integration buffers (e.g., of an EDI component of a deblurring circuit) at the end of the corresponding CIS integration period. As a specific example, referring again to Fig.11 , the EVS pixel of EVS row N (corresponding to having Fig.11 1197) may be reset in a period 1192 extending between time t5 and time t6. As shown, period 1192 occurs after (a) the end of exposure period 1197 for the corresponding CIS pixel and (b) the end of accumulation period 1198 for the EVS pixel aligned with exposure period 1197 at block 1001 of method 1000.

[0199] At blocks 1009 to 1011, Fig.10 The method 1000 then captures EVS pixel data using EVS pixels, reads the EVS pixel data from the EVS pixels to the deblurring circuit, and accumulates the EVS pixel data using one or more EVS integration buffers, respectively. Blocks 1009 to 1011 may be substantially similar to blocks 1002 to 1004 of the method 1000 described above. Therefore, for the sake of brevity, detailed discussion of blocks 1009 to 1011 is substantially omitted herein.

[0200] For example and clarity, refer to Fig.11 1198, at blocks 1009-1011, method 1000 captures, reads out, and accumulates EVS pixel data during an accumulation period 1199 that runs outside (e.g., after) the exposure period 1197 of the corresponding CIS pixel. Thus, method 1000 captures, reads out, and accumulates EVS pixel data (e.g., event data) corresponding to a time during which CIS pixel data was not captured by the corresponding CIS pixel. As discussed in more detail below, the EVS pixel data captured and accumulated during the accumulation period 1199 can be used in video interpolation calculations (in conjunction with (i) CIS pixel data captured during the exposure period 1197 and / or (ii) EVS pixel data captured during the accumulation period 1198) to generate interpolation information.

[0201] At block 1011 or block 1012 of method 1000, the accumulated EVS pixel data may be output to a downstream application processor, such as to an EVS frame buffer of the application processor. For example, a running sum of events and / or an index of the running sum multiplied by a contrast threshold may be output to a downstream application processor and / or loaded into an EVS frame buffer at a timing corresponding to the time at which the running sum is stored in the first integration buffer and / or the time at which the index is calculated. As another example, the running sum and / or index may be output to a downstream application processor and / or loaded into an EVS frame buffer at a specified timing, such as at the end of an accumulation period (e.g., at Fig.11 The running sum and / or index may be output row by row to a downstream application processor and / or loaded into an EVS frame buffer. Additionally or alternatively, loading (a) the running sum (e.g., stored in a first integration buffer) and / or (b) the index (e.g., output from an index calculation block) into an application processor and / or a corresponding EVS frame buffer may be gated or otherwise controlled via a trigger and corresponding switch, as discussed in more detail below.

[0202] For example, refer again to Figure 6 , 10 and 11, during the accumulation period 1199( Fig.11), the EVS pixel data captured by the EVS pixels of EVS row N at block 1009 and read out at block 1010 may be tracked in a running sum maintained by the counter 673 and first integration buffer 674 of the EDI calculation block 671 of the deblurring circuit 670. The running sum may then be (a) output to a downstream application processor or (b) first fed to the product calculation block 675 and the exponent calculation block 676 and then output to an application processor, such as an application processor external to a hybrid image sensor incorporating the deblurring circuit 670.

[0203] At block 1012, method 1000 then outputs the accumulated EVS pixel data and deblurred image data to a video interpolation block. Fig.10 At blocks 1002 to 1004 of method 1000, Fig.11 During the accumulation period 1198 described in , EVS pixel data may be captured, read out, accumulated and / or output to the EVS frame buffer of the application processor by the EVS pixels of EVS row N. The accumulated EVS pixel data may be output to the EVS frame buffer of the application processor continuously (e.g., when it is stored in the EVS integration buffer and / or output from the exponential calculation block of the deblurring circuit) or at a preset timing (e.g., at an interpolation timing point, at the beginning of the exposure period of the corresponding CIS pixel, at the end of the exposure period of the corresponding CIS pixel and / or at the end of the accumulation period of the corresponding EVS pixel). In addition, during Fig.10 At blocks 1009 to 1011 of method 1000, Fig.11 During the accumulation period 1199 described in , EVS pixel data may be captured, read out, accumulated, and output to the EVS frame buffer of the application processor by the EVS pixels of EVS row N. The accumulated EVS pixel data may be output to the EVS frame buffer of the application processor continuously (e.g., when it is stored in the EVS integration buffer and / or output from the exponential calculation block of the deblurring circuit) or at a preset timing (e.g., at an interpolation timing point, at the beginning of the accumulation period of the corresponding EVS pixel, and / or at the end of the accumulation period of the corresponding EVS pixel). At box 1012, at or after the end of the accumulation period 1199 (representing the interpolation timing point), the accumulated EVS pixel data from both the accumulation period 1198 and the accumulation period 1199 may be output from the EVS frame buffer of the application processor to the video interpolation calculation block, for example, by reading out portions of the EVS pixel data row by row from the EVS frame buffer of the application processor. More specifically, during Fig.10 At block 1012 of method 1000, the accumulated EVS pixel data in the EVS frame buffer of the application processor may be fed to the video interpolation frame calculation block. The video frame calculation block may additionally receive the pixel data output from the deblurring circuit at block 1007 of method 1000 and / or at the video interpolation frame calculation block. Fig.10The latent image frame L(s) (representing deblurred image data) is read from the CIS key frame buffer of the application processor to the video interpolation block at box 1007 or box 1012.

[0204] At block 1013, method 1000 then calculates one or more interpolated image / video frames. In some embodiments, calculating the interpolated image / video frame may include, for example, calculating the interpolated image / video frame by performing a multiplication of the image / video frames from time t s The interpolated image / video frame at a given time t is calculated by incrementing the latent image frame L(s) (corresponding to the beginning of each CIS integration period) from the beginning of the latent image frame L(s) to all events at a given time t. More specifically, calculating the interpolated image / video frame may include using (a) the latent image frame L(s) output at block 1007, (b) all accumulated EVS pixel data or a subset thereof output at block 1012, and (c) the above equation 15 to calculate the interpolated image / video frame at a given time t.

[0205] As a given example, method 1000 may use Equation 15, the latent image frame L(s), and all accumulated EVS pixel data output at block 1012 to calculate the value corresponding to Fig.11 Additionally or alternatively, method 1000 may use equation 15, the latent image frame L(s), all accumulated EVS pixel data accumulated during an accumulation period aligned with the CIS exposure period (e.g., accumulation period 1198), and a subset of EVS pixel data accumulated during an accumulation period extending beyond the CIS exposure period (e.g., accumulation period 1199) to calculate a pixel corresponding to Fig.11 The subset of EVS pixel data may include up to the time corresponding to the second interpolated image frame of time t11 shown in FIG. Fig.11 EVS pixel data is captured, read out and accumulated at a time point in each accumulation period of time t11 of the bottom accumulation period shown in FIG.

[0206] Although blocks 1001 through 1013 of method 1000 are described and illustrated in a particular order, Fig.10 The method 1000 of the present invention is not limited thereto. In other embodiments, all of the blocks 1001 to 1013 of the method 1000 or a subset of one or more thereof may be performed in a different order. In these and other embodiments, all of the blocks 1001 to 1013 or a subset of any thereof may be performed before, during, and / or after all of the other blocks 1001 to 1013 or a subset of any thereof. In addition, it will be readily appreciated by those skilled in the art that the method 1000 may be altered and still remain within these and other embodiments of the present technology. For example, in some embodiments, all of the blocks 1001 to 1013 or a subset of one or more thereof may be omitted and / or repeated.

[0207] As a specific example, in some embodiments, block 1008 of method 1000 may be omitted. For example, for each EVS pixel row, the EVS pixels may be continuously enabled to detect events occurring in the external scene between time ts (representing the start time of the integration period of the corresponding CIS pixel) and the interpolation timing point corresponding to the end of the associated integration period. For example, for example, reference is made to Fig.11 The EVS pixels of EVS row N may remain enabled to capture time t0 (representing time t of integration period 1197). s ) and the end of accumulation period 1199, is not deactivated or reset during period 1192. This can reduce the likelihood that EVS data occurring during period 1192 is not detected by the EVS pixels of EVS row N.

[0208] As another example, although the interpolated image frames are described above at box 1013 as latent image frames that each correspond to a time occurring after the CIS integration period of the corresponding CIS pixel ends, method 1000 is not limited to this. For example, at box 1013, method 1000 may calculate one or more latent image frames corresponding to one or more times occurring while the CIS integration period of the corresponding CIS pixel is ongoing. In such embodiments, EVS pixel data accumulated at times occurring outside the integration period may not be used in the video interpolation calculations. In other words, the video interpolation calculations used to generate one or more latent image frames corresponding to times occurring during the integration period of the corresponding CIS pixel may be based on (i) the latent image frames L(s) and (ii) all or a subset of the EVS pixel data accumulated during the integration period, for example, without taking into account EVS pixel data captured and accumulated after the end of the CIS integration period. A specific example of this case is described below with reference to Figures 14 to 15B describe.

[0209] Fig.14 is a flow chart illustrating a method 1400 of operating an imaging system according to various embodiments of the present technology. For example, method 1400 may be a method of (i) performing (e.g., on-chip) deblurring of CIS data and / or (ii) video interpolation. Method 1400 is illustrated as a series of boxes 1401 to 1409 or steps. All boxes 1401 to 1409 or a subset of one or more thereof may be performed by a device or component of an imaging system configured according to various embodiments of the present technology. For example, all boxes 1401 to 1409 or a subset of one or more thereof may be performed by a hybrid image sensor, a CIS pixel of a pixel array, an EVS pixel of an event-driven sensing array, a common control block, a row / column control circuit system, a column readout circuit system, a column scan readout circuit system, a deblurring block or circuit, and / or an application processor. All boxes 1401 to 1409 of method 1400 or a subset of one or more thereof may be performed according to the above. Figures 1 to 13B In fact, some of the blocks 1401 to 1409 of method 1400 are referred to below. Fig.15A and 15B describe.

[0210] As shown, method 1400 includes: aligning CIS pixel data with EVS pixel data on a row-by-row basis (box 1401); capturing CIS pixel data during corresponding exposure / integration periods and capturing EVS pixel data during corresponding accumulation periods aligned with the exposure / integration periods (box 1402); reading out EVS pixel data from EVS pixels (box 1403); accumulating EVS pixel data in one or more EVS integration buffers (e.g., of a hybrid image sensor) and / or an EVS frame buffer (e.g., of a downstream application processor) (box 1404); reading out CIS pixel data at the end of the integration period (box 1405); deblurring the CIS pixel data using the accumulated EVS pixel data to produce a latent image frame L(s) (box 1406); and outputting the latent image frame L(s) to an application processor (e.g., a CIS key frame buffer of an application processor) and / or an image signal processor (box 1407). Boxes 1401 to 1407 may be used in conjunction with the above. Fig.10 The blocks 1001 to 1007 of the method 1000 are the same or at least substantially similar. Therefore, for the sake of brevity, the detailed discussion of blocks 1401 to 1407 is basically omitted here.

[0211] Referring again to block 1404, method 1400 may continue from block 1404 to block 1408 to output the accumulated EVS pixel data. For example, during the accumulation period of the EVS pixels, Fig.14At blocks 1402 to 1404 of method 1400, EVS pixel data may be captured, read out, and accumulated by the EVS pixels. At block 1408, method 1400 may in turn output the accumulated EVS pixel data to an EVS frame buffer of a downstream application processor. For example, a running sum of events and / or an index of the product of the running sum and a contrast threshold may be output to a downstream application processor and / or loaded into an EVS frame buffer at a timing corresponding to the time at which the running sum is stored in a first integration buffer and / or the time at which the index is calculated. As another example, the running sum and / or index may be output to a downstream application processor and / or loaded into an EVS frame buffer at a specified timing, such as at an interpolation timing point. The running sum and / or index of the result of the first integration may be output to a downstream application processor and / or loaded into an EVS frame buffer row by row. Additionally or alternatively, loading of (a) a running sum (e.g., stored in a first integral buffer) and / or (b) an exponent (e.g., output from an exponential calculation block) into an application processor and / or a corresponding EVS frame buffer may be gated or otherwise controlled via a trigger and corresponding switch, as discussed in more detail below.

[0212] After (i) loading the accumulated EVS pixel data corresponding to the interpolation time into the EVS frame buffer of the downstream application processor at block 1408 and (ii) accumulating the event data until the end of the accumulation period aligned with the exposure period of the corresponding CIS pixel at blocks 1402-1404, method 1400 (at block 1408) may output the accumulated EVS pixel data from the EVS frame buffer of the application processor to a video interpolation calculation block (e.g., of the application processor). For example, the accumulated EVS pixel data stored in the EVS frame buffer and corresponding to the interpolation frame may be output to the video interpolation calculation block at a timing corresponding to the time when the deblurred image data is (a) output from the deblurring circuit to the CIS key frame buffer (e.g., of the application processor) and / or (b) provided to the video interpolation calculation block.

[0213] As a specific example, consider Fig.15A , which illustrates a timing diagram 1595a according to various embodiments of the present technology. Fig.14 and 15A At block 1401 of method 1400, an EVS pixel of EVS row N, a corresponding portion of an EVS integration buffer (e.g., of a deblurring circuit of a hybrid image sensor), and / or an EVS frame buffer (e.g., of a downstream application processor) and / or a corresponding portion of a CIS key frame buffer may be processed simultaneously with the corresponding CIS pixel and over a period extending over time t -1 With time t 0The EVS pixels of EVS row N and the corresponding CIS pixels are reset simultaneously to align the CIS integration period 1597 of the CIS pixels with the accumulation period 1598 of the EVS pixels of row N. Fig.15A Time t 0 , method 1400 may continue at block 1402 to capture CIS pixel data during integration period 1597 and corresponding EVS pixel data during accumulation period 1598. At blocks 1403 and 1404 of method 1400, EVS pixel data may be read out, accumulated, and stored in one or more EVS integration buffers (e.g., of a deblurring circuit of a hybrid image sensor).

[0214] At box 1408, EVS pixel data captured by the EVS pixels of EVS row N and accumulated between time t0 and time t5 may be output and stored in an EVS frame buffer (e.g., of an application processor), for example (a) by streaming the accumulated EVS pixel data to an EVS frame buffer between time t0 and time t5 or (b) by reading the accumulated EVS pixel data (e.g., row by row) out to the EVS frame buffer at or after time t5. As discussed above, in some embodiments, a trigger and corresponding switch may be used to control when the accumulated EVS pixel data is stored to the EVS frame buffer. For example, a trigger may be triggered to selectively enable a switch at time t0 so that the accumulated EVS pixel data may be streamed from the first integration buffer or the exponential calculation block into the EVS frame buffer. Then, at time t5 (corresponding to Fig.15A The trigger may deactivate the switch so that the accumulated EVS pixel data stored in the first integration buffer or output from the index calculation block after time t5 (e.g., between (i) time t5 and (ii) time t7 or t8) is not loaded into the EVS frame buffer. As another example, the trigger may be at time t5 (corresponding to the interpolation timing point of the EVS pixel row N in FIG. 1 ). Fig.15A The EVS pixel data stored in the first integration buffer and / or output from the index calculation block is read out to the EVS frame buffer row by row (at an interpolation timing point of the EVS pixel row N in the frame).

[0215] like Fig.15A, for EVS row N, the interpolation timing point occurs at time t5 and occurs in the middle of the accumulation period 1598. Therefore, the interpolation timing point of EVS row N occurs at a timing in the middle of the integration period 1597 of the corresponding CIS pixel. In fact, the integration period of the corresponding pixel extends from time t0 to time t7 and may cover the entire frame time (for example, under low light conditions). Thus, the accumulated EVS pixel data in the first integration buffer and / or output from the index calculation block may be output and stored to an EVS frame buffer (for example, of a downstream application processor) at a timing that occurs before the CIS pixel data captured by the corresponding CIS pixel is read out from the corresponding CIS pixel. In other embodiments, for example, under bright light conditions, the integration period 1597 of the corresponding pixel may extend from time t0 to a time that occurs before time t7, so that the integration period 1597 covers less than the entire frame time. In some such embodiments, the accumulated EVS pixel data in the first integration buffer and / or output from the exponential calculation block may be output and stored to an EVS frame buffer (e.g., of a downstream application processor) at the same timing as or after the time when the CIS pixel data captured by the corresponding CIS pixel is read out from the corresponding CIS pixel.

[0216] Return to reference Fig.14 , at block 1409, method 1400 may continue to calculate an interpolated video / image frame based on (a) the accumulated EVS pixel data output at block 1408 and (b) the latent image frame L(s). As discussed above, the latent image frame L(s) may be calculated and output at block 1407 based on (i) the CIS pixel data captured at block 1402 and read out from the CIS pixels at block 1405 at the end of the corresponding integration period and (ii) the EVS pixel data captured and accumulated during the accumulation period corresponding to the integration period. The calculation of the interpolated video / image frame may be similar to that discussed above. Fig.10 The calculation of the interpolated video / image frame at block 1013 of method 1000 is the same or at least substantially similar. Therefore, for the sake of brevity, a detailed description of the calculation of the interpolated video / image frame at block 1409 of method 1400 is omitted here.

[0217] Although blocks 1401 through 1409 of method 1400 are described and illustrated in a particular order, Fig.14The method 1400 of is not limited thereto. In other embodiments, all of the blocks 1401 to 1409 of the method 1400 or a subset of one or more thereof may be performed in a different order. In these and other embodiments, all of the blocks 1401 to 1409 or a subset of any of them may be performed before, during, and / or after all of the other blocks 1401 to 1409 or a subset of any of them. In addition, it will be readily appreciated by those skilled in the art that the method 1400 may be altered and still remain within these and other embodiments of the present technology. For example, in some embodiments, all of the blocks 1401 to 1409 or a subset of one or more thereof may be omitted and / or repeated.

[0218] As a specific example, blocks 1408 and 1409 may be repeated to generate more than one interpolated video / image frame. Fig. 15B Description and Fig.15A The timing diagram 1595a is substantially similar to the timing diagram 1595b. Fig. 15B 1504. As shown in FIG. 1505 , the accumulated EVS pixel data may be output at block 1408 of method 1400 at a plurality of different times throughout the illustrated integration period, as shown by arrows 1502-1504. For example, for EVS row N, EVS pixel data accumulated between time t0 and time t5 (corresponding to arrow 1502) may be loaded into the EVS frame buffer in a first instance of block 1408, EVS pixel data accumulated between time t0 and time t7 (corresponding to arrow 1503) may be loaded into the EVS frame buffer in a second instance of block 1408, and EVS pixel data accumulated between time t0 and time t9 (corresponding to arrow 1504) may be loaded into the EVS frame buffer in a third instance of block 1408. The accumulated EVS pixel data corresponding to each of arrows 1502-1504 may be (a) stored in a different EVS frame buffer (e.g., of an application processor) and / or (b) provided to a video interpolation computation block and used in Fig.14 The three interpolated video / image frames are calculated at block 1409 of method 1400. In some embodiments, loading the accumulated EVS pixel data into each of the different EVS frame buffers may be performed, for example, using one or more triggers and one or more corresponding switch gating (discussed in more detail below).

[0219] Fig.16is a flow chart illustrating another method 1600 of operating an imaging system according to various embodiments of the present technology. For example, method 1600 may be a method of (i) performing (e.g., on-chip) deblurring of CIS data, (ii) performing (e.g., on-chip) rolling shutter distortion correction, and / or (iii) video interpolation. Method 1600 is illustrated as a series of boxes 1601 to 1615 or steps. All boxes 1601 to 1615 or a subset of one or more thereof may be performed by a device or component of an imaging system configured according to various embodiments of the present technology. For example, all boxes 1601 to 1615 or a subset of one or more thereof may be performed by a hybrid image sensor, a CIS pixel of a pixel array, an EVS pixel of an event-driven sensing array, a common control block, a row / column control circuit system, a column readout circuit system, a column scan readout circuit system, a deblurring block or circuit, and / or an application processor. All boxes 1601 to 1615 of method 1600 or a subset of one or more thereof may be performed according to the above. Figures 1 to 15B In fact, several of the blocks 1601 to 1615 of method 1600 are referred to below. Fig.17A and 17B describe.

[0220] Method 1600 begins with block 1601, which resets EVS pixels and corresponding portions of one or more EVS integration buffers (e.g., one or more EDI integration buffers and / or one or more RSDC integration buffers). In some embodiments, resetting the EVS pixels and corresponding portions of the EVS integration buffers may include resetting the EVS pixels and corresponding portions of the EVS integration buffers simultaneously and / or during a period that occurs before the start of an event accumulation period that precedes an integration period for the corresponding CIS pixels. In these and other embodiments, corresponding portions of one or more EVS frame buffers and / or one or more CIS key frame buffers (e.g., of a downstream application processor) may also be reset at block 1601, e.g., simultaneously with corresponding portions of the EVS pixels and / or EVS integration buffers. The reset may be similar to that described above with reference to Fig.10 The reset performed at block 1001 of the described method 1000 is substantially similar. Therefore, for the sake of brevity, a detailed discussion of the reset performed at block 1601 is substantially omitted herein. Compared to the reset performed at block 1001 described above, the reset performed at block 1601 may omit resetting the corresponding CIS pixel and / or resetting the CIS key frame buffer.

[0221] Fig.17A A timing diagram 1795a is shown illustrating various embodiments of the present technology. Fig.16 and 17A At block 1601 of method 1600, EVS pixels of EVS row N may be arranged to extend over time t -7 With time t-6 The EVS pixels of EVS row N may be reset during period 1792 between the periods 1792 and before the accumulation period 1796 for the EVS pixels of EVS row N. Portions of one or more EDI integration buffers corresponding to the EVS pixels of EVS row N, portions of one or more RSDC integration buffers corresponding to the EVS pixels of EVS row N, and / or portions of one or more EVS frame buffers (e.g., of an application processor) corresponding to the EVS pixels of EVS row N may be reset during period 1792 and / or concurrently with the resetting of the EVS pixels of EVS row N.

[0222] Reference again Fig.16 At blocks 1602 to 1604, method 1600 then captures, reads out, and accumulates EVS pixel data for each EVS pixel within a corresponding accumulation period. The corresponding accumulation period may precede the integration period of the corresponding CIS pixel. For example, the accumulation period 1796 of the EVS pixel corresponding to EVS row N is Fig.17A 1796 is shown as preceding the integration period 1797 of the CIS pixel corresponding to the EVS pixel of EVS row N. In the illustrated example, the integration / exposure period of the corresponding CIS pixel begins in the middle of the interpolated frame. Therefore, the accumulation period 1796 precedes the start of the integration period 1797 of the CIS pixel corresponding to the EVS pixel of EVS pixel row N.

[0223] Reading out EVS data from the EVS pixel at block 1603 of method 1600 may be performed after and / or concurrently with performing block 1602. For example, when an EVS pixel detects an event, the event may be read out from the EVS pixel. When an event is read out from the EVS pixel, the EVS pixel may be reset and thereby enabled to detect subsequent events. Reading out the EVS data may include an RSDC component that reads out the EVS data to a computational block of a deblurring circuit. Additionally or alternatively, reading out the EVS data may include an EDI component that reads out the EVS data to a computational block of a deblurring circuit. In these and other embodiments, reading out the EVS data may include reading out the EVS data using a scanning readout technique, such as described above with reference to Fig.13A and 13B The scanning readout technique is described in more detail.

[0224] The accumulation performed at block 1604 of method 1600 may include, for each EVS pixel, accumulating the EVS pixel data read out from the EVS pixel during a corresponding accumulation period. Fig.17A The accumulation period 1796) may extend from (i) the beginning of the interpolation frame (eg, time t -7 ) and (ii) the start of the image frame (eg, time t0). Alternatively, the corresponding accumulation period may extend between (i) the start of the interpolation frame (eg, time t -7 ) and (ii) the reset period before the start of an image frame (e.g. Fig.17A The period 1793) starts (for example, time t -1 ). Block 1604 may be performed after and / or concurrently with execution of blocks 1602 and / or 1603, as shown by an arrow returning from block 1604 to block 1602. Additionally or alternatively, block 1604 may be performed by an EDI component and / or an RSDC component of a deblurring circuit (e.g., of a hybrid image sensor).

[0225] As discussed above, accumulating EVS data may include, for each EVS pixel, calculating and maintaining a running sum of events, such as using a counter and an integrating buffer (e.g., of a deblurring circuit of a hybrid image sensor). Additionally, accumulating EVS data may include, for each EVS pixel, (a) calculating the product of the running sum and a contrast threshold parameter and (b) determining an exponent of the product.

[0226] At block 1605, method 1600 may then output the accumulated EVS pixel data. For example, during the accumulation period of the EVS pixels, Fig.16 At blocks 1602 to 1604 of method 1600, EVS pixel data may be captured, read out, and accumulated by the EVS pixels. At block 1605, method 1600 may in turn output the accumulated EVS pixel data to an EVS frame buffer of a downstream application processor. For example, a running sum of events stored in an integration buffer and / or an index of the product of each pixel may be output to a downstream application processor and / or loaded into an EVS frame buffer at a timing corresponding to the time at which the running sum is stored in the integration buffer and / or the time at which the index is calculated. As another example, the running sum and / or index may be output to a downstream application processor and / or loaded into an EVS frame buffer at a specified timing, such as at the end of a corresponding accumulation period, at the beginning of a corresponding exposure period, at an interpolation timing point, etc. The running sum and / or index may be output to a downstream application processor and / or loaded into an EVS frame buffer row by row. Additionally or alternatively, loading of (a) a running sum (e.g., stored in an integral buffer) and / or (b) an exponent (e.g., output from an exponential calculation block) into an application processor and / or a corresponding EVS frame buffer may be gated or otherwise controlled via flip-flops and corresponding switches, as described above (e.g., with reference to Fig.14 ) and are discussed in more detail below.

[0227] Additionally or alternatively, outputting the accumulated EVS pixel data at block 1605 may include outputting all of the accumulated EVS pixel data, or a subset thereof, to a latent frame calculation block of a deblurring circuit to, for example, perform rolling shutter distortion correction of the corresponding CIS pixel data. In these and other embodiments, outputting the accumulated EVS pixel data may include, for example, outputting the accumulated EVS pixel data from one or more EVS frame buffers to a video interpolation block (e.g., of an application processor) at a timing corresponding to the time when a CIS key frame is output to the video interpolation block. In some embodiments, the accumulated EVS pixel data may be output from the EDI integration buffer, RSDC buffer, exponential calculation block, and / or EVS frame buffer at block 1605 as part of a row-by-row readout of the accumulated EVS pixel data that occurs before CIS integration begins (e.g., as determined by Fig.17A 1705 as shown).

[0228] As a specific example, refer again to Fig.17A , (i) captured by the EVS pixel of EVS row N+2, (ii) read out, and (iii) over a period extending approximately over time t -4 With t 1 All EVS pixel data accumulated during the EVS accumulation period between time t0 and time t1 may be stored in the EVS integration buffer, output to the EVS frame buffer, and / or provided to the video interpolation calculation block of the application processor. Additionally or alternatively, the EVS pixel data captured by the EVS pixels of EVS row N+2 between time t0 (corresponding to the start of the image frame) and time t1 may be stored in the RSDC integration buffer and output to the latent frame calculation block of the corresponding deblurring circuit to enable the latent frame calculation block to perform rolling shutter distortion correction on the CIS pixel data captured by the EVS pixels corresponding to the EVS pixels of EVS row N+2 during a subsequent integration period (extending between time t2 and time t9).

[0229] At blocks 1606-1613, method 1600 then: aligns CIS pixel data with EVS pixel data on a row-by-row basis (block 1606); captures CIS pixel data during corresponding exposure / integration periods and captures EVS pixel data during corresponding accumulation periods aligned with the exposure / integration periods (block 1607); reads out EVS pixel data from EVS pixels (block 1608); accumulates EVS pixel data in, for example, (i) one or more EVS integration buffers (e.g., of a hybrid image sensor) and / or an EVS frame buffer (e.g., of a downstream application processor) (Block 1609); read out the CIS pixel data at the end of the integration period (Block 1610); deblur the CIS pixel data using the accumulated EVS pixel data to produce a latent image frame L(s) (Block 1611); correct the rolling shutter distortion of the CIS pixel data using all or a subset of the EVS pixel data accumulated between the start of the image frame and the start of the corresponding integration period to produce a latent image frame L(0) (Block 1612); and output the latent image frame L(s) and / or the latent image frame L(0) to an application processor (e.g., a CIS key frame buffer of the application processor) and / or an image signal processor (Block 1613). Blocks 1606 to 1611 may be the same as those described above. Fig.10 Blocks 1001 to 1006 of method 1000 and Fig.14 The blocks 1401 to 1406 of the method 1400 are the same or at least substantially similar. Therefore, for the sake of brevity, the detailed discussion of blocks 1606 to 1611 is basically omitted here.

[0230] Correcting the rolling shutter distortion of the CIS data at block 1612 of method 1600 may include using the accumulated EVS data stored in the corresponding portion of the RSDC integration buffer at block 1604 to correct the rolling shutter distortion of the CIS data. More specifically, the accumulated EVS data stored in the RSDC integration buffer of the deblurring circuit between the start time t0 of the image frame and the start of the corresponding exposure period of the CIS pixel aligned with the integration period of the image frame may be multiplied by a contrast threshold. The deblurring circuit may in turn determine an exponent of the product of the accumulated EVS data and the contrast threshold, and may output the exponent to a latent frame calculation block of the deblurring circuit. Thereafter, the exponent may be used to correct the rolling shutter distortion of the CIS data (e.g., the CIS data captured by the CIS pixel at block 1607 and / or the deblurred CIS data from block 1611), for example, using equation 20 above to determine the corresponding latent image frame L(0).

[0231] Outputting the deblurred and / or rolling shutter distortion corrected image data at block 1613 of method 1600 may include outputting one or more latent image frames (e.g., latent image frame L(s) and / or latent image frame L(0)) calculated at blocks 1611 and / or 1612, e.g., in addition to or in lieu of outputting the raw CIS data read out from the CIS pixels at block 1610 and / or the raw EVS data generated and read out from the EVS pixels at blocks 1607 and 1608. Outputting the deblurred and / or rolling shutter distortion corrected image data may include outputting the deblurred image data from the image sensor to a downstream application processor. As a specific example, outputting the deblurred and / or rolling shutter distortion corrected image data may include outputting the deblurred and / or rolling shutter distortion corrected image data to a CIS key frame buffer and / or a video interpolation calculation block of a downstream application processor. In some embodiments, a trigger and corresponding switch may be used to gate or control loading of the deblurred and / or rolling shutter distortion corrected image data into the CIS key frame buffer. Additionally or alternatively, outputting the deblurred and / or rolling shutter distortion corrected image data may include outputting the deblurred and / or rolling shutter distortion corrected image data to an image signal processor, such as for previewing the deblurred and / or rolling shutter distortion corrected image data.

[0232] Referring again to block 1607, method 1600 may continue from block 1607 to block 1614 to output the accumulated EVS pixel data. Fig.16 At blocks 1607 to 1609 of method 1600, during the accumulation period of the EVS pixels, EVS pixel data may be captured, read out, and accumulated by the EVS pixels. At block 1614, method 1600 may in turn output the accumulated EVS pixel data to an EVS frame buffer of a downstream application processor. For example, a running sum of events and / or an index of the running sum multiplied by a contrast threshold may be output to a downstream application processor and / or loaded into an EVS frame buffer at a timing corresponding to the time at which the running sum is stored in a first integration buffer and / or the time at which the index is calculated. As another example, the running sum and / or index may be output to a downstream application processor and / or loaded into an EVS frame buffer at a specified timing, such as at an interpolation timing point. The running sum and / or index may be output to a downstream application processor and / or loaded into an EVS frame buffer line by line. Additionally or alternatively, loading of (a) the running sum (e.g., stored in the first integration buffer) and / or (b) the exponent (e.g., output from the exponent calculation block) into the application processor and / or the corresponding EVS frame buffer may be gated or otherwise controlled via a flip-flop and corresponding switch, as described above with reference to Fig.14 and is discussed in more detail below.

[0233] After (i) loading the accumulated EVS pixel data corresponding to the interpolation time into the EVS frame buffer of the downstream application processor at block 1614 and (ii) accumulating event data until the end of the accumulation period aligned with the exposure period of the corresponding CIS pixel at blocks 1607-1609, method 1600 (at block 1614) may output the accumulated EVS pixel data from the EVS frame buffer to a video interpolation calculation block (e.g., of the application processor). For example, the accumulated EVS pixel data stored in the EVS frame buffer and corresponding to the interpolation time may be output to the video interpolation calculation block at a timing corresponding to the time when the deblurred image data is (a) output from the deblurring circuit to the CIS key frame buffer (e.g., of the application processor) and / or (b) provided to the video interpolation calculation block.

[0234] As a specific example, see also Fig.16 and 17A At block 1606 of method 1600, an EVS pixel of EVS row N, a corresponding portion of an EVS integration buffer (e.g., of a deblurring circuit of a hybrid image sensor), and / or an EVS frame buffer (e.g., of a downstream application processor) and / or a corresponding portion of a CIS key frame buffer may be processed simultaneously with the corresponding CIS pixel and over a period extending over time t -1 With time t 0 The EVS pixels of EVS row N and the corresponding CIS pixels are reset simultaneously to align the CIS integration period 1797 of the CIS pixels with the accumulation period 1798 of the EVS pixels of EVS row N. Fig.17A Time t 0 , the method 1600 may continue at block 1607 to capture CIS pixel data during an integration period 1797 and corresponding EVS pixel data during an accumulation period 1798. At blocks 1608 and 1609 of the method 1600, the EVS pixel data may be read out, accumulated, and stored in one or more EVS integration buffers, such as an EDI integration buffer of a deblurring circuit of a hybrid image sensor.

[0235] At box 1614, EVS pixel data captured by the EVS pixels of EVS row N and accumulated between time t0 and time t5 may be output and stored in an EVS frame buffer (e.g., of an application processor), for example (a) by streaming the accumulated EVS pixel data to an EVS frame buffer between time t0 and time t5 or (b) by reading the accumulated EVS pixel data (e.g., row by row) out to the EVS frame buffer at or after time t5. As discussed above, in some embodiments, a trigger and corresponding switch may be used to control when the accumulated EVS pixel data is stored to the EVS frame buffer. For example, the trigger may be triggered to selectively enable the switch at time t0 (corresponding to the beginning of exposure period 1797 of the CIS pixel corresponding to the EVS pixel of EVS row N) so that the accumulated EVS pixel data may be streamed from the first integration buffer or the exponential calculation block to the EVS frame buffer. Then, at time t5 (corresponding to the beginning of exposure period 1797 of the CIS pixel corresponding to the EVS pixel of EVS row N), the accumulated EVS pixel data may be read out to the EVS frame buffer. Fig.17A The trigger may deactivate the switch so that the accumulated EVS pixel data stored in the first integration buffer or output from the index calculation block after time t5 (e.g., between (i) time t5 and (ii) time t7 or t8) is not loaded into the EVS frame buffer. As another example, the trigger may be at time t5 (corresponding to the interpolation timing point of the EVS pixel row N in FIG. 1 ). Fig.17A The EVS pixel data stored in the first integration buffer and / or output from the index calculation block is read out to the EVS frame buffer row by row (at an interpolation timing point of the EVS pixel row N in the frame).

[0236] like Fig.17A , for EVS row N, the interpolation timing point occurs at time t5 and occurs in the middle of the accumulation period 1798. Therefore, the interpolation timing point of EVS row N occurs at a timing in the middle of the integration period 1797 of the corresponding CIS pixel. In fact, the integration period of the corresponding pixel extends from time t0 to time t7 and may cover the entire frame time (for example, under low light conditions). Thus, the accumulated EVS pixel data in the first integration buffer and / or output from the index calculation block may be output and stored to an EVS frame buffer (for example, of a downstream application processor) at a timing that occurs before the CIS pixel data captured by the corresponding CIS pixel is read out from the corresponding CIS pixel. In other embodiments, for example, under bright light conditions, the integration period 1797 of the corresponding pixel may extend from time t0 to a time that occurs before time t7, so that the integration period 1797 covers less than the entire frame time. In some such embodiments, the accumulated EVS pixel data in the first integration buffer and / or output from the exponential calculation block may be output and stored to an EVS frame buffer (e.g., of a downstream application processor) at the same timing as the time when the CIS pixel data captured by the corresponding CIS pixel is read out from the corresponding CIS pixel, or at a timing that occurs after that time.

[0237] Return to reference Fig.16 , at block 1615, method 1600 may continue to calculate an interpolated video / image frame based on (a) the EVS pixel data accumulated at block 1604 and output at block 1605, (b) the EVS pixel data accumulated at block 1609 and output at block 1614, and (c) the latent image frame L(s) and / or the latent image frame L(0) calculated at blocks 1611 and 1612, respectively, and output at block 1613. For example, referring to Fig.17A , calculating the interpolated video / image frame may include using (i) the accumulated EVS pixel data output at box 1605, (ii) the accumulated EVS pixel data output at box 1614, (iii) the latent image frame L(0) output at box 1613 and (iv) the above-mentioned equation 21 to calculate the interpolated image frame as the latent image frame L(t5).

[0238] Although blocks 1601 through 1615 of method 1600 are described and illustrated in a particular order, Fig.16 The method 1600 of is not limited thereto. In other embodiments, all of the blocks 1601 to 1615 of the method 1600 or a subset of one or more thereof may be performed in a different order. In these and other embodiments, all of the blocks 1601 to 1615 or a subset of any of them may be performed before, during, and / or after all of the other blocks 1601 to 1615 or a subset of any of them. In addition, it will be readily appreciated by those skilled in the art that the method 1600 may be altered and still remain within these and other embodiments of the present technology. For example, in some embodiments, all of the blocks 1601 to 1615 or a subset of one or more thereof may be omitted and / or repeated.

[0239] As a specific example, blocks 1605, 1614, and / or 1615 may be repeated to generate more than one interpolated video / image frame. Fig. 17B Description and Fig.17A The timing diagram 1795a is substantially similar to the timing diagram 1795b. Fig. 17B , the accumulated EVS pixel data may be output at block 1605 of method 1600 at a time corresponding to arrow 1705 shown in timing diagram 1795b. For example, for EVS row N, at approximately time t -6 With time t -11705) may be loaded into the EVS frame buffer at block 1605 of method 1600. Additionally, the accumulated EVS pixel data may be output at a plurality of different times throughout the illustrated integration period, as shown by arrows 1702-1704. For example, for EVS row N, EVS pixel data accumulated between time t0 and time t3 (corresponding to arrow 1702) may be loaded into the EVS frame buffer in a first instance of block 1614, EVS pixel data accumulated between time t0 and time t6 (corresponding to arrow 1703) may be loaded into the EVS frame buffer in a second instance of block 1614, and EVS pixel data accumulated between time t0 and time t8 (corresponding to arrow 1704) may be loaded into the EVS frame buffer in a third instance of block 1614. The accumulated EVS pixel data corresponding to each of arrows 1702-1704 may be (a) stored in a different EVS frame buffer (e.g., of an application processor) and / or (b) provided to a video interpolation calculation block and used in Fig.16 The three interpolated video / image frames are calculated at block 1615 of method 1600. The accumulated EVS pixel data corresponding to arrow 1705 may be (a) loaded into all different EVS frame buffers or a subset thereof and / or (b) provided to the video interpolation frame calculation block and used to calculate the three interpolated video / image frames at block 1615. In some embodiments, loading the accumulated EVS pixel data into each of the different EVS frame buffers may be performed, for example, using one or more triggers and one or more corresponding switch gating (discussed in more detail below).

[0240] As another specific example, in some embodiments, resetting the EVS pixels, the corresponding portion of the EVS integration buffer, the corresponding portion of the RSDC integration buffer, and / or the corresponding portion of the EVS frame buffer may be omitted from block 1606 of method 1600. For example, for each EVS pixel row, the EVS pixels may be continuously enabled to detect the corresponding CIS pixel during the integration period (e.g., Fig.17A The events that occur in the external scene between the beginning of the corresponding accumulation period before the integration period 1797) and the end of the integration period. For example and clarity, refer to Fig.17A In the embodiment of the present invention, the EVS pixels of EVS row N of EVS row N may remain enabled to capture EVS pixel data for the entire duration between the start of accumulation period 1796 and time t7 corresponding to the end of integration period 1797, and are not deactivated or reset during period 1793. This may reduce the likelihood that EVS data occurring during period 1793 is not detected by the EVS pixels of EVS row N.

[0241] As yet another example, although the interpolated image frame is referred to above Figures 16 to 17BDescribed as latent image frames each corresponding to a time occurring within (or during) a CIS integration period of a corresponding CIS pixel, but method 1600 is not limited thereto. For example, at block 1615, method 1600 may calculate one or more latent image frames corresponding to one or more times occurring before the CIS integration period of the corresponding CIS pixel. Additionally or alternatively, at block 1615, method 1600 may calculate one or more latent image frames corresponding to one or more times occurring after the CIS integration period of the corresponding pixel. In such embodiments, the EVS pixel may be enabled to capture EVS pixel data at a time occurring after the end of the integration period, so that the EVS pixel data may be accumulated and provided to the video interpolation frame calculation block to calculate the corresponding interpolated video / image frame.

[0242] Fig.18 18 is a partial schematic diagram illustrating an imaging system 1840 configured in accordance with various embodiments of the present technology. As shown, the imaging system 1840 includes a system processor 1830, such as an application processor. The system processor 1830 includes a first frame buffer 1843, a second frame buffer 1844, a third frame buffer 1846, a video interpolation calculation block 1845, and an image signal processor (ISP) component 1852. The first frame buffer 1843 is configured to store one or more key frames of CIS data 1821. The CIS key frame may include raw CIS data captured by, for example, CIS pixels of an upstream hybrid image sensor (not shown) coupled to the system processor 1830. Additionally or alternatively, the CIS key frame may include, for example, one or more latent image frames (e.g., latent image frame L(s) including deblurred CIS data and / or latent image frame L(0) including deblurred and rolling shutter distortion corrected CIS data) output to the system processor 1830 by a deblurring circuit (not shown) of the upstream hybrid image sensor (not shown). The second frame buffer 1844 may be configured to store one or more frames of accumulated EVS data 1871. The accumulated EVS data may include EVS data accumulated before, during, and / or after an integration / exposure period corresponding to a CIS key frame.

[0243] The video interpolation calculation block 1845 may include a CIS / EVS synchronization block 1845a, an event-oriented deblurring block 1845b, and / or a video interpolation block 1845c. The CIS / EVS synchronization block 1845a may be configured to synchronize the accumulated EVS data output from the second frame buffer 1844 with the CIS key frames output from the first frame buffer 1843. After the accumulated EVS data is synchronized with the CIS key frames, the event-oriented deblurring block 1845b may be configured to deblur the CIS key frames using all or a subset of the accumulated EVS data. In addition, the video interpolation block 1845c may be configured to calculate one or more interpolated video / image frames based at least in part on the accumulated EVS data and the CIS data of the key frames. As discussed above, the interpolated video / image frame may include one or more latent images calculated based on, for example, latent image L(s), latent image L(0), and / or accumulated EVS data.

[0244] In at least some embodiments where the accumulated EVS data is synchronized upstream with the CIS key frame (e.g., by resetting the EVS pixel and the corresponding CIS pixel simultaneously), at least a portion of the CIS / EVS synchronization performed by the CIS / EVS synchronization block 1845a may be skipped or omitted. Additionally or alternatively, in at least some embodiments where the CIS key frame is deblurred upstream using the accumulated EVS data (e.g., by a hybrid image sensor coupled to the system processor 1830), at least a portion of the deblurring process performed by the event-directed deblurring block 1845b may be skipped or omitted.

[0245] The third frame buffer 1846 is configured to store N frames of CIS data output from the video interpolation calculation block 1845, where N represents the interpolation ratio. Therefore, when N is 4, the third frame buffer 1846 may be configured to store four frames of CIS data output from the video interpolation calculation block 1845. In some embodiments, the four frames of CIS data may include at least one CIS key frame (e.g., latent image L(s) and / or latent image L(0)) and up to three interpolated video / image frames (e.g., three latent images L(t), e.g., corresponding to three different times). In other embodiments, the four frames of CIS data may include up to four interpolated video / image frames (e.g., four latent images L(t), e.g., corresponding to four different times).

[0246] like Fig.18 , CIS data 1821 (e.g., key frames of CIS data) may be scaled down (e.g., to reduce resolution) and provided to ISP component 1852 for preview. Additionally or alternatively, frames output from third frame buffer 1846 may be provided to ISP component 1852 for storage and / or playback.

[0247] In the illustrated embodiment, the system processor 1830 further includes a first trigger check block 1842a and a second trigger check block 1842b that are each responsive to the trigger 1841. In some embodiments, the trigger 1841 can be used to control when the CIS data 1821 and the EVS data 1871 are provided to and / or loaded in the first frame buffer 1843 and the second frame buffer 1844, respectively. For example, a pre-processor (not shown) may, for example, perform data analysis on the EVS data captured by the EVS pixels of the upstream hybrid image sensor (not shown) to identify when motion has occurred within the external scene. In response to identifying motion in the external scene, the trigger 1841 may enable the first trigger check block 1842a (e.g., a first switch or a first multiplexer) and the second trigger check block 1842b (e.g., a second switch or a second multiplexer) to pass the CIS data 1821 to the first frame buffer 1843 and the EVS data 1871 to the second frame buffer 1844, respectively.

[0248] As another example, the trigger 1841 may be triggered or activated at a specified timing. For example, the trigger 1841 may be triggered to selectively enable the first trigger check block 1842a at a timing corresponding to the time when the corresponding deblurring circuit outputs the deblurred image data (e.g., latent image frame L(s)) and / or the deblurred and rolling shutter distortion corrected image data (e.g., latent image frame L(0)) to the system processor 1830 for storage in the first frame buffer 1843. Additionally or alternatively, the trigger 1841 may be triggered to selectively enable the second trigger check block 1842b at a timing corresponding to the time when the accumulated EVS pixel data used in the video interpolation calculation should be loaded into the second frame buffer 1844 (e.g., at the beginning of the exposure period, at the end of the exposure period, at the beginning of the integration period, at the end of the integration period, at the end of the reset period, at the interpolation timing, etc.). In other words, the trigger 1841 and the corresponding first trigger check block 1842a and second trigger check block 1842b can be used to selectively load CIS data and EVS data into the first frame buffer 1843 and the second frame buffer 1844 at a given time and / or within a given period.

[0249] Fig.19 1 is a partial schematic diagram illustrating an imaging system 1940 configured in accordance with various embodiments of the present technology. As shown, the imaging system 1940 includes a system processor 1930, such as an application processor. The system processor 1930 includes a processor that is associated with Fig.18The system processor 1930 of the imaging system 1840 is substantially similar to the selected components of the system processor 1830. For example, the system processor 1930 includes a first frame buffer 1943, a second frame buffer 1944, a third frame buffer 1946, a video interpolation calculation block 1945, an image signal processor (ISP) component 1952, and a first trigger check block 1942a and a second trigger check block 1942b responsive to the trigger 1941. Therefore, in view of the above reference Fig.18 A detailed description of substantially similar components of imaging system 1840 is described, and a detailed description of each of these components of imaging system 1940 is substantially omitted herein for the sake of brevity.

[0250] like Fig.19 As shown in FIG. 1 , the imaging system 1940 has Fig.18 For example, the third frame buffer 1946 is configured to store N+1 frames of image data (rather than N+1 frames that can be stored in the image buffer). Fig.18 In addition, the frames output from the third frame buffer 1946 to the ISP component 1952 can be reduced and previewed. Fig.18 In the imaging system 1840 of FIG. 1840, the CIS data 1821 may be downscaled when it is input into the imaging system 1840 before being provided to the ISP component 1852 (e.g., before being stored in the first frame buffer 1843 and / or the third frame buffer 1846 and before being processed by the video interpolation calculation block 1845). In some embodiments, when the input frame rate (e.g., 7.5 fps) is less than the display frame rate of the preview functionality, a preview function may be used. Fig.19 In these and other embodiments, when the input frame rate (e.g., 30 fps) is greater than or equal to the display frame rate of the preview functionality, the preview function may be used. Fig.18 The arrangement of imaging system 1840 is illustrated in FIG.

[0251] Fig. 20 2050 ("VFI pipeline 2050") is a video interpolation pipeline 2050 ("VFI pipeline 2050") configured in accordance with various embodiments of the present technology. In some embodiments, the VFI pipeline 2050 may be a neural network processing unit. In these and other embodiments, the VFI pipeline 2050 may be used to generate interpolated video / image frames. For example, the VFI pipeline 2050 may be used to combine spatially dense CIS image data with temporally dense EVS data to generate slow motion video. The CIS image data may be captured using an active image sensor, and the EVS data may be captured using a separate event vision sensor. Alternatively, the CIS image data may be captured using CIS pixels of a hybrid image sensor, and the EVS data may be captured using EVS pixels of a hybrid image sensor.

[0252] As shown, the CIS data 2021 may be input into the VFI pipeline 2050 via a multiplexer 2042 and stored in a first frame buffer 2053 (e.g., a circular buffer). As discussed in more detail below, the multiplexer 2042 (or switch) may be controlled using a trigger 2041 of the VFI pipeline 2050. Additionally or alternatively, the CIS data 2021 may be provided to a preview ISP 2062 for previewing the CIS data 2021. In some embodiments, the CIS data 2021 includes raw CIS data. In other embodiments, the CIS data 2021 includes deblurred and / or rolling shutter distortion corrected CIS data, such as one or more latent image frames (e.g., latent image frame L(s) and / or latent image frame L(0)). For example, an upstream deblurring circuit (not shown), such as an upstream hybrid image sensor (not shown) coupled to the VFI pipeline 2050 , may be configured to deblur the CIS data and / or correct rolling shutter distortion of the CIS data and thereafter output the CIS data 2021 to the VFI pipeline 2050 .

[0253] The EVS data 2022 may be input into the VFI pipeline 2050 and stored in a second frame buffer 2054 (e.g., a circular buffer). The EVS data 2022 may include raw EVS data. Additionally or alternatively, the EVS data 2022 may include accumulated EVS data accumulated by an upstream hybrid image sensor (not shown), such as a deblurring circuit of the upstream hybrid image sensor.

[0254] The EVS data 2022 stored to the second frame buffer 2054 may be output to a preprocessor block 2055, which is configured to preprocess events in the EVS data 2022 for further processing in the VFI pipeline 2050. As shown, the preprocessor block 2055 includes an activity monitor block 2055a, a decoder block 2055b, and a denoiser block 2055c. The denoiser block 2055c may be configured to denoise the EVS data 2022, and the decoder block 2055b may be configured to decode the EVS data 2022 for interpretation by the activity monitor block 2055a and / or other components of the VFI pipeline 2050. In embodiments where the events in the EVS data 2022 are not encoded, the decoder block 2055b may be omitted. The activity monitor block 2055a is configured to analyze the EVS data to identify motion in the external scene. When motion is identified in the EVS data 2022 by the activity monitor block 2055a, the pre-processor block 2055 may activate the trigger 2041 to enable the multiplexer 2042 to allow the CIS data 2021 to pass to the first frame buffer 2053. The CIS data 2021 and the EVS data 2022 may be buffered in the first frame buffer 2053 and the third frame buffer 2056, respectively, for synchronization.

[0255] In some embodiments, trigger 2041 may be automatically triggered when motion is recognized in the external scene. In these and other embodiments, trigger 2041 may be manually triggered, e.g., in response to recognition of motion in the external scene and / or independently of motion recognized in the external scene. In these and other embodiments, trigger 2041 may be triggered based on a timer (e.g., after a preset duration has elapsed), e.g., in response to recognition of motion in the external scene and / or independently of motion recognized in the external scene.

[0256] As another example, the trigger 2041 may be triggered or activated at a specified timing. For example, the trigger 2041 may be triggered to selectively enable the multiplexer 2042 at a timing corresponding to the time when the corresponding deblurring circuit outputs the deblurred image data (e.g., latent image frame L(s)) and / or the deblurred and rolling shutter distortion corrected image data (e.g., latent image frame L(0)) to the processor 2057 for storage in the first frame buffer 2053. Additionally or alternatively, the pre-processor block 2055 may be used to gate or control when the EVS data and / or the accumulated EVS data stored in the second frame buffer 2054 is loaded into the third frame buffer 2056, such as at the beginning of the exposure period, at the end of the exposure period, at the beginning of the integration period, at the end of the integration period, at the end of the reset period, at the interpolation timing, etc. In other words, the flip-flop 2041, the multiplexer 2042 and / or the pre-processor block 2055 may be used to selectively enable the CIS data and the EVS data to be loaded into the first frame buffer 2053 and the third frame buffer 2056, respectively, at a given time and / or within a given period.

[0257] The CIS data 2021 stored in the first frame buffer 2053 and the EVS data 2022 preprocessed by the preprocessor block 2055 and stored in the third frame buffer 2056 may be output to one or more processors 2057 (e.g., one or more CPUs, GPUs, NPUs, and / or DSPs) of the VFI pipeline 2050. As shown, the processor 2057 includes an EVS / CIS synchronization block 2057a, a threshold calibration block 2057b, a deblurring and / or rolling shutter distortion correction block 2057c, and / or a video interpolation block 2057d. The EVS / CIS synchronization block 2057a may be configured to synchronize the CIS data output from the first frame buffer 2053 with the corresponding EVS data 2022 output from the third frame buffer 2056. The threshold calibration block 2057b is configured to calibrate the contrast threshold. The rolling shutter distortion correction block 2057c is configured to use the EVS data 2022 to deblur the CIS data 2021 and / or correct the rolling shutter distortion of the CIS data 2021 to, for example, generate a latent image frame L(s) and / or a latent image frame L(0). The video interpolation block 2057d is configured to interpolate one or more additional video / image frames using the deblurred and / or rolling shutter distortion corrected CIS data 2021 and all or a subset of the EVS data 2022. As discussed above, the interpolated video / image frames can be used to generate slow motion video.

[0258] The interpolated video / image frames may be output from the processor 2057 (e.g., from the video interpolation block 2057d) to a fourth frame buffer 2058. In some embodiments, the fourth frame buffer 2058 may be a ping-pong buffer capable of reading out one interpolated video / image frame to the ISP component 2052 while another video / image frame is interpolated. The ISP component 2052 may be configured to output the interpolated video / image frames to an MPEG encoder, which in turn may be configured to provide the encoded interpolated video / image frames to a memory for storage.

[0259] The VFI pipeline 2050 may include or be embodied by various components of the imaging system. In some embodiments, the VFI pipeline 2050 may be embodied by an image sensor, such as a hybrid image sensor that includes a deblurring circuit. In such embodiments, the image sensor may include on-chip deblurring capability, on-chip rolling shutter distortion correction capability, on-chip contrast threshold calibration capability, and / or on-chip video interpolation capability. In other embodiments, the VFI pipeline 2050 may be embodied by an off-chip processor, such as an application processor positioned downstream of one or more image sensors. In such embodiments, the imaging system may include off-chip deblurring capability, off-chip rolling shutter distortion correction capability, off-chip contrast threshold calibration capability, and / or off-chip video interpolation capability. In other embodiments, the VFI pipeline 2050 may be embodied in part by one or more image sensors (e.g., a hybrid image sensor) and in part by an off-chip processor (e.g., an application processor downstream of the hybrid image sensor). In such embodiments, all deblurring processes or a subset thereof, all rolling shutter distortion correction processes or a subset thereof, all contrast threshold calibration processes or a subset thereof, and / or all video interpolation processes and / or a subset thereof may be performed on-chip, while all deblurring processes or a subset thereof, all rolling shutter distortion correction processes or a subset thereof, all contrast threshold calibration processes or a subset thereof, and / or all video interpolation processes and / or a subset thereof may be performed off-chip.

[0260] C. in conclusion

[0261] The above detailed description of the embodiments of the present technology is not intended to be exhaustive or to limit the present technology to the precise form disclosed above. Although specific embodiments and examples of the present technology are described above for illustrative purposes, it should be recognized by those skilled in the relevant art that various equivalent modifications may be within the scope of the present technology. For example, although the steps are presented in the order given above, alternative embodiments may perform the steps in a different order. In addition, the various embodiments described herein may also be combined to provide additional embodiments.

[0262] It should be understood from the foregoing that the specific embodiments of the present technology have been described herein for illustrative purposes, but the well-known structures and functions are not shown or described in detail so as not to make the description of the embodiments of the present technology unclear unnecessarily. If any material incorporated herein by reference conflicts with the present disclosure, the present disclosure shall prevail. Where the context permits, singular or plural terms may also include plural or singular terms, respectively. In addition, unless the word "or" is clearly limited to mean excluding only a single item of other items in a list of two or more items, the use of "or" in this list should be interpreted as including any single item in the list of (a), all items in the list of (b) or any combination of items in the list of (c). In addition, as used herein, the phrase "and / or" as in "A and / or B" relates to only A, only B and both A and B. In addition, the use of the terms "including", "comprising", "having" and "having" throughout the text means including at least the stated features to not exclude any greater number of identical features and / or other features of additional types. In addition, as used herein, the phrases "based on", "depending on", "due to", and "in response to" should not be interpreted as referring to a closed set of conditions. For example, without departing from the scope of the present disclosure, the exemplary steps described as "based on condition A" may be based on both condition A and condition B. In other words, as used herein, the phrase "based on" should be interpreted in the same manner as the phrase "based at least in part on" or the phrase "based at least in part on". Moreover, the terms "connected" and "coupled" are used interchangeably herein and refer to both direct and indirect connections or couplings. For example, where the context permits, element A "connected" or "coupled" to element B may refer to (i) A directly "connected" or directly "coupled" to B and / or (ii) A indirectly "connected" or indirectly "coupled" to B.

[0263] It should also be understood from the foregoing that various modifications can be made without departing from the present disclosure or the present technology. For example, it should be understood by those of ordinary skill in the art that the various components of the present technology can be further divided into sub-components, or the various components and functions of the present technology can be combined and integrated. In addition, certain aspects of the present technology described in the context of a specific embodiment may also be combined or eliminated in other embodiments. In addition, although the advantages associated with certain embodiments of the present technology have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily need to exhibit such advantages to fall within the scope of the present technology. Therefore, the present disclosure and associated technology may encompass other embodiments that are not explicitly shown or described herein.

Claims

1. An imaging system, comprising: an event-driven sensing array comprising one or more event vision sensor pixels, wherein each event vision sensor pixel of the one or more event vision sensor pixels is configured to capture event data corresponding to contrast information of light incident on the event vision sensor pixel; a pixel array comprising a plurality of CMOS image sensor pixels arranged in one or more CMOS image sensor pixel rows, wherein each CMOS image sensor pixel of the plurality of CMOS image sensor pixels is configured to capture CMOS image sensor data corresponding to an intensity of light incident on the CMOS image sensor pixel; a deblurring circuit configured to deblur the CMOS image sensor data captured by the plurality of CMOS image sensor pixels using a first portion of the event data captured by the one or more event vision sensor pixels; and A system processor is configured to interpolate a video frame using the deblurred CMOS image sensor data and all or a subset of the event data.

2. The imaging system of claim 1 , wherein the deblurring circuit is configured to (a) accumulate the event data, (b) deblur the complementary metal oxide semiconductor image sensor data using the accumulated event data corresponding to the first portion of the event data, and (c) output the accumulated event data corresponding to a second portion of the event data to the system processor.

3. The imaging system of claim 2, wherein the accumulated event data corresponding to the second portion of the event data comprises an exponential of the product of (a) a running sum of the second portion of the event data and (b) a contrast threshold.

4. The imaging system of claim 2 , wherein each of the plurality of CMOS image sensor pixels is configured to capture the CMOS image sensor data during an integration period, wherein the deblurring circuit is configured to accumulate the second portion of the event data during the integration period, and wherein the deblurring circuit is further configured to output the accumulated event data corresponding to the second portion of the event data to the system processor at an interpolation timing point occurring during the integration period.

5. The imaging system of claim 2 , wherein each of the plurality of CMOS image sensor pixels is configured to capture the CMOS image sensor data during an integration period, wherein the deblurring circuit is configured to accumulate the second portion of the event data during an accumulation period prior to the integration period, and wherein the deblurring circuit is further configured to output the accumulated event data corresponding to the second portion of the event data to the system processor before the integration period begins.

6. The imaging system of claim 2 , wherein each of the plurality of CMOS image sensor pixels is configured to capture the CMOS image sensor data during an integration period, wherein the deblurring circuit is configured to accumulate the second portion of the event data during a period after an end of the integration period, and wherein the deblurring circuit is configured to output the accumulated event data corresponding to the second portion of the event data to the system processor at an interpolation timing point occurring after an end of the integration period.

7. The imaging system of claim 1 , wherein each CMOS image sensor pixel of the plurality of CMOS image sensor pixels is configured to capture the CMOS image sensor data during an integration period, and wherein the deblurring circuit is further configured to correct rolling shutter distortion of the CMOS image sensor data using a second portion of the event data accumulated by the deblurring circuit before the start of the integration period.

8. The imaging system of claim 7, wherein all of the event data or the subset thereof comprises the second portion of the event data, and wherein the system processor is configured to further use the rolling shutter distortion corrected complementary metal oxide semiconductor image sensor data to interpolate the video frame.

9. The imaging system of claim 1, further comprising: (i) a first frame buffer configured to store key frames of the complementary metal oxide semiconductor image sensor data; and (ii) a second frame buffer configured to store a frame of accumulated event data corresponding to the key frame of CMOS image sensor data.

10. The imaging system of claim 9, further comprising: (i) a first switch; and (ii) a trigger operable to control the first switch to selectively enable the first frame buffer to receive the key frame of the complementary metal oxide semiconductor image sensor data.

11. The imaging system of claim 10, further comprising a second switch, wherein the trigger is further operable to control the second switch to selectively enable the second frame buffer to receive the frame of accumulated event data.

12. The imaging system of claim 10, further comprising: A pre-processor block is configured to (a) analyze the accumulated event data output from the second frame buffer to identify motion in an external scene and (b) control the trigger to selectively enable the first switch based at least in part on the identification of the motion in the external scene.

13. The imaging system of claim 12, wherein the pre-processor block is further configured to decode and / or denoise the accumulated event data.

14. The imaging system of claim 9, further comprising a third frame buffer configured to store the interpolated video frames output from the system processor.

15. The imaging system of claim 14, wherein the third frame buffer comprises a ping-pong buffer.

16. The imaging system of claim 1, further comprising a hybrid image sensor including the event-driven sensing array and the pixel array, wherein the system processor is external to the hybrid image sensor.

17. The imaging system of claim 16, wherein the hybrid image sensor comprises the deblurring circuit.

18. A method of operating an imaging system, the method comprising: accumulating, during an integration period of a complementary metal oxide semiconductor image sensor pixel of the imaging system, first event data captured using one or more event vision sensor pixels of the imaging system corresponding to the complementary metal oxide semiconductor image sensor pixel; outputting at least a portion of the accumulated first event data to an event vision sensor frame buffer; deblurring CMOS image sensor data captured by the CMOS image sensor pixels during the integration period using the accumulated first event data; and A video frame is interpolated based at least in part on the deblurred CMOS image sensor data and the portion of the accumulated first event data.

19. The method of claim 18, wherein accumulating the first event data comprises accumulating the first event data during a first accumulation period, and wherein the method further comprises aligning the first accumulation period with the integration period such that the first accumulation period and the integration period begin simultaneously.

20. The method of claim 19, wherein aligning the first accumulation period with the integration period comprises resetting the one or more event vision sensor pixels and the complementary metal oxide semiconductor image sensor pixels simultaneously and prior to (i) accumulating the first event data.

21. The method of claim 18, wherein: The method further comprises accumulating second event data captured using the one or more event vision sensor pixels during a period after the end of the integration period, and outputting at least a portion of the accumulated second event data at an interpolation timing point occurring after the end of the integration period, Wherein interpolating the video frame includes interpolating the video frame based at least in part on the portion of the accumulated second event data.

22. The method of claim 21, further comprising resetting the one or more event vision sensor pixels at the end of the integration period and prior to the period.

23. The method of claim 18, wherein outputting at least the portion of the accumulated first event data to the event visual sensor frame buffer comprises outputting at least the portion of the accumulated first event data to the event visual sensor frame buffer at an interpolation timing point occurring during the integration period.

24. The method of claim 18, wherein: The method further comprises accumulating second event data captured using the one or more event vision sensor pixels during a period prior to the start of the integration period, and outputting at least a portion of the accumulated second event data to the event vision sensor frame buffer at the end of the period and before the beginning of the integration period, Wherein interpolating the video frame includes interpolating the video frame based at least in part on the portion of the accumulated second event data.

25. The method of claim 24, further comprising resetting the one or more event vision sensor pixels and the event vision sensor frame buffer prior to the start of the period.

26. The method of claim 18, further comprising decoding and / or denoising at least the portion of the accumulated first event data stored to the event vision sensor frame buffer.

27. The method of claim 18, further comprising: identifying motion in an external scene based on analysis of at least said portion of said accumulated first event data stored to said event vision sensor frame buffer; and A trigger of the imaging system is activated such that at least a portion of the CMOS image sensor data captured by the CMOS image sensor pixels during the integration period is stored to a CMOS image sensor frame buffer of the imaging system.

Images