Hybrid image sensor with on-chip image deblurring and rolling shutter distortion correction

Abstract

Hybrid image sensors (and associated systems, devices, and methods) with on-chip image deblurring and rolling shutter distortion correction are disclosed herein. In one embodiment, an image sensor includes: (a) an event driven sensing array including one or more event vision sensor (EVS) pixels, each EVS pixel configured to capture event data corresponding to contrast information of light incident on the EVS pixel; (b) a pixel array comprising a plurality of CMOS image sensor (CIS) pixels arranged in one or more CMOS image sensor (CIS) pixel rows, each CIS pixel configured to capture CIS data corresponding to an intensity of light incident on the CIS pixel; and (c) a rolling shutter distortion correction circuit configured to correct rolling shutter distortion of the CIS data using the event data.

Description

[0001] CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 597,638, filed on November 9, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to image sensors. For example, several embodiments of the present technology relate to hybrid image sensors that utilize event-based vision sensor (EVS) data to deblur and correct rolling shutter distortion of on-chip complementary metal oxide semiconductor (CMOS) image sensor (CIS) data before outputting the deblurred and rolling shutter corrected image frames (e.g., to downstream components of a corresponding imaging system). Background Art

[0004] Image sensors have become ubiquitous and are now widely used in digital cameras, cell phones, video cameras, as well as in medical, automotive and other applications. As image sensors are integrated into a wider range of electronic devices, it is desirable to enhance their functionality, performance metrics, and the like in as many ways as possible (e.g., resolution, power consumption, dynamic range, etc.) both through device architecture design and image acquisition processing.

[0005] A typical image sensor operates in response to image light from an external scene being incident on the image sensor. The image sensor includes an array of pixels having photosensitive elements (e.g., photodiodes) that absorb a portion of the incident image light and generate image charges after absorbing the image light. The image charge generated by the pixel light can be measured as an analog output image signal on a column bit line, which varies depending on the incident image light. In other words, the amount of image charge generated is proportional to the intensity of the image light, and the image light is read out as an analog image signal from the column bit line and converted into a digital value to provide information representing the external scene. Summary of the invention

[0006] On the one hand, the present disclosure provides an image sensor, comprising: an event-driven sensing array, which includes one or more event vision sensor (EVS) pixels, wherein each of the one or more EVS pixels is configured to capture event data corresponding to contrast information of light incident on the EVS pixel; a pixel array, which includes a plurality of CMOS image sensor (CIS) pixels arranged into one or more CIS pixel rows, wherein each of the plurality of CIS pixels is configured to capture CIS data corresponding to the intensity of light incident on the CIS pixel; a rolling shutter distortion correction circuit, which is configured to generate rolling shutter distortion-corrected image data by correcting rolling shutter distortion of the CIS data captured by the plurality of CIS pixels using the event data captured by the one or more EVS pixels; and a physical interface, which can be used to output the rolling shutter distortion-corrected image data from the image sensor.

[0007] On the other hand, the present disclosure provides an image sensor, comprising: an event-driven sensing array, which includes one or more event vision sensor (EVS) pixels, wherein each of the one or more EVS pixels is configured to capture event data corresponding to contrast information of light incident on the EVS pixel; a pixel array, which includes a plurality of CMOS image sensor (CIS) pixels, wherein each of the plurality of CIS pixels is configured to capture CIS data corresponding to the intensity of light incident on the CIS pixel; and a rolling shutter distortion correction circuit, which includes: a counter, which for each of the one or more EVS pixels can be used to calculate a running sum of events detected by the EVS pixel during a period extending from the beginning of an image frame to the beginning of an exposure period of one or more CIS pixels of the plurality of CIS pixels corresponding to the EVS pixel; and an integration buffer, which is configured to store the running sum corresponding to the EVS pixel for each of the one or more EVS pixels.

[0008] On the other hand, the present disclosure provides a method for operating an image sensor, the method comprising: during an exposure period corresponding to an image frame, using CIS pixels in one or more CMOS image sensor (CIS) pixel rows of a pixel array of the image sensor to capture CIS data, wherein the CIS data corresponds to the intensity of light incident on the CIS pixels during the exposure period; during a period extending between the beginning of the image frame and the beginning of the exposure period, using one or more event vision sensor pixels of an event-driven sensing array of the image sensor to capture event vision sensor (EVS) data, wherein the EVS data includes events detected by the one or more EVS pixels during the period, and wherein each event represents a temporal contrast of light incident on the one or more EVS pixels exceeding a threshold; and (i) correcting rolling shutter distortion of the CIS data based on the EVS data and (ii) using a rolling shutter distortion correction (RSDC) circuit of the image sensor within the image sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Non-limiting and non-exhaustive embodiments of the present technology are described below with reference to the following figures, wherein like or similar reference numerals are used to refer to all like or similar components unless otherwise specified.

[0010] Figure 1 is a partial schematic diagram of an EVS pixel configured according to various embodiments of the present technology.

[0011] Figure 2 are plots illustrating (a) exposure period, (b) sequence of event pulses formed into a time-continuous signal, (c) sum of the sequence of event pulses over time, and (d) integral of the sum of the sequence of event pulses over time.

[0012] Figure 3 is a partial schematic diagram illustrating an example of an imaging system with off-chip image deblurring.

[0013] Figure 4A is a partial schematic diagram of a stacked hybrid complementary metal oxide semiconductor (CMOS) image sensor (CIS) and event-based vision sensor (EVS) system configured in accordance with various embodiments of the present technology.

[0014] Figure 4B yes Figure 4A A partial schematic diagram of a specific example of a system.

[0015] Figure 4C is a partial schematic diagram of a 4x4 pixel cluster configured in accordance with various embodiments of the present technology.

[0016] Figure 5is a partial schematic block diagram of an image sensor configured in accordance with various embodiments of the present technology.

[0017] Figure 6 is a partial schematic diagram of a deblurring and rolling shutter distortion correction circuit configured in accordance with various embodiments of the present technology.

[0018] Figure 7 is a diagram illustrating a depiction of an adjacent image frame of a CIS pixel array in accordance with various embodiments of the present technology.

[0019] Figure 8 is a diagram illustrating a depiction of an adjacent image frame of a CIS pixel array according to various other embodiments of the present technology.

[0020] Fig. 9 is a partial schematic diagram of another deblurring and rolling shutter distortion correction circuit configured in accordance with various embodiments of the present technology.

[0021] Fig.10 is a flow chart illustrating a method of operating an image sensor according to various embodiments of the present technology.

[0022] Fig.11 and 12 is a timing diagram corresponding to a method of operating an image sensor and / or a corresponding imaging system according to various embodiments of the present technology.

[0023] Fig.13A is a partial schematic diagram illustrating an event-driven sensing array and a row-scan readout scheme configured in accordance with various embodiments of the present technology.

[0024] Fig. 13B is the use of various embodiments according to the present technology Fig.13A The line scan readout scheme is from Fig.13A Plotting of detected events from event-driven sensing array readout.

[0025] Fig.14 is a partial schematic diagram illustrating an imaging system configured in accordance with various embodiments of the present technology.

[0026] Those skilled in the art will appreciate that the elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the figures may be exaggerated relative to other elements to aid in understanding various aspects of the present technology. In addition, common but well-known elements or methods that are useful or required in commercially feasible embodiments are generally not depicted in the figures or described in detail below so as not to unnecessarily obscure the description of various aspects of the present technology. DETAILED DESCRIPTION

[0027] The present disclosure relates to hybrid image sensors with on-chip image capabilities (e.g., on-chip event-oriented deblurring capabilities with rolling shutter distortion correction). For example, several embodiments of the present technology described in detail below relate to image sensors that (a) synchronize CIS data captured by CIS pixels with corresponding EVS / event data captured by one or more EVS pixels, (b) accumulate event data within an event accumulation period aligned with the exposure period of the CIS pixels, and (c) use the accumulated event data to deblur the CIS data to produce deblurred image / video data that is corrected for rolling shutter distortion and can then be output from the image sensor. In the following description, specific details are stated to provide a thorough understanding of aspects of the present technology. However, those skilled in the relevant art should recognize that the systems, devices, and techniques described herein may be practiced without one or more of the specific details stated herein or with other methods, components, materials, etc.

[0028] Reference throughout this specification to an "example" or "embodiment" means that a particular feature, structure, or characteristic described in connection with the example or embodiment is included in at least one example or embodiment of the present technology. Thus, the use of the phrases "for example," "as an example," or "embodiment" herein does not necessarily all refer to the same example or embodiment and is not necessarily limited to the particular example or embodiment discussed. Furthermore, the features, structures, or characteristics of the present technology described herein may be combined in any suitable manner to provide additional examples or embodiments of the present technology.

[0029] For ease of description, spatially relative terms (e.g., "below," "below," "above," "below," "above," "up," "top," "bottom," "left," "right," "center," "middle," and the like) may be used herein to describe the relationship of one element or feature relative to one or more other elements or features, as illustrated in the figures. It should be understood that, in addition to the orientation depicted in the figures, spatially relative terms are intended to cover different orientations of the device or system in use or operation. For example, if the device or system illustrated in the figures is rotated, turned, or flipped around a horizontal axis, an element or feature described as being "below" or "below" or "below" one or more other elements or features may be oriented to be "above" one or more other elements or features. Therefore, the exemplary terms "below" or "below" are non-limiting and may cover both above and below orientations. The device or system may be additionally or alternatively oriented in a manner other than that illustrated in the figures (e.g., rotated 90 degrees around a vertical axis or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly. In addition, it will also be understood that when an element is referred to as being “between” two other elements, it can be the only element between the other two elements, or one or more intervening elements may also be present.

[0030] Throughout this specification, several terms of the art are used. These terms have their ordinary meaning in the art, unless specifically defined herein or the context of their use clearly indicates otherwise. It should be noted that element names and symbols (e.g., Si vs. silicon) are used interchangeably throughout this document; however, both have the same meaning.

[0031] A. Overview

[0032] The active pixel sensor employs a pixel array for capturing an intensity image / video of an external scene. More specifically, the pixels are used to obtain CIS information (e.g., intensity information) corresponding to light from the external scene incident on the pixels. The CIS information obtained during an integration period is read out at the end of the integration period and used to generate a corresponding intensity image of the external scene.

[0033] The pixels of an active pixel sensor typically have a globally defined integration time. Thus, the pixels in an array of an active pixel sensor typically have the same integration time, and each pixel in the array is typically converted into a digital signal regardless of its content (e.g., whether there have been changes in the external scene captured by the pixel since the pixel was last read out). Thus, relatively large amounts of memory and power are required to operate an active pixel sensor at a high frame rate. Therefore, due in part to memory and power constraints, it is difficult to use active pixel sensors alone to obtain intensity images / videos of an external scene at ultra-high frame rates.

[0034] Furthermore, when motion or other changes occur in the external scene during the integration period, motion artifacts may be observed as blur in the resulting intensity image of the external scene. Blurring may be particularly prominent in low light conditions where longer exposure times are used. Thus, active pixel image sensors are not inherently good at obtaining sharp intensity images / videos of highly dynamic scenes.

[0035] In contrast, an event vision sensor (e.g., an event-driven sensor or a dynamic vision sensor) employs EVS pixels that can be used to obtain non-CIS information (e.g., contrast information, intensity changes, event data) corresponding to light from an external scene incident on those EVS pixels. The event vision sensor reads out the EVS pixels and / or converts the corresponding pixel signals into digital signals only when the EVS pixels detect a change in the external scene (e.g., an event). In other words, the EVS pixels of the event vision sensor that do not detect a change in the external scene are not read out, and / or the pixel signals corresponding to such EVS pixels are not converted into digital signals (thereby saving power). Therefore, each EVS pixel of the event vision sensor can be independent of the other EVS pixels of the event vision sensor, and only the EVS pixels that detect a change in the external scene need to be read out and / or have their corresponding pixel signals converted into digital signals. Therefore, unlike active pixel sensors with synchronized integration times, event vision sensors are not affected by limited dynamic range and can accurately capture high-speed motion. Therefore, event vision sensors are generally more robust than active pixel sensors in weak lighting conditions and / or highly dynamic scenes because they are not affected by under / overexposure or motion blur associated with synchronized shutters. In other words, event vision sensors can be used to provide ultra-high frame rates and accurately capture high-speed motion.

[0036] A hybrid image sensor employs a pixel array comprising a combination of: (i) active (CIS) pixels, which can be used to obtain CIS information corresponding to light from an external scene; and (ii) EVS pixels, which can be used to obtain non-CIS information corresponding to light from an external scene. Such a hybrid image sensor is thus capable of simultaneously capturing (a) an intensity image / video of an external scene and (b) events occurring within the external scene. The event data captured by the EVS pixels can be used to address / mitigate (i) the low frame rate intensity image problem discussed above and (ii) the blurring effects inherent in intensity images captured using CIS pixels in the presence of motion. For example, using an event-based double integral (EDI) model, a high frame rate intensity image / video of an external scene can be reconstructed from a single (e.g., blurred) intensity image and its event sequence.

[0037] For clarity and understanding, a description of the EDI model is provided here. The instantaneous intensity (or irradiance / flux) at a pixel (x, y) at any time t is related to the rate at which photons arrive at that pixel (x, y) and is called the latent image L xy (t). Latent image L xy (t) is not directly output from the hybrid image sensor corresponding to the pixel (x, y). Instead, the hybrid image sensor outputs: (i) an intensity image (e.g., a blurred image) representing multiple latent images (including the latent image L) captured by one or more CIS pixels during the exposure period xy(t)); and (ii) a sequence of events that record intensity changes between latent images captured during the exposure period. A description of how events can be detected is provided below, followed by how the detected events can be used to obtain a latent image L of a pixel. xy Because each pixel of the image sensor can be processed individually, the subscripts x and y are omitted from the following equations and variables for readability. However, it should be understood that the latent image L of a pixel in the pixel array at a given time xy (t) represents the latent image L of the entire pixel array at the given time. F (t). Therefore, the latent image L of the entire pixel array at any time F (t) can be determined by determining the latent image L of each pixel in the array at the given time xy (t) to obtain.

[0038] Figure 1 1 is a partial schematic diagram of an EVS pixel 100 configured in accordance with various embodiments of the present technology. The EVS pixel 100 is also referred to herein as an “event sensing front end circuit.” As shown, the EVS pixel 100 includes a photosensor 101, a logarithmic amplifier 102, a buffer 103, a difference detector 104, and an up / down comparator 105.

[0039] The photosensor 101 is configured to photogenerate an image charge (photocurrent) in response to incident light. The photocurrent photogenerated by the photosensor 101 at time t is proportional to the latent image L(t) at time t (eg, the irradiance of the incident light), as indicated by the following equation 1:

[0040] Equation 1: I photo (t)∝L(t)

[0041] As discussed above, the latent image L(t) represents the instantaneous intensity at the EVS pixel 100 at time t related to the rate at which photons arrive at the EVS pixel 100 .

[0042] The photocurrent generated by the photosensor 101 is fed to the logarithmic amplifier 102. The logarithmic amplifier 102 in turn converts (a) the photocurrent which is linearly proportional to the latent image L(t) into (b) a voltage V which logarithmically depends on the latent image L(t). FE , as indicated by the following equation 2:

[0043] Equation 2: V FE ∝ln[I photo (t)])∝ln[L(t)]

[0044] Temporal contrast (also referred to herein as “linear contrast”) is defined as the change in light contrast at an EVS pixel 100 (eg, at photosensor 101) relative to a reference time t0 and is provided by the following Equation 3:

[0045] Equation 3:

[0046] The difference detector 104 of the EVS pixel 100 is used to monitor the temporal contrast of light incident on the photosensor 101. More specifically, when reset, the difference detector 104 samples the voltage V at the reference time t0. FE And then generate the output V shown by the following equation 4 O :

[0047] Equation 4:

[0048] The output V of the difference detector 104 O Tracking voltage V FE The voltage V at the reference time t0 over time FE As shown in the following equation 5, as the voltage V FE As time goes by, the output V of the difference detector 104 O The corresponding change in is proportional to the logarithm of the temporal contrast:

[0049] Equation 5:

[0050] The output V of the difference detector 104 O is fed to the up / down comparator 105. The up / down comparator 105 in turn compares the output V O The corresponding threshold voltage V is given by Equation 6 below +TH and V -TH , where C log-TH is a contrast threshold parameter that determines whether an event should be recorded. As shown by the following equation 7, when the output V of the difference detector 104 O Exceeding the threshold voltage V +TH When the output ΔV of the difference detector 104 is ΔV, the upward comparator detects an event. As shown by the following equation 8, when the output ΔV of the difference detector 104 is ΔV, the upward comparator detects an event. O Less than the threshold voltage V -TH , the down comparator detects an event.

[0051] Equation 6: V ±TH = ±β·C log-TH

[0052] Equation 7:

[0053] Equation 8:

[0054] Using the log ratio rule, Equations 7 and 8 provide Equations 9 and 10, respectively, that specify when the EVS pixel 100 detects an event:

[0055] Equation 9: ln[L(t i )]≥ln[L(t i-1 )]+C log-TH

[0056] Equation 10: ln[L(t i )]≤ln[L(t i-1 )]-C log-TH

[0057] In the above equation, time t i corresponds to the time when each event is detected, and time t i-1 Indicates the timestamp of the previous event. When an event is triggered in an EVS pixel, L(t i-1 ) is updated to a new intensity level (e.g., by resetting the difference detector 104 so that the difference detector 104 newly samples the voltage V FE ). Therefore, at time t i The detected event indicates that the i-1 The previous event detected occurred when the contrast threshold parameter C log-TH In other words, each detected event indicates a latent image (current latent image L(t i ) and the previous latent image L(t i-1 )) between the intensity changes. Therefore, the above equations 9 and 10 provide where c is equal to C log-TH And p i is the following equation 11 for the event polarity:

[0058] Equation 11: ln[L(t i )]=ln[L(t i-1 )]+c·p i

[0059] Every time t i The event polarity p i Given by the following equation 12. The polarity p of +1 i The irradiance of light incident on the photosensor 101 increases, and the polarity p of -1 i This indicates that the irradiance of light incident on the photosensor 101 is reduced.

[0060] Equation 12:

[0061] The events detected at each time ti can be modeled using a unit pulse (Dirac function δ) multiplied by the corresponding polarity pi. The detected events can be defined as continuous time functions. For example, Figure 2 Various plots 210, 212, 214, 216, and 218 are illustrated. Plot 210 illustrates the exposure period of a row of CIS pixels in a pixel array for one image frame. A rolling shutter is used such that the exposure period of the rows of pixels in the pixel array is staggered with the readout. The exposure period of the top two rows of the CIS pixel array extends between times t0 and t 0+T and the exposure period of the bottom two rows of the CIS pixel array extends to time t s With t s+T Graph 212 illustrates the sequence of events detected by the EVS pixels corresponding to the bottom two rows of the CIS pixel array. More specifically, at time t0 and time t s+T The sequence of events detected between is plotted as a time-continuous signal e(t) comprising a sequence of unit pulses. The time-continuous signal e(t) is modeled by the following equation 13:

[0062] Equation 13: e(t) = p i ·δ(tt i )

[0063] Because each event detected by an EVS pixel indicates a change between latent images captured at different times, the proportional change in intensity at the pixels of the bottom two rows of the CIS pixel array over the exposure period of those rows of pixels can be represented by the intensity of the corresponding EVS pixel at time t s The sum (or combination) of events detected between t and time t provides, as shown in Equation 14 below:

[0064] Equation 14: Figure 2 The plot 214 illustrates a time-continuous signal E(t) representing the exposure period corresponding to the bottom two rows of the CIS pixel array at time t s With time t s+T Between Figure 2 The sum of events in the time-continuous signal e(t) of the plot 212 .

[0065] When ln[L(t i-1 )] is known, using the above equations 11 and 12, ln[L(t i )]. Thus, given a sequence of events specified by a time-continuous signal e(t) and assuming that c in Equation 11 above remains constant, it is possible (in the linear domain) to sThe latent image L(t) of the bottom two rows of CIS pixels of the array at any given time t is determined by incrementing the starting latent image L(s) at time t over all events up to time t, as shown by the following equation 15:

[0066] Equation 15:

[0067] As discussed above with reference to Equation 1, the photocurrent generated by photosensor 101 is proportional to the latent image L(t) (or irradiance) at time t. Therefore, the latent image L(t) is proportional to the irradiance at time t. s With time t s+T The integral over the exposure period between t corresponds to the “charge” and can be related to the frame information captured by the CIS pixels of the frame-based image sensor. Furthermore, as discussed above, the blurred intensity image captured by the frame-based image sensor can be viewed as a latent image sequence extending over time t s With time t s+T The integral of the exposure period between (during which events are accumulated) or the integral of the combination of multiple latent images captured by the frame-based image sensor within the exposure period. Therefore, using the above equation 15, the blurred frame B captured by the frame-based image sensor can be expressed by the following equation 16:

[0068] Equation 16: Figure 2 Description of the drawing 216 Figure 2 The time continuous signal E(t) is plotted 214 from time t s To time t s+T The integral within the time.

[0069] The above equation 16 is called the EDI model and provides the blur frame B captured by the frame-based image sensor and the time t s (corresponding to the start of the frame / exposure period of the CIS pixel). This relationship can be rearranged to find the latent image L(s), as shown by the following Equation 17:

[0070] Equation 17:

[0071] The latent image L(s) in Equation 17 above is interpreted as a deblurred frame based on (a) a blurry frame B captured by a frame-based image sensor and (b) events detected by the EVS pixels across corresponding exposure periods. In other words, because the events detected by the EVS pixels during the exposure period indicate changes between the latent images captured by one or more active (CIS) pixels during the exposure period, the detected events can be used to perform event-directed deblurring.

[0072] As discussed above, a rolling shutter can be used to capture and read out CIS information. For example, Figure 2 The exposure period of the top two rows of the CIS pixel array shown in the drawing 210 starts at time t0, and time t0 starts at time t1. s Before. Therefore, at least some of the CIS information captured by the top two rows of CIS pixels is captured at a different time than at least some of the CIS information captured by the bottom two rows of CIS pixels. Thus, when (a) the CIS information captured by the top two rows and the bottom two rows of CIS pixels of the CIS pixel array are read out and (b) the latent image L(0) of the top two rows of CIS pixels and the latent image L(s) of the bottom two rows of CIS pixels are determined using the above equation 17, the latent image L obtained from the CIS information captured by the CIS pixels of the entire array is F (s) may include rolling shutter distortion (e.g., due to the presence of a variable number of pixels in the imaged scene between time t0 and time t s movement between them).

[0073] Events detected by the EVS pixels corresponding to the bottom two rows of the pixel array can be used to correct this distortion. More specifically, at time t0 and time t s Events detected by EVS pixels between Figure 2 218) can be used to provide rolling shutter correction for the CIS information captured by the bottom two rows of CIS pixels. Specifically, at time t0 (corresponding to the start of the exposure period for the top two rows of pixels of the CIS pixel array) and time t s The proportional change in intensity during the period between time t0 and time t s The sum (or combination) of events detected by corresponding EVS pixels between is provided as shown in Equation 18 below:

[0074] Equation 18:

[0075] Figure 2 The plot 218 illustrates the time continuous signal E'(t), which represents the time between time t0 and time t s Between Figure 2 The sum of events in the time-continuous signal e(t) of the plot 212 .

[0076] As discussed above, when ln[L(t i-1 )] is known, equations 11 and 12 can be used to determine ln[L(t i )]. Therefore, given that at time t0 and time t sThe sequence of events specified by the time continuous signal e(t) between time and time t0 and assuming that c in Equation 11 above remains constant can be expressed (in the linear domain) by (i) s The CIS pixel at time t is determined by integrating / incrementing over all events detected by the corresponding EVS pixel and (ii) multiplying the accumulated events by the starting latent image L(0). s The latent image L(s) at time , as shown by the following equation 19:

[0077] Equation 19:

[0078]

[0079] Therefore, in view of the above equations 17 and 19, the latent image L(0) of the CIS pixel can be solved and obtained using the following equation 20:

[0080] Equation 20:

[0081] L(0) in Equation 20 above corresponds to the deblurred rolling shutter distortion corrected latent frame. Therefore, the latent frame L for the entire pixel array is F (0) can be obtained using the latent frame L(0) of the individual CIS pixels in the array.

[0082] Many event-directed deblurring solutions use an active image sensor to capture CIS data and a separate event vision sensor to capture EVS data. However, such a dual sensor configuration has several disadvantages, such as parallax introduced due to the uncombined sensors, complexity of spatial and temporal synchronization of the CIS data and the EVS data, and increased cost (e.g., due to the need for two pairs of lenses, packaging, etc.). Although these disadvantages can be overcome by using a hybrid image sensor, all existing event-directed deblurring solutions known to the inventors of the present disclosure output the CIS data and the EVS data to an application processor external to the hybrid image sensor so that the application processor performs off-chip event-directed deblurring. Such off-chip event-directed deblurring solutions and rolling shutter correction solutions have several additional disadvantages, many of which are discussed below with reference to FIG. Figure 3 Discussion.

[0083] Figure 33 is a partial schematic diagram illustrating an example of an imaging system 320 that performs off-chip event-directed deblurring. As shown, CIS data 321 and EVS data 322 are output from (a) a hybrid image sensor (not shown) of imaging system 320 or (b) an active pixel sensor (not shown) and a separate event vision sensor (not shown) of imaging system 320 to an application processor 323 of imaging system 320. Application processor 323 is configured to (a) perform event-directed deblurring of CIS data 321 using EVS data 322 and (b) output a deblurred image frame to image signal processor 352.

[0084] One disadvantage of an off-chip event-oriented deblurring solution is that a relatively large amount of memory is required. For example, to perform event-oriented deblurring, the application processor 323 uses a first buffer 324, a second buffer 325, a third buffer 326, and a fourth buffer 327. More specifically, the first buffer 324 is used to store two frames of CIS image data. One of the frames is used to synchronize CIS data 321 with EVS data 322 at the CIS / EVS synchronization block 328 of the application processor 323, and the other of the frames is deblurred by the application processor 323 using EVS data 322 at the deblurring block 329 of the application processor 323. In one example, for a 12-megapixel image sensor, the size of the first buffer 324 may be approximately 44MB. The second buffer 325 is used to store the EVS data 322 before decoding, and the third buffer 326 is used to store the EVS data 322 after decoding. In one example, the second buffer 325 may be configured to store the EVS data 322 for approximately 50ms and thus may be approximately 90MB in size. Thus, depending on the decoding, the size of the third buffer 326 may be approximately 100 to 400 MB. The fourth buffer 327 may be configured to store deblurred image frames and thus may be approximately 22 MB in size for a 12 megapixel image sensor. Thus, continuing with the above example, the application processor 323 may require approximately 550 MB of memory to perform event-directed deblurring of the CIS data 321 using the EVS data 322.

[0085] Other disadvantages of off-chip event-based deblurring solutions include: (i) relatively large latency and (ii) inability to support real-time video. Figure 3 As shown in FIG. 3 , CIS data 321 and EVS data 322 are output to an external application processor 323 (causing a 1-frame delay). In addition, the application processor 323 introduces additional delay when performing event-directed deblurring calculations. For example, it may take up to 1 second for the application processor 323 to process a 12-megapixel image. Figure 3 The described off-chip event-directed deblurring solution is only able to process still images, which means that this solution cannot support real-time video.

[0086] Other disadvantages of off-chip event-directed deblurring solutions include (i) requiring relatively high input / output (IO) bandwidth between the image sensor and the external application processor and (ii) consuming relatively large amounts of power. Figure 3 , both CIS data 321 and EVS data 322 are output from the image sensor to an application processor 323, which requires a relatively high IO throughput (e.g., about 20 Gbps for a 12 megapixel, 30fps image sensor). This high IO throughput and long data processing time (e.g., up to 1 second, as discussed above) results in a relatively large amount of power being consumed.

[0087] Another disadvantage of off-chip event-directed deblurring solutions is the complexity of the interface between the image sensor and downstream components of the imaging system. For example, instead of outputting a deblurred image frame, Figure 3 The image sensor (not shown) of the imaging system 320 outputs raw CIS data 321 and raw EVS data 322, which requires a relatively complex application processor 323 and an interface corresponding to the image sensor.

[0088] To address these issues, several embodiments of the present application described herein generally relate to hybrid image sensors with on-chip image deblurring and rolling shutter distortion correction capabilities. For example, several embodiments of the present technology described in detail below relate to image sensors with the following on-chip capabilities: (a) synchronization of CIS data captured using active (CIS) pixels and EVS data captured using EVS pixels; (b) image deblurring; and / or (c) rolling shutter distortion correction. In some embodiments, on-chip image deblurring may include on-chip event-directed deblurring. In these and other embodiments, on-chip rolling shutter distortion correction may include on-chip event-directed rolling shutter distortion correction.

[0089] The present technology is expected to provide several advantages. For example, compared to the off-chip event-oriented deblurring solution discussed above, the present technology is expected to reduce or minimize: (a) the amount of memory required to perform image deblurring and rolling shutter distortion correction; (b) the latency associated with performing image deblurring and rolling shutter distortion correction; (c) the required IO bandwidth / throughput; and / or (d) the amount of power required to perform image deblurring and rolling shutter distortion correction. Thus, in addition to still image processing, the present technology is expected to support real-time video. In addition, because image deblurring and rolling shutter distortion correction are performed on-chip (e.g., completely or partially within the image sensor and / or without the need to first output or need to first output raw CIS data and / or raw EVS data from the image sensor), image sensors configured according to various embodiments of the present technology are able to output deblurred rolling shutter distortion corrected image frames (e.g., in addition to or in place of raw CIS data and / or raw EVS data), which means that the interface between such image sensors and downstream components of the corresponding imaging system can be simplified compared to the off-chip event-oriented deblurring solution discussed above.

[0090] B. Hybrid image sensor with on-chip image deblurring and rolling shutter correction and associated systems, devices and Selected embodiments of the method

[0091] Figure 4A 4 is a partial schematic diagram of a stacked complementary metal oxide semiconductor (CMOS) image sensor (CIS) having an event-based vision sensor (EVS) system 430 ("stacked system 430" or "image sensor 430") configured in accordance with various embodiments of the present technology. As shown, the stacked system 430 includes a first die 432, a second die 434, and a third die 436 stacked and coupled together in a stacked chip scheme. In some embodiments, the first die 432, the second die 434, and the third die 436 are semiconductor dies including suitable semiconductor materials, such as silicon. In the illustrated embodiment, the first die 432 (also referred to herein as the "top die") includes a pixel array 438. The third die 436 (also referred to herein as the "bottom die") includes an image readout circuitry 446 (also referred to herein as the "image readout mixed signal circuitry"). The image readout circuitry 446 can be coupled to the pixel array 438 of the first die 432 through a column level connection 440 for normal image readout. In some embodiments, column level connections 440 for normal image readout are implemented from column bit lines of pixel array 438 with through silicon vias (TSVs) extending between first die 432 and third die 436 and routed through second die 434 .

[0092] In some embodiments, the pixel array 438 is a two-dimensional (2D) array including a plurality of pixel units (also referred to as "pixels"), each of which includes at least one photosensor (e.g., at least one photodiode) exposed to incident light. As shown in the illustrated embodiment, the pixels are arranged in rows and columns. Some of the pixels may be configured as CMOS image sensor (CIS) pixels, which are configured to acquire image data of a person, location, object, etc., which may then be used to reproduce images and / or videos of the person, location, object, etc. For example, each CIS pixel is configured to photogenerate an image charge in response to incident light. After each CIS pixel acquires its image charge, the corresponding analog image charge data may be read out by the image readout circuitry 446 in the third die 436 through the column bit lines. In some embodiments, the image charge from each row of the pixel array 438 may be read out in parallel by the image readout circuitry 446 through the column bit lines. As discussed in more detail below, other of the pixels of the pixel array 438 may be configured as event vision sensor (EVS) pixels.

[0093] The image readout circuitry 446 in the third die 436 may include amplifiers for normal image readout and processing, analog / digital converter (ADC) circuitry, associated analog support circuitry, associated digital support circuitry, and the like. In some embodiments, the image readout circuitry 446 may also include event-driven readout circuitry, as will be described in more detail below. In operation, photogenerated analog image charge signals are read out from the pixel cells of the pixel array 438, amplified in the image readout circuitry 446 and converted into digital values. In some embodiments, the image readout circuitry 446 may read out a row of image data at a time. In other examples, the image readout circuitry 446 may read out the image data using various other techniques (not illustrated) such as serial readout or full parallel readout of all pixels simultaneously. The image data may be stored or even manipulated by applying post-image effects such as cropping, rotation, red-eye removal, adjusting brightness, adjusting contrast, and the like.

[0094] In the illustrated embodiment, the second die 434 (also referred to herein as the "middle die") includes an event-driven sensing array 442 coupled to at least some of the pixels (e.g., EVS pixels) of the pixel array 438 in the first die 432. In some embodiments, the event-driven sensing array 442 is coupled to the pixels of the pixel array 438 by hybrid bonding between the first die 432 and the second die 434. The event-driven sensing array 442 may include an array of event-driven circuits. In some embodiments, each of the event-driven circuits in the event-driven sensing array 442 is coupled to at least one of a plurality of pixels of the pixel array 438 by hybrid bonding between the first die 432 and the second die 434 to asynchronously detect an event occurring in light incident on the pixel array 438 in accordance with the teachings of the present disclosure.

[0095] In some embodiments, the corresponding event detection signal is generated by an event-driven circuit (e.g., similar to Figure 1 The event detection signal may be received and processed by the event-driven peripheral circuit system 444. In some embodiments, the event-driven peripheral circuit system 444 is arranged in the second die 434 around the periphery of the event-driven sensing array 442, such as Figure 4A Displayed in. Figure 4A The embodiment illustrated in FIG. 4 also illustrates column level connections 440 for normal image readout being routed through the second die 434 between the first die 432 and the third die 436 .

[0096] Figure 4B yes Figure 4A A partial schematic diagram of a specific example of a stacked system 430. Figure 4B As shown in FIG. 4 , the stacked system 430 includes a pixel array 438 ( Figure 4B 438 is shown in the figure), the event-driven circuit 400 of the event-driven sensing array 442 on the second die 434, and the image readout circuit system 446 on the third die 436. The image readout circuit system 446 includes an analog-to-digital converter 451 ("ADC 451"), an image signal processor 452, a scanning readout circuit system 453, an event signal processor 454, a synchronous communication interface 455 (e.g., a mobile industry processor interface (MIPI) transmitter and / or receiver), and various auxiliary circuits 456. As discussed in more detail below, the image readout circuit system 446 may also include a deblurring circuit (e.g., for performing event-directed deblurring and / or rolling shutter distortion correction of CIS data).

[0097] Figure 4BThe portion of the pixel array 438 shown in 438 corresponds to a 4x4 cluster of pixels in the pixel array 438. This cluster may be repeated across the pixel array 438. In the illustrated embodiment, fifteen (15) pixels in the cluster are configured as active (CIS) pixels 435 to capture CIS information (e.g., intensity information) corresponding to light incident on the photosensors of these pixels. Additionally, one pixel in the cluster is configured as an EVS pixel 437 to capture non-CIS information (e.g., contrast information, event data) corresponding to light incident on the photosensor of the EVS pixel 437. Figure 4C illustrate Figure 4B 4x4 pixel clusters, where CIS pixels 435 are arranged in a Bayer pattern to capture CIS (frame) information corresponding to light incident on the cluster, and EVS pixels 437 are arranged to detect EVS (asynchronous event) information corresponding to light incident on the cluster. Of course, in other embodiments of the present technology, CIS pixels 435 may be arranged in another pattern besides a Bayer pattern.

[0098] Reference again Figure 4B , independently read out the clustered CIS pixels 435 and the clustered EVS pixels 437. More specifically, the CIS information captured by the CIS pixels 435 is read out to the ADC 451 on the third die 436 through the second die 434 using the corresponding row / column control circuit system (not shown). The non-CIS information captured by the EVS pixels 437 is read out to the event-driven circuit 400 on the second die 434 using the corresponding row / column control circuit system (not shown), and the events detected by the event-driven circuit 400 are read out by the scanning readout circuit system 453 on the third die 436. The CIS information captured by the CIS pixels 435 of the pixel array 438 is frame-based and can be read out from the CIS pixels 435 row by row at the end of the exposure period. In contrast, the non-CIS information captured by the EVS pixels 437 is used by the event-driven circuit 400 to asynchronously detect / trigger events, and the events can be read out according to the row scanning readout scheme or the column scanning readout scheme. The row scan readout scheme is discussed in more detail below.

[0099] In some embodiments, the row / column control circuitry corresponding to the CIS pixel 435 may be allocated on the same or different die as the die on which the ADC 451 is allocated (e.g., the third die 436). In these and other embodiments, the row / column control circuitry corresponding to the EVS pixel 437 may be allocated on the same or different die as the die on which the scanning readout circuitry 453 is allocated (e.g., the third die 436). In these and other embodiments, the ADC 451 and / or the row / column control circuitry corresponding to the CIS pixel 435 may be allocated on the same or different die as the die on which the scanning readout circuitry 453 and / or the row / column control circuitry corresponding to the EVS pixel 437 is allocated (e.g., the third die 436).

[0100] In the illustrated embodiment, EVS pixels 437 are dedicated to capturing non-CIS (EVS) information, while CIS pixels 435 are dedicated to capturing CIS information. In other embodiments, one or more of EVS pixels 437 and / or CIS pixels 435 may be switched between being configured to capture CIS information and non-CIS information. This may enable stacked system 430 to operate in the following modes: a CIS-only mode, in which all pixels 435 and pixels 437 are used to capture CIS information; an EVS-only mode, in which all pixels 435 and pixels 437 are used to capture non-CIS (EVS) information; and / or a mixed CIS and EVS mode, in which a first subset of pixels 435, 437 are used to capture CIS information and a second subset of pixels 435, 437 are used to capture non-CIS (EVS) information.

[0101] In some embodiments, the event driven circuit 400 on the second die 434 has the same die size as the 4x4 pixel cluster on the first die 432. In other embodiments, the event driven circuit 400 may have a different die size than the 4x4 pixel cluster. Additionally or alternatively, although the ratio of CIS pixels to EVS pixels in the 4x4 pixel cluster is 15:1, other ratios of CIS pixels to EVS pixels (e.g., 14:2, 12:4, 8:8, 4:12, 2:14, 15:1) are possible and within the scope of the present technology. Furthermore, although Figure 4B The EVS pixels 457 correspond to 4x4 pixel clusters, but other arrangements (e.g., EVS pixels corresponding to 1X1 pixel clusters, 4X2 pixel clusters, etc.) are possible and within the scope of the present technology. Figure 4BIn one embodiment, a row of EVS pixels (e.g., a row including EVS pixels 457) corresponds to four rows of CIS pixels, but other arrangements are possible and within the scope of the present technology. For example, each row of EVS pixels may correspond to (a) one row of CIS pixels, (b) two rows of CIS pixels, (c) three rows of CIS pixels, or (d) more than four rows of CIS pixels. Examples of other CIS pixel resolutions and EVS pixel resolutions are described in co-applied, co-pending and commonly assigned applications entitled “METHODS FOR OPERATING HYBRID IMAGE SENSORS HAVING DIFFERENT CIS-TO-EVS RESOLUTIONS,” which have been incorporated herein by reference in their entirety above.

[0102] As discussed above, event data captured using EVS pixels may be used to perform event-directed deblurring and / or rolling shutter distortion correction of CIS (frame) information captured using CIS pixels. To this end, Figure 5 is a partially schematic block diagram of an image sensor 530 that may include on-chip image deblurring and / or rolling shutter correction capabilities and configured in accordance with various embodiments of the present technology. The image sensor 530 may be the above Figure 4A and / or Figure 4B Examples of stacked system 430 or other image sensors configured in accordance with the present technology.

[0103] As shown, image sensor 530 includes a CIS pixel array 538 (e.g., similar to Figure 4A , Figure 4B and / or Figure 4C 438) and / or an event-driven sensing array 542 (e.g., similar to Figure 4A , Figure 4B and / or Figure 4C The image sensor 530 further includes (a) a row / column control circuit system 561 and a column readout circuit system 563 for controlling the operation and readout of the CIS pixels included in the pixel array 538 and (b) a row / column control circuit system 562 for controlling the operation and readout of the EVS pixels included in the event-driven sensing array 542. The image sensor 530 may optionally include a pre-processing circuit 564 for performing various operations (e.g., denoising) on ​​the EVS data read out from the EVS pixels of the event-driven sensing array 542.

[0104] The image sensor 530 further includes a common control block 568 for synchronizing the operation of the pixel array 538 with the operation of the event-driven sensing array 542. More specifically, although the CIS pixels of the pixel array 538 and the EVS pixels of the event-driven sensing array 542 themselves include row / column control circuitry and are independently read by their own readout circuitry, the common control block synchronizes the operation (e.g., reset, exposure start time, exposure end time) of the CIS pixels, EVS pixels, row / column control circuitry, and / or readout circuitry. This synchronization is described below with reference to Figures 10 to 12 Describe in more detail.

[0105] In some embodiments, the image sensor 530 may include a first multiplexer 565, a second multiplexer 566, and / or a third multiplexer 567. As shown, the first multiplexer 565, the second multiplexer 566, and the third multiplexer 567 may be controlled using a deblur enable signal deblurEN. When the deblur enable signal deblurEN is not asserted (e.g., in a low or '0' state), the first multiplexer 565 and the third multiplexer 567 may stream CIS data (e.g., raw intensity image frames, blurred intensity image frames) to the image signal processor 552 of the image sensor 530, e.g., instead of streaming raw CIS data to the deblur and rolling shutter distortion correction circuit 570 ("deblur circuit 570" or "rolling shutter distortion correction circuit 570") of the image sensor 530. Image signal processor 552 may in turn provide the CIS data to a synchronous communication interface 555 a (eg, a MIPI interface / transmitter), for example, for output from image sensor 530 .

[0106] Additionally or alternatively, when the deblur enable signal deblurEN is not asserted (e.g., in a low or '0' state), the second multiplexer 566 may stream the EVS data to the column scan readout circuitry 553, e.g., instead of streaming the EVS data to the deblur circuit 570. The column scan readout circuitry 553 may, in turn, provide the EVS data to an event signal processor 554 of the image sensor 530, and the event signal processor 554 may provide the EVS data to a synchronous communication interface 555b (e.g., a MIPI interface / transmitter), e.g., for output from the image sensor 530.

[0107] In the illustrated embodiment, the synchronous communication interface 555a and the synchronous communication interface 555b may be independent physical interfaces. Alternatively, the synchronous communication interface 555a and the synchronous communication interface 555b may be merged. For example, CIS data and EVS data may be output from the image sensor 530 via a shared synchronous communication interface 555 (e.g., a shared MIPI interface, a shared virtual channel, an embedded line).

[0108] Referring again to the first multiplexer 565, the second multiplexer 566, and the third multiplexer 567, when the deblur enable signal deblurEN is asserted (e.g., in a high or '1' state), the first multiplexer 565 is enabled to stream CIS information read from the CIS pixels of the pixel array 538 into the deblur circuit 570, and the second multiplexer 566 is enabled to stream EVS information read from the EVS pixels of the event-driven sensing array 542 into the deblur circuit 570. For example, when the deblur enable signal deblurEN is asserted, the EVS information read from the EVS pixels of the event-driven sensing array 542 may be continuously streamed into the deblur circuit 570 via the second multiplexer 566 (e.g., when the CIS pixels of the pixel array 538 integrate photo-generated charge during an exposure period). Additionally or alternatively, when the CIS information is read out from the CIS pixels of the pixel array 538 after an exposure period, the digitized CIS information may be streamed into the de-obfuscation circuit 570 via the first multiplexer 565 .

[0109] The deblurring circuit 570 may (a) compute a fused image / video stream from the CIS data and the EVS data received via the first multiplexer 565 and the second multiplexer 566, respectively, and (b) may output the fused image / video stream into the third multiplexer 567 for streaming to the image signal processor 552. The fused image / video stream may then be provided from the image signal processor 552 to the synchronous communication interface 555a for output from the image sensor 530. The fusion calculations performed by the deblurring circuit 570 may target deblurring of CIS frame information captured by the CIS pixels of the pixel array 538, correcting rolling shutter artifacts, and / or creating interpolated video frames. On-chip deblurring of CIS frame information and rolling shutter correction of CIS frame information are described below with reference to Figures 6 to 14 Discuss in more detail.

[0110] In embodiments where image sensor 530 is a stacked system, components of deblurring circuit 570 may be located on one or more of the dies (e.g., the top die, the middle die, or the bottom die) of the stacked system. As a specific example, image sensor 530 may be substantially similar to the above-described Figure 4A and 4B 4 , and the deblurring circuit 570 of the image sensor 530 may be located on the third (or bottom) die of the image sensor 530 .

[0111] In some embodiments, the Figure 5565, the second multiplexer 566 and / or the third multiplexer 567 shown in FIG. In at least some of these embodiments, the image sensor 530 may be substantially similar to Figure 5 5B , the image sensor 530 described in FIG. 5B operates in the manner of how it would operate if the deblur enable signal deblurEN was permanently in an asserted state (e.g., by always streaming CIS information and EVS information into the deblur circuit 570 so that the deblur circuit 570 computes a fused image / video stream). Additionally or alternatively, the image sensor 530 may be configured to output raw CIS data and / or EVS data. For example, the image sensor 530 may output raw CIS data and / or EVS data in addition to or in lieu of outputting a fused image / video stream based on the raw CIS data and EVS data. Furthermore, although Figure 5 The image signal processor 552 of the image sensor 530 illustrated in FIG. 5 is configured to process the fused image / video stream output by the deblurring circuit 570 when the deblurring enable signal deblurEN is asserted, but in other embodiments, the image signal processor 552 may be configured to process the CIS information read out from the CIS pixels of the pixel array 538 before the deblurring circuit 570 calculates the fused image / video stream. In such embodiments, after processing the CIS information read out from the CIS pixels of the pixel array 538, the image signal processor 552 may output the processed CIS information to the deblurring circuit 570 so that the deblurring circuit 570 calculates the fused image / video stream based on the processed CIS information (e.g., not based on the original CIS information).

[0112] Figure 6 is a partial schematic diagram of a deblurring and rolling shutter distortion correction circuit 670 (“deblurring circuit 670” or “rolling shutter distortion correction circuit 670”) configured in accordance with various embodiments of the present technology. The deblurring circuit 670 may be Figure 5 670 or other deblurring circuitry configured in accordance with the present technology. As shown, the deblurring circuit 670 includes a rolling shutter distortion correction and event-based double integral (EDI) calculation block 671 ("calculation block 671") and a latent frame calculation block 672. The operation of at least a portion of the calculation block 671 can be performed by at least generally following the time-continuous signal e(t) above in equation 13 (an example of which is shown in FIG. Figure 2 In other words, whenever an event is triggered and a pixel is read out from the EVS, the operation of at least a portion of the computation block 671 may be enabled. The operation of the latent frame computation block 672 may be timed by the control signal line_sync. The control signal line_sync may be controlled by a common control block (e.g., Figure 5The control signal line_sync may be provided / controlled by a common control block 568 of the image sensor 530. In some embodiments, the control signal line_sync may correspond to the duration of the exposure period T, as shown in equations 16 and 17 above. In these and other embodiments, the control signal line_sync may correspond to the duration extending between (i) the beginning of the exposure period of the first row of CIS pixels in the CIS pixel array and (ii) the end of the exposure period of the last row of CIS pixels in the CIS pixel array.

[0113] In the illustrated embodiment, the calculation block 671 includes a plurality of EDI components. The EDI components of the calculation block 671 include a counter 673, a first integral buffer 674, a product calculation block 675, an exponent calculation block 676, an integral calculation block 677, and a second integral buffer 678. The counter 673 is also referred to herein as the "first counter", the product calculation block 675 is also referred to herein as the "first product calculation block", and the exponent calculation block 676 is also referred to herein as the "first exponent calculation block". The first integral buffer 674 and the second integral buffer 678 are also referred to herein as "EDI integral buffers".

[0114] The counter 673 may be an integer counter (e.g., an up / down counter) or another suitable type of counter for calculating a running sum of events detected by each EVS pixel during an exposure period of the CIS pixel corresponding to the EVS pixel, and the first integral buffer 674 may be configured to track / store the running sum. More specifically, the counter 673 and the first integral buffer 674 may be configured to calculate a first inner integral of the EDI model described above. For example, when event data is detected by this EVS pixel during an exposure period of the CIS pixel corresponding to the EVS pixel and streamed into the deblurring circuit 670, the counter 673 may be enabled via the control signal EVS_CLK, which may cause the counter 673 to increment or decrement (depending on the polarity of the detected event) the running sum maintained by the first integral buffer 674 for this EVS pixel. For each EVS pixel, the running sum maintained over time by the first integral buffer 674 is equal to E(t) shown in Equation 14 above, an example of which is Figure 2 214. For this implementation, the buffer size of the first integration buffer 674 may be relatively small. As a specific example, assuming that the duration of the exposure period of the CIS pixel corresponding to the EVS pixel is 33 ms, the first integration buffer 674 may include a buffer size of approximately 10 bits / pixel. In some embodiments, a floating point calculator may be used instead of the counter 673. In such embodiments, the first integration buffer 674 may have a larger or smaller buffer size.

[0115] The product calculation block 675, the exponent calculation block 676, the integral calculation block 677 and the second integral buffer 678 may be configured to calculate the second outer integral of the above-mentioned EDI model. For example, the product calculation block 675 may multiply the running sum stored in the first integral buffer 674 by the contrast threshold parameter c, which is equal to C log-TH (described above with reference to equations 6-12 and 15-17) and assumed to remain constant. Therefore, for each EVS pixel, the output of product calculation block 675 may be equal to c∑ i∈[s,t] p i Thereafter, for each EVS pixel, the exponent calculation block 676 may determine the exponent of the output of the product calculation block 675, thereby obtaining exp(c∑ i∈[s,t] p i ). The integral calculation block 677 may continuously integrate the output of the index calculation block 676 over time for each EVS pixel. More specifically, the integral calculation block 677 may be s (corresponding to the start of the exposure period of the CIS pixel corresponding to the corresponding EVS pixel) to time t (at time t s+T Each of the outputs of the index calculation block 676 is integrated at the end of the exposure period corresponding to the CIS pixel corresponding to the corresponding EVS pixel. The second integration buffer 678 can track / store the results of this time-continuous integration, which are each equal to the end of each corresponding exposure period. The example result of the EVS pixel output by the integral calculation block 677 is Figure 2 This is shown in Figure 216.

[0116] Each of the calculations performed by the product calculation block 675, the exponent calculation block 676, the integral calculation block 677, and the second integral buffer 678 may be performed in a floating point representation, such as a 9-bit mantissa and a 4-bit exponent. In addition, although the operation of the counter 673 and / or the first integral buffer 674 may be clocked by the control signal EVS_CLK, the product calculation block 675, the exponent calculation block 676, the integral calculation block 677, and / or the second integral buffer 678 may be enabled to continuously perform their respective operations over time. As a specific example, in some embodiments, the operation of the integral calculation block 677 is neither clocked by the control signal EVS_CLK nor triggered by an event. Specifically, the integral calculation block 677 is configured to continuously integrate the output of the exponential calculation block 676 over time, at least at times t corresponding to the start and stop times of the corresponding exposure period, respectively. s With time t s+T In these embodiments, the operations of the product calculation block 675 and / or the exponent calculation block 676 may be clocked or enabled by the control signal EVS_CLK to continuously perform their respective operations over time.

[0117] Because the events detected at each EVS pixel during the corresponding exposure period are accumulated by the EDI component of the calculation block 671, the raw EVS data input into the calculation block 671 of the deblurring circuit 670 during the corresponding exposure period may be discarded once the events of the raw EVS data are accumulated by the EDI component of the calculation block 671. Therefore, in some embodiments, the second integration buffer 678 only stores / maintains the accumulated results of the integral calculation block 677, which means that the second integration buffer 678 may have a relatively small buffer size (e.g., about 9MB, e.g., about 8.625MB or about 13 bits / pixel) compared to the buffer used in the off-chip event-oriented deblurring solution. In addition, because the raw EVS data may be discarded instead of being output from the image sensor corresponding to the deblurring circuit 670, the IO throughput and power consumption may be reduced compared to the off-chip event-oriented deblurring solution in which the raw EVS data is output from the image sensor to an external application processor. In other embodiments of the present technology, after the events in the raw EVS data are accumulated, all of the raw EVS data or a subset thereof may be stored and / or output from the image sensor.

[0118] After the exposure period ends, the CIS data may be read out from the CIS pixels of the image sensor and streamed into the latent frame calculation block 672 of the deblurring circuit 670. At this point, the latent frame calculation block 672 may deblur the CIS data by combining / fusing the CIS data with the accumulated EVS data stored in the second integration buffer 678 of the calculation block 671. More specifically, for one or more of the CIS pixels, the latent frame calculation block 672 may calculate the respective corresponding time t by performing the operations specified in the above equation 17 using the CIS data captured by the one or more CIS pixels and the corresponding EVS data accumulated in the second integration buffer 678. s (indicating the start of the corresponding exposure period) of one or more latent image frames L(s). Additionally or alternatively, for one or more of the CIS pixels, the latent frame calculation block 672 may calculate, for each corresponding to time t, by performing the operations specified in the above equation 15 using the corresponding latent image frame L(s). s With time t s+T(representing the end of the corresponding exposure period) between one or more latent image frames L(t) at time t. Because the CIS data can be read directly into the latent frame calculation block 672 at or after the end of the exposure period and because the EVS data accumulated in the second integration buffer 678 is readily available and is aligned with the CIS data at this time (as discussed in more detail below), no CIS frame buffer is required to perform on-chip deblurring using the deblurring circuit 670. Therefore, in some embodiments, the deblurring circuit 670 and / or the corresponding image sensor may lack a CIS frame buffer. In other embodiments, the deblurring circuit 670 and / or the corresponding image sensor may include a CIS frame buffer, such as in embodiments where raw CIS data plus fused image / video data can be output.

[0119] As discussed above, a rolling shutter can be used to capture and read out CIS data from a CIS pixel array. Figure 7 790 is a diagram illustrating two sequential image frames (frame i and frame i+1) of a CIS pixel array according to various embodiments of the present technology. The CIS data corresponding to frame i and the CIS data corresponding to frame i+1 are each captured using a rolling shutter. Thus, referring to frame i shown on the left side of the diagram 790 as an example, the exposure period of the first two rows of CIS pixels of the CIS pixel array may be at time The exposure period of the last two rows of CIS pixels in the CIS pixel array may be at time In other words, due to the use of a rolling shutter, there is a delay between the time when the exposure period of the first two pixel rows begins and the time when the exposure period of each of the other pixel rows (including the last two pixel rows) begins. Figure 6 and 7 , in the EDI component using the computing block 671 for the Rolling shutter distortion may exist in the latent image frame L(s) and the latent image frame L(t) calculated by the CIS pixels of the pixel row in the CIS pixel array during the exposure period starting at time t, especially when there is a rolling shutter distortion in the external scene at time t. Thus, in embodiments where a rolling shutter is used, Figure 6 The computation block 671 may additionally include a rolling shutter distortion correction component that may be used to correct rolling shutter distortion of the latent image frames L(s) and L(t).

[0120] like Figure 6As shown in FIG. 6 , the rolling shutter distortion correction component of the calculation block 671 may include a counter 679 (also referred to herein as a “second counter”), an integral buffer 680 (also referred to herein as a “third integral buffer”), a product calculation block 681 (also referred to herein as a “second product calculation block”), and an exponent calculation block 682 (also referred to herein as a “second exponent calculation block”). The integral buffer 680 is also referred to herein as a rolling shutter distortion correction (RSDC) integral buffer.

[0121] Counter 679 may be an integer counter (eg, an up / down counter) or a counter for calculating the time between time t0 (corresponding to the start of the first exposure period of a given image frame) and time t s Another suitable type of counter that can be used to store / maintain, for each EVS pixel, a running sum of events detected by each EVS pixel between time t0 and time t1 (representing the start of the exposure period of the CIS pixel corresponding to the EVS pixel for a given image frame). Additionally, the integration buffer 680 can be configured to store / maintain, for each EVS pixel, a running sum of events detected by each EVS pixel between time t0 and time t2. s More specifically, the counter 679 and the integration buffer 680 may be configured to calculate, for each EVS pixel, the time between (i) time t0 and (ii) the start of the exposure period of the CIS pixel corresponding to the EVS pixel (also referred to herein as time t s ) is the integral of events that occur between .

[0122] For example, consider Figure 7 The last two CIS pixel rows shown in frame i of the drawing 790. Figure 6 and 7 , when the EVS pixels corresponding to the last two CIS pixel rows are at time With time When an event is detected between t and t, the corresponding event data may be read out from the EVS pixel into the computation block 671 of the deblurring circuit 670. The counter 679 of the deblurring circuit 670 may in turn be enabled (e.g., via the control signal EVS_CLK or another control signal) to increment or decrement (depending on the polarity of the detected event) the running sum maintained by the integration buffer 680 for this EVS pixel. The running sum of the EVS pixel over time is equal to E'(t) shown in Equation 18 above, an example of which is Figure 2 In some embodiments, at time Starting with (corresponding to the start of the exposure period of the CIS pixels in the last two rows of CIS pixels), events detected by the EVS pixels may then be accumulated by the EDI component of computation block 671 (e.g., rather than the rolling shutter distortion correction component of computation block 671).

[0123] Reference again Figure 6, in some embodiments, the buffer size of the integration buffer 680 can be relatively small. As a specific example, assuming a readout speed of 30 frames / second, the maximum delay between the start of the first exposure period of a given frame and the start of the last exposure period of the given frame can be about 33ms. Continuing with this example, assuming (a) a 12-megapixel CIS pixel array is used, (b) a 3-megapixel EVS pixel array is used, and (c) an EVS readout rate of about 50 nanoseconds / row is used (meaning that the entire EVS pixel array can be read out once every 33ms), the counter 679 and the integration buffer 680 are expected to accumulate up to 400 events between the start of the first exposure period of a given frame and the start of the last exposure period of the given frame. Because the events have positive or negative polarity, the possible values ​​of the running sum calculated by the counter 679 and the integration buffer 680 for each EVS pixel can range from about -400 to about +400. Thus, in some embodiments, the integration buffer 680 can include a buffer size of about 10 bits / pixel. In these and other embodiments, a floating point calculator may be used in place of counter 679. In such embodiments, integration buffer 680 may have a buffer size that is larger or smaller than 10 bits / pixel.

[0124] The product calculation block 681 may multiply the running sum stored in the integration buffer 680 by a contrast threshold parameter c, which is equal to C log-TH (described above with reference to equations 6-12, 15-17, 19, and 20) and assumed to remain constant. Therefore, for each EVS pixel, the output of product calculation block 681 may be equal to c∑ i∈[0,t] p i Thereafter, for each EVS pixel, the exponent calculation block 682 may determine the exponent of the output of the product calculation block 681, thereby obtaining exp(c∑ i∈[0,t] p i ). The exponent of the output of the product calculation block 681 determined by the exponent calculation block 682 may be output to the latent frame calculation block 672 of the deblurring circuit 670.

[0125] In some embodiments, the calculations performed by the product calculation block 681 and / or the exponent calculation block 682 may be performed in a floating point representation, such as a 9-bit mantissa and a 4-bit exponent. Additionally or alternatively, although the operation of the counter 679 and / or the integral buffer 680 may be timed by the control signal EVS_CLK or another control signal, the product calculation block 681 and / or the exponent calculation block 682 may be enabled to continuously perform their respective operations over time. Alternatively, the operation of the product calculation block 681 and / or the exponent calculation block 682 may also be timed by the control signal EVS_CLK.

[0126] As discussed above, after the exposure period of the CIS pixel ends, the CIS data captured by the CIS pixel can be read out to the latent frame calculation block 672 of the deblurring circuit 670. According to the above discussion, the latent frame calculation block 672 can deblur the CIS data by combining / fusing the CIS data with the accumulated EVS data stored in the second integration buffer 678 of the calculation block 671. In addition, the latent frame calculation block 672 can correct the rolling shutter distortion of the CIS data using the corresponding output of the exponential calculation block 682 from the calculation block 671. More specifically, the latent frame calculation block 672 can calculate the corresponding time t by performing the operation specified in the above equation 17 for each EVS pixel. s In addition, the latent frame calculation block 672 may calculate the latent image frame L(0) corresponding to time t0 (the beginning of the corresponding image frame, e.g., the beginning of the first exposure period of the corresponding image frame) for each pixel using: (i) the latent image frame L(s); (ii) the corresponding output from the index calculation block 682; and / or (iii) the operation specified in the above equation 20. The latent image frame L(0) may correspond to the deblurred rolling shutter distortion corrected CIS data and may be output from the latent frame calculation block 672 to an image signal processor of the corresponding imaging system.

[0127] Reference again Figure 7 , frame i+1 does not overlap with frame i. In other words, time (indicating the start of the first exposure period of frame i+1) occurs at time (indicating the end of the last exposure period of frame i). As long as this relationship between immediately adjacent frames (e.g., frame i and frame i+1) is maintained, a single (e.g., only one) instance of counter 679 and integration buffer 680 may be used Figure 6 671 of the deblurring circuit 670. However, this relationship limits the maximum frame rate that can be used by the corresponding image sensor.

[0128] For example, the maximum frame rate that can be used by the corresponding image sensor can be increased by starting the first exposure period of the second frame before the end of the last exposure period of the first frame. Figure 8 is a plot 895 illustrating two immediately adjacent image frames (frame i and frame i+1). As shown, a rolling shutter is used to enable time (indicating the start of the first exposure period of frame i+1) occurs at time (denoting the end of the last exposure period of frame i). Therefore, the exposure period of frame i+1 of at least some CIS pixel rows of the CIS pixel array may overlap in time with the exposure period of frame i of at least some other CIS pixel rows of the CIS pixel array. Figure 8 The time displayed in With time During the period between , this arrangement needs to track both (a) events corresponding to frame i and (b) events corresponding to frame i+1. Implementing a ping-pong buffer into the rolling shutter distortion correction component of the deblurring circuit can achieve this functionality.

[0129] For example, Fig. 9 is a partial schematic diagram of a deblurring and rolling shutter distortion correction circuit 970 (“deblurring circuit 970” or “rolling shutter distortion correction circuit 970”) configured in accordance with various embodiments of the present technology. The deblurring circuit 970 may be Figure 5 An example of a deblurring circuit 570 of the present invention or other deblurring circuits configured in accordance with the present technology. As shown, the deblurring circuit 970 is generally similar to Figure 6 The deblurring circuit 670. Therefore, similar reference numbers are Figure 6 and 9 To represent identical or at least substantially similar components, and in view of the detailed description of deblurring circuit 670 provided above, a detailed description of deblurring circuit 970 is substantially omitted herein for the sake of brevity.

[0130] like Fig. 9 , the deblurring circuit 970 includes a rolling shutter distortion correction and event-based double integral (EDI) calculation block 971 ("calculation block 971") and a latent frame calculation block 972. The calculation block 971 includes EDI components (e.g., a counter 973, a first integral buffer 974, a product calculation block 975, an exponent calculation block 976, an integral calculation block 977, and a second integral buffer 978). The counter 973 is also referred to herein as the "first counter", the product calculation block 975 is also referred to herein as the "first product calculation block", and the exponent calculation block 976 is also referred to herein as the "first exponent calculation block". The first integral buffer 974 and the second integral buffer 978 are also referred to herein as "EDI integral buffers".

[0131] The computation block 971 of the deblurring circuit 970 further includes a rolling shutter distortion correction component. Figure 6 The rolling shutter distortion correction component of the computation block 671 of the deblurring circuit 670 is compared to Fig. 9The rolling shutter distortion correction component of the calculation block 971 of the deblurring circuit 970 includes a ping-pong buffer. More specifically, the rolling shutter distortion correction component includes a routing switch 983, a counter 979a (also referred to herein as a "second counter"), a counter 979b (also referred to herein as a "third counter"), an integral buffer 980a (also referred to herein as a "third integral buffer"), an integral buffer 980b (also referred to herein as a "fourth integral buffer"), and a multiplexer 984. The integral buffers 980a and 980b are also referred to herein as "rolling shutter distortion correction (RSDC) integral buffers".

[0132] As shown, the rolling shutter distortion component of the calculation block 971 also includes a product calculation block 981a (also referred to herein as a “second product calculation block”), a product calculation block 981b (also referred to herein as a “third product calculation block”), an exponent calculation block 982a (also referred to herein as a “second exponent calculation block”), and an exponent calculation block 982b (also referred to herein as a “third exponent calculation block”). In other embodiments, the rolling shutter distortion component may include a single (e.g., only one) instance of the product calculation block 981 and / or a single (e.g., only one) instance of the exponent calculation block 982. In such embodiments, the product calculation block 981 and the exponent calculation block 982 may be positioned downstream of the multiplexer 984, such as between the multiplexer 984 and the latent frame calculation block 972. Continuing with this example, the product calculation block 981 and the exponent calculation block 982 may be configured to perform operations on a running sum output from the integral buffer 980a or the integral buffer 980b via the multiplexer 984.

[0133] The counters 979a and 979b of the deobfuscation circuit 970 may be substantially similar to Figure 6 The counter 679 of the deblurring circuit 670, and the integral buffers 980a and 980b may be substantially similar to Figure 6 The integral buffer 680 of the deblurring circuit 670. In addition, the product calculation block 981a and the product calculation block 981b can be substantially similar to Figure 6 The product calculation block 681 of the deblurring circuit 670, and the exponent calculation block 982a and the exponent calculation block 982b may be substantially similar to Figure 6 The exponential calculation block 682 of the deblurring circuit 670. Therefore, in view of the above reference Figure 6 A detailed description of each of these rolling shutter distortion correction components of deblurring circuit 970 is omitted here for the sake of brevity, as a detailed description of similar components of deblurring circuit 670 is provided.

[0134] In the illustrated embodiment, the counter 979a, the integral buffer 980a, the product calculation block 981a, and the exponent calculation block 982a (collectively referred to herein as the "first set of rolling shutter distortion correction (RSDC) components") may correspond to a different frame than the counter 979b, the integral buffer 980b, the product calculation block 981b, and the exponent calculation block 982b (collectively referred to herein as the "second set of RSDC components"). For example, referring together Figure 8 and 9 , the first set of RSDC components can be used to accumulate Figure 8 The events detected during the first frame (frame i) shown on the left in FIG; and the second set of RSDC components can be used to accumulate Figure 8 The events detected during the second frame (frame i+1) are shown on the right side of FIG. Thereafter, the first set of RSDC components can be used to accumulate events detected during the third frame (frame i+2; Figure 8 The second set of RSDC components can be used to accumulate events detected during the fourth frame (frame i+3; Figure 8 (not shown); and so on. Therefore, continuing the above example, at time The start time of the exposure period corresponding to frame i (eg Figure 8 The timing of the bottom two pixel rows of the CIS pixel array shown in ) events detected by EVS pixels between 1 and 2 can be accumulated using the first set of RSDC components. The start time of the exposure period corresponding to frame i+1 (eg Figure 8 The timing of the bottom two pixel rows of the CIS pixel array shown in ) events detected by EVS pixels between can be accumulated using a second set of RSDC components.

[0135] Routing events detected by EVS pixels to the appropriate set of RSDC components may be handled via routing switch 983. More specifically, routing switch 983 is configured to receive a control signal ping_pong. In some embodiments, control signal ping_pong may be generated by a common control block (e.g., Figure 5 Alternatively, the control signal ping_pong may be provided and / or controlled by another control block (e.g., Figure 5 The image sensor 530 may be provided / controlled by the column control circuit system 562 and / or the column scanning readout circuit system 553).

[0136] The control signal ping_pong may be used to control to which counter (counter 979a or counter 979b) an event detected by an EVS pixel is routed via routing switch 983. Figure 8 and 9 ,exist Figure 8 The time displayed in With time Between, the control signal ping_pong( Fig. 9 ) can transition to or maintain a first state (e.g., asserted state, high state, "1" state). Thus, at time The start time of the exposure period corresponding to frame i (eg Figure 8 The timing of the bottom two pixel rows of the CIS pixel array shown in ) between the events detected by the EVS pixels can be routed to the counter 979a via the routing switch 983. Figure 8 The time displayed in With time Between, the control signal ping_pong( Fig. 9 ) may transition to or remain in a second state (e.g., a deasserted state, a low state, a "0" state). The start time of the exposure period corresponding to frame i+1 (eg Figure 8 The timing of the bottom two pixel rows of the CIS pixel array shown in ) can be routed to counter 979b via routing switch 983. Events detected by this EVS pixel during the exposure period of the CIS pixel corresponding to the EVS pixel can be routed to counter 973 of the EDI component of computation block 971, as described above with reference to Figure 6 The description of the EDI component of the computation block 671 of the defuzzification circuit 670 is consistent.

[0137] Referring now to the multiplexer 984, the control signal ping_pong may be used to control which input into the multiplexer 984 (e.g., which output of the exponential calculation block 982a and the exponential calculation block 982b) is output from the multiplexer 984 (e.g., to the latent frame calculation block 972). In the illustrated embodiment, when the control signal ping_pong transitions or remains in a first state, the output of the exponential calculation block 982b may be routed to the latent frame calculation block 972 via the multiplexer 984. On the other hand, when the control signal ping_pong transitions or remains in a second state, the output of the exponential calculation block 982a may be routed to the latent frame calculation block 972 via the multiplexer 984. Thus, when the detected event is routed to the counter 979a via the routing switch 983, the output of the exponential calculation block 982b may be passed to the latent frame calculation block 972 via the multiplexer 984. Additionally, when the detected event is routed to counter 979b via routing switch 983, the output of index calculation block 982a may be passed to latent frame calculation block 972 via multiplexer 984. In this manner, the ping-pong buffer implements rolling shutter distortion correction for CIS data corresponding to two different frames that at least partially overlap in time.

[0138] Fig.10 10 is a flow chart illustrating a method 1000 of operating an image sensor according to various embodiments of the present technology. For example, method 1000 may be a method of performing on-chip deblurring and / or rolling shutter distortion correction of CIS data. Method 1000 is illustrated as a series of boxes 1001 to 1012 or steps. All boxes 1001 to 1012 or a subset of one or more thereof may be performed by a device or component of an image sensor configured according to various embodiments of the present technology. For example, all boxes 1001 to 1012 or a subset of one or more thereof may be performed by a CIS pixel of a pixel array, an EVS pixel of an event-driven sensing array, a common control block, a row / column control circuit system, a column readout circuit system, a column scan readout circuit system and / or a deblurring block or circuit. All boxes 1001 to 1012 of method 1000 or a subset of one or more thereof may be performed according to the above-described Figures 1 to 9 In fact, several blocks 1001 to 1012 of method 1000 are referred to below. Figures 6 to 13B describe.

[0139] Method 1000 begins with block 1001, which resets the EVS pixels and / or integration buffers of the corresponding deblurring circuit at the beginning of a frame. In some embodiments, resetting the EVS pixels and integration buffers may include resetting the EVS pixels and integration buffers simultaneously. In some embodiments, resetting the EVS pixels and / or integration buffers may be performed at least in part using a common control block of the corresponding image sensor. For example, the common control block may synchronize the operation of the deblurring block with the beginning of an image frame, such as by using one or more control signals.

[0140] In these and other embodiments, resetting the EVS pixels may include resetting all or a subset of the EVS pixels. As a specific example, resetting the EVS pixels at block 1001 may include resetting all EVS pixels at the start of a frame. As another specific example, resetting the EVS pixels at block 1001 may include resetting a subset of the EVS pixels, such as the EVS pixels corresponding to the CIS pixel rows having ended / elapsed exposure periods in the last frame.

[0141] In these and other embodiments, resetting the integration buffers may include resetting all or a subset of the integration buffers of the deobfuscation circuit (e.g., Figure 6 The first integral buffer 674, the second integral buffer 678 and / or the integral buffer 680 of the deblurring circuit 670; Fig. 9 970 of the deblurring circuit 970. As a specific example, resetting the integration buffers may include resetting all integration buffers of the deblurring circuit (e.g., Figure 6 A first integral buffer 674, a second integral buffer 678 and an integral buffer 680 of a deblurring circuit 670; Fig. 9 As another specific example, resetting the integration buffers may include resetting all integration buffers of the deblurring circuit corresponding to a given frame (e.g., Figure 6 A first integral buffer 674, a second integral buffer 678 and an integral buffer 680 of a deblurring circuit 670; Fig. 9 As yet another specific example, resetting the integration buffers may include resetting one or more RSDC integration buffers (e.g., Figure 6The integral buffer 680 of the deblurring circuit 670, for example, does not reset the first integral buffer 674 and / or the second integral buffer 678; Fig. 9 The integral buffer 980a and / or the integral buffer 980b of the deblurring circuit 970 is reset, for example, without resetting the first integral buffer 974 and / or the second integral buffer 978).

[0142] Fig.11 1195 is a timing diagram of five EVS pixel rows (EVS row N through EVS row N+4) and five corresponding CIS pixel rows in accordance with various embodiments of the present technology. Each of EVS pixel rows N through N+4 is illustrated directly below a corresponding one of the CIS pixel rows. Fig.11 Also illustrated are (a) reset periods 1193 and 1196, (b) exposure period 1197 for CIS pixels of five illustrated CIS pixel rows, and (c) event accumulation period 1198 for EVS pixels of EVS row N to EVS row N+4. Fig.10 and 11 For clarity and example, at block 1001 of method 1000, all five EVS pixel rows N to N+4 of the EVS pixels and / or one or more RSDC integration buffers of the corresponding deblurring circuits may be extended over time t -1 to time t0 (corresponding to the beginning of the illustration frame) during the reset period 1196 ( Fig.11 ) during reset period 1196. In some embodiments, one or more EDI integration buffers corresponding to the defuzzification circuit may also be reset during reset period 1196.

[0143] Reference again Fig.10 At block 1002, method 1000 then captures EVS data (also referred to herein as "EVS pixel data" or "event data") for each EVS pixel. Capturing EVS data may include detecting events occurring in an external scene using the EVS pixels. Referring again to Fig.11 , for purposes of example and clarity, the EVS pixels of all five EVS rows N through EVS row N+4 are configured to detect events during corresponding EVS accumulation periods 1198. As shown, each EVS accumulation period 1198 extends from time t0 (indicating the beginning of the illustrated frame) to the end of the exposure period 1197 of the corresponding CIS pixel. Referring to EVS row N+2 as an example, the EVS accumulation period extends from time t0 to time t7 (corresponding to the end of the exposure period 1197 of the CIS pixel corresponding to the EVS pixel of EVS row N+2). As discussed in more detail below, events detected by the EVS pixels of EVS row N+2 between time t0 and time t2 (corresponding to the beginning of the exposure period 1197 of the CIS pixel corresponding to the EVS pixel of EVS row N+2) may be detected in method 1000 ( Fig.10) and accumulated in one or more RSDC integration buffers of the deblurring circuit at block 1004 of method 1000. Additionally, events detected by the EVS pixels of EVS row N+2 between time t2 and time t7 may be read out from the EVS pixels at block 1007 of method 1000 and accumulated in one or more EDI integration buffers of the deblurring circuit at block 1008 of method 1000.

[0144] At block 1003, the method 1000 then reads out the EVS data from the EVS pixel. Block 1003 may be performed after and / or simultaneously with the execution of block 1002. For example, when an EVS pixel detects an event, the event may be read out from the EVS pixel. When an event is read out from the EVS pixel, the EVS pixel may be reset and thereby enabled to detect subsequent events. Reading out the EVS data may include reading out the EVS data to an RSDC component of a computational block of a deblurring circuit. Additionally or alternatively, reading out the EVS data may include reading out the EVS data to an EDI component of a computational block of a deblurring circuit. In these and other embodiments, reading out the EVS data may include reading out the EVS data using a scanning readout technique, such as the scanning readout technique described in more detail below with reference to block 1007 of the method 1000.

[0145] At block 1004, method 1000 then accumulates, for each EVS pixel, the time between the start of the frame (eg, time t0) and the start of the exposure period of the CIS pixel corresponding to this EVS pixel (eg, time t s ). Block 1004 may be performed after and / or concurrently with performing blocks 1002 and / or 1003, as shown by the arrow returning from block 1004 to block 1002. Additionally or alternatively, block 1004 may be performed by an RSDC component of an on-chip deblurring circuit.

[0146] As discussed above, accumulating EVS data for rolling shutter distortion correction may include, for each EVS pixel, for example, using a counter and an integration buffer (e.g.: Figure 6 An integral buffer 680; Fig. 9 The running sum of the events can be calculated and maintained by integrating buffer 980a and / or integrating buffer 980b of the deblurring circuit. In addition, accumulating EVS data can include, for each EVS pixel, (a) calculating the product of the running sum and the contrast threshold parameter and (b) determining an exponent of the product. The exponent of the product can be output to the latent frame calculation block of the deblurring circuit.

[0147] As discussed above, for each EVS pixel, the start of the frame (eg, time t0) and the start of the exposure period of the CIS pixel corresponding to this EVS pixel (eg, time t s) between the EVS pixels captured by the EVS data is read out from the EVS pixels and accumulated in one or more RSDC integration buffers of the deblurring circuit. Fig.10 and 11 For example and clarity, at block 1004 of method 1000, events detected by the EVS pixel of EVS row N+1 between time t0 and time t1 (representing the start of exposure period 1197 of the CIS pixel corresponding to the EVS pixel of EVS row N+1) may be accumulated in the RSDC integration buffer of the deblurring circuit. An example of an RSDC integration buffer includes Figure 6 The integral buffer 680, Fig. 9 The integral buffer 980a and Fig. 9 In addition, continue Fig.11 In the example, events detected by the EVS pixels of EVS row N+2 between time t0 and time t2 (indicating the start of exposure period 1197 for the CIS pixels of the EVS pixels corresponding to EVS row N+2) may be accumulated in the RSDC integration buffer of the deblurring circuit; events detected by the EVS pixels of EVS row N+3 between time t0 and time t3 (indicating the start of exposure period 1197 for the CIS pixels of the EVS pixels corresponding to EVS row N+3) may be accumulated in the RSDC integration buffer of the deblurring circuit; and events detected by the EVS pixels of EVS row N+4 between time t0 and time t4 (indicating the start of exposure period 1197 for the CIS pixels of the EVS pixels corresponding to EVS row N+4) may be accumulated in the RSDC integration buffer of the deblurring circuit. Fig.11 In the embodiment illustrated in FIG. 1 , the event detected by the EVS pixel of EVS pixel row N at and after time t0 occurs during the exposure period 1197 of the CIS pixel corresponding to the EVS pixel of EVS pixel row N. Therefore, according to the following Fig.10 As discussed in blocks 1005 through 1008 of method 1000 , events detected by EVS pixels of EVS pixel row N may be accumulated in one or more EDI integration buffers of the deblurring circuit (eg, not in an RSDC integration buffer of the deblurring circuit).

[0148] At block 1005, Fig.10 The method 1000 then aligns the CIS pixel data with the corresponding EVS pixel data for an image deblurring operation. In some embodiments, aligning the CIS pixel data with the corresponding EVS pixel data may be performed at least in part using a common control block of the corresponding image sensor. For example, the common control block may synchronize the operation of the row / column control circuitry and / or the deblurring block of the image sensor, such as by using one or more control signals.

[0149] As mentioned above Figure 6 and 9 As discussed, the EDI component of the deblurring circuit may be configured to (a) integrate the events to compute a running total integral of the events detected by the EVS pixels, (b) store the running total in a first integral buffer (e.g., Figure 6 A first integral buffer 674; Fig. 9 (c) exponentially integrating the product of (i) the running sum and (ii) the contrast threshold, and (d) storing the result of the integration in a second integration buffer (e.g., Figure 6 A second integral buffer 678; Fig. 9 Therefore, in order to ensure that the results of the integration of the running sum stored in the first integration buffer of the EDI component of the deblurring circuit and the running sum stored in the second integration buffer correspond only to events detected by the EVS pixel during the exposure period of the CIS pixel corresponding to the EVS pixel, the EDI integration buffer of the deblurring circuit (e.g.: Figure 6 A first integral buffer 674 and a second integral buffer 678; Fig. 9 The first integral buffer 974 and the second integral buffer 978) may be reset before the exposure period of the CIS pixel corresponding to the EVS pixel begins. In some embodiments, the first integral buffer and / or the second integral buffer may be reset simultaneously with the corresponding CIS pixel.

[0150] Aligning the CIS pixel data with the corresponding EVS pixel data at block 1005 may include aligning / synchronizing (a) the timing of the EDI integration buffer reset of the deblurring circuit and (b) the exposure period of one or more rows of corresponding CIS pixels. In some embodiments, aligning the timing of the EDI integration buffer reset of the deblurring circuit with the exposure period may include aligning the exposure periods with each other and / or aligning the exposure periods with the event accumulation in the EDI integration buffer such that the exposure periods and the event accumulation in the EDI integration buffer have the same start time t s Start and / or end at the same time t s+T For example, before the exposure period of the CIS pixel begins, the CIS pixels of one or more CIS pixel rows may be reset at the same time as (a) each other and / or (b) one or more EDI integration buffers. Thus, the exposure period of the CIS pixel and the accumulation of events in the EDI integration buffer may begin at the same time as each other after the CIS pixel and the EDI integration buffer are reset. In addition, assuming that the exposure period and the period during which events are accumulated in the EDI integration buffer have the same duration, aligning the start times of the exposure period and the accumulation of events in the EDI integration buffer with each other may also align their stop times.

[0151] For clarity and understanding of the alignment performed at block 1005 of method 1000, consider Fig.11 and 12 . Fig.12 A timing diagram 1290 is shown illustrating various embodiments of the present technology. Fig.12 , timing diagram 1290 illustrates the Fig.11 4N-3 . In the illustrated embodiment, each of the EVS pixel rows N, N+1, and N+2 corresponds to four of the twelve CIS pixel rows shown. For example, EVS pixel row N corresponds to CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3. In other words, the EVS data captured by the EVS pixels of EVS pixel row N can be used for event-directed deblurring and rolling shutter distortion correction of the CIS data captured by the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3.

[0152] The CIS data captured by the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2 and 4N-3 are read out synchronously on a frame basis after each corresponding exposure period ends. In many active pixel sensors, the CIS data is read out row by row. In such a configuration, different exposure period start and stop times are typically used for different rows. For example, in many active pixel sensors, the CIS pixels of CIS pixel row 4N will have a first exposure period that starts and stops at a different time than the second exposure period for the CIS pixels of CIS pixel row 4N-1. When CIS pixel row 4N and CIS pixel row 4N-1 correspond to the same EVS pixel row, this may be a problem for event-oriented deblurring, because the misalignment between the first exposure period and the second exposure period means that the start and / or stop time of the event accumulation in the EDI integration buffer of the deblurring circuit corresponding to the EVS pixel of the EVS pixel row will be different from the start and / or stop time of the first exposure period and / or the second exposure period. Therefore, the EVS data captured by the EVS pixels of the EVS pixel row will be misaligned with the CIS data captured by the CIS pixels of the CIS pixel row 4N and / or CIS pixel row 4N- 1. This misalignment may affect the accuracy and / or efficacy of event-directed deblurring operations performed on the CIS data and / or require additional storage / processing after data capture and / or readout to align the CIS data with the EVS data.

[0153] To address this issue, at block 1005 of method 1000, the exposure periods of CIS pixel rows corresponding to the same EVS pixel row may be aligned with each other and the event accumulation in the EDI integration buffer of the deblurring circuit. Fig.12, CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 all correspond to EVS pixel row N. Thus, at block 1005, exposure periods 1197 for CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 may be aligned with (a) each other and (b) a portion of an event accumulation period 1198 for EVS row N. The portion of event accumulation period 1198 may correspond to a period in which events detected by EVS pixels of EVS pixel row N are accumulated in an EDI integration buffer of a deblurring circuit (e.g., not in an RSDC integration buffer of a deblurring circuit). Thus, for EVS pixels of EVS row N, exposure periods 1197 for CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 and event accumulation in an EDI integration buffer of a deblurring circuit may each begin at time t0. Additionally, because the duration of exposure period 1197 is the same as the duration of event accumulation in the EDI integration buffer, the alignment of the start time at time t0 can align the end time of exposure period 1197 with the end time of event accumulation in the EDI integration buffer at time t5 for the EVS pixels of EVS row N. In this way, the EVS data captured by the EVS pixels of EVS pixel row N are aligned with the CIS data captured by the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3.

[0154] Continue to refer Fig.11 and 12 , the alignment between the exposure period 1197 and the accumulation of events in the EDI integration buffer can be achieved by resetting the CIS pixels of the CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 simultaneously and / or before the start time t0. Fig.11 , the CIS pixels of the CIS pixel rows 4N, 4N-1, 4N-2 and 4N-3 can each be reset in the reset period 1193 before the start time t0 of the corresponding exposure period 1197.

[0155] In addition, in order to ensure that the deblurring circuit is used to deblur the CIS pixel rows 4N, 4N-1, 4N-2 and 4N-3 ( Fig.12 ) is based only on events detected by the EVS pixels of EVS row N during exposure periods 1197 of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3, and the EDI integration buffers of the deblurring circuits corresponding to the EVS pixels of EVS row N (e.g.: Figure 6 The first integral buffer 674 and / or the second integral buffer 678; Fig. 9The first integral buffer 974 and / or the second integral buffer 978 of the exposure period 1197 may be reset before the start time t0 of the exposure period 1197, for example, (i) between time t-1 and time t0. Fig.11 and / or (ii) at the same time as the CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2 and 4N-3.

[0156] Other EVS pixel rows (e.g., EVS pixel rows N+1, N+2, N+3, N+4, etc.) and other CIS pixel rows (e.g., 4N+1, 4N+2, ..., 4N+8, etc.) may be operated in a similar manner. For example, CIS pixel rows 4N+5, 4N+6, 4N+7, and 4N+8 ( Fig.12 ) can be reset simultaneously with each other and / or in the reset period 1193 before the start time t2 of the exposure period 1197 of these CIS pixels. In addition, the portion of the EDI integration buffer of the deblurring circuit corresponding to the EVS pixel of EVS row N+2 can be reset before the start time t2 of the exposure period 1197, such as (i) between time t1 and time t2. Fig.11 and / or (ii) at the same time as the CIS pixels of CIS pixel rows 4N+5, 4N+6, 4N+7 and 4N+8.

[0157] Reference again Fig.10 , method 1000 may then, at block 1006, (a) capture CIS data using CIS pixels during corresponding exposure periods and (b) capture EVS data using EVS pixels. For example, referring again to Fig.11 , the CIS pixels of the CIS pixel row corresponding to EVS pixel row N may integrate the photogenerated charge during the exposure period 1197 extending from the start time t0 to the stop time t5. At the same time, the EVS pixels of the EVS pixel row N may asynchronously detect events during the aligned portion of the EVS accumulation period 1198 that also extends from the start time t0 to the stop time t5. As another example, the CIS pixels of the CIS pixel row corresponding to EVS pixel row N may integrate the photogenerated charge during the corresponding exposure period 1197 extending from the start time t1 to the stop time t6. At the same time, the EVS pixels of EVS pixel row N+1 may asynchronously detect events during the aligned portion of the EVS accumulation period 1198 that also extends from the start time t1 to the stop time t5. The EVS pixels of EVS pixel row N+1 may also asynchronously detect events during the portion of the EVS accumulation period 1198 extending from the start time t0 of the frame and the start time t1 of the exposure period 1197 of the CIS pixels corresponding to EVS pixel row N+1. According to Fig.10As discussed in blocks 1001 - 1004 of method 1000 , events detected by EVS pixels of EVS pixel row N+1 during the period between time t0 and time t1 may be accumulated in an RSDC integration buffer of a deblurring circuit.

[0158] In some embodiments, the EVS pixels may be selectively enabled to capture EVS data during the corresponding event accumulation period 1198 (eg, selectively enabled to detect events). Fig.11 , the EVS pixel of the EVS pixel row N may be enabled at time t0 (or shortly before) so that the EVS pixel is configured to detect an event occurring during the event accumulation period 1198 between time t0 and time t5. Additionally or alternatively, the EVS pixel of the EVS pixel row N may be disabled at a stop time t5 (or shortly thereafter) of the event accumulation period 1198 and / or at other times outside the corresponding event accumulation period. Selectively disabling the EVS pixel at the stop time t5 (or shortly thereafter) of the corresponding event accumulation period 1198 and / or at other times outside the corresponding event accumulation period of this EVS pixel may save power.

[0159] In other embodiments, the EVS pixels may remain enabled to capture EVS data at times outside of the corresponding event accumulation period. Fig.11 The EVS pixels of EVS pixel row N in may remain activated and enabled to detect events occurring after time t5 corresponding to the end of event accumulation period 1198, such as detecting an event occurring after a time t5 corresponding to the end of event accumulation period 1198. Fig.11 1192. Such events detected by the EVS pixels of EVS pixel row N after time t5 may be: (a) discarded without being accumulated by the deblurring circuitry of the corresponding image sensor; or (b) used in other operations of the image sensor (e.g., video frame interpolation, rolling shutter distortion correction of CIS data of the next image frame).

[0160] exist Fig.10 At block 1007 of the method 1000, the method 1000 then reads out the events detected by the EVS pixels of the event-driven sense array. Block 1007 may be performed when block 1006 is performed. For example, when an EVS pixel detects an event, the event may be read out from the EVS pixel. When an event detected by an EVS pixel is read out from the EVS pixel, the EVS pixel may be reset and thereby enabled to detect subsequent events. In these and other embodiments, reading out the events detected by the EVS pixels may be substantially similar to reading out the events detected by the EVS pixels at block 1003 of the method 1000.

[0161] A single EVS pixel may detect hundreds of events within a single event accumulation period. In some embodiments, each of these events may be read out from the EVS pixel and / or provided to the deblurring circuitry of the corresponding image sensor for accumulation (e.g., in one or more RSDC integration buffers of block 1003, in one or more EDI integration buffers of block 1007). Thus, for a single CIS frame, a relatively large amount of EVS data may be provided to the deblurring circuitry for accumulation and subsequent use in event-directed deblurring and / or rolling shutter distortion correction of the CIS data corresponding to the CIS frame.

[0162] In some cases, this large amount of EVS data can complicate and / or slow down the deblurring and / or rolling shutter distortion correction calculations performed by the deblurring circuitry of the corresponding image sensor, which can be inappropriate or unacceptable for certain applications. Therefore, in some embodiments, the EVS data can be read out from the EVS pixels using a progressive scan readout. More specifically, the image sensor can progressively scan / step through the event-driven sensing array and take a uniform amount of time to read out each EVS pixel row. In this way, the scanning readout can limit the number of EVS readouts per CIS frame, which can simplify and / or speed up the deblurring calculations performed by the deblurring circuitry.

[0163] For the sake of clarity and example, consider Fig.13A and 13B , which illustrate (i) an example event-driven sensing array 1342 and (ii) a corresponding plot 1305 of detected events read out from the event-driven sensing array 1342 for a single CIS frame, respectively. Fig.13A , the event-driven sense array 1342 includes a plurality of EVS pixels 00 to 57 arranged in a plurality of rows Row_0 to Row_5 and a plurality of columns Col_0 to Col_7. EVS pixels 12, 13, 35, 42, 53, 54, and 55 in the sense array 1342 have detected an event.

[0164] To read out the EVS data captured by EVS pixels 00 to 57 of the event-driven sense array 1342, the image sensor may cycle through rows Row_0 to Row_5 of the event-driven sense array 1342, row by row, spending the same amount of time at each row to read out events from the EVS pixels of this row. For example, although none of the EVS pixels 00 to 07 of row Row_0 have detected an event, the image sensor may spend a fixed / preset amount of time (e.g., 50ns) at Row_0 before moving to row Row_1. Then, at row Row_1, the image sensor may spend the same fixed amount of time (e.g., 50ns) to read out events detected by EVS pixels 10 to 17. In the illustrated example, EVS pixels 12, 13, and 15 in row Row_1 have detected an event. Thus, during the fixed / preset amount of time allocated to row Row_1, the image sensor may (i) read out events detected by EVS pixels 12, 13, and 15 and / or (ii) reset EVS pixels 12, 13, and 15 so that they are enabled to detect subsequent events. At the end of the fixed / preset amount of time allocated to row Row_1, the image sensor may move to row Row_2 of the event-driven sense array 1342 to read out events detected by EVS pixels 20 to 27 of row Row_2, if any. Fig. 13B The drawing 1305 shows the Fig.13A The event drives the sensing array 1342 to perform a scan to read out the results.

[0165] The above-described row-by-row scan readout scheme thus limits the total number of EVS readouts per CIS frame and keeps the time required to scan each row in the event-driven sensing array constant within each scan cycle. For example, given that (i) the event-driven sensing array has 2,000 rows of EVS pixels and (ii) the preset amount of time for reading out each row of EVS pixels in the event-driven sensing array is 50 ns, the scanning readout requires 100 μs (2,000 rows of EVS pixels x 50 ns) to scan the entire event-driven sensing array once. Therefore, assuming that the duration of the CIS exposure period is 32 ms, the scanning readout can limit the number of EVS readouts per EVS pixel to 320 (320 ms divided by 100 μs) for each CIS readout. In other words, each CIS frame readout can correspond to a maximum of 320 EVS readouts per EVS pixel. This can limit the amount of EVS data fed to the deblurring circuit, which can simplify the calculations performed by the deblurring circuit and / or speed up the availability of the final results of such calculations.

[0166] In the example of row-by-row scan readout described above, the image sensor spends a fixed / preset amount of time reading out detected events (if any) at each EVS pixel row. As discussed above, this can keep the total time required to scan the entire event-driven sensing array once constant within each cycle of scan readout. In other embodiments, the image sensor can skip EVS pixel rows where no events are detected. For example, referring again to Fig.13A , none of the EVS pixels 20 to 27 of row Row_2 of the event-driven sense array 1342 have detected an event. Therefore, during scanning readout, the image sensor may spend a preset amount of time (e.g., 50 ns) reading out events detected by EVS pixels 12, 13, and 15 of row Row_1, skip row Row_2, and then spend the next preset amount of time (e.g., 50 ns) reading out events detected by EVS pixels 35 of row Row_3. In such embodiments, the total time taken for the scanning readout cycle to pass through the entire event-driven sense array 1342 may vary across cycles depending on which of rows Row_0 to Row_5 include EVS pixels that have detected an event.

[0167] Reference again Fig.10 , method 1000 may then, at block 1008, accumulate, for each EVS pixel of the event-driven sense array, EVS data read out from the EVS pixel during the corresponding event accumulation period. Block 1008 may be performed when blocks 1006 and / or 1007 are performed, as shown by the arrow returning from block 1008 to block 1006. Additionally or alternatively, block 1008 may be performed by an on-chip deblurring circuit of the image sensor.

[0168] As discussed above, accumulating EVS data at block 1008 may include accumulating detected events in an EDI integration buffer of a deblurring circuit for each EVS pixel. For example, accumulating EVS data at block 1008 may include calculating and maintaining a running sum of events for each EVS pixel, for example using a counter (e.g., of a deblurring circuit of a hybrid image sensor) and a first integration buffer. Additionally, accumulating EVS data may include, for each EVS pixel, (a) calculating the product of the running sum and a contrast threshold parameter, (b) determining an exponent of the product, and (c) continuously integrating the exponent over a corresponding EVS accumulation period. The result of the integration for each EVS pixel may be stored in a second integration buffer (e.g., of a deblurring circuit of a hybrid image sensor).

[0169] As discussed above, for each EVS pixel, EVS data is accumulated within a corresponding portion of an event accumulation period. For example, for each EVS pixel, a corresponding portion of the first integration buffer and a corresponding portion of the second integration buffer may be reset before the exposure period of the CIS pixel corresponding to the EVS pixel begins. The reset of the corresponding portions of the first integration buffer and the second integration buffer may reset (i) the running sum stored in the corresponding portion of the first integration buffer and (ii) the integration results stored in the corresponding portion of the second integration buffer. Event accumulation in the first integration buffer and the second integration buffer may then be enabled at the beginning of the exposure period of the CIS pixel corresponding to the EVS pixel, and thereafter disabled at the end of the exposure period, such that the integration results stored in the corresponding portion of the second integration buffer at the end of the exposure period correspond only to events detected by the EVS pixel during the exposure period of the CIS pixel corresponding to the EVS pixel. In some embodiments, once event data has been accumulated, the raw event data may be discarded.

[0170] Reference again Fig.11 For clarity and example, the first integration buffer and the portion of the second integration buffer corresponding to the EVS pixels of EVS pixel row N may be reset during a reset period 1193 prior to the start time t0 of the exposure period 1197 of the CIS pixels of the EVS pixels of EVS pixel row N. At time t0, the event accumulation in the portions of the first integration buffer and the second integration buffer of the deblurring circuit may be enabled for the EVS pixels of EVS pixel row N. When an event is detected by an EVS pixel during the exposure period 1197, the corresponding running count / sum maintained in the corresponding portion of the first integration buffer is updated. In addition, the running sum is multiplied by the contrast threshold, the exponent of the resulting product is calculated, and the exponent is integrated. The integration result of the EVS pixels corresponding to EVS pixel row N is stored to the corresponding portion of the second integration buffer.

[0171] exist Fig.10At box 1009 of method 1000, the CIS data is then read out at the end of the corresponding exposure period. In some embodiments, reading out the CIS data may include reading out the CIS data into a deblurring circuit, such as into a latent frame calculation block of the deblurring circuit. In these and other embodiments, reading out the CIS data may include reading out the CIS data from CIS pixels in a row or a row group. For example, in an embodiment in which multiple CIS pixel rows correspond to the same EVS pixel row, the CIS data captured by the CIS pixels of multiple CIS pixel rows may be read out together / simultaneously at the end of or after the corresponding exposure period. In some embodiments, reading out the CIS data may include skipping the readout of one or more CIS pixel rows and / or columns (for example, for reducing the resolution of the CIS data, matching the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array, and / or reducing the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array. The mismatch between). In these and other embodiments, reading out the CIS data may include merging one or more CIS pixel rows and / or columns (e.g., to reduce the resolution of the CIS data, match the resolution of EVS data captured by the EVS pixels of the event-driven sensing array, and / or reduce the mismatch between the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array). Additional details regarding (a) skipping readout of one or more CIS pixel rows and / or columns and / or (b) merging one or more CIS pixel rows and / or columns during readout are provided in co-application, co-pending and commonly assigned application entitled “METHODS FOR OPERATING HYBRID IMAGE SENSOR SHAVING DIFFERENT CIS-TO-EVS RESOLUTIONS,” which is incorporated herein by reference in its entirety.

[0172] Reference again Fig.11 and 12 For the sake of clarity and example, CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 all correspond to EVS pixel row N and Fig.10 After alignment at block 1005 of FIG. 1 , there is a common exposure period 1197 extending between time t0 and time t5. Thus, at or after time t5, CIS data captured by CIS pixels of CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 can be read out from the CIS pixels simultaneously. This is also done via Fig.11Additionally or alternatively, one or more of the CIS pixel rows 4N, 4N-1, 4N-2, and 4N-3 may be skipped or merged together during readout.

[0173] At block 1010, method 1000 then deblurs the CIS data (read out from the CIS pixel at block 1009) using the accumulated EVS data generated at block 1008 and stored in the corresponding portion of the second integration buffer. In some embodiments, deblurring the CIS data may include combining the CIS data with the accumulated EVS data to calculate one or more latent image frames, such as (a) a start time t corresponding to an exposure period of the CIS pixel; s The latent image frame L(s) of (b) corresponds to one or more other latent image frames L(t) at one or more other times t along the exposure period of the CIS pixels. In some embodiments, combining the CIS data with the accumulated EVS data may include interpolating the EVS data to generate additional EVS data corresponding to additional EVS pixel rows and / or columns (e.g., to increase the resolution of the EVS data, match the resolution of the CIS data captured by the CIS pixels of the CIS pixel array, and / or reduce the mismatch between the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels of the event-driven sensing array). Additional details regarding interpolating EVS data corresponding to additional EVS pixel rows and / or columns are provided in (a) co-application, co-pending and commonly assigned application entitled “METHODS FOROPERATING HYBRID IMAGE SENSORS HAVING DIFFERENT CIS-TO-EVS RESOLUTIONS” and (b) co-application, co-pending and commonly assigned application entitled “HYBRID IMAGE SENSORS WITH VIDEO FRAME INTERPOLATION,” each of which is incorporated herein by reference in its entirety.

[0174] At block 1011, the method 1000 then corrects the rolling shutter distortion of the CIS data using the accumulated EVS data stored in the corresponding portion of the RSDC integration buffer at block 1004. More specifically, the RSDC integration buffer (e.g., Figure 6 An integral buffer 680; Fig. 9The accumulated EVS data in the integration buffer 980a or the integration buffer 980b of block 1006 may be multiplied by a contrast threshold. The deblurring circuit may in turn determine an exponent of the product of the accumulated EVS data and the contrast threshold, and may output the exponent to a latent frame calculation block of the deblurring circuit. Thereafter, the exponent may be used to correct rolling shutter distortion of CIS data (e.g., CIS data captured by CIS pixels at block 1006 and / or deblurred CIS data from block 1010), for example, using equation 20 above to determine a corresponding latent image frame L(0).

[0175] At block 1012, method 1000 then outputs the deblurred and rolling shutter distortion corrected image data from the image sensor. Outputting the deblurred and rolling shutter distortion corrected image data may include outputting the one or more latent image frames calculated at block 1011, e.g., in addition to or in lieu of outputting the raw CIS data read out from the CIS pixels at block 1009 and / or the raw EVS data generated and read out from the EVS pixels at block 1007.

[0176] Fig.11 The timing diagram 1195 provides Fig.10 1000 . For example, the reset period 1196 of the timing diagram 1195 corresponds to the period before the frame starts at time t0. During this time, all EVS pixels or subsets thereof of EVS rows N to N+4 and / or all integration buffers or subsets thereof (e.g., RSDC integration buffers and / or EDI integration buffers) of the deblurring circuit may be reset. The EVS pixels of EVS rows N to N+4 may each detect an event between time t0 (corresponding to the start of the frame) and the end time of the exposure period 1197 of the CIS pixel corresponding to this EVS pixel. The events detected by this EVS pixel during the period between time t0 and the start of the exposure period 1197 of the CIS pixel corresponding to the EVS pixel may be read out and accumulated in the RSDC integration buffer of the deblurring circuit. The events detected by this EVS pixel during the exposure period 1197 of the CIS pixel corresponding to the EVS pixel may be read out and accumulated in one or more EDI integration buffers of the deblurring circuit. Thus, corresponding portions of one or more EDI integration buffers may be reset during corresponding reset periods 1193 prior to the start of an exposure period. Additionally, CIS pixels corresponding to EVS pixels may be reset during reset periods 1193 so that events accumulated in corresponding portions of the EDI integration buffers during the exposure period are aligned with CIS data captured by the CIS pixels during the exposure period. Fig.11The arrow 1199 in shows that at the end of each exposure period 1197, CIS data can be read out from the corresponding CIS pixels in rows, deblurred using the EVS data accumulated in the EDI integration buffer of the deblurring circuit, and corrected for rolling shutter distortion using the EVS data accumulated in the RSDC integration buffer of the deblurring circuit.

[0177] Although blocks 1001 through 1012 of method 1000 are described and illustrated in a particular order, Fig.10 The method 1000 of the present invention is not limited thereto. In other embodiments, all of the blocks 1001 to 1012 of the method 1000 or a subset of one or more thereof may be performed in a different order. In these and other embodiments, all of the blocks 1001 to 1012 or a subset of any thereof may be performed before, during, and / or after all of the other blocks 1001 to 1012 or a subset of any thereof. In addition, it will be readily appreciated by those skilled in the art that the method 1000 may be altered and still remain within these and other embodiments of the present technology. For example, in some embodiments, all of the blocks 1001 to 1012 or a subset of one or more thereof may be omitted and / or repeated.

[0178] As another example, in other embodiments of the present technology, method 1000 may include additional blocks. As a specific example, method 1000 may include a decision block in which method 1000 determines whether a deblur enable signal deblurEN is asserted. If method 1000 determines that deblur enable signal deblurEN is asserted, method 1000 may continue in a manner generally consistent with the discussion of blocks 1001 to 1012 described above. On the other hand, if method 1000 determines that deblur enable signal deblurEN is not asserted, method 1000 reads out CIS data from CIS pixels and EVS data from EVS pixels into an image signal processor and an event signal processor, respectively (e.g., for output from an image sensor via a synchronous communication interface), e.g., instead of reading CIS data and EVS data into a deblur circuit to cause the deblur circuit to calculate deblurred and rolling shutter distortion corrected image data.

[0179] Fig.1414 is a partial schematic diagram illustrating an imaging system 1420 configured in accordance with various embodiments of the present technology. As shown, the imaging system 1420 includes an image sensor 1430 with on-chip event-directed deblurring and rolling shutter distortion correction capabilities. More specifically, the CIS data 1421 and the EVS data 1422 may be aligned / synchronized with each other and the start of a frame, such as using a common control block 1468. Thereafter, as the CIS pixels capture the CIS data 1421, the EVS data 1422 may be read out from the EVS pixels using a row scan readout scheme 1400 and accumulated in a computation block 1471 to produce accumulated event data. The accumulated event data may be stored in a corresponding buffer 1478, such as an RSDC integration buffer, a first EDI integration buffer, and / or a second EDI integration buffer. The CIS data 1421 may then be read out from the CIS pixels row by row or in groups of rows at the end of corresponding exposure periods, streamed into latent frame computation block 1472, deblurred using corresponding portions of the accumulated event data stored in buffer 1478, and corrected for rolling shutter distortion using other corresponding portions of the accumulated event data stored in buffer 1478. The deblurred and rolling shutter distortion corrected image data may then be output from image sensor 1430, such as to one or more downstream components of imaging system 1420.

[0180] vs. performing off-chip image deblurring Figure 3 Compared with the imaging system 320, Fig.14 The imaging system 1420 of the present invention provides several advantages. For example, instead of approximately 550MB of buffer space, the imaging system 1420 can use approximately 12MB to 17MB of buffer space (depending on the number of RSDC component groups used in the deblurring circuit) to perform image deblurring and rolling shutter distortion correction. In addition, the CIS data 1421 and the EVS data 1422 do not need to be output from the image sensor 1430 of the imaging system 1420. Specifically, the imaging system 1420 can output deblurred and rolling shutter distortion corrected image data, which represents a significant reduction in required IO bandwidth / throughput and (therefore) power consumption over the imaging system 320. Moreover, because the imaging system 1420 can perform the deblurring calculations on the image sensor 1430, many of the delays that exist in the imaging system 320 can be reduced / eliminated in the imaging system 1420, which means that in addition to processing still images, the imaging system 1420 can also support real-time video. Furthermore, because imaging system 1420 outputs deblurred image data, the interface between image sensor 1430 of imaging system 1420 and downstream components is relatively simple and easy to work with, especially compared to the interface required for imaging system 320 .

[0181] C. in conclusion

[0182] The above detailed description of the embodiments of the present technology is not intended to be exhaustive or to limit the present technology to the precise form disclosed above. Although specific embodiments and examples of the present technology are described above for illustrative purposes, it should be recognized by those skilled in the relevant art that various equivalent modifications may be within the scope of the present technology. For example, although the steps are presented in the order given above, alternative embodiments may perform the steps in a different order. In addition, the various embodiments described herein may also be combined to provide additional embodiments.

[0183] It should be understood from the foregoing that the specific embodiments of the present technology have been described herein for illustrative purposes, but the well-known structures and functions are not shown or described in detail so as not to make the description of the embodiments of the present technology unclear unnecessarily. If any material incorporated herein by reference conflicts with the present disclosure, the present disclosure shall prevail. Where the context permits, singular or plural terms may also include plural or singular terms, respectively. In addition, unless the word "or" is clearly limited to mean excluding only a single item of other items in a list of two or more items, the use of "or" in this list should be interpreted as including any single item in the list of (a), all items in the list of (b) or any combination of items in the list of (c). In addition, as used herein, the phrase "and / or" as in "A and / or B" relates to only A, only B and both A and B. In addition, the use of the terms "including", "comprising", "having" and "having" throughout the text means including at least the stated features to not exclude any greater number of identical features and / or other features of additional types. In addition, as used herein, the phrases "based on", "depending on", "due to", and "in response to" should not be interpreted as referring to a closed set of conditions. For example, without departing from the scope of the present disclosure, the exemplary steps described as "based on condition A" may be based on both condition A and condition B. In other words, as used herein, the phrase "based on" should be interpreted in the same manner as the phrase "based at least in part on" or the phrase "based at least in part on". Moreover, the terms "connected" and "coupled" are used interchangeably herein and refer to both direct and indirect connections or couplings. For example, where the context permits, element A "connected" or "coupled" to element B may refer to (i) A directly "connected" or directly "coupled" to B and / or (ii) A indirectly "connected" or indirectly "coupled" to B.

[0184] It should also be understood from the foregoing that various modifications can be made without departing from the present disclosure or the present technology. For example, it should be understood by those of ordinary skill in the art that the various components of the present technology can be further divided into sub-components, or the various components and functions of the present technology can be combined and integrated. In addition, certain aspects of the present technology described in the context of a specific embodiment may also be combined or eliminated in other embodiments. In addition, although the advantages associated with certain embodiments of the present technology have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily need to exhibit such advantages to fall within the scope of the present technology. Therefore, the present disclosure and associated technology may encompass other embodiments that are not explicitly shown or described herein.

Claims

1. An image sensor, comprising: an event-driven sensing array comprising one or more event vision sensor pixels, wherein each event vision sensor pixel of the one or more event vision sensor pixels is configured to capture event data corresponding to contrast information of light incident on the event vision sensor pixel; a pixel array comprising a plurality of CMOS image sensor pixels arranged in one or more CMOS image sensor pixel rows, wherein each CMOS image sensor pixel of the plurality of CMOS image sensor pixels is configured to capture CMOS image sensor data corresponding to an intensity of light incident on the CMOS image sensor pixel; a rolling shutter distortion correction circuit configured to generate rolling shutter distortion corrected image data by correcting rolling shutter distortion of the complementary metal oxide semiconductor image sensor data captured by the plurality of complementary metal oxide semiconductor image sensor pixels using the event data captured by the one or more event vision sensor pixels; and A physical interface is operable to output the rolling shutter distortion corrected image data from the image sensor.

2. The image sensor of claim 1 , wherein the rolling shutter distortion correction circuit comprises a counter operable to calculate a running sum of events detected in event data captured by an event vision sensor pixel during a period extending from the beginning of a frame to the beginning of an exposure period of one or more of the plurality of complementary metal oxide semiconductor image sensor pixels corresponding to an event vision sensor pixel of the one or more event vision sensor pixels. The image sensor of claim 2 , wherein the counter comprises an integer counter. The image sensor of claim 3 , wherein the integer counter comprises an up / down counter.

5. The image sensor of claim 2, wherein the counter is clocked by events in the event data captured by the event vision sensor pixels during the period extending from the start of the frame to the start of the exposure period.

6. The image sensor of claim 2, wherein the rolling shutter distortion correction circuit further comprises a buffer configured to store the running sum.

7. The image sensor of claim 2, wherein the rolling shutter distortion correction circuit further comprises a product calculation block configured to calculate a product of the running sum and a contrast threshold parameter.

8. The image sensor of claim 7, wherein the rolling shutter distortion correction circuit further comprises an exponent calculation block configured to calculate an exponent of the product.

9. The image sensor of claim 1, wherein the rolling shutter distortion correction circuit comprises a ping-pong buffer.

10. The image sensor of claim 9, wherein the ping-pong buffer comprises: a routing switch; a first integrating buffer having an input coupled to the first output of the routing switch; a second integrating buffer having an input coupled to the second output of the routing switch; and a multiplexer having a first input coupled to the output of the first integrating buffer and a second input coupled to the output of the second integrating buffer.

11. The image sensor of claim 10, wherein the rolling shutter distortion correction circuit further comprises: a first counter coupled between the first output of the routing switch and the input of the first integration buffer; and a second counter coupled between the second output of the routing switch and the input of the second integration buffer.

12. The image sensor of claim 1 , wherein the rolling shutter distortion correction circuit is a deblurring and rolling shutter distortion correction circuit, and wherein the deblurring and rolling shutter distortion correction circuit is further configured to generate deblurred image data by deblurring the complementary metal oxide semiconductor image sensor data captured by the plurality of complementary metal oxide semiconductor image sensor pixels using the event data captured by the one or more event vision sensor pixels.

13. The image sensor of claim 12 , wherein the deblurring and rolling shutter distortion correction circuit comprises: (a) a counter operable to calculate a running sum of events detected in event data captured by one or more of the plurality of complementary metal oxide semiconductor image sensor pixels corresponding to an event visual sensor pixel in the one or more event visual sensor pixels during an exposure period of the one or more complementary metal oxide semiconductor image sensor pixels; and (b) an integration buffer configured to store the running sum.

14. The image sensor of claim 13 , wherein the integration buffer is a first integration buffer, and wherein the deblurring and rolling shutter distortion correction circuit further comprises: (a) a product calculation block configured to calculate a product of the running sum and a contrast threshold; (b) an exponent calculation block configured to calculate an exponent of the product; and (c) an integral calculation block configured to calculate an integral of the exponent of the product over time.

15. The image sensor of claim 1, wherein the rolling shutter distortion correction circuit comprises a latent frame calculation block, and wherein the rolling shutter distortion corrected image data comprises a latent frame of the complementary metal oxide semiconductor image sensor data corresponding to the beginning of an image frame.

16. An image sensor comprising: an event-driven sensing array comprising one or more event vision sensor pixels, wherein each event vision sensor pixel of the one or more event vision sensor pixels is configured to capture event data corresponding to contrast information of light incident on the event vision sensor pixel; a pixel array comprising a plurality of CMOS image sensor pixels, wherein each CMOS image sensor pixel of the plurality of CMOS image sensor pixels is configured to capture CMOS image sensor data corresponding to an intensity of light incident on the CMOS image sensor pixel; and A rolling shutter distortion correction circuit comprising— a counter operable to calculate, for each of the one or more event vision sensor pixels, a running sum of events detected by the event vision sensor pixel during a period extending from the beginning of an image frame to the beginning of an exposure period of one or more of the plurality of complementary metal oxide semiconductor image sensor pixels corresponding to the event vision sensor pixel, and An integration buffer is configured to store, for each event vision sensor pixel of the one or more event vision sensor pixels, the running sum corresponding to the event vision sensor pixel.

17. The image sensor according to claim 16, wherein: The counter is a first counter, the integration buffer is a first integration buffer, the period is a first period, the image frame is a first image frame, and the exposure period is a first exposure period; and The rolling shutter distortion correction circuit further comprises— A ping-pong buffer having a routing switch, the first integral buffer, a second integral buffer and a multiplexer; and A second counter which can be used to calculate, for each of the one or more event vision sensor pixels, a running sum of events detected by the event vision sensor pixel during a second time period extending from the beginning of a second image frame to the beginning of a second exposure period of the complementary metal oxide semiconductor image sensor pixel of the plurality of complementary metal oxide semiconductor image sensor pixels corresponding to the event vision sensor pixel.

18. The image sensor of claim 16, wherein the rolling shutter distortion correction circuit further comprises— a product calculation block configured to calculate, for each of the one or more event visual sensor pixels, a product of (i) the running sum corresponding to the event visual sensor pixel and (ii) a contrast threshold parameter; and An exponent calculation block is configured to calculate, for each event vision sensor pixel of the one or more event vision sensor pixels, an exponent corresponding to the product of the event vision sensor pixel.

19. The image sensor according to claim 16, wherein: The rolling shutter distortion correction circuit is a deblurring and rolling shutter distortion correction circuit; The counter is a first counter and the integration buffer is a first integration buffer; and The deblurring and rolling shutter distortion correction circuit further comprises— a second counter operable to calculate, for each of the one or more event vision sensor pixels, a running sum of events detected by the event vision sensor pixel during the exposure period of the one or more of the plurality of complementary metal oxide semiconductor image sensor pixels corresponding to the event vision sensor pixel, a second integration buffer configured to store, for each of the one or more event vision sensor pixels, the running sum calculated by the second counter corresponding to the event vision sensor pixel, a product calculation block configured to calculate, for each of the one or more event visual sensor pixels, a product of (i) the running sum calculated by the second counter corresponding to the event visual sensor pixel and (ii) a contrast threshold parameter; and An exponent calculation block is configured to calculate, for each event vision sensor pixel of the one or more event vision sensor pixels, an exponent corresponding to the product of the event vision sensor pixel.

20. A method of operating an image sensor, the method comprising: capturing CMOS image sensor data using CMOS image sensor pixels in one or more CMOS image sensor pixel rows of a pixel array of the image sensor during an exposure period corresponding to an image frame, wherein the CMOS image sensor data corresponds to an intensity of light incident on the CMOS image sensor pixels during the exposure period; capturing event vision sensor data using one or more event vision sensor pixels of an event-driven sensing array of the image sensor during a period extending between the start of the image frame and the start of the exposure period, wherein the event vision sensor data comprises events detected by the one or more event vision sensor pixels during the period, and wherein each event represents a temporal contrast of light incident on the one or more event vision sensor pixels exceeding a threshold; and The complementary metal oxide semiconductor image sensor data is corrected for rolling shutter distortion (i) based on the event vision sensor data and (ii) within the image sensor using rolling shutter distortion correction circuitry of the image sensor.

21. The method of claim 20, wherein the time period is a first time period, and wherein the method further comprises resetting the one or more event vision sensor pixels during a second time period immediately preceding the first time period.

22. The method of claim 20, wherein the time period is a first time period, and wherein the method further comprises resetting one or more rolling shutter distortion correction integration buffers of the rolling shutter distortion correction circuit within a second time period immediately preceding the first time period, such that one or more values ​​corresponding to the one or more event visual sensor pixels and stored in the one or more rolling shutter distortion correction integration buffers are in a reset state at the beginning of the image frame.

23. The method of claim 20, further comprising calculating, for each of the one or more event vision sensor pixels, a running sum of events detected by the event vision sensor pixel over the time period.

24. The method of claim 23, further comprising calculating the product of the running sum and a contrast threshold parameter for each of the one or more event visual sensor pixels.

25. The method of claim 24, further comprising calculating an exponent of the product for each of the one or more event vision sensor pixels.

26. The method of claim 20, wherein: The event visual sensor data is first event visual sensor data; The method further includes capturing second event vision sensor data using the one or more event vision sensor pixels during the exposure period; the second event vision sensor data comprising events detected by the one or more event vision sensor pixels during the exposure period; and Each event in the second event vision sensor data represents a temporal contrast of light incident on the one or more event vision sensor pixels exceeding the threshold.

27. The method of claim 26, wherein: The period is a first period; The rolling shutter distortion correction circuit is a deblurring and rolling shutter distortion correction circuit; and The method further includes resetting one or more event double-integration integration buffers of the deblurring and rolling shutter distortion correction circuit during the first time period and in a second time period immediately before the start of the exposure period, so that one or more values ​​corresponding to the one or more event visual sensor pixels and stored in the one or more event double-integration integration buffers are in a reset state at the start of the exposure period.

28. The method of claim 27, further comprising resetting the complementary metal oxide semiconductor image sensor pixels at the same timing as resetting the one or more event double integration buffers.

29. The method of claim 26, further comprising: calculating, for each event vision sensor pixel of the one or more event vision sensor pixels, a running sum of events detected by the event vision sensor pixel during an exposure period; calculating, for each event vision sensor pixel of the one or more event vision sensor pixels, a product of the running sum and a contrast threshold parameter; calculating an exponent of the product for each event vision sensor pixel of the one or more event vision sensor pixels; and An integral of the index over the exposure period is calculated for each of the one or more event vision sensor pixels.

30. The method of claim 26, wherein: The rolling shutter distortion correction circuit is a deblurring and rolling shutter distortion correction circuit; The method further includes generating deblurred image data using the deblurring and rolling shutter distortion correction circuit, wherein generating the deblurred image data includes (i) being based on the second event vision sensor data and (ii) deblurring the complementary metal oxide semiconductor image sensor data internally within the image sensor; Correcting the CMOS image sensor data for rolling shutter distortion includes calculating a latent frame included in the CMOS image sensor data; and The latent frame corresponds to the intensity of light incident on the complementary metal oxide semiconductor image sensor pixel at the beginning of the image frame.

31. The method of claim 20, further comprising reading out the event vision sensor data from the one or more event vision sensor pixels as part of a readout operation that cyclically scans the event-driven sensing array on a row-by-row basis.

32. The method of claim 31, wherein the readout operation is configured to spend a preset amount of time at each event vision sensor pixel row of the event-driven sensing array to read out corresponding event vision sensor data.

33. A method according to claim 31, wherein the readout operation is configured to: (a) spend a preset amount of time on each event vision sensor pixel row of the event-driven sensing array having at least one event vision sensor pixel storing unread event vision sensor data; and (b) skip event vision sensor pixel rows in which no event vision sensor pixels store unread event vision sensor data.

Images