Method for operating hybrid image sensor with different CIS and EVS resolutions

Abstract

Methods (and associated systems, devices, and methods) for operating hybrid image sensors with different CIS and EVS resolutions are disclosed herein. In one embodiment, an imaging system includes a hybrid image sensor including: (a) an event driven sensing array including one or more event vision sensor (EVS) pixels arranged in one or more rows of EVS pixels and configured to capture EVS data having an EVS resolution; and (b) a pixel array comprising a plurality of CMOS image sensor (CIS) pixels arranged in one or more CMOS CIS pixel rows and configured to capture CIS data having a CIS resolution. The imaging system may further include control circuitry configured to adjust the CIS resolution of the CIS data and / or the EVS resolution of the EVS data such that a mismatch between the CIS resolution and the EVS resolution is reduced.

Description

[0001] CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 597,638, filed on November 9, 2023, which is incorporated herein by reference in its entirety.

[0003] This application contains subject matter related to commonly-applied, co-pending and commonly assigned U.S. Patent Application No. ______, filed by ______ and entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR,” which is incorporated herein by reference in its entirety.

[0004] This application contains subject matter related to co-applied, co-pending and commonly assigned U.S. Patent Application No. ______, filed on ______ and entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR AND ROLLINGSHUTTER DISTORTION CORRECTION,” which is incorporated herein by reference in its entirety.

[0005] This application contains subject matter related to commonly-applied, co-pending and commonly-assigned U.S. Patent Application No. ______, filed by ______ and entitled “HYBRIDIMAGE SENSORS WITH VIDEO FRAME INTERPOLATION,” which is incorporated herein by reference in its entirety.

[0006] This application contains subject matter related to commonly-applied, co-pending and commonly assigned U.S. Patent Application No. ______, filed by ______ and entitled “HYBRID IMAGE SENSORS WITH ADJUSTABLE CONTRAST THRESHOLDS,” which is incorporated herein by reference in its entirety. Technical Field

[0007] The present disclosure relates generally to image sensors. For example, several embodiments of the present technology relate to hybrid image sensors having active pixels (e.g., CMOS image sensor (CIS) pixels) and event vision sensor (EVS) pixels and associated methods of operating such hybrid image sensors to address different CIS and EVS resolutions. Background Art

[0008] Image sensors have become ubiquitous and are now widely used in digital cameras, cell phones, video cameras, as well as in medical, automotive and other applications. As image sensors are integrated into a wider range of electronic devices, it is desirable to enhance their functionality, performance metrics, and the like in as many ways as possible (e.g., resolution, power consumption, dynamic range, etc.) both through device architecture design and image acquisition processing.

[0009] A typical image sensor operates in response to image light from an external scene being incident on the image sensor. The image sensor includes an array of pixels having photosensitive elements (e.g., photodiodes) that absorb a portion of the incident image light and generate image charges after absorbing the image light. The image charge generated by the pixel light can be measured as an analog output image signal on a column bit line, which varies depending on the incident image light. In other words, the amount of image charge generated is proportional to the intensity of the image light, and the image light is read out as an analog image signal from the column bit line and converted into a digital value to provide information representing the external scene. Summary of the invention

[0010] One aspect of the present application relates to a method of operating an imaging system including a hybrid image sensor, the method comprising: adjusting a first resolution associated with CIS pixel data captured using multiple CMOS image sensor (CIS) pixels of the hybrid image sensor, wherein adjusting the first resolution comprises adjusting the first resolution to reduce a mismatch between the first resolution and a second resolution associated with EVS pixel data captured using one or more EVS pixels of the hybrid image sensor, and wherein the one or more EVS pixels correspond to the multiple CIS pixels; or adjusting the second resolution, wherein adjusting the second resolution comprises adjusting the second resolution to reduce the mismatch between the first resolution and the second resolution.

[0011] Another aspect of the present application relates to an imaging system comprising: a hybrid image sensor comprising an event-driven sensing array comprising one or more event vision sensor (EVS) pixel rows, wherein each of the one or more EVS pixels is configured to capture EVS data corresponding to contrast information of light incident on the EVS pixel, and wherein the EVS data captured using the one or more EVS pixels of the event-driven sensing array has an EVS resolution; and a pixel array comprising a plurality of CMOS image sensor (CIS) pixels arranged into one or more CIS pixel rows, wherein each of the plurality of CIS pixels is configured to capture CIS data corresponding to the intensity of light incident on the CIS pixel, and wherein the CIS data captured using the plurality of CIS pixels of the pixel array has a CIS resolution; and a control circuit system configured to adjust the CIS resolution of the CIS data and / or the EVS resolution of the EVS data so that the mismatch between the CIS resolution and the EVS resolution is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Non-limiting and non-exhaustive embodiments of the present technology are described below with reference to the following figures, wherein like or similar reference numerals are used to refer to all like or similar components unless otherwise specified.

[0013] Figure 1A is a partial schematic diagram of a stacked hybrid complementary metal oxide semiconductor (CMOS) image sensor (CIS) and event-based vision sensor (EVS) system configured in accordance with various embodiments of the present technology.

[0014] Figure 1B yes Figure 1A A partial schematic diagram of a specific example of a system.

[0015] Figure 1C is a partial schematic diagram of a 4x4 pixel cluster configured in accordance with various embodiments of the present technology.

[0016] Figure 2 is a partial schematic diagram of an EVS pixel configured according to various embodiments of the present technology.

[0017] Figure 3 is a table illustrating example CIS and EVS pixel row correspondences according to various embodiments of the present technology.

[0018] Figure 4 is a flow chart illustrating a method of operating an imaging system including a hybrid image sensor in accordance with various embodiments of the present technology.

[0019] Figures 5A to 5Cis a partial schematic diagram illustrating a method for reducing or eliminating a resolution mismatch between a CIS and an EVS according to various embodiments of the present technology.

[0020] Figures 6A to 6C is a partial schematic diagram illustrating another method for reducing or eliminating the mismatch between CIS and EVS resolutions according to various embodiments of the present technology.

[0021] Figures 7A to 7B is a partial schematic diagram illustrating yet another method for reducing or eliminating the mismatch between the CIS and EVS resolutions according to various embodiments of the present technology.

[0022] Figure 8 is a partial schematic diagram illustrating yet another method for reducing or eliminating the mismatch between the CIS and EVS resolutions according to various embodiments of the present technology.

[0023] Those skilled in the art will appreciate that the elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the figures may be exaggerated relative to other elements to aid in understanding various aspects of the present technology. In addition, common but well-known elements or methods that are useful or required in commercially feasible embodiments are generally not depicted in the figures or described in detail below so as not to unnecessarily obscure the description of various aspects of the present technology. DETAILED DESCRIPTION

[0024] The present disclosure relates to hybrid image sensors and associated systems, devices, and methods. For example, several embodiments of the present technology described in detail below relate to hybrid image sensors having active pixels (e.g., CMOS image sensor (CIS) pixels) and event vision sensor (EVS) pixels and associated methods of operating such hybrid image sensors to address different CIS and EVS resolutions. In the following description, specific details are stated to provide a thorough understanding of aspects of the present technology. However, those skilled in the relevant art should recognize that the systems, devices, and techniques described herein may be practiced without one or more of the specific details stated herein or with other methods, components, materials, etc.

[0025] Reference throughout this specification to an "example" or "embodiment" means that a particular feature, structure, or characteristic described in connection with the example or embodiment is included in at least one example or embodiment of the present technology. Thus, the use of the phrases "for example," "as an example," or "embodiment" herein does not necessarily all refer to the same example or embodiment and is not necessarily limited to the particular example or embodiment discussed. Furthermore, the features, structures, or characteristics of the present technology described herein may be combined in any suitable manner to provide additional examples or embodiments of the present technology.

[0026] For ease of description, spatially relative terms (e.g., "below," "below," "above," "below," "above," "up," "top," "bottom," "left," "right," "center," "middle," and the like) may be used herein to describe the relationship of one element or feature relative to one or more other elements or features, as illustrated in the figures. It should be understood that, in addition to the orientation depicted in the figures, spatially relative terms are intended to cover different orientations of the device or system in use or operation. For example, if the device or system illustrated in the figures is rotated, turned, or flipped around a horizontal axis, an element or feature described as being "below" or "below" or "below" one or more other elements or features may be oriented to be "above" one or more other elements or features. Therefore, the exemplary terms "below" or "below" are non-limiting and may cover both above and below orientations. The device or system may be additionally or alternatively oriented in a manner other than that illustrated in the figures (e.g., rotated 90 degrees around a vertical axis or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly. In addition, it will also be understood that when an element is referred to as being “between” two other elements, it can be the only element between the other two elements, or one or more intervening elements may also be present.

[0027] Throughout this specification, several terms of the art are used. These terms have their ordinary meaning in the art, unless specifically defined herein or the context of their use clearly indicates otherwise. It should be noted that element names and symbols (e.g., Si vs. silicon) are used interchangeably throughout this document; however, both have the same meaning.

[0028] A. Overview

[0029] The active pixel sensor employs a pixel array for capturing an intensity image / video of an external scene. More specifically, the pixels are used to obtain CIS information (e.g., intensity information) corresponding to light from the external scene incident on the pixels. The CIS information obtained during an integration period is read out at the end of the integration period and used to generate a corresponding intensity image of the external scene.

[0030] The pixels of an active pixel sensor typically have a globally defined integration time. Thus, the pixels in an array of an active pixel sensor typically have the same integration time, and each pixel in the array is typically converted into a digital signal regardless of its content (e.g., whether there have been changes in the external scene captured by the pixel since the pixel was last read out). Thus, relatively large amounts of memory and power are required to operate an active pixel sensor at a high frame rate. Therefore, due in part to memory and power constraints, it is difficult to use active pixel sensors alone to obtain intensity images / videos of an external scene at ultra-high frame rates.

[0031] Furthermore, when motion or other changes occur in the external scene during the integration period, motion artifacts may be observed as blur in the resulting intensity image of the external scene. Blurring may be particularly prominent in low light conditions where longer exposure times are used. Thus, active pixel image sensors are not inherently good at obtaining sharp intensity images / videos of highly dynamic scenes.

[0032] In contrast, an event vision sensor (e.g., an event-driven sensor or a dynamic vision sensor) employs EVS pixels that can be used to obtain non-CIS information (e.g., contrast information, intensity changes, event data) corresponding to light from an external scene incident on those EVS pixels. The event vision sensor reads out the EVS pixel and / or converts the corresponding pixel signal into a digital signal only when the EVS pixel detects a change in the external scene (e.g., an event). In other words, the EVS pixels of the event vision sensor that have not detected a change in the external scene are not read out, and / or the pixel signals corresponding to such EVS pixels are not converted into digital signals (thereby saving power). Therefore, each EVS pixel of the event vision sensor can be independent of the other EVS pixels of the event vision sensor, and only the EVS pixels that detect a change in the external scene need to be read out and / or have their corresponding pixel signals converted into digital signals. Therefore, unlike active pixel sensors with synchronized integration times, event vision sensors are not affected by limited dynamic range and can accurately capture high-speed motion. Therefore, event vision sensors are generally more robust than active pixel sensors in weak lighting conditions and / or highly dynamic scenes because they are not affected by under / overexposure or motion blur associated with synchronized shutters. In other words, event vision sensors can be used to provide ultra-high frame rates and accurately capture high-speed motion.

[0033] A hybrid image sensor employs a pixel array comprising a combination of: (i) active (CIS) pixels, which can be used to obtain CIS information corresponding to light from an external scene; and (ii) EVS pixels, which can be used to obtain non-CIS information corresponding to light from an external scene. Such a hybrid image sensor is thus capable of simultaneously capturing (a) an intensity image / video of an external scene and (b) events occurring within the external scene. The event data captured by the EVS pixels can be used to address / mitigate (i) the low frame rate intensity image problem discussed above and (ii) the blurring effects inherent in intensity images captured using CIS pixels in the presence of motion. For example, using an event-based double integral (EDI) model, a high frame rate intensity image / video of an external scene can be reconstructed from a single (e.g., blurred) intensity image and its event sequence. A description of how the EDI model and EVS pixels can be used to provide (e.g., on-chip) event-directed deblurring and / or rolling shutter distortion correction is provided in: (1) co-application, co-pending and commonly assigned application entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR” and (2) co-application, co-pending and commonly assigned application entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR AND ROLLING SHUTTER DISTORTION CORRECTION”, each of which has been incorporated herein by reference in its entirety above.

[0034] Hybrid image sensors typically use a different number of EVS pixels than CIS pixels. More specifically, in a pixel array within a hybrid image sensor, only a small subset of pixels are typically used as EVS pixels to obtain non-CIS information corresponding to light from an external scene, while the remaining pixels are typically used as CIS pixels to obtain CIS information corresponding to light from an external scene. This is because using more pixels as EVS pixels often results in a degradation of CIS image quality.

[0035] As a specific example, consider a 4x4 pixel array. A hybrid image sensor may employ one of the pixels as an EVS pixel, and the remaining pixels as CIS pixels. In this example, when the EVS pixel is used to obtain non-CIS information of the remaining pixels in the 4x4 array (e.g., for performing image deblurring, rolling shutter distortion correction, video interpolation, etc.), the CIS and EVS pixel resolutions are 15:1. In this example, the CIS and EVS resolutions may alternatively be represented by pixel rows or pixel columns. More specifically, an EVS pixel is positioned in a single pixel row and a single pixel column of a 4x4 array and may be used to obtain non-CIS information of 4 rows of CIS pixels and 4 columns of CIS pixels. Therefore, the CIS and EVS resolutions may be represented as 4:1 CIS and EVS pixel row resolutions or 4:1 CIS and EVS pixel column resolutions. This large CIS and EVS resolution mismatch may complicate downstream calculations (e.g., image deblurring calculations, rolling shutter distortion calculations, etc.) and / or cause the accuracy of these calculations to be lower than expected.

[0036] Furthermore, continuing with the 4x4 pixel array example above, the 4x4 pixel array may include CIS pixels corresponding to different colors. For example, a first subset of CIS pixels may correspond to red, a second subset of CIS pixels may correspond to green, a third subset of CIS pixels may correspond to blue, and / or a fourth subset of CIS pixels may correspond to transparent (or monochrome, where light of any color is permitted to illuminate the photodiode corresponding to the CIS pixel). In some examples, the EVS pixels may correspond to transparent (or monochrome). In such examples, when the non-CIS information obtained by the EVS pixels is used for the remaining CIS pixels of the 4x4 array, it is assumed that the event is triggered in each color to which these remaining CIS pixels correspond (this is not true in all cases).

[0037] In other examples, the EVS pixel may correspond to only one color (e.g., red, green, or blue). In these examples, when the non-CIS information obtained by the EVS pixel is used for the remaining CIS pixels of the 4x4 array, the non-CIS information corresponding to one color may be used for CIS pixels corresponding to a different color (which may result in inaccuracies). In other examples, the 4x4 array may include more than one EVS pixel corresponding to different colors from each other (e.g., red, green, blue, or transparent). In these examples, the non-CIS information obtained by the EVS pixel corresponds to more than one color, which may result in inaccuracies when used for the remaining CIS pixels of the 4x4 array, which may correspond to the same or different colors.

[0038] To address the above-mentioned problems, the present technology generally relates to hybrid image sensors having active pixels (e.g., CMOS image sensor (CIS) pixels) and event vision sensor (EVS) pixels and associated methods of operating such hybrid image sensors to address different CIS and EVS resolutions. For example, several embodiments of the present technology described in detail below relate to various methods of reducing or eliminating CIS and EVS resolution mismatches. In one example, an EVS pixel or EVS pixel row may correspond to a transparent (or operated as a monochrome EVS pixel). Continuing with this example, CIS information obtained by a CIS pixel corresponding to a particular color (e.g., red, green, or blue) and / or CIS information obtained by a CIS pixel corresponding to a transparent (or operated as a monochrome CIS pixel) may be (i) read out from the array by skipping the readout of one or more CIS pixel rows and / or one or more CIS pixel columns and (ii) combined with non-CIS information obtained by the EVS pixel / EVS pixel row (e.g., as part of an image deblurring calculation, rolling shutter distortion calculation, etc.).

[0039] In another example, an EVS pixel or EVS pixel row may correspond to red, green, blue, or another color other than transparent. Continuing with this example, CIS information obtained only by CIS pixels corresponding to such a particular color (e.g., red, blue, green, or another color other than transparent) may be (i) read out from the array by skipping the readout of one or more CIS pixel rows and / or one or more CIS pixel columns and (ii) combined with non-CIS information obtained by the EVS pixel / EVS pixel row (e.g., as part of an image deblurring calculation, rolling shutter distortion calculation, etc.). In other words, in this example, the color channels between the CIS pixel and the EVS pixel may be matched to each other during the readout of the CIS pixel.

[0040] In yet another example, an EVS pixel or EVS pixel row may correspond to a transparent (or operated as a monochrome EVS pixel). Continuing with this example, CIS information obtained by CIS pixels corresponding to one or more colors (e.g., red, blue, green, transparent) may be merged vertically across CIS pixel rows and / or horizontally across CIS pixel columns to reduce or eliminate CIS and EVS pixel resolution mismatches, CIS and EVS pixel row resolution mismatches, and / or CIS and EVS pixel column resolution mismatches.

[0041] In yet another example, an EVS pixel or EVS pixel row may correspond to red, green, or blue (or another color other than transparent / monochrome). Continuing with this example, CIS information obtained by CIS pixels corresponding to such a particular color (e.g., red, blue, green, or another color other than transparent / monochrome) may be merged vertically across CIS pixel rows and / or horizontally across CIS pixel columns to (a) reduce or eliminate CIS and EVS pixel resolution mismatches, CIS and EVS pixel row resolution mismatches, and / or CIS and EVS pixel column resolution mismatches and (b) match color channels between CIS pixels and EVS pixels.

[0042] In yet another example, non-CIS information for additional EVS pixel rows and / or additional EVS pixel columns may be interpolated using non-CIS information obtained from EVS pixels of one or more EVS pixel rows and / or EVS pixel columns to reduce or eliminate CIS and EVS pixel resolution mismatches, CIS and EVS pixel row resolution mismatches, and / or CIS and EVS pixel column resolution mismatches. As a specific example, two EVS pixel rows may each include fewer EVS pixel columns than CIS pixel columns included in one or more corresponding CIS pixel rows. In such examples, interpolation may be used on non-CIS information obtained from EVS pixels of two EVS pixel rows to reduce or eliminate CIS and EVS pixel resolution mismatches, CIS and EVS pixel row resolution mismatches, and / or CIS and EVS pixel column resolution mismatches.

[0043] Additional details regarding hybrid image sensors configured according to various embodiments of the present technology and various methods of reducing or eliminating CIS and EVS resolution mismatches are provided below. Figures 1A to 8 supply.

[0044] B. Selected embodiments of hybrid image sensors and associated methods of operation

[0045] Figure 1A1 is a partial schematic diagram of a stacked complementary metal oxide semiconductor (CMOS) image sensor (CIS) having an event-based vision sensor (EVS) system 130 ("stacked system 130" or "image sensor 130") configured in accordance with various embodiments of the present technology. As shown, the stacked system 130 includes a first die 132, a second die 134, and a third die 136 stacked and coupled together in a stacked chip scheme. In some embodiments, the first die 132, the second die 134, and the third die 136 are semiconductor dies including suitable semiconductor materials, such as silicon. In the illustrated embodiment, the first die 132 (also referred to herein as the "top die") includes a pixel array 138. The third die 136 (also referred to herein as the "bottom die") includes an image readout circuitry 146 (also referred to herein as the "image readout mixed signal circuitry"). The image readout circuitry 146 can be coupled to the pixel array 138 of the first die 132 through a column level connection 140 for normal image readout. In some embodiments, column level connections 140 for normal image readout are implemented from column bit lines of pixel array 138 with through silicon vias (TSVs) extending between first die 132 and third die 136 and routed through second die 134 .

[0046] In some embodiments, the pixel array 138 is a two-dimensional (2D) array including a plurality of pixel cells (also referred to as "pixels"), each of which includes at least one photosensor (e.g., at least one photodiode) exposed to incident light. As shown in the illustrated embodiment, the pixels are arranged in rows and columns. Some of the pixels may be configured as CMOS image sensor (CIS) pixels, which are configured to acquire image data of a person, location, object, etc., which may then be used to reproduce an image and / or video of the person, location, object, etc. For example, each CIS pixel is configured to photogenerate an image charge in response to incident light. After each CIS pixel of the pixel array 138 acquires its image charge, the corresponding analog image charge data may be read out by the image readout circuitry 146 in the third die 136 through the column bit lines. In some embodiments, the image charge from each row of the pixel array 138 may be read out in parallel by the image readout circuitry 146 through the column bit lines. As discussed in more detail below, other of the pixels of the pixel array 138 may be configured as event vision sensor (EVS) pixels.

[0047] The image readout circuitry 146 in the third die 136 may include amplifiers for normal image readout and processing, analog / digital converter (ADC) circuitry, associated analog support circuitry, associated digital support circuitry, etc. In some embodiments, the image readout circuitry 146 may also include event-driven readout circuitry, as will be described in more detail below. In operation, photogenerated analog image charge signals are read out from the pixel cells of the pixel array 138, amplified and converted into digital values ​​in the image readout circuitry 146. In some embodiments, the image readout circuitry 146 may read out a row of image data at a time. In other examples, the image readout circuitry 146 may read out image data using various other techniques (not illustrated) such as serial readout or full parallel readout of all pixels at the same time. The image data may be stored or even manipulated by applying post-image effects (e.g., cropping, rotating, removing red eye, adjusting brightness, adjusting contrast, etc.).

[0048] In the illustrated embodiment, the second die 134 (also referred to herein as the "middle die") includes an event-driven sensing array 142 coupled to at least some pixels (e.g., EVS pixels) of the pixel array 138 in the first die 132. In some embodiments, the event-driven sensing array 142 is coupled to at least some pixels of the pixel array 138 through a hybrid bond between the first die 132 and the second die 134. The event-driven sensing array 142 may include an array of event-driven circuits. In some embodiments, each of the event-driven circuits in the event-driven sensing array 142 is coupled to at least one of a plurality of pixels of the pixel array 138 through a hybrid bond between the first die 132 and the second die 134 to asynchronously detect an event occurring in light incident on the pixel array 138 in accordance with the teachings of the present disclosure.

[0049] In some embodiments, the corresponding event detection signal is generated by an event-driven circuit in the event-driven sensing array 142. The event detection signal can be received and processed by the event-driven peripheral circuit system 144. In some embodiments, the event-driven peripheral circuit system 144 is arranged in the second die 134 around the periphery of the event-driven sensing array 142, such as Figure 1A Displayed in. Figure 1A The embodiment illustrated in FIG. 1 also illustrates column level connections 140 for normal image readout being routed through the second die 134 between the first die 132 and the third die 136 .

[0050] Figure 1B yes Figure 1A A partial schematic diagram of a specific example of a stacked system 130. Figure 1B As shown in FIG. 1 , the stacked system 130 includes a pixel array 138 ( Figure 1B138), an event-driven circuit 100 of an event-driven sensing array 142 on a second die 134, and an image readout circuit system 146 on a third die 136. The image readout circuit system 146 includes an analog-to-digital converter 151 ("ADC 151"), an image signal processor 152, a scanning readout circuit system 153, an event signal processor 154, a synchronous communication interface 155 (e.g., a mobile industry processor interface (MIPI) transmitter and / or receiver), and various auxiliary circuits 156. As discussed in more detail below, the image readout circuit system 146 may also include a deblurring circuit (e.g., for performing event-directed deblurring of CIS data).

[0051] Figure 1B The portion of the pixel array 138 shown in 1 corresponds to a 4x4 cluster of pixels in the pixel array 138. This cluster may be repeated across the pixel array 138. In the illustrated embodiment, fifteen (15) pixels in the cluster are configured as active (CIS) pixels 135 to capture CIS information (e.g., intensity information) corresponding to light incident on the photosensors of these pixels. Additionally, one pixel in the cluster is configured as an EVS pixel 137 to capture non-CIS information (e.g., contrast information, event data) corresponding to light incident on the photosensor of the EVS pixel 137. Figure 1C illustrate Figure 1B A specific example of a 4x4 pixel cluster in which the CIS pixels 135 are arranged in a Bayer pattern to capture CIS (frame) information corresponding to light incident on the cluster, and the EVS pixels 137 are arranged to detect EVS (asynchronous event) information corresponding to light incident on the cluster.

[0052] In the illustrated example, the EVS pixel 137 may be positioned below a transparent color filter. Thus, light of any color may be permitted to pass through the transparent color filter and illuminate the photosensor corresponding to the EVS pixel 137. In other words, the EVS pixel 137 may be a monochrome EVS pixel. In other embodiments, the EVS pixel 137 may be positioned below a color filter corresponding to a specific (e.g., only one) color (e.g., red, green, or blue). Continuing with this example, only light of a specific color may be permitted to pass through the color filter and illuminate the photosensor corresponding to the EVS pixel 137.

[0053] Reference again Figure 1B, independently read out the clustered CIS pixels 135 and the clustered EVS pixels 137. More specifically, the CIS information captured by the CIS pixels 135 is read out to the ADC 151 on the third die 136 through the second die 134 using the corresponding row / column control circuit system (not shown). The non-CIS information captured by the EVS pixels 137 is read out to the event-driven circuit 100 on the second die 134 using the corresponding row / column control circuit system (not shown), and the events detected by the event-driven circuit 100 are read out by the scanning readout circuit system 153 on the third die 136. The CIS information captured by the CIS pixels 135 of the pixel array 138 is frame-based and can be read out from the CIS pixels 135 row by row (or using another technique) at the end of the exposure period. In contrast, the non-CIS information captured by the EVS pixels 137 is used by the event-driven circuit 100 to asynchronously detect / trigger events, and the events can be read out according to the row scanning readout scheme or the column scanning readout scheme. Examples of row scan readout schemes are discussed in detail in: (1) co-application, co-pending and commonly assigned application entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR” and (2) co-application, co-pending and commonly assigned application entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR AND ROLLING SHUTTER DISTORTION CORRECTION”, each of which has been incorporated herein by reference in its entirety above.

[0054] In some embodiments, the row / column control circuitry corresponding to the CIS pixel 135 may be allocated on the same or different die as the die on which the ADC 151 is allocated (e.g., the third die 136). In these and other embodiments, the row / column control circuitry corresponding to the EVS pixel 137 may be allocated on the same or different die as the die on which the scanning readout circuitry 153 is allocated (e.g., the third die 136). In these and other embodiments, the ADC 151 and / or the row / column control circuitry corresponding to the CIS pixel 135 may be allocated on the same or different die as the die on which the scanning readout circuitry 153 and / or the row / column control circuitry corresponding to the EVS pixel 137 is allocated (e.g., the third die 136).

[0055] In the illustrated embodiment, EVS pixels 137 are dedicated to capturing non-CIS (EVS) information, while CIS pixels 135 are dedicated to capturing CIS information. In other embodiments, one or more of EVS pixels 137 and / or CIS pixels 135 may be switched between being configured to capture CIS information and non-CIS information. This may enable stacked system 130 to operate in the following modes: a CIS-only mode, in which all pixels 135 and pixels 137 are used to capture CIS information; an EVS-only mode, in which all pixels 135 and pixels 137 are used to capture non-CIS (EVS) information; and / or a mixed CIS and EVS mode, in which a first subset of pixels 135, 137 are used to capture CIS information and a second subset of pixels 135, 137 are used to capture non-CIS (EVS) information.

[0056] In some embodiments, the event driven circuit 100 on the second die 134 has the same die size as the 4x4 pixel cluster on the first die 132. In other embodiments, the event driven circuit 100 may have a different die size than the 4x4 pixel cluster. Additionally or alternatively, although the ratio of CIS pixels to EVS pixels in the 4x4 pixel cluster is 15:1, other ratios of CIS pixels to EVS pixels (e.g., 14:2, 12:4, 8:8, 4:12, 2:14, 15:1) are possible and within the scope of the present technology. Furthermore, although Figure 1B The EVS pixels 157 correspond to 4x4 pixel clusters, but other arrangements (e.g., EVS pixels corresponding to 1X1 pixel clusters, 4X2 pixel clusters, etc.) are possible and within the scope of the present technology. Figure 1B In one embodiment, a row of EVS pixels (e.g., a row including EVS pixels 157) corresponds to four rows of CIS pixels, but other arrangements are possible and within the scope of the present technology. For example, each row of EVS pixels may correspond to (a) one row of CIS pixels, (b) two rows of CIS pixels, (c) three rows of CIS pixels, or (d) more than four rows of CIS pixels.

[0057] Figure 2 2 is a partial schematic diagram of an EVS pixel 200 configured according to various embodiments of the present technology. The EVS pixel 200 is also referred to herein as an "event sensing front-end circuit." It should be understood that the EVS pixel 200 may be Figure 1A The event-driven circuit in the event-driven sensing array 142, Figure 1B 1 or one of the other EVS pixels configured according to various embodiments of the present technology. As shown, the EVS pixel 200 includes a photosensor 201, a logarithmic amplifier 202, a buffer 203, a difference detector 204, and an up / down comparator 205.

[0058] The photosensor 201 is configured to photogenerate image charge (photocurrent) in response to incident light. The photocurrent photogenerated by the photosensor 201 is fed into the logarithmic amplifier 202. The logarithmic amplifier 202 in turn transduces the photocurrent into a voltage V FE .

[0059] The difference detector 204 of the EVS pixel 200 is used to monitor the temporal contrast of light incident on the photosensor 201. More specifically, when reset, the difference detector 204 samples the voltage V at the reference time t0. FE And then the output V O The output V of the difference detector 204 is O After that, the tracking voltage V FE The voltage V at the reference time t0 over time FE changes.

[0060] The output V of the difference detector 204 O is fed to the up / down comparator 205. The up / down comparator 205 in turn compares the output V O The corresponding threshold voltage V +TH and V -TH When the output V of the difference detector 204 O Exceeding the threshold voltage V +TH When the output ΔV of the difference detector 204 is O Less than the threshold voltage V -TH When an event is triggered in the EVS pixel 200, the event may be the readout of the EVS pixel 200 and the difference detector 204 may be reset so that the difference detector 204 resamples the voltage V FE In this manner, EVS pixel 200 may be used to detect events (eg, motion) occurring within an external scene.

[0061] As discussed above, hybrid image sensors typically employ EVS pixels to obtain non-CIS information, which can be combined with CIS information captured by corresponding CIS pixels to, for example, perform image deblurring, rolling shutter distortion correction, etc. on the CIS information. Although typically arranged in EVS pixel rows and EVS pixel columns, the EVS pixels may be dispersed across the pixel array of the hybrid image sensor and interspersed among more CIS pixels arranged in more CIS pixel rows and / or more CIS pixel columns. Each EVS pixel, EVS pixel row, and / or EVS pixel column may therefore correspond to one or more CIS pixels, CIS pixel rows, and / or CIS pixel columns, respectively. Therefore, there may be a CIS and EVS resolution mismatch in such hybrid image sensors.

[0062] For example, Figure 33 is a table 330 illustrating example CIS and EVS pixel row correspondences according to various embodiments of the present technology. As shown, 3 EVS pixel rows (EVS pixel row N, EVS pixel row N+1, and EVS pixel row N+2) correspond to 12 CIS pixel rows (CIS pixel row 4N to CIS pixel row 4N+11). More specifically, the EVS pixels of EVS pixel row N are used to capture non-CIS information of the CIS pixels of CIS pixel row 4N, CIS pixel row 4N+1, CIS pixel row 4N+2, and CIS pixel row 4N+3. Similarly, the EVS pixels of EVS pixel row N+1 are used to capture non-CIS information of CIS pixels of CIS pixel row 4N+4, CIS pixel row 4N+5, CIS pixel row 4N+6, and CIS pixel row 4N+7; and the EVS pixels of EVS pixel row N+2 are used to capture non-CIS information of CIS pixels of CIS pixel row 4N+8, CIS pixel row 4N+9, CIS pixel row 4N+10, and CIS pixel row 4N+11. Therefore, each EVS pixel row can correspond to more CIS pixel rows.

[0063] In these and other embodiments, the number of EVS pixels in an EVS pixel row may be different from the number of CIS pixels in one or more of the corresponding CIS pixel rows. For example, EVS pixel row N may include a first number of EVS pixels that is different from the number of CIS pixels included in CIS pixel row 4N, CIS pixel row 4N+1, CIS pixel row 4N+2, and / or CIS pixel row 4N+3. In such examples, the number of EVS pixel columns in EVS pixel row N may be different from the number of CIS pixel columns included in CIS pixel row 4N, CIS pixel row 4N+1, CIS pixel row 4N+2, and / or CIS pixel row 4N+3.

[0064] Additionally or alternatively, the EVS pixels in an EVS pixel row may correspond to a different color than one or more CIS pixels in one or more corresponding CIS pixel rows. For example, EVS pixel row N may include a first EVS pixel corresponding to transparent (or monochrome) and / or a second EVS pixel corresponding to a particular color (e.g., red, green, or blue). Continuing with this example, one or more CIS pixels in CIS pixel row 4N may correspond to a different color than the first EVS pixel and / or the second EVS pixel. Thus, there may be a mismatch between the EVS color channels and the CIS color channels.

[0065] Methods for operating a hybrid imaging system to reduce or eliminate CIS and EVS resolution mismatches and / or color channel mismatches are described below. Figures 4 to 8 More specifically, Figure 44 is a flow chart illustrating a method 440 of operating an imaging system including a hybrid image sensor according to various embodiments of the present technology. For example, the method 440 may be a method of reducing, minimizing, or eliminating CIS and EVS resolution mismatches (e.g., matching, better matching, aligning, better aligning CIS data resolution and EVS data resolution) and / or a method of reducing, minimizing, or eliminating mismatches between EVS color channels and CIS color channels. The method 440 is illustrated as a series of blocks 441 to 443 or steps. All blocks 441 to 443 or a subset of one or more thereof may be performed by a device or component of an imaging system (e.g., a device or component of a hybrid image sensor). For example, all blocks 441 to 443 or a subset of one or more thereof may be performed by a CIS pixel of a pixel array, an EVS pixel of an event-driven sensing array, and / or a control circuit system of a hybrid image sensor and / or imaging system (e.g., a common control block, a row / column control circuit system, a column readout circuit system, a column scan readout circuit system, a row scan readout circuit system, etc.). All blocks 441 to 443 of method 440 or a subset of one or more of them may be performed according to the description of Figures 1 to 3 above and / or the description below. In fact, block 442 of method 440 is referred to below. Figure 3 Description, and block 443 of method 440 is referred to below Figures 5A to 8 describe.

[0066] Method 440 begins with block 441, which resets one or more EVS pixels and / or one or more CIS pixels of the pixel array. In some embodiments, resetting the EVS pixels and / or CIS pixels may include aligning CIS pixel data with corresponding EVS pixel data. Aligning the CIS pixel data with the corresponding EVS pixel data may include aligning / synchronizing the timing of exposure periods of one or more rows of CIS pixels with event accumulation periods of one or more corresponding EVS pixels. Additional details about aligning / synchronizing the timing of exposure periods of one or more rows of CIS pixels with event accumulation periods of one or more corresponding EVS pixels are provided in the co-application, co-pending and commonly assigned application entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR”, which has been incorporated herein by reference in its entirety above.

[0067] At block 442, the method 440 then (a) captures CIS data using the CIS pixels during the corresponding exposure period and (b) captures EVS data using the EVS pixels. Figure 3For example, CIS pixels of CIS pixel row 4N to CIS pixel row 4N+3 corresponding to EVS pixel row N may integrate charge photogenerated in the CIS pixels during a corresponding exposure period. At the same time, the EVS pixels of EVS pixel row N may asynchronously detect events during a corresponding and concurrent EVS accumulation period that share the same start and stop times as the CIS pixels of CIS pixel row 4N to CIS pixel row 4N+3.

[0068] At block 443, method 440 then reads out the events detected by the EVS pixels of the event-driven sense array and reads out the CIS data obtained using one or more of the CIS pixels. At least a portion of block 443 may be performed when block 442 is performed. For example, when an EVS pixel detects an event, the event may be read out from the EVS pixel. When an event detected by an EVS pixel is read out from the EVS pixel, the EVS pixel may be reset and thereby enabled to detect subsequent events.

[0069] A single EVS pixel may detect hundreds of events within a single event accumulation period. Therefore, for a single CIS frame, a relatively large amount of EVS data may be generated. In some cases, this large amount of EVS data may complicate and / or slow down calculations (e.g., deblurring calculations) performed by the image sensor and / or downstream processor, which may be inappropriate or unacceptable for certain applications. Therefore, in some embodiments, EVS data may be read out from the EVS pixel using a progressive scan readout. More specifically, the image sensor may scan / step through the event-driven sensing array row by row and take a uniform amount of time to read out each EVS pixel row. In this way, the scanning readout may limit the number of EVS readouts per CIS frame, which may simplify and / or speed up calculations, downstream calculations. Additional details on the progressive scan readout technique are provided in the co-application, co-pending and co-assigned application entitled “HYBRID IMAGE SENSORS WITH ON-CHIP IMAGE DEBLUR”, which has been incorporated herein by reference in its entirety above.

[0070] In some embodiments, reading out CIS data obtained using one or more CIS pixels may include reading out CIS data from CIS pixels in rows or in groups of rows. For example, in embodiments where multiple CIS pixel rows correspond to the same EVS pixel row, CIS data captured by CIS pixels of the multiple CIS pixel rows may be read out together / simultaneously at or after the end of the corresponding exposure period.

[0071] In these and other embodiments, reading out EVS pixel data and reading out CIS data may include reducing, minimizing, or eliminating CIS and EVS resolution mismatches and / or mismatches between CIS color channels and EVS color channels. For example, reading out CIS data from CIS pixels of a pixel array may include skipping readout of one or more CIS pixel rows and / or one or more CIS pixel columns to, for example, reduce the resolution of the CIS data, match the resolution of the EVS data captured by the corresponding EVS pixels, and / or reduce, minimize, or eliminate a mismatch between the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the corresponding EVS pixels. Figures 5A to 5C Three such examples are described in accordance with various embodiments of the present technology. Figures 5A to 5C , the EVS pixels of EVS pixel row #1 correspond to the CIS pixels of CIS pixel row #1 to CIS pixel row #4. In addition, in the illustrated example, the EVS pixels of EVS pixel row #1 are operated as monochrome pixels (e.g., corresponding to transparency) to permit light of any color to illuminate one or more photosensors corresponding to the EVS pixels. The CIS pixels of CIS pixel row #1 to CIS pixel row #4 may include CIS pixels corresponding to specific colors (e.g., red, blue, green, etc.) and / or CIS pixels operated as monochrome pixels (e.g., corresponding to transparency).

[0072] First reference Figure 5A , although CIS pixel rows #1 to CIS pixel rows #4 each correspond to EVS pixel row #1, only the CIS pixels of CIS pixel rows #1 and CIS pixel row #2 are read out. In other words, the readout of the CIS pixels of CIS pixel rows #3 and CIS pixel rows #4 can be skipped. Therefore, the resolution of the CIS data captured by the CIS pixels corresponding to the EVS pixels of EVS pixel row #1 can be reduced in order to reduce, minimize and / or eliminate the mismatch between the resolution of the CIS data and the resolution of the corresponding EVS data obtained by the EVS pixels of EVS pixel row #1. As discussed above, the CIS pixels of CIS pixel rows #1 and CIS pixel rows #2 may include CIS pixels corresponding to specific colors (e.g., red, blue, green, etc.) and / or CIS pixels operated as monochrome pixels (e.g., corresponding to transparent).

[0073] Reference Figure 5B , only the CIS pixels in CIS pixel row #3 and CIS pixel row #4 are read out. In other words, Figure 5ACompared to the example described in , the readout of the CIS pixels of CIS pixel row #1 and CIS pixel row #2 can be skipped. Therefore, the resolution of the CIS data captured by the CIS pixels corresponding to the EVS pixels of EVS pixel row #1 can be reduced in order to reduce, minimize and / or eliminate the mismatch between the resolution of the CIS data and the resolution of the corresponding EVS data obtained by the EVS pixels of EVS pixel row #1. As discussed above, the CIS pixels of CIS pixel row #3 and CIS pixel row #4 may include CIS pixels corresponding to specific colors (e.g., red, blue, green, etc.) and / or CIS pixels operated as monochrome pixels (e.g., corresponding to transparent). In a specific example, the CIS pixels of CIS pixel row #3 and CIS pixel row #4 may only include CIS pixels corresponding to specific colors (e.g., red, blue, green, etc.), even if the EVS pixels of EVS pixel row #1 are operated as monochrome EVS pixels.

[0074] Reference Figure 5C , only the CIS pixels in CIS pixel row #1 and CIS pixel row #3 are read out. In other words, Figure 5A and 5B Compared to the example described in , the readout of the CIS pixels of CIS pixel row #2 and CIS pixel row #4 can be skipped. Therefore, the resolution of the CIS data captured by the CIS pixels corresponding to the EVS pixels of EVS pixel row #1 can be reduced in order to reduce, minimize and / or eliminate the mismatch between the resolution of the CIS data and the resolution of the corresponding EVS data obtained by the EVS pixels of EVS pixel row #1. As discussed above, the CIS pixels of CIS pixel row #1 and CIS pixel row #3 may include CIS pixels corresponding to specific colors (e.g., red, blue, green, etc.) and / or CIS pixels operated as monochrome pixels (e.g., corresponding to transparent). In a specific example, the CIS pixels of CIS pixel row #1 and CIS pixel row #3 may only include CIS pixels corresponding to specific colors (e.g., red, blue, green, etc.), even if the EVS pixels of EVS pixel row #1 are operated as monochrome EVS pixels.

[0075] remove Figures 5A to 5COther readout skipping techniques other than those shown in are possible and within the scope of the present technology. For example, only the CIS pixels of CIS pixel row #2 and CIS pixel row #4 may be read out, while the readout of the CIS pixels of CIS pixel row #1 and CIS pixel row #3 may be skipped. As another example, in addition to or in lieu of skipping the readout of one or more CIS pixel rows in CIS pixel row #1 to CIS pixel row #4, the readout of one or more CIS pixel columns in CIS pixel row #1 to CIS pixel row #4 may be skipped. It is expected that skipping the readout of one or more CIS pixel columns will also reduce, minimize and / or eliminate the mismatch between CIS and EVS pixel resolutions and / or CIS and EVS pixel column resolutions.

[0076] Figures 6A to 6C Three additional examples of skipping readout of one or more CIS pixel rows and / or one or more CIS pixel columns (e.g., to reduce the resolution of CIS data, match the resolution of EVS data captured by corresponding EVS pixels, and / or reduce, minimize, or eliminate a mismatch between the resolution of CIS data captured by CIS pixels and the resolution of EVS data captured by corresponding EVS pixels) are described in accordance with various embodiments of the present technology. Figures 6A to 6C , the EVS pixels of EVS pixel row #1 and EVS pixel row #2 correspond to the CIS pixels of CIS pixel row #1 to CIS pixel row #4. In addition, in the illustrative example, the EVS pixels of EVS pixel row #1 and EVS pixel row #2 correspond to specific colors (e.g., red, green and / or blue) to allow only light of this / these specific colors to illuminate one or more photosensors corresponding to the EVS pixels. The CIS pixels of CIS pixel row #1 to CIS pixel row #4 may include CIS pixels corresponding to specific colors (e.g., red, blue, green, etc.) and / or CIS pixels operated as monochrome pixels (e.g., corresponding to transparent). As a specific example, Figures 6A to 6B 4 and the EVS pixels of EVS pixel rows #1 and EVS pixel rows #2 are arranged in a Bayer pattern. It should be understood that the Bayer pattern is used only as an example. In fact, in other embodiments, the EVS pixels and / or CIS pixels may be arranged in any other suitable color filter array pattern and / or may correspond to colors other than red, green, and blue.

[0077] First reference Fig. 6A, although CIS pixel row #1 to CIS pixel row #4 correspond to EVS pixel row #1 and EVS pixel row #2 respectively, only the CIS pixels of CIS pixel row #1 and CIS pixel row #2 are read out. In other words, the readout of the CIS pixels of CIS pixel row #3 and CIS pixel row #4 can be skipped (e.g., to reduce, minimize and / or eliminate the CIS and EVS resolution mismatch). In addition, as described above with reference to Figure 5A Compared to the example discussed, only the CIS pixels of CIS pixel row #1 and CIS pixel row #2 corresponding to the colors matching the colors corresponding to the EVS pixels of EVS pixel row #1 and EVS pixel row #2 are read out. Specifically, in the illustrated example, the EVS pixels of EVS pixel row #1 correspond to red and green. Therefore, only those CIS pixels in CIS pixel row #1 corresponding to red and green (which may be all CIS pixels or a subset thereof included in CIS pixel row #1) are read out, so that the color channels read out from CIS pixel row #1 match the color channels of the EVS pixels of EVS pixel row #1. In addition, in the illustrated example, the EVS pixels of EVS pixel row #2 correspond to green and blue. Therefore, only those CIS pixels in CIS pixel row #2 corresponding to green and blue (which may be all CIS pixels or a subset thereof included in CIS pixel row #2) are read out, so that the color channels read out from CIS pixel row #2 match the color channels of the EVS pixels of EVS pixel row #2. Fig. 6A Such readout in the example illustrated in may be achieved by skipping readout of one or more CIS pixels in one or more CIS pixel rows and / or one or more CIS pixel columns.

[0078] Reference Figure 6B , although CIS pixel rows #1 to CIS pixel rows #4 correspond to EVS pixel rows #1 and EVS pixel rows #2, respectively, only the CIS pixels of CIS pixel rows #3 and CIS pixel rows #4 are read out. In other words, the readout of the CIS pixels of CIS pixel rows #1 and CIS pixel rows #2 may be skipped (e.g., to reduce, minimize and / or eliminate CIS and EVS resolution mismatch). In addition, as described above with reference to Figure 5BCompared to the example discussed, only the CIS pixels of CIS pixel row #3 and CIS pixel row #4 corresponding to the colors matching the colors corresponding to the EVS pixels of EVS pixel row #1 and EVS pixel row #2 are read out. Specifically, in the illustrated example, the EVS pixels of EVS pixel row #1 correspond to red and green. Therefore, only those CIS pixels in CIS pixel row #3 corresponding to red and green (which may be all CIS pixels or a subset thereof included in CIS pixel row #3) are read out, so that the color channels read out from CIS pixel row #3 match the color channels of the EVS pixels of EVS pixel row #1. In addition, in the illustrated example, the EVS pixels of EVS pixel row #2 correspond to green and blue. Therefore, only those CIS pixels in CIS pixel row #4 corresponding to green and blue (which may be all CIS pixels or a subset thereof included in CIS pixel row #4) are read out, so that the color channels read out from CIS pixel row #4 match the color channels of the EVS pixels of EVS pixel row #2. Figure 6B Such readout in the example illustrated in may be achieved by skipping readout of one or more CIS pixels in one or more CIS pixel rows and / or one or more CIS pixel columns.

[0079] Reference Figure 6C , although CIS pixel rows #1 to CIS pixel rows #4 correspond to EVS pixel rows #1 and EVS pixel rows #2 respectively, only the CIS pixels of CIS pixel rows #2 and CIS pixel rows #3 are read out. In other words, the readout of the CIS pixels of CIS pixel rows #1 and CIS pixel rows #4 can be skipped (for example, to reduce, minimize and / or eliminate CIS and EVS resolution mismatch). In addition, only the CIS pixels of CIS pixel rows #2 and CIS pixel rows #3 corresponding to the colors that match the colors corresponding to the EVS pixels of EVS pixel rows #1 and EVS pixel rows #2 are read out. Specifically, in the illustrated example, the EVS pixels of EVS pixel row #1 correspond to green and blue. Therefore, only those CIS pixels in CIS pixel row #2 corresponding to green and blue (which may be all CIS pixels or a subset thereof included in CIS pixel row #2) are read out so that the color channels read out from CIS pixel row #2 match the color channels of the EVS pixels of EVS pixel row #1. Additionally, in the illustrated example, the EVS pixels of EVS pixel row #2 correspond to red and green. Therefore, only those CIS pixels in CIS pixel row #3 that correspond to red and green (which may be all CIS pixels included in CIS pixel row #3 or a subset thereof) are read out so that the color channels read out from CIS pixel row #3 match the color channels of the EVS pixels of EVS pixel row #2. Figure 6CSuch readout in the example illustrated in may be achieved by skipping readout of one or more CIS pixels in one or more CIS pixel rows and / or one or more CIS pixel columns.

[0080] remove Figures 6A to 6C Other readout skipping techniques other than those shown in are possible and within the scope of the present technology. For example, only the CIS pixels of CIS pixel row #1 and CIS pixel row #4 can be read out, while the readout of the CIS pixels of CIS pixel row #2 and CIS pixel row #3 can be skipped. As another example, only the CIS pixels of CIS pixel row #1 and CIS pixel row #3 can be read out, while the readout of the CIS pixels of CIS pixel row #2 and CIS pixel row #4 can be skipped. As yet another example, only the CIS pixels of CIS pixel row #2 and CIS pixel row #4 can be read out, while the readout of the CIS pixels of CIS pixel row #1 and CIS pixel row #3 can be skipped. In any of these additional examples, the CIS pixels can be read out so that the readout CIS color channel matches the color channel of the corresponding EVS pixel.

[0081] In addition to or in lieu of skipping readout of one or more CIS pixels, one or more CIS pixel rows and / or one or more CIS column pixels, in Figure 4 Reducing, minimizing, or eliminating CIS and EVS resolution mismatches and / or mismatches between CIS color channels and EVS color channels at block 443 of method 440 may include merging one or more CIS pixel rows and / or columns. Fig. 7A and 7B Two such examples are described in accordance with various embodiments of the present technology.

[0082] First reference Fig. 7A , the EVS pixels of EVS pixel row #1 correspond to the CIS pixels of CIS pixel row #1 to CIS pixel row #4. In addition, the EVS pixels of EVS pixel row #1 operate as monochrome pixels (e.g., corresponding to transparent) to allow light of any color to illuminate one or more photosensors corresponding to the EVS pixels. The CIS data obtained by one or more CIS pixels of one or more of CIS pixel rows #1 to #4 can be vertically merged to reduce, minimize or eliminate CIS and EVS resolution mismatch. Fig. 7AIn the specific example described in , CIS data obtained using CIS pixels of all four CIS pixel rows #1 to #4 are vertically merged so that the resolution of the CIS data is better aligned and / or matched with the resolution of the EVS data obtained using the EVS pixels of EVS pixel row #1. Additionally or alternatively, CIS data obtained using CIS pixels of one or more of CIS pixel rows #1 to #4 may be horizontally merged. For example, in a scenario where there are fewer EVS pixel columns in EVS pixel row #1 than in one or more of CIS pixel rows #1 to #4, horizontal merging may, for example, reduce, minimize, or eliminate CIS and EVS pixel column mismatches.

[0083] Reference Figure 7B , the EVS pixels of EVS pixel row #1 and EVS pixel row #2 correspond to specific colors (e.g., red, green, and / or blue) to permit only light of this / these specific colors to illuminate one or more photosensors corresponding to the EVS pixels. More specifically, the EVS pixels of EVS pixel row #1 correspond to red and green, and the EVS pixels of EVS pixel row #2 correspond to green and blue. In the illustrative example, the CIS pixels of CIS pixel row #1 to CIS pixel row #4 may include CIS pixels corresponding to specific colors (e.g., red, green, and / or blue). More specifically, the CIS pixels of CIS pixel rows #1 and #3 correspond to red and green, and the CIS pixels of CIS pixel rows #2 and #4 correspond to green and blue. As a specific example, Figure 7B 4 and the EVS pixels of EVS pixel rows #1 and EVS pixel rows #2 are arranged in a Bayer pattern. It should be understood that the Bayer pattern is used only as an example. In fact, in other embodiments, the EVS pixels and / or CIS pixels may be arranged in any other suitable color filter array pattern and / or may correspond to colors other than red, green, and blue.

[0084] As shown in the illustrative example, CIS data obtained from one or more CIS pixels of CIS pixel rows #1 and #3 can be vertically combined to reduce, minimize, or eliminate the CIS and EVS resolution mismatch between the CIS data captured by the CIS pixels of these CIS pixel rows and the EVS data obtained using the EVS pixels of EVS pixel row #1. Also as shown, this combining scheme can ensure that the color channels read from CIS pixel rows #1 and #3 match the color channels of the EVS pixels of EVS pixel row #1. Additionally, CIS data obtained from one or more CIS pixels of CIS pixel rows #2 and #4 can be vertically combined to reduce, minimize, or eliminate the CIS and EVS resolution mismatch between the CIS data captured by the CIS pixels of these CIS pixel rows and the EVS data obtained using the EVS pixels of EVS pixel row #2. As shown, this combining scheme can ensure that the color channels read from CIS pixel rows #2 and #4 match the color channels of the EVS pixels of EVS pixel row #2. Similar to the example discussed above with reference to Fig. 7A In addition to or instead of vertical combining, the CIS data obtained using one or more CIS pixels of CIS pixel rows #1 to #4 can be horizontally combined. For example, in a scenario where the number of EVS pixel columns in EVS pixel row #1 is less than the number of CIS pixel columns in one or more of CIS pixel rows #1 to #4, horizontal combining can, for example, reduce, minimize, or eliminate the CIS and EVS pixel column mismatch.

[0085] In addition to or instead of the following, reducing, minimizing, or eliminating the CIS and EVS resolution mismatch at block 443 of method 440 in Figure 4 can include interpolating the EVS data: (1) skipping the readout of one or more CIS pixels, one or more CIS pixel rows, and / or one or more CIS pixel columns and / or (2) vertically and / or horizontally combining the CIS data captured using one or more CIS pixels of one or more CIS pixel rows and / or one or more CIS pixel columns. Figure 8 Illustrate one such example according to various embodiments of the present technology.

[0086] As Figure 8 shown, the EVS pixels of EVS pixel row #1 correspond to the CIS pixels of CIS pixel rows #1 to CIS pixel row #4. Additionally, in this example, each of CIS pixel rows #1 to #4 includes m CIS pixel columns, while EVS pixel row #1 includes k EVS pixel columns, where k is less than m (k < m). EVS pixel row #2 is also shown in Figure 8 and is illustrated with k EVS pixel columns.

[0087] EVS data captured using the EVS pixels of EVS pixel row #1 and EVS pixel row #2 may be used to interpolate additional EVS data corresponding to additional rows and / or columns of EVS pixel data. More specifically, if Figure 8 , EVS data captured using the EVS pixels of EVS pixel row #1 and EVS pixel row #2 may be used to interpolate additional EVS data corresponding to three additional EVS pixel rows (illustrated as EVS pixel rows #2' to #4') between EVS pixel row #1 (EVS pixel row #1') and EVS pixel row #2 (EVS pixel row #5'). Thus, after interpolation, the EVS data corresponding to EVS pixel rows #1' to #4' may correspond to CIS data corresponding to CIS pixel rows #1 to #4, respectively. In other words, the EVS data captured using the EVS pixels of EVS pixel row #1 and EVS pixel row #2 may be used to interpolate additional EVS data corresponding to additional EVS pixel rows to increase the resolution of the EVS data, match the resolution of the CIS data captured by the CIS pixels, and / or reduce a mismatch between the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels.

[0088] Additionally or alternatively, EVS data captured using EVS pixels of EVS pixel row #1 and / or EVS pixel row #2 may be used to interpolate additional columns of EVS data. More specifically, if Figure 8, EVS data captured using the EVS pixels of EVS pixel row #1 and / or EVS pixel row #2 can be used to interpolate additional EVS data corresponding to additional EVS pixel columns of EVS pixel row #1 and / or EVS pixel row #2. Specifically, EVS data captured using the EVS pixels of EVS pixel row #1 and / or EVS pixel row #2 can be used to interpolate additional EVS data corresponding to additional EVS pixel columns to increase the number of columns of EVS data of EVS pixel row #1 and / or EVS pixel row #2 from k to m. Thus, after interpolation, EVS data corresponding to EVS pixel row #1' and / or EVS pixel row #5' can correspond to m columns, which match the number of CIS pixel columns in each of CIS pixel rows #1 to #4. Similarly, EVS data captured using the EVS pixels of EVS pixel row #1 and / or EVS pixel row #2 can be used to interpolate additional EVS data corresponding to additional EVS pixel columns of additional EVS pixel rows #2' to #4'. In other words, EVS data captured using EVS pixels of EVS pixel row #1 and / or EVS pixel row #2 may be used to interpolate additional EVS data corresponding to additional EVS pixel columns to increase the resolution of the EVS data, match the resolution of CIS data captured by the CIS pixels, and / or reduce the mismatch between the resolution of the CIS data captured by the CIS pixels and the resolution of the EVS data captured by the EVS pixels.

[0089] Although blocks 441 through 443 of method 440 are described and illustrated in a particular order, Figure 4 The method 440 is not limited in this regard. In other embodiments, all of the frames 441 to 443 of the method 440 or a subset of one or more thereof may be performed in a different order. In these and other embodiments, all of the frames 441 to 443 or a subset of any of them may be performed before, during, and / or after all of the other frames 441 to 443 or a subset of any of them. In addition, it should be readily apparent to those skilled in the art that the method 440 may be altered and still remain within these and other embodiments of the present technology. For example, in some embodiments, all of the frames 441 to 443 or a subset of one or more thereof may be omitted and / or repeated. As a specific example, the EVS pixel and the CIS pixel may be reset at frame 441 at different times from each other and / or at a time that causes them to be misaligned with each other (e.g., at the time of data capture and / or at the time of readout).

[0090] C. in conclusion

[0091] The above detailed description of the embodiments of the present technology is not intended to be exhaustive or to limit the present technology to the precise form disclosed above. Although specific embodiments and examples of the present technology are described above for illustrative purposes, it should be recognized by those skilled in the relevant art that various equivalent modifications may be within the scope of the present technology. For example, although the steps are presented in the order given above, alternative embodiments may perform the steps in a different order. In addition, the various embodiments described herein may also be combined to provide additional embodiments.

[0092] It should be understood from the foregoing that the specific embodiments of the present technology have been described herein for illustrative purposes, but the well-known structures and functions are not shown or described in detail so as not to make the description of the embodiments of the present technology unclear unnecessarily. If any material incorporated herein by reference conflicts with the present disclosure, the present disclosure shall prevail. Where the context permits, singular or plural terms may also include plural or singular terms, respectively. In addition, unless the word "or" is clearly limited to mean excluding only a single item of other items in a list of two or more items, the use of "or" in this list should be interpreted as including any single item in the list of (a), all items in the list of (b) or any combination of items in the list of (c). In addition, as used herein, the phrase "and / or" as in "A and / or B" relates to only A, only B and both A and B. In addition, the use of the terms "including", "comprising", "having" and "having" throughout the text means including at least the stated features to not exclude any greater number of identical features and / or other features of additional types. In addition, as used herein, the phrases "based on", "depending on", "due to", and "in response to" should not be interpreted as referring to a closed set of conditions. For example, without departing from the scope of the present disclosure, the exemplary steps described as "based on condition A" may be based on both condition A and condition B. In other words, as used herein, the phrase "based on" should be interpreted in the same manner as the phrase "based at least in part on" or the phrase "based at least in part on". Moreover, the terms "connected" and "coupled" are used interchangeably herein and refer to both direct and indirect connections or couplings. For example, where the context permits, element A "connected" or "coupled" to element B may refer to (i) A directly "connected" or directly "coupled" to B and / or (ii) A indirectly "connected" or indirectly "coupled" to B.

[0093] It should also be understood from the foregoing that various modifications can be made without departing from the present disclosure or the present technology. For example, it should be understood by those of ordinary skill in the art that the various components of the present technology can be further divided into subcomponents, or the various components and functions of the present technology can be combined and integrated. In addition, certain aspects of the present technology described in the context of a particular embodiment may also be combined or eliminated in other embodiments. In addition, although the advantages associated with certain embodiments of the present technology have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily need to exhibit such advantages to fall within the scope of the present technology. Therefore, the present disclosure and associated technology may encompass other embodiments that are not explicitly shown or described herein.

Claims

1. A method of operating an imaging system including a hybrid image sensor, the method comprising: adjusting a first resolution associated with complementary metal oxide semiconductor image sensor pixel data captured using a plurality of complementary metal oxide semiconductor image sensor pixels of the hybrid image sensor, wherein adjusting the first resolution comprises adjusting the first resolution to reduce a mismatch between the first resolution and a second resolution associated with event vision sensor pixel data captured using one or more event vision sensor pixels of the hybrid image sensor, and wherein the one or more event vision sensor pixels correspond to the plurality of complementary metal oxide semiconductor image sensor pixels; or The second resolution is adjusted, wherein adjusting the second resolution includes adjusting the second resolution to reduce the mismatch between the first resolution and the second resolution.

2. The method according to claim 1, wherein: The plurality of complementary metal oxide semiconductor image sensor pixels are arranged into a plurality of complementary metal oxide semiconductor image sensor pixel rows; The method includes adjusting the first resolution; and Adjusting the first resolution includes skipping readout of CMOS image sensor pixel data corresponding to at least one CMOS image sensor pixel row of the plurality of CMOS image sensor pixel rows such that the first resolution is reduced.

3. The method according to claim 1, wherein: The plurality of complementary metal oxide semiconductor image sensor pixels are arranged into a plurality of complementary metal oxide semiconductor image sensor pixel columns; The method includes adjusting the first resolution; and Adjusting the first resolution includes skipping readout of CMOS image sensor pixel data corresponding to at least one CMOS image sensor pixel column of the plurality of CMOS image sensor pixel columns such that the first resolution is reduced.

4. The method according to claim 1, wherein: The plurality of complementary metal oxide semiconductor image sensor pixels are arranged into a plurality of complementary metal oxide semiconductor image sensor pixel rows and a plurality of complementary metal oxide semiconductor image sensor pixel columns; The one or more event vision sensor pixels include event vision sensor pixels operated as monochrome event vision sensor pixels; The method includes adjusting the first resolution; and Adjusting the first resolution includes reading out complementary metal oxide semiconductor image sensor pixel data corresponding to multiple color channels, while skipping the reading out of complementary metal oxide semiconductor image sensor pixel data corresponding to at least one complementary metal oxide semiconductor image sensor pixel row among the multiple complementary metal oxide semiconductor image sensor pixel rows and / or at least one complementary metal oxide semiconductor image sensor pixel column among the multiple complementary metal oxide semiconductor image sensor pixel columns, so that the first resolution is reduced.

5. The method according to claim 1, wherein: The one or more event vision sensor pixels include event vision sensor pixels corresponding to a specific color; The method includes adjusting the first resolution; and Adjusting the first resolution includes reading out complementary metal oxide semiconductor image sensor pixel data corresponding to a color channel that matches the specific color while skipping readout of complementary metal oxide semiconductor image sensor pixel data corresponding to a color channel that does not match the specific color.

6. The method according to claim 1, wherein: The plurality of complementary metal oxide semiconductor image sensor pixels are arranged into a plurality of complementary metal oxide semiconductor image sensor pixel rows; The method includes adjusting the first resolution; and Adjusting the first resolution includes merging CMOS image sensor pixel data captured by CMOS image sensor pixels located in CMOS image sensor pixel rows that are different from one another among the plurality of CMOS image sensor pixel rows.

7. The method according to claim 6, wherein: The one or more event vision sensor pixels include event vision sensor pixels corresponding to a specific color; and Merging the CMOS image sensor pixel data captured by the CMOS image sensor pixels positioned in the different CMOS image sensor pixel rows includes merging CMOS image sensor pixel data corresponding to color channels that match the particular color.

8. The method according to claim 1, wherein: The plurality of complementary metal oxide semiconductor image sensor pixels are arranged into a plurality of complementary metal oxide semiconductor image sensor pixel columns; The method includes adjusting the first resolution; and Adjusting the first resolution includes merging CMOS image sensor pixel data captured by CMOS image sensor pixels positioned in CMOS image sensor pixel columns that are different from one another among the plurality of CMOS image sensor pixel columns.

9. The method according to claim 1, wherein: The one or more event vision sensor pixels are arranged into a plurality of event vision sensor pixel rows; The method includes adjusting the second resolution; and Adjusting the second resolution includes interpolating additional event vision sensor pixel data corresponding to at least one additional event vision sensor pixel row using event vision sensor pixel data captured by event vision sensor pixels of at least two event vision sensor pixel rows of the plurality of pixel rows.

10. The method of claim 1, wherein: The one or more event vision sensor pixels are arranged into a plurality of event vision sensor pixel rows and a plurality of event vision sensor pixel columns; The method includes adjusting the second resolution; and Adjusting the second resolution includes interpolating additional event vision sensor pixel data corresponding to at least one additional event vision sensor pixel column using event vision sensor pixel data captured by event vision sensor pixels of one or more event vision sensor pixel rows of the plurality of pixel rows.

11. An imaging system comprising: A hybrid image sensor comprising— an event-driven sensing array comprising one or more event vision sensor pixels arranged into one or more event vision sensor pixel rows, wherein each of the one or more event vision sensor pixels is configured to capture event vision sensor data corresponding to contrast information of light incident on the event vision sensor pixel, and wherein the event vision sensor data captured using the one or more event vision sensor pixels of the event-driven sensing array has an event vision sensor resolution, and a pixel array comprising a plurality of CMOS image sensor pixels arranged in one or more CMOS image sensor pixel rows, wherein each CMOS image sensor pixel of the plurality of CMOS image sensor pixels is configured to capture CMOS image sensor data corresponding to an intensity of light incident on the CMOS image sensor pixel, and wherein the CMOS image sensor data captured using the plurality of CMOS image sensor pixels of the pixel array has a CMOS image sensor resolution; and A control circuit system is configured to adjust the CMOS image sensor resolution of the CMOS image sensor data and / or the EVS resolution of the EVS data so that a mismatch between the CMOS image sensor resolution and the EVS resolution is reduced.

12. The imaging system of claim 11, wherein: The control circuitry is configured to adjust the CMOS image sensor resolution of the CMOS image sensor data; and To adjust the CMOS image sensor resolution, the control circuitry is configured to skip readout of CMOS image sensor data captured using CMOS image sensor pixels of at least one of the one or more CMOS image sensor pixel rows.

13. The imaging system of claim 11, wherein: The plurality of complementary metal oxide semiconductor image sensor pixels are further arranged into one or more complementary metal oxide semiconductor image sensor pixel columns; The control circuitry is configured to adjust the CMOS image sensor resolution of the CMOS image sensor data; and To adjust the CMOS image sensor resolution, the control circuitry is configured to skip readout of CMOS image sensor data captured using CMOS image sensor pixels of at least one of the one or more CMOS image sensor pixel columns.

14. The imaging system of claim 11, wherein: An event vision sensor pixel of the one or more event vision sensor pixels is operated as a monochrome event vision sensor pixel; The plurality of complementary metal oxide semiconductor image sensor pixels are further arranged into one or more complementary metal oxide semiconductor image sensor pixel columns; The control circuitry is configured to adjust the CMOS image sensor resolution of the CMOS image sensor data; To adjust the CMOS image sensor resolution, the control circuit system is configured to read out CMOS image sensor data captured using CMOS image sensor pixels corresponding to a plurality of colors while skipping readout of CMOS image sensor data captured using CMOS image sensor pixels corresponding to at least one of the one or more CMOS image sensor pixel rows and / or at least one of the one or more CMOS image sensor pixel columns.

15. The imaging system of claim 11, wherein: An event vision sensor pixel of the one or more event vision sensor pixels corresponds to a specific color; The control circuitry is configured to adjust the CMOS image sensor resolution of the CMOS image sensor data; In order to adjust the complementary metal oxide semiconductor image sensor resolution, the control circuit system is configured to read out complementary metal oxide semiconductor image sensor data captured using complementary metal oxide semiconductor image sensor pixels corresponding to the specific color, while skipping the readout of complementary metal oxide semiconductor image sensor data captured using complementary metal oxide semiconductor image sensor pixels corresponding to another color different from the specific color, so that the color channels of the readout complementary metal oxide semiconductor image sensor data match the color channels of the event vision sensor pixels.

16. The imaging system of claim 11, wherein: Control circuitry is configured to adjust the CMOS image sensor resolution of the CMOS image sensor data; and To adjust the CMOS image sensor resolution, the control circuit system is configured to merge CMOS image sensor data captured by CMOS image sensor pixels belonging to two or more CMOS image sensor pixel rows of the one or more CMOS image sensor pixel rows.

17. The imaging system of claim 16, wherein: An event vision sensor pixel of the one or more event vision sensor pixels corresponds to a particular color; and The CMOS image sensor pixels belonging to the two or more CMOS image sensor pixel rows each correspond to the specific color.

18. The imaging system of claim 11, wherein: The plurality of complementary metal oxide semiconductor image sensor pixels are further arranged into one or more complementary metal oxide semiconductor image sensor pixel columns; The control circuitry is configured to adjust the CMOS image sensor resolution of the CMOS image sensor data; and To adjust the CMOS image sensor resolution, the control circuit system is configured to merge CMOS image sensor data captured by CMOS image sensor pixels belonging to two or more CMOS image sensor pixel columns of the one or more CMOS image sensor pixel columns.

19. The imaging system of claim 11, wherein: The control circuitry is configured to adjust the event vision sensor resolution of the event vision sensor data; and To adjust the event vision sensor resolution, the control circuit system is configured to interpolate additional event vision sensor data corresponding to at least one additional event vision sensor pixel row using event vision sensor data captured via event vision sensor pixels of two or more of the one or more event vision sensor pixel rows.

20. The imaging system of claim 11, wherein: The control circuitry is configured to adjust the event vision sensor resolution of the event vision sensor data; and To adjust the event vision sensor resolution, the control circuit system is configured to interpolate additional event vision sensor data corresponding to at least one additional event vision sensor pixel column using event vision sensor data captured via event vision sensor pixels of at least one of the one or more event vision sensor pixel rows.

Images