Image sensor and method for luminous flux-to-digital conversion and pixel-by-pixel exposure encoding

Abstract

A method of luminous flux-to-digital conversion for pixels of an image sensor and the image sensor are provided. The method includes receiving light for a pixel in the image sensor while the pixel is exposed to light for an exposure period, the pixel providing an output that is a function of the received light, performing a linear fit on a transient component of the pixel output, the transient component being determined using a periodic reference voltage, and outputting the linear fit value as a digital representation of the luminous flux of the received light. The image sensor includes a first port for receiving an exposure code that constitutes the exposure of the pixel in the array, and a second port for readout of the digital representation value, where a linear fit is performed on the transient component of the readout, the transient component being determined using the periodic reference voltage, and the linear fit value is the digital representation value.

Description

[Technical Field]

[0001] The following relates generally to imaging, and more particularly to image sensors and methods for luminous flux-to-digital conversion and pixel-by-pixel exposure encoding. [Background technology]

[0002] Consumer cameras typically use computational imaging techniques to digitally enhance images through the use of software post-processing, achieving both high fidelity and low cost. For example, one technique is to combine multiple shots using different camera settings into a single enhanced image with high dynamic range (HDR). However, this post-processing-based approach can encounter many challenges, failing, for example, in the presence of fast motion and / or rapidly changing lighting. Such conditions are often present in autonomous driving, drone imaging, action camera applications, or when the lighting itself is actively controlled (e.g., depth sensing). Summary of the Invention

[0003] In one aspect, a method for luminous flux-to-digital conversion for an image sensor is provided, the method including receiving light for a pixel in the image sensor while the pixel is exposed to light for an exposure period, the pixel providing an output that is a function of the received light; performing a linear fit to a transient component of the pixel output, the transient component being determined using a periodic reference voltage; and outputting the linear fit value as a digital representation of the luminous flux of the received light.

[0004] In a particular instance of the method, the pixel output is determined by comparing the transient component with a periodic reference voltage using a binary comparator.

[0005] In another instance of the method, the line fitting comprises linear regression.

[0006] In yet another instance of the method, the timestamps of the transient components of the binary comparator output are used to estimate the instantaneous incident flux, which is equal to the slope of the fitted line.

[0007] In yet another example of the method, the binary comparator outputs a comparison over multiple periods of a periodic reference voltage.

[0008] In yet another example of the method, the periodic reference voltage is a sinusoidal reference voltage.

[0009] In yet another instance of the method, the sinusoidal voltage waveform has one or fewer sinusoidal periods for the duration of the exposure time.

[0010] In yet another example of the method, the sinusoidal reference voltage is generated using a resonance-based circuit.

[0011] In yet another example of the method, multiple transient components are generated within a single exposure period, and the multiple transient components within the single exposure period are used to determine a line in a line fitting.

[0012] In yet another example of the method, the number of cycles of the periodic reference voltage is selected to detect multiple flux values ​​during the exposure period.

[0013] In another aspect, there is provided a coded exposure pixel image sensor comprising an array of coded exposure pixels, each of which receives light, and a set of readout circuits for converting the light received at the pixels into a digital representation value that is a function of the received light, the image sensor further comprising a first port for receiving an exposure code that constitutes the exposure of a pixel in the array, and a second port for readout of the digital representation value of the pixel, wherein a linear fit is performed on a transient component of the pixel readout, the transient component being determined using a periodic reference voltage, and the linear fit value being the digital representation value.

[0014] In the particular case of an image sensor, the exposure code is received contemporaneously with the partial exposure readout of the pixel.

[0015] In another case of an image sensor, during each sub-exposure period, a digital representation is generated of only a subset of pixels that require an exposure code update.

[0016] In yet another example of an image sensor, the image sensor includes a stacked wafer fabrication, and the digital representation values ​​are generated on the underlying wafer for each pixel or group of pixels.

[0017] In yet another instance of the image sensor, the image sensor further comprises pixel exposure circuitry for generating different pixel exposure codes or for performing decompression of received pixel exposure codes.

[0018] In another aspect, a method is provided for high speed imaging over an exposure period using a coded exposure pixel image sensor, the exposure period being subdivided into a plurality of partial exposures, the image sensor including an array of coded exposure pixels, each coded exposure pixel including one or more taps, the method including: receiving, at each coded exposure pixel, an exposure code for each of the partial exposures; receiving light at each coded exposure pixel and converting the light into photo-generated charge; selectively integrating, for each partial exposure, the photo-generated charge onto one of the taps of the coded exposure pixel based on the exposure code of such pixel; exposing the photo-generated charge from one of the taps of the coded exposure pixel during successive partial exposures for each coded exposure pixel in turn; and performing a parallel readout of the collective output photo-generated charge after the exposure period.

[0019] In a particular instance of the method, each coding exposure pixel includes two taps, and the photo-generated charge is exposed from the first of the two taps.

[0020] In another instance of the method, the method further includes generating frames of high-speed video, the frames being generated after the exposure period from parallel readout of the collective output photo-generated charges by spatially demultiplexing the frames into multiple images, the number of such images equal to the number of partial exposures.

[0021] In yet another example of the method, the array of coded exposure pixels is arranged into subsets of coded exposure pixels of programmable size, the size of the subsets varying based on local motion or speed of illumination.

[0022] In yet another instance of the method, the speed of the readout increases or decreases based on the motion or changing lighting present in the scene being captured.

[0023] These and other embodiments are contemplated and described herein. It should be understood that the foregoing summary has set forth exemplary aspects of the systems and methods to aid those skilled in the art in understanding the following detailed description.

[0024] The features of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0025] [Figure 1] 10A and 10B are diagrams illustrating single-slope ADC (SS-ADC) pixel output processing in comparison with the image sensor of the present embodiment for single comparison and multiple comparison. [Figure 2] FIG. 2 is a block diagram of an example image sensor of the present embodiment and various encoding strategies implemented in the image sensor. [Figure 3] 1 is an example circuit diagram of an encoding exposure pixel and readout path, and a corresponding timing diagram. [Figure 4] FIG. 1 shows experimentally measured signal-to-noise ratio (SNR) and pixel charge transfer contrast for the image sensor of this example. [Figure 5] Figure 1 shows the raw experimentally measured output and the reconstructed HDR image, which is mapped into low-dynamic-range (LDR) space by logarithmic compression. [Figure 6] 10 is a comparative analysis table showing a comparison of the image sensor of this embodiment with other image sensors with wide dynamic range readout and / or coded exposure. [Figure 7] FIG. 10 is a diagram showing an example of a micrograph of the image sensor of this embodiment manufactured using a 110 nm CIS process. [Figure 8] FIG. 10 is a diagram showing a simulation result comparing the SNR of the image sensor of this embodiment with various VREF waveforms. [Figure 9A]FIG. 10 shows a comparison of quantization using different reference voltage waveforms. [Figure 9B] FIG. 1 is a diagram showing a prototype test PCB on which the present embodiment is implemented. [Figure 10] 10A and 10B are diagrams illustrating an example of application of the image sensor of this embodiment to direct / indirect light intensity imaging. [Figure 11] Figure 1 illustrates simplified multispectral imaging, which goes beyond RGB to yield a custom spectrum of a scene. [Figure 12] FIG. 10 is a diagram showing an example of application of depth-gated imaging to the image sensor of this embodiment. [Figure 13] 1 is a flowchart of a method for luminous flux-to-digital conversion for a pixel of an image sensor, according to one embodiment. [Figure 14] FIG. 1 shows block diagrams of equivalent single-port architectures for (a) uncoded, (b) iToF, and (c) CEP image sensors, and (d) a block diagram of equivalent dual-port architecture for the CEP image sensor of the present embodiment. [Figure 15] 1 is a block diagram of a dual-port Coded Exposure Pixel (CEP) image sensor, according to an embodiment; [Figure 16] (a) is a diagram showing a readout path incorporating coded exposure pixels and RFDC, and (b) is a timing diagram for simultaneous operation of both ports of the CEP sensor. [Figure 17A] FIG. 1 is a diagram showing a high-speed imaging configuration for TPM using the CEP sensor of this embodiment. [Figure 17B] FIG. 1 shows coded exposure, demultiplexed image, and upscaled image for a CEP sensor. [Figure 18] 1 is a diagram of a superpixel and an example of its principle of operation; [Figure 19] Figure 1 shows a block diagram of the sensor and the programming strategy for spatially varying exposure speed. [Figure 20]10A-10C show experimental results in which the sensor of the present example captured fast motion using spatially varying exposure speeds. [Figure 21] (a) shows a conventional pixel readout path using a conventional ADC, such as a single-slope ADC, to estimate light intensity; (b) shows pixel voltage waveforms in a CIS during the exposure and readout phases of a frame, illustrating how the maximum digitized light flux value is determined by the pixel's maximum full well capacity and exposure time. [Figure 22] (a) Energy-efficient high dynamic range flux estimation in a conventional pixel using a single-comparison flux-to-digital converter (FDC), and (b) a multiple-comparison regression-based flux-to-digital converter (RFDC). [Figure 23A] FIG. 1 shows the signal paths of both the SS-ADC and the FDC. [Figure 23B] 10A-10C are diagrammatic representations of pixel output and reference voltage waveforms during SS-ADC and FDC operation, respectively. [Figure 23C] 10A-10C are diagrammatic representations of pixel output and reference voltage waveforms during SS-ADC and FDC operation, respectively. [Figure 24] FIG. 10 illustrates that a noisy pixel voltage can be sampled multiple times during a cross. [Figure 25] (a) shows a sinusoidal reference voltage with up to 4 and up to 20 crosses during one exposure; (b) shows an SNR comparison between RFDCs with increasing numbers of sinusoidal crosses ranging from 1 to 32 during exposure over a wide beam range; (c) shows an SNR plot comparing a single cross FDC with an RFDC with up to 4 crosses; and (d) shows an SNR plot comparing a single cross FDC with an RFDC with up to 20 crosses. [Figure 26](a) is a block diagram of a VLSI-implemented CIS, and (b) is a schematic diagram of a dual-tap coded exposure pixel. [Figure 27A] FIG. 1 is a diagram showing an FDC / RFDC readout path. [Figure 27B] FIG. 1 is a diagram illustrating a schematic of a strong-arm latch comparator. [Figure 27C] FIG. 1 is a timing diagram of exposure and readout using an FDC. [Figure 28] Figure 1 shows a simplified basic implementation of the sinusoidal reference voltage generation concept for FDC using a digital pulse PWM INPUT controlled by an FPGA. DETAILED DESCRIPTION OF THE INVENTION

[0026] The embodiments will now be described with reference to the drawings. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Additionally, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will understand that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Additionally, the description should not be considered to limit the scope of the embodiments described herein.

[0027] Various terms used throughout this specification may be read and understood as follows, unless the context indicates otherwise: "or," as used throughout, is inclusive as if written "and / or." Singular articles and pronouns, as used throughout, include their plurals, and vice versa. Similarly, pronouns of a certain gender include their opposite pronouns, and therefore pronouns should not be construed as limiting anything described herein to use, implementation, performance, etc., by a single gender. "Exemplary" should be understood as "illustrative" or "exemplifying," and not necessarily "preferred" over other examples. Additional definitions of terms may be set forth herein. These may also apply to the preceding and following cases of these terms, as will be understood from reading this specification.

[0028] The modules, units, components, servers, computers, terminals, engines, or devices illustrated herein that execute instructions may include or otherwise be accessible by computer-readable media, such as storage media, computer storage media, or data storage devices (removable and / or non-removable), e.g., magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and that can be accessed by an application, module, or both. Such computer storage media may be part of, accessible to, or connectable to the device. Furthermore, unless the context clearly dictates otherwise, any processor or controller described herein may be implemented as a single processor or as multiple processors. Multiple processors may be arrayed or distributed, and any processing function referred to herein, while a single processor may be illustrated, may be performed by one processor or multiple processors. Any method, application, or module described herein may be implemented using computer-readable / executable instructions, which may be stored or otherwise maintained by such computer-readable media and executed by one or more processors.

[0029] The following relates generally to imaging, and more particularly to image sensors and methods for luminous flux-to-digital conversion and pixel-by-pixel exposure encoding.

[0030] As used herein, "pixel-wise" generally refers to operations or functions on a per-pixel basis, i.e., for each pixel. However, it should be understood that in some cases, pixel-wise may include operations or functions for small groups of pixels.

[0031] Some imaging environments require much stricter time integration of (1) in-pixel processing, (2) pixel readout, and (3) post-capture enhancement. For example, this occurs when there is fast motion and / or rapidly changing lighting, as in autonomous driving, drone imaging, and action camera applications, or when the lighting itself is actively controlled (e.g., in the case of depth sensing). In some cases, "coded" computational image sensors can be used to address this, using both fine (i.e., per pixel) and coarse (i.e., per pixel cluster) programmable exposure control. Some of these sensors can provide spatial exposure control for single-shot HDR imaging, but may require multiple analog-to-digital (ADC) types and / or numerous pre- and post-processing steps (e.g., adaptive per-pixel exposure control, HDR reconstruction, etc.). Other coded sensors may support various computational imaging techniques (e.g., robust depth imaging, compressed sensing), but these conventional ADCs do not provide HDR readout. Other HDR sensors can digitize the pixel output during exposure before it saturates, but generally do not provide encoding.

[0032] Other approaches to coded exposure pixel (CEP) image sensors for high dynamic range (HDR) imaging use several scene-adaptive high dynamic range (HDR) imaging techniques, including techniques operating at a frame rate and techniques operating at a partial exposure rate, both of which utilize coded exposure pixels. These techniques utilize coded exposure to prevent pixel saturation by the end of the frame's exposure phase. However, these approaches generally exhaust the functionality of coded exposure pixels only for HDR imaging, thus limiting the use of coded exposure for other computational and / or imaging applications. In addition, the sequential exposure and readout operations within a frame for such CEP sensors typically result in a blank period during the readout phase. The photo-generated charge generated during this phase is inevitably discarded, resulting in information loss.

[0033] Other approaches have used high-speed dual-tap coded-exposure data-memory pixels (DMPs). While these DMPs may allow for fast transfer of photogenerated charge, they typically scan the entire pixel array row-by-row during each sub-exposure, subsequently limiting the sub-exposure speed. This approach has not allowed for high-speed imaging with such pixels.

[0034] To overcome the challenges of the above approaches, the present embodiment provides a dual-port architecture for a CMOS image sensor (CIS) that can perform the following functions jointly (e.g., simultaneously or at different times) or independently (e.g., separately, each in its own capacity): Coded Exposure Pixel (CEP) imaging for applications such as region-of-interest (ROI) coded imaging, high-speed imaging, spatially velocity-varying imaging, and other CEP applications; and / or ·Flux-to-Digital (FDC) readout for applications such as Return Flux-to-Digital (RFDC), HDR readout, low resolution readout, and exposure-synchronous in-frame readout.

[0035] For CEP imaging, rapid pixel-by-pixel exposure encoding of a region of interest (ROI) allows such imaging to be performed quickly. Enabled by a random-access row decoder, the provided sensor architecture allows code updates only at pixels or rows of interest. Therefore, the overall duration of a partial exposure can be very short (e.g., as short as 80 ns in a specific experimental prototype when only one row of pixel codes is updated per subframe). This embodiment allows high-speed imaging to be performed using the sensor architecture. The sensor architecture also enables spatially variable-speed imaging capabilities.

[0036] With respect to FDC, the sensor of the present embodiment can include an array of flux-to-digital converters (FDCs), which can be column-shared, column-parallel, or in-pixel, the latter being more practical in stacked wafer image sensor implementations. With respect to CEP, the present embodiment can use recursive flux-to-digital conversion (RFDC), which is a special case of FDC in which regression or another technique is used to estimate the output. Also advantageously, the present embodiment can enable HDR readout and / or low-resolution readout (e.g., binary readout). Also advantageously, the present embodiment can enable exposure-concurrent in-frame readout.

[0037] The image sensor of this embodiment provides dual ports that allow not only separate engagement of the above two functionalities, but also the simultaneous synergistic combination of coded exposure pixel (CEP) imaging, which receives input through a port that may be referred to as port 1, and FDC readout, which provides output through a port that may be referred to as port 2.

[0038] In some cases, when using only one of the above-mentioned functionalities separately, other sensor architecture configurations are possible and useful, such as: CEP imaging in combination with conventional ADC readout or any other ADC, or Conventional intensity sensing with CEP pixels or any other pixel type combined with FDC readout.

[0039] To enable functionality combined with conventional ADC readout, the sensor architecture can include, for example, a bank of conventional single-slope (SS) or successive-approximation-register (SAR) analog-to-digital conversion (ADC) units that can be used for conventional intensity readout. For purposes of combining with conventional intensity sensing, the sensor CEP pixels can be configured like regular intensity pixels, or any other pixel type can be included and combined with FDC readout.

[0040] This embodiment provides an image sensor with an ADC-free flux readout scheme capable of outputting one or more digital HDR flux samples per frame. Such output can be based on the binary output of a comparator against a periodic reference waveform (e.g., a sinusoidal reference) readout at, for example, 26 kHz during exposure. This readout scheme, which may be referred to as recursive flux-to-digital conversion (RFDC), offers a wider dynamic range compared to other ADC architectures. This readout scheme is generally applicable to any pixel design and advantageously simplifies digital processing to a simple pixel-by-pixel regression. Furthermore, this embodiment of the image sensor can support pixel-by-pixel exposure encoding with updates as fast as 80 nanoseconds (ns), more than two orders of magnitude faster than other approaches. This functionality enables the capture of light with rapidly varying intensities and is independent of and complementary to RFDC schemes.

[0041] Single-slope (SS) ADCs are the architecture of choice for most CMOS image sensors (CIS), primarily due to their compactness and linearity, but they typically have limited dynamic range, speed, and energy efficiency. An SS-ADC consists of a comparator and a shared global ramp generator. As shown in Figure 1(a), the voltage corresponding to the integrated photo-generated charge is sampled at the end of the exposure period and compared to a ramp reference voltage by the SS-ADC. The comparator output transient is then time-stamped and converted to a digital number corresponding to the pixel intensity. However, the pixel's maximum full-well capacity (FWC) limits the sensor's dynamic range for high light fluxes, and its output is only available at the end of the exposure. Furthermore, SS-ADCs are generally neither particularly fast nor energy-efficient; the ramp generator often limits the ADC speed and dominates its power requirements.

[0042] In contrast, the RFDC scheme of this embodiment can estimate the light flux while the pixel is being exposed, as shown in Figures 1(a) through 1(d). Figures 1(a) through 1(d) show SS-ADC versus RFDC, using single and multiple comparisons. RFDC uses only a comparator with a sinusoidal reference voltage or any other periodic waveform reference voltage. In the version of the RFDC scheme shown in Figure 1(b), a timestamp of the comparator output transient is used to estimate the instantaneous incident flux (flux equals the slope of a line). High flux values ​​above a predetermined threshold trigger the comparator early in the exposure period, before the pixel saturates. This allows the scene to be captured with a wider dynamic range than other approaches. Because RFDC operates during exposure, no extra readout time is typically required. In the case of Figure 1(b), the sensor can advantageously operate with a sinusoidal voltage waveform with a sinusoidal period of one or fewer for the duration of the exposure time.

[0043] A sinusoidal waveform serves as a good approximation of the theoretically derived optimal reference voltage shape for HDR. Contrary to ramps or other time-varying waveforms employed in SS-ADCs, a low-noise sinusoidal voltage can be generated using negligible power. For example, it can be generated by a resonant-based circuit, such as a resonantly clocked LC tank (inductor and capacitor resonant circuit) with small resistive losses, or by a resonantly generated rectangular waveform, and may or may not be low-pass filtered. In the first example above, generating a higher-frequency sinusoidal signal for more robust and / or faster flux estimation requires only smaller values ​​of L or C, and therefore incurs a relatively small additional energy cost. We exploit this fact to further boost the SNR of the measured flux by using a multi-period sinusoidal reference voltage. Figure 1(c) shows that such an approach generates multiple comparator output transients within one exposure period, which can then be used to improve flux estimation through linear fitting. Depending on the flux, different numbers of voltage crossings may occur. In this case, low-flux pixels cross the sinusoidal reference voltage the maximum number of times (four times in the example shown), while high-flux pixels cross it at least once. Linear regression was used for the line fitting because it reduces the effect of reset noise (corresponding to an offset in the fitted line) and serves as a viable alternative to traditional correlated-double-sampling (CDS) techniques. Extending this further, as shown in Figure 1(d), the number of cycles of the sinusoid can be increased to allow for piecewise linear fitting, allowing multiple flux values ​​to be detected before resetting the pixel. This increase potentially allows for scene motion estimation within a single frame, as long as the pixel does not saturate.

[0044] In addition to the RFDC readout scheme, the image sensor of this example enables high-speed pixel-by-pixel encoding, allowing it to capture light with rapidly varying intensities. Figures 2(a) and 2(b) show block diagrams of an example image sensor and various encoding strategies implemented on the chip. Figure 2(a) illustrates the block diagram of an example image sensor with two respective ports: (1) a high-speed per-pixel exposure binary code input at a maximum rate of 12.5M pixel rows per second, and (2) a comparator output at a maximum effective rate of 26k binary pixel array readout per second, used to estimate the in-frame HDR light flux. Each port is addressable by an independent random-access row decoder.

[0045] In our demonstration experiments, up to 870 full-frame comparisons were performed at 30 fps with an IC consuming only 8.56 mW. Energy efficiency is expected to be even higher if pixels and comparators are 3D stacked with low interconnect capacitance. A wafer-stacked 1-bit binary frame buffer can be used to detect and send only the comparator transient timestamps, thereby reducing the output data rate below that of other image sensors. In some cases, linear regression may add very little power overhead because it requires computing the dot product of two low-dimensional vectors (e.g., 3-5 dimensions in Figures 1c and 1d).

[0046] In this example, pixel-by-pixel exposure encoding can be implemented using a two-tap pixel design based on indirect time-of-flight (iToF) pixels, but with two substantial differentiators: The iToF pixel only performs fixed temporal encoding (as in FIG. 2(b)), whereas in some cases the embodiment includes per-pixel charge sorting circuitry between Tap 1 and Tap 2, enabling per-pixel spatial exposure programmability. Because iToF pixels operate with phase measurements, a moderate charge transfer contrast of 60-70% is typically sufficient for iToF pixels. In some cases, however, this embodiment enables applications with more stringent contrast requirements, exceeding 90-95%, limiting the coding fractional exposure rate to approximately 12.5 MHz for higher operating iToF frequencies.

[0047] Additionally, many spatially encoded computational image sensors offer coarse-resolution encoding for pixel clusters (such as the top center of Figure 2(b)), e.g., for a 16 × 16 pixel grid. This effectively reduces spatial resolution by a large factor (e.g., by a factor of 256). In contrast, the pixel encoding in this example can be performed at native spatial resolution (such as the bottom center of Figure 2(b)), scaling well for stacked wafer technologies by local time multiplexing for small pixel groups without requiring extensive 3D interconnects or large temporal lags at each pixel. Furthermore, the encoded partial exposure rate in this example is up to 320 times higher than the fastest previous pixel-by-pixel exposure encoding implementations due to the random-access nature of the code update port (such as in the bottom of Figure 2(b)). In contrast, many encoded imaging techniques require sparse, temporary code updates (e.g., only a single row or a few rows at a time). As shown in Figure 2(a), to facilitate high-speed, low-power code delivery, the custom code generator can locally generate three common code types without having to expend power to retrieve them off-chip. It also has the ability to decompress codes received from elsewhere to reduce the incoming code data rate. In some cases, the image sensor can include exposure code circuitry that can locally generate different types of pixel exposure codes and / or perform decompression of received codes.

[0048] Figure 3 illustrates an example circuit diagram of the encoding exposure pixel and readout path, as well as the corresponding timing diagram. The pixel encoding and RFDC processes may be staggered or interleaved in time, providing additional flexibility for various computational imaging techniques.

[0049] The experimentally measured RFDC signal-to-noise ratio (SNR) and pixel charge transfer contrast are shown in Figure 4. A total dynamic range of 95 dB was measured with four VREF-VIN crossovers per RFDC output, and the SNR ranged between approximately 10 and 42 dB.

[0050] Figure 5 shows the experimentally measured raw output and the reconstructed HDR image mapped to low dynamic range (LDR) space using 4-comparison RFDC with logarithmic compression. The RFDC output is split into four images to show the output at four different VREF-VIN crossovers, one image for each VREF half-cycle during the exposure. The insets in Figure 5 show the dark and light regions of the scene mapped into different 8-bit LDR spaces.

[0051] Figure 6 shows a comparative analysis table showing comparisons with other image sensors with wide dynamic range readout and / or coded exposure for example experiments.

[0052] Figure 7 shows a micrograph of an image sensor fabricated with a 110 nm CIS process. A 3.67 mW flux-to-digital converter bank has a very small footprint and was used to obtain all experimental results (a SAR ADC is included for comparison).

[0053] FIG. 8 illustrates simulation results comparing the SNR of RFDC with various example VREF waveforms, including but not limited to sinusoidal waveforms.

[0054] FIG. 9A shows a comparison of quantization using different example reference voltage waveforms, and FIG. 9B shows a prototype test PCB implementing this embodiment.

[0055] Figures 10, 11, and 12 show example applications of the single-shot coded exposure image sensor (measured experimentally using a test chip with the same coded pixel design). Figure 10 shows a direct / indirect light intensity imaging application of the sensor, enabling imaging in highly scattering, reflective, or refractive media, and, depending on the application, avoiding and / or utilizing multipath light propagation. Charges generated by direct incident light (i.e., once reflected) and indirect incident light are split into two taps based on ray optics. Unlike the arbitrary path of indirect light, direct light travels through an "epipolar plane" from the projector to the camera. Using the pixel code as shown, charges generated by direct and indirect incident light are split into two taps.

[0056] Figure 11 shows a simplified example of multispectral imaging, where a custom spectrum of a scene can be obtained beyond RGB. Within a single frame, five LEDs with different wavelengths, λ1 through λ5, are individually illuminated, and the five code matrices are time-multiplexed across a 2x2 pixel tile and synchronously transmitted to the sensor. Five images at five wavelengths are extracted from eight taps in each 2x2 pixel tile by demultiplexing.

[0057] Figure 12 shows an example of the sensor's depth-gated imaging application. The programmable pixel sensor allows capturing two modalities: intensity and depth information, demonstrating depth-range selective imaging capability.

[0058] As shown in example experiments, the image sensor example of this embodiment provides an ADC-free light flux readout scheme that uses regression against a sine wave reference binary comparator output that is read out at 26 KHz during exposure to create one or more digital HDR light flux samples per frame. Additionally, an 80 ns sparse update per-pixel exposure encoding scheme is provided for imaging applications with fast intensity changes. The sensor has been demonstrated for HDR imaging, exhibiting a per-pixel exposure encoding rate that is over 300 times faster than other approaches.

[0059] 13 illustrates a method for light flux-to-digital conversion for a pixel of an image sensor, according to one embodiment. In block 302, the image sensor receives light flux for a pixel while the pixel is exposed to light. In block 304, a processor in the image sensor performs a straight-line fit to the transient component of the light flux relative to a multi-period sinusoidal reference voltage using linear regression. In block 306, the processor outputs the straight-line fit to storage or to interface with another device as a digital representation of the received light flux.

[0060] In some cases, multiple transients are generated within a single exposure period. The multiple transients in a single exposure period can then be used to determine the line in the line fit.

[0061] The pixel array of an image sensor resembles a random-access memory (RAM), as both function as a storage medium holding data that is accessed as needed. Because of this similarity in purpose, there is considerable overlap in their operation and design principles. In a CMOS image sensor (CIS), each pixel accumulates light and converts it into an electrical charge, which is then sequentially read out, similar to the operation of a single-port RAM. Just as data is accessed in a RAM array via word lines, rows of pixels in a CMOS image sensor can be thought of similarly. This involves a single access point for data readout, with each row accessed once at a time. However, unlike a RAM, where both updating and reading the memory elements must occur through electrical ports, in a CIS, incident light updates pixel values, and digitization occurs through the readout section. Therefore, a single-port equivalent architecture has generally been sufficient for traditional CIS architectures, as shown in Figure 14. FIG. 14 shows block diagrams of equivalent single-port architectures for (a) uncoded, (b) iToF, and (c) CEP image sensors, as well as (d) a block diagram of equivalent dual-port architecture for the CEP image sensor of this embodiment.

[0062] Coded exposure pixel (CEP) sensors receive additional input in the form of an exposure code. Indirect time-of-flight (iToF) sensors are generally the simplest form of CEP sensors. In such an arrangement, each pixel in the array receives a globally shared modulation signal as its exposure code, as shown in Figure 14(b), and no row address control is required for coded exposure. In this way, iToF sensors can operate within the framework of a single-port equivalent architecture.

[0063] The limitations of single-port architectures become apparent for coded-exposure pixel sensors, which require coded exposure either by pixel group or pixel by pixel. In such setups, multiple exposure codes must be sent to the pixel array, as shown in Figure 14(c), and the pixels must be digitized using the same row scanner or row decoder. Such sensors divide a frame into separate exposure and readout phases. The sensor row decoder is used to send codes to each pixel during the exposure phase, followed by row selection and digitization using an ADC during the readout phase. This sequence creates a blank period in the frame during the readout phase, during which no photons are captured from the scene. This design not only suffers from poor optical efficiency, but can also lead to limitations in computational imaging, such as being limited to high-speed burst imaging rather than continuous high-speed video.

[0064] In a specific embodiment, a dual-port architecture for a CEP image sensor is provided, as shown in Figure 14(d). Within this sensor, separate row decoders for exposure code delivery and readout facilitate simultaneous coding exposure and readout. This design eliminates issues associated with blanking periods observed in existing CEP sensors and improves light efficiency.

[0065] The per-pixel exposure encoding utilizes a two-tap pixel design inspired by indirect time-of-flight (iToF) pixels. Other approaches for iToF pixels span time-encoding frequencies, also commonly known as modulation frequencies, exceeding 300 MHz. However, these pixels generally do not employ standard pinned photodiode (PPD)-based pixels. Instead, the pixels rely on specialized pixel technology optimized for ToF, which increases manufacturing costs and compromises traditional intensity imaging performance. Although PPD-based iToF pixels have achieved time-encoding / modulation frequencies up to 75 MHz, the fastest reported per-pixel exposure encoding frequency remains at only 39 kHz.

[0066] In general, there are two key differences between iToF pixels and CEPs. iToF pixels only perform fixed-time encoding, as depicted in Figure 2(b). In contrast, this embodiment allows for per-pixel charge sorting between tap 1 and tap 2, facilitating pixel-by-pixel spatial exposure programmability. A moderate charge transfer contrast of 60-70% is sufficient for iToF pixels, due to its operation for phase measurements. In contrast, CEPs can support applications requiring stringent contrast requirements of over 90-95%.

[0067] As described herein, the CEP can achieve a photo-generated charge transfer rate of 80 ns with a tap contrast of over 96%, but the maximum exposure code update rate remains constrained to 39 kHz. This occurs because during each sub-exposure, all 320 rows of the CEP sensor are sequentially accessed to transfer the photo-generated charge to their respective taps. This results in a minimum sub-exposure time of 80 ns × 320 = 25.6 μs, or a maximum sub-exposure rate of 39 kHz.

[0068] Examination of exposure codes for various computational imaging applications reveals that not all per-pixel exposure codes require updating with each sub-exposure, as shown in Figure 2(b). In this example, the decoder in port 1 of the dual-port image sensor provides random access to each row, allowing for sparse exposure coding with fast updates. The camera only needs to access rows where the exposure code for one of the pixels in that row has changed. This approach minimizes the number of rows accessed with each sub-exposure, potentially to just one row, thereby reducing the sub-exposure time to the charge transfer time of the CEP required for a specified contrast.

[0069] Example experiments show that the CEP achieves a sufficiently high tap contrast of 96% within a charge transfer time of 80 ns. As a result, the fast-update CEP sensor of this example can achieve a partial exposure time as short as 80 ns (= 1 / 12.5 MHz). Therefore, the CEP sensor can reach an exposure encoding frequency of 12.5 MHz at a tap contrast of 96%, which is similar to a PPD-type iToF pixel that exhibits a modulation frequency of 75 MHz at a tap contrast of 60%.

[0070] 15 is a block diagram of a dual-port coded exposure pixel (CEP) image sensor 1500 according to an embodiment. This sensor 1500 incorporates two ports: Port 1 serves as an input port for exposure codes, and Port 2 serves as an output for pixel readout. In one example, the sensor includes a 640x480 array of dual-tap coded exposure pixels.

[0071] For port 1, in one example, the array exposure code can be distributed through a bank of 20 1:32 deserializers. Each deserializer operates in dual-data-rate (DDR) mode and can receive codes at up to 400 Mbps with a 200 MHz clock. Instead of a traditional row scanner circuit, the sensor 1500 can incorporate a row decoder, enabling random row access for fast-updating exposure codes. Consequently, in this case, the 9-bit row address should be accompanied by mask data. The array exposure code can be supplied externally or generated on-chip using a custom code generator block. This block can generate three distinct types of masks: pseudorandom, repeating tile, and sliding pattern.

[0072] In the case of port 2, in one example, such a port can function as an output port employed to read out pixel values. Sensor 1500 incorporates two data converter banks: (1) a conventional successive approximation (SAR) analog-to-digital converter (ADC) and (2) an RFDC. Each ADC / RFDC within a bank can be shared with two adjacent columns. Each ADC can operate, for example, at a clock frequency of 10 MHz, yielding a sampling rate of 769,230 samples per second. This sampling rate allows the ADC bank to reach a maximum frame rate of 400 FPS while digitizing both taps of the entire pixel array. The SAR ADC achieves a signal-to-quantization noise ratio of 68.6 dB, corresponding to 10.6 effective number of bits (ENOB).

[0073] The sensor 1500 can also incorporate an alternative HDR digitization method that utilizes the RFDC described herein. The RFDC has a significantly smaller footprint and in some cases includes only comparators. For example, it can operate at 30 FPS while making 870 comparisons per frame and achieve a total dynamic range of 95 dB. The reference voltage for each RFDC is globally shared and supplied from an external power supply.

[0074] The sensor 1500's dual-port architecture enables simultaneous exposure encoding and readout, thereby eliminating blank periods within a frame. Figure 16(a) shows a readout path incorporating the encoding exposure pixel and RFDC. The operational timing diagram shown in Figure 16(b) illustrates the simultaneous operation of both ports. Contrary to other pixel-wise encoding exposure image sensor architectures, during each sub-exposure, the sensor 1500 does not scan all rows, but instead selects only those rows that require an exposure code update. For example, Figure 16(b) shows in its upper part that a set of r rows, R[0] through R[r-1], are accessed in sub-exposure SUBEXP[2], which need not be sequentially ordered. Concurrent with the encoding exposure, a high dynamic range (HDR) readout employing RFDC occurs, as shown in the lower part of the timing diagram in Figure 16(b). In one example, for every frame, the RFDC can produce M 1-bit frame outputs by scanning all rows from 0 to V-1 (in this example, V equals 480).

[0075] In this way, the dual-port CEP image sensor 1500 provides an array of dual-tap coded exposure pixels (e.g., 640 x 480). This sensor 1500 addresses the limitations of single-port CEP architectures, significantly eliminating inefficient blanking periods and enabling simultaneous exposure coding and readout. Furthermore, it enables sparse exposure coding with fast updates, significantly increasing the partial exposure speed, e.g., to 12.5 MHz. The sensor 1500 includes two readout modes: a SAR ADC and RFDC. The SAR ADC achieves a frame rate of 400 FPS with a SQNR of 68.6 dB, while the RFDC can capture HDR images with up to 95 dB in a single shot.

[0076] Traditionally, the spatial and temporal resolutions (i.e., frame rates) of conventional image sensors have been fixed. As a result, scenes with motion faster than the frame rate appear blurred in the final image. To mitigate this problem, high-speed cameras and burst imaging sensors have been utilized. However, such high-speed cameras consume significant power due to the increased data rate, and existing burst imaging sensors face challenges such as a high pixel pitch and low fill factor due to the large number of taps per pixel. In contrast, the high partial exposure rate of the CEP image sensor in this embodiment enables the introduction of a temporal pixel multiplexing (TPM) paradigm for imaging, which enables the simultaneous capture of high-speed, high-resolution images. This is done by exposing multiple pixels in a rapid sequence, effectively performing a fast sequential phase exposure, and then parallel readout of these multiple pixels at a low output rate.

[0077] In fixed data throughput systems, there is a trade-off between spatial and temporal resolution. For high-speed applications, two approaches can be compared: (1) capture at high temporal resolution and apply spatial super-resolution in post-processing, or (2) capture at high spatial resolution and use temporal interpolation for the desired frame rate. The former approach is more appropriate because high frame rates are essential for faithfully capturing dynamic scenes, and temporal interpolation struggles to meet these requirements. Also, in high-speed settings, prioritizing temporal resolution during capture can lead to stronger super-resolution results downstream. High frame rates reduce the presence of motion blur and prevent the occurrence of temporal aliasing artifacts, both of which are challenges faced by super-resolution techniques. Ultimately, a linear increase in spatial resolution by a factor of n is only achieved with a quadratic reduction in temporal sampling (1 / n 2 ), which is a significant trade-off, especially at high speeds. Therefore, increasing the frame rate during capture is advantageous for accurately capturing high-speed events, enabling faithful dynamic scene capture, and enhancing spatial super-resolution in post-processing.

[0078] Figure 17A shows a high-speed imaging configuration for a TPM using the CEP sensor of this example. The pixel array is partitioned into m × n pixel tiles, called superpixels, and the exposure duration is subdivided into m × n sub-exposures. During each sub-exposure, a single pixel from the superpixel is configured to collect light at TAP1, while the remaining pixels accumulate light at TAP2. Consequently, each pixel in the tile captures a different 1 / (m × n) fraction of the exposure time. This configuration does not consume additional power compared to a conventional camera because all pixels are digitized at the same readout rate, e.g., 60 fps or 360 fps.

[0079] Following 25 partial exposures at 4.7 kHz, a digitized TAP1 image (320 × 320 pixels) is shown on the left side of Figure 17B. The inset highlights a 20 × 20 pixel neighborhood, and the small rectangles represent 5 × 5 pixel tiles, with each pixel capturing a unique fraction of the exposure time: 1 / 25. The TAP1 image was demultiplexed into m × n = 25 images, each with reduced spatial resolution (64 × 64 pixels) but enhanced temporal resolution (4700 fps), as depicted in the center of Figure 17B. These demultiplexed images were then upscaled using an image super-resolution solution. One of the demultiplexed images, Image 15, was upscaled and displayed on the right side of Figure 17B.

[0080] Generally, there are two significant limitations to the application of CEP sensors. First, due to the exposure dead time during ADC1 readout, CEP sensors are limited to high-speed burst imaging rather than continuous high-speed video. Photons incident on the pixel during the readout phase are discarded, resulting in a dead time. The dual-port sensor of this embodiment overcomes this drawback by using simultaneous coded exposure and readout capabilities, thereby eliminating the dead time and enabling continuous high-speed video capture via TPM. Second, any other taps, such as the second tap of a dual-tap CEP, are generally unused, and photo-generated charge collected in TAP2 is discarded. In most cases, TAP2 is exposed (m-1) times longer than TAP1, and therefore reaches saturation by the end of the frame. However, in this invention, FDC readout can be used with this dual-tap CEP sensor to obtain additional high-spatial resolution information about the scene, thereby increasing the fidelity of the scene capture, leading to improved noise robustness compared to single-tap implementations. Thus, it combines the benefits of dual-tap coded exposure and FDC readout while eliminating the alignment, calibration, and bulk issues associated with other systems.

[0081] High-speed CEP imaging not only facilitates fast motion capture through its flexible coded exposure scheme, but also reduces power consumption due to its low readout rate compared to high-frame-rate cameras. Enabled by on-chip integration of pixel-by-pixel coded exposure, such image sensors offer a cost-effective, compact, and scalable solution for achieving high sensor exposure rates while providing output compatible with conventional standard video-rate imaging pipelines.

[0082] In some cases, coded exposure pixel (CEP) imaging can be used to adapt to fast scene changes by utilizing superpixels of programmable size / speed, both locally and globally, and a conventional scalable-rate ADC (not necessarily FDC / RFDC, as described herein). In one example implementation, each superpixel contains up to 8x8 pixels exposed in rapid sequence within a single frame period, enabling up to a 64x boost in exposure rate over the output rate without a corresponding increase in ADC power. At 360 fps readout, this equates to over 23,000 exposures / second while using only 24.5 mW, at the expense of a slight loss of local resolution, but only in areas of fast change in the scene. The scalable-rate ADC allows scaling down to a 30 fps "slow" mode, achieving an additional 1 / 12 savings in output data and camera digital power, as well as a 30% savings in ADC power.

[0083] Modern applications demand image sensors capable of capturing high-speed, high-quality images at low cost while adapting to rapidly changing motion dynamics and illumination. Highly reconfigurable sensors featuring per-pixel programmable exposure and motion-adaptive frame rates typically suffer from either high speeds and uniformly low spatial resolution, or high spatial resolution and low speeds, respectively. The image sensor in this example can be used to bridge this gap, offering the flexibility to be programmed using software: (1) selectively trading spatial resolution for a high exposure rate in parts of the sensor array where fast motion is present, without compromising spatial resolution or output data rate elsewhere in the scene, and (2) dynamically adapting the readout rate based on global motion information. To achieve this, the sensors each include superpixels with programmable non-uniform exposure rates and a programmable adaptive-rate ADC.

[0084] Low-cost image sensors, such as those found in most mobile phones, have relatively slow exposure times and are not tolerant of fast motion or rapidly changing lighting, leading to degradation of image quality. High-speed image sensors avoid motion artifacts, but increase camera costs due to the hardware resources required to process and store the high-speed video output. The software-defined image sensor in this example combines the best of both worlds: it is both motion-tolerant and low-cost, capable of 23,040 exposures per second at a maximum readout rate of only 360 fps, without the corresponding increase in ADC / digital power.

[0085] FIG. 18 shows a schematic of a superpixel and an example of its principle of operation. Conventional image sensors typically employ uniform exposure of H×V pixels and a fixed-rate ADC, resulting in limited software programmability that constrains imaging speed, as shown in the top part of FIG. 18. The software-defined sensor of this example (shown in the center of FIG. 18) employs superpixels with programmable exposure rates, each containing an individually sized cluster of dual-tap pixels (shown in the bottom part of FIG. 18) that are programmable with N×N exposure phases, where N can be, for example, 1, 2, 4, or 8. A superpixel is a set of (1)N 2 The incoming light is deserialized by (1) sequentially exposing Tap 1 of each of its constituent pixels over a fast partial exposure interval, and (2) holding the photo-generated charge until the end of the exposure time, when all Tap 1 values ​​are digitized. The size and location of each superpixel can be programmed based on local motion or changing illumination information in the scene, trading local spatial resolution for local exposure speed without unnecessarily reducing spatial resolution in other regions of a static or slow-moving image. When nothing else is available, local motion / illumination can be estimated by comparing two subsequent partial exposure results accumulated at Taps 1 and 2 and calculating an optical flow map. To save power and reduce computation, these can be sparsely sampled, for example, using only one pixel in each 4 × 4 pixel subarray (as shown in Figure 18, center).

[0086] Figure 19 shows a block diagram of the sensor and the programming strategy for spatially varying exposure rates. In this example, a 640 × 480 pixel 110 nm CIS image sensor (Figure 19, left) contains a dynamically constructed mosaic of superpixels whose size and speed vary spatially. Figure 19 (right) illustrates the principle of sensor operation in a forward-facing automotive camera example. Regions near the edge of the sensor observe the fastest motion; therefore, the largest 8 × 8 superpixels are used in that region, boosting the local exposure rate by 64 times from a 360 fps readout rate to 23,040 exposures / s. The superpixel size gradually decreases toward the center of the scene, where a single unit pixel is exposed at the lowest exposure rate of 360 exp / s. The square-sized superpixels are optimally mosaicked together using quadtree tiling. The ADC readout rate is determined by externally supplied global motion information (e.g., camera motion from the accelerometer / velocimeter) and can be scaled down to 30 fps for slower motion, reducing the output data and camera digital power by a factor of 12.

[0087] Figure 20 shows experimental results in which our sensor captured high-speed motion using spatially varying exposure rates. The original scene is depicted in Figure 20 (top, left), where two high-speed fans are turned off so that the fan blades are clearly visible. The image in Figure 20 (top, right) was captured by the sensor using simultaneous uniform exposure for all pixels at 360 exposures / s and shows motion blur degradation from both fans. Figure 20 (bottom, left) shows how the sensor then adaptively reconfigures to capture the high-speed fan (approximately 2200 RPM) with 2 × 2 superpixels (highlighted square on the left) and the even faster fan (greater than 6600 RPM) with 8 × 8 superpixels (highlighted square on the right), with a 4x and 64x exposure rate speedup, respectively. The corresponding locally boosted exposure rates are 1,440 exposures / s and 23,040 exposures / s, respectively. This approach proportionally reduces the spatial resolution in these two local regions (by 1 / 2 and 1 / 8 in both the H and V dimensions, respectively), but does not adversely affect spatial resolution elsewhere in the scene, so these areas are captured at full spatial resolution (640x480) and nominal speed (360 FPS). Figure 20 (bottom, right) shows how the two low-resolution areas can be upsampled to full resolution using an appropriate upsampling technique.

[0088] As discussed herein, high dynamic range (HDR) imaging techniques exist that utilize coded exposure pixels (CEPs) to dynamically adjust the exposure code, thereby avoiding saturation. While these techniques significantly increase the inherent dynamic range of the sensor, this adjustment is necessary due to the limited readout dynamic range associated with conventional ADCs. Consequently, coded exposure is not available for other computational imaging applications.

[0089] Based on what has been provided herein, an inherently HDR quantization technique can be used that uses a flux-to-digital converter (FDC), particularly one of its special cases, the recursive FDC (RFDC). The FDC / RFDC constitutes one port of a dual-port image sensor that allows simultaneous exposure encoding and HDR readout. This architecture frees up the sensor's exposure encoding capabilities for computational imaging tasks other than HDR imaging.

[0090] Analog-to-digital converters (ADCs) play a vital role in a variety of data acquisition applications, including imaging, audio capture, instrumentation, biomedical sensing, industrial automation, and environmental monitoring. Simply put, an ADC quantizes an input analog voltage into a digital number without detailed knowledge of the input voltage waveform's characteristics. By constraining the input signal with bandwidth and amplitude requirements, ADC performance can be improved, allowing for tradeoffs among area, power, noise, and speed. For example, high-frequency signals in high-speed communications are digitized using very fast but relatively low-precision ADCs with a lower effective number of bits (ENOB), while digitizing low-frequency analog signals, such as temperature sensing, requires a slower but more accurate (higher ENOB) ADC.

[0091] An ADC is an integral part of a CMOS image sensor (CIS) and converts the light flux information converted by the pixels into analog voltages into a digital value. Traditionally, a conventional CIS uses a bank of ADCs arranged around the periphery of the pixel array. However, in some cases, stacked-wafer CIS can be used, with an ADC per pixel or per group of pixels. This approach allows for higher pixel density, lower power consumption, and better signal-to-noise ratios, as well as high dynamic range (HDR) imaging. In some cases, when stacked wafers are used in its fabrication, the image sensor can be arranged so that a digital representation of the incoming light flux can be generated on the bottom wafer for each pixel or group of pixels (ideally a small group of pixels).

[0092] Most CIS utilize conventional ADC architectures, where the physical layout size and sampling speed are constrained by the pixel pitch and frame readout rate, respectively, with the goal of minimizing quantization noise compared to thermal noise from transistors in the pixel's analog readout path.

[0093] Figure 21(a) shows a conventional pixel readout path, using a conventional ADC such as a single-slope ADC to estimate light intensity. Figure 21(b) shows the pixel voltage waveform in a CIS during the exposure and readout phases of a frame, illustrating how the maximum digitized luminous flux value is determined by the pixel's maximum saturation capacity and exposure time. The signal path in Figure 21(a) includes a photodiode that generates electrons, a tap that accumulates the photo-generated charge, and a source follower that buffers the readout line, which connects to either a column-parallel or per-pixel ADC. Among all ADC architectures, single-slope (SS) ADCs, as shown in Figure 21(a) (bottom left), have become the most common choice for CISs, primarily due to their compactness and linearity, but typically offer limited dynamic range, speed, and energy efficiency. SS-ADCs include a comparator and a shared global ramp generator.

[0094] Figure 21(b) illustrates the principle of operation, as the SS-ADC plots the pixel voltage over a frame period. The voltage corresponding to the integrated photo-generated charge is sampled at the end of the exposure period and compared by the SS-ADC to a ramp reference voltage. The comparator output transient timestamp is converted into a digital number corresponding to the pixel intensity. However, the pixel's maximum well-saturation capacity significantly limits the sensor's dynamic range for high light flux levels. Also, the SS-ADC output is only available at the end of the exposure period, as shown in Figure 21(b). Furthermore, SS-ADCs are not particularly fast or energy-efficient, and the ramp generator is often the limiting factor in terms of ADC speed and power consumption.

[0095] In conventional image sensors, all pixels are exposed for the same duration, and the ADC measures the light intensity at each pixel at the end of the exposure. For example, in the case of a SS-ADC, the timestamp of the comparator output transient is converted into a digital number corresponding to the pixel intensity and a reference voltage. However, the pixel's maximum well-saturation capacity significantly limits the sensor's dynamic range for high light flux levels, and readout path noise limits its performance for low light flux levels. In some approaches, the possibility of sampling and digitizing the photo-generated charge in the pixel during the exposure period itself enables energy-efficient single-shot high dynamic range (HDR) imaging.

[0096] This embodiment provides energy-efficient, high-dynamic-range flux estimation in conventional pixels using a single-comparison flux-to-digital converter (FDC) as shown in Figure 22(a) and a multiple-comparison recursive flux-to-digital converter (RFDC) as shown in Figure 22(b). This approach is called flux-to-digital conversion (FDC) because it directly measures flux instead of light intensity, as is done in conventional ADCs. In FDC, the timing at which photo-generated charge is sampled after the start of exposure depends directly on the incident flux value; for example, pixels capturing higher flux values ​​are digitized sooner than pixels capturing lower flux values. Some approaches provide high-dynamic-range readouts but can be prohibitively expensive to manufacture because they rely on novel pixels that require additional engineering modifications. Additionally, none of these approaches offer a unified solution for small pixel pitch, global shutter, and low-latency quantization.

[0097] One specific FDC method introduces a partial exposure rate scene-adaptive HDR imaging technique that combines coded exposure pixels with FDC for high flux values ​​and conventional ADC for low flux values. Although this is a powerful HDR imaging technique, it exhausts the coded exposure capacity of the pixel to perform HDR FDC. Therefore, the coded exposure feature cannot be used for computational imaging applications other than HDR imaging (and vice versa).

[0098] Embodiments of the present invention provide a low-power, HDR ADC-free recursive flux-to-digital converter (RFDC) implemented as part of a dual-port CIS, where both the CEP (port 1) and the FDC / RFDC (port 2) can operate independently, performing HDR readout while freeing up coding exposure capacity for other computational imaging tasks.

[0099] Single-photon avalanche diodes (SPADs) have proven to be a promising option for HDR luminous flux measurement due to their inherent ability to operate in photon-counting mode. With the potential to detect extremely low luminous flux levels down to a single photon, SPAD pixels offer the ability to measure high light levels without saturating. An in-pixel counter can be used to measure the frequency of arriving photons over a wide luminous flux range, with higher pulse frequencies corresponding to higher luminous flux values ​​and lower pulse frequencies corresponding to lower luminous flux values. This technique offers a very wide dynamic range, limited only by the regeneration time of the quench circuit and the dark current of the SPAD. However, compared to conventional CMOS imagers, SPAD pixels suffer from a larger pixel pitch, higher power-demanding peripheral circuitry, and more expensive manufacturing techniques.

[0100] CMOS image sensors using flux readout have also been used. Some CISs implement flux readout using only an FDC, while others combine an FDC with an ADC or incorporate other imaging modalities such as coded exposure pixels. For example, a digital pixel array is used to generate a digital pulse corresponding to the incident flux value after the start of exposure. The digital pixel allows the sensor to quantize the flux over a wide dynamic range with improved energy efficiency. However, the in-pixel PMOS of these CISs limits the photodetector selection and increases the pixel pitch.

[0101] In contrast, CISs feature active pixel sensor arrays with dual-gain pixels and a triple quantization scheme used for HDR imaging. This scheme includes a per-pixel FDC for quantizing high flux values ​​and a per-pixel ADC for linearly quantizing two taps of different conversion gains at each pixel. While this architecture benefits from the energy efficiency of stacked technology, it requires both an FDC and an ADC to digitize the entire intensity range, and readout can only begin at the end of the exposure. In another example, CISs can be used to utilize PMOS-free coded-exposure pixel arrays, introducing HDR imaging techniques that combine coded-exposure pixels with an FDC for high flux values ​​and a conventional ADC for low flux values. While this sensor uses embedded photodiodes to achieve a relatively small pixel pitch, it relies on the coded-exposure capability of the pixel. Additionally, the readout process requires both an FDC and an ADC, and therefore the output is only available at the end of the exposure, increasing power consumption and latency.

[0102] Embodiments of the present disclosure can use an ADC-free flux readout scheme that can output one or more digital HDR flux samples per frame based on the binary output of a sinusoidal reference comparator. Such a readout scheme can be referred to as recursive flux-to-digital conversion (RFDC). Such an approach offers a much wider dynamic range than conventional ADC architectures, is generally applicable to any (non-SPAD) pixel design, and simplifies the digital processing to a simple pixel-by-pixel linear regression. Such an approach has been verified with a CMOS image sensor (CIS). In one example, the CIS has a 640 x 480 pixel resolution array and samples light with a global shutter to avoid rolling shutter artifacts.

[0103] The FDC techniques presented here address the challenges encountered by existing FDC and SS-ADC approaches in CIS while using only simple circuit building blocks such as comparators and oscillators. As shown in Figure 22(a), the first technique uses a comparator with a half-period sinusoidal waveform as a reference voltage that varies from the pixel saturation level to the reset level. This ensures that each pixel must cross the reference voltage and thus be digitized before saturating. While this technique offers the advantages of a wider inherent dynamic range and lower implementation complexity, it is susceptible to reset noise present in the pixel tap voltage at the start of exposure.

[0104] To overcome these drawbacks, the recursive FDC (RFDC), as shown in Figure 22(b), utilizes a comparator with multiple periods (two periods in this example) of a sinusoidal waveform whose voltage ranges from the pixel saturation level to the reset level. This technique also has the potential to enable intraframe motion estimation. With multiple periods, the FDC samples the pixel output corresponding to the same flux value multiple times. These samples are then used to estimate the slope of the pixel voltage, which is the flux value of the light collected by the pixel. Because the slope calculation does not rely on the reset level of the pixel voltage, the recursive approach eliminates reset noise from the estimated flux, thereby improving SNR performance.

[0105] Unlike SS-ADC, where readout occurs after pixel exposure is complete, both of these FDC techniques estimate the light flux while the pixel is being exposed, allowing for digitization of the pixel before it is saturated. Because these FDC techniques operate during exposure, no extra readout time is required, reducing latency.

[0106] Compared to any analog signal digitized by a conventional ADC, the output voltage of a CIS pixel tap is further constrained. To illustrate how a pixel signal is readout in a CIS, Figure 23A shows the signal path from the photodiode to the ADC comparator and various noise sources along that path. When an incident photon strikes the photodiode, a photogenerated electron-hole pair is created, the probability of which is determined by the pixel's quantum efficiency, η. Typically, the photogenerated holes are discarded, and the electrons are collected at charge collection sites called taps. Through an analog buffer, the amount of photogenerated charge, represented by the pixel voltage, is sent as input to the ADC comparator. This approach allows for the conversion of the incident photon counts into a digital representation.

[0107] More specifically, the photo-generated charge is transferred to the pixel tap C FDThe charge is collected using in-pixel source followers (SFs) before being sampled by ADCs at the periphery of the pixel array. Comparators within the data converter then compare the sampled value with a known reference voltage waveform, VREF, to produce a stream of one-bit comparisons per clock cycle, which are processed to estimate the analog value.

[0108] The signal path in Figure 23A is similar for both the SS-ADC and the proposed FDC, with the key difference being the comparator reference voltage waveform VREF. Figure 23B illustrates a diagrammatic representation of the pixel output and reference voltage waveforms during SS-ADC operation, and Figure 23C illustrates a diagrammatic representation of the pixel output and reference voltage waveforms during FDC operation, this time assuming a noise-free system. The SS-ADC operates after the pixel is exposed for a fixed period of time, while the FDC readout occurs in parallel with the exposure. In the FDC, the exposure is divided into N sub-exposures, and the pixel voltage VPIX[n] is compared with VREF[n] at the end of each sub-exposure n. The following equations show the number of photo-generated electrons measured by the SS-ADC[n] and the number of photo-generated electrons measured by the FDC, corresponding to a log2(N)-bit digital number n:

number

[0109] where n represents the comparison index (e.g., SS-ADC1->0 and FDC0->1) when the comparator output toggles, N is the total number of comparisons, FWC is the maximum saturation capacity in terms of the number of electrons, VREF[n] is the reference voltage during the nth comparison, VSAT is the saturation voltage, and VRST is the reset voltage of the pixel tap. Both SS-ADC and FDC are analyzed assuming a constant incident flux, which is valid when there is no fast motion in the scene. Unlike SS-ADC, FDC does not need to wait until the end of the exposure period to digitize the flux. Instead, it quantifies the flux as soon as it reaches the level set by the reference voltage waveform, allowing it to capture a scene with a higher dynamic range. This also allows for reduced motion artifacts, especially for bright elements in the scene. Motion artifacts depend on pixel brightness and can be corrected for during post-processing in a digital processor, if necessary, similar to how motion blur correction is performed.

[0110] In reality, no electronic signal path is without noise. For a specific analysis, five main noise sources can be considered in the pixel readout path: (1) The input of the CIS-ADC is also affected by thermal noise, since it is buffered from the pixel tap through an analog amplifier to the ADC input. Resistors and transistor channels in the signal path add thermal noise to the signal. This noise is referred to herein as readout noise, V n read The main contributor is the noise V n from the in-pixel source follower. SF , and the comparator input-referred noise, V n , including clock jitter comp (2) Noise due to silicon surface defects and charge leakage through pixel taps is accumulated and becomes the dark current In dark (3)V n rstrepresents the power supply noise and the thermal noise due to the reset transistor, which is sampled at the pixel tap during the reset between two exposures. (4) The noise from the comparator reference voltage generator is V n ref (5) The pixel voltage has additional noise due to the stochastic nature of photons, known as photon shot noise, which is expressed as In shot It is expressed as: The photon shot noise follows a Poisson distribution, and the noise power is equal to the square root of the signal power. Traditionally, CIS-ADCs are designed to keep the quantization noise below the readout noise across the entire pixel voltage range. As a result, at higher intensities, the SNR of the digitized pixel output is dominated by the photon shot noise. Understanding and mitigating these noise sources can be used to achieve high-quality HDR imaging. Traditional ADCs are typically over-designed to account for the photon shot noise at higher intensities, leading to suboptimal use of resources.

[0111] To accurately simulate the performance of the SS-ADC and FDC methods, statistical noise models are employed because these models can capture the behavior of regression-type methods more effectively than analytical methods. For statistical analysis, the luminous flux, exposure time, and voltage scales are normalized so that a constant luminous flux of one unit produces an average photon count equal to the pixel's maximum saturation capacity within a unit exposure time. The following equations summarize the signal and noise models utilized in the example simulations:

number

[0112] In the above equation, EP[n] represents the total number of photogenerated electrons during sub-exposure n for a photodiode with quantum efficiency η = 1. F[n] represents the normalized flux during sub-frame n, and N represents the total number of sub-exposures in one exposure. The term ED[n] represents the total number of electrons accumulated in the tap due to leakage and dark current Id during sub-exposure n. To introduce shot noise into the ideal photon flux and dark current in the pixel tap, a Poisson probability distribution with mean μ, denoted as Pois(μ), is employed. EC[n] represents the number of electrons collected in the tap by the end of sub-exposure n. Reset noise is sampled at the beginning of exposure as EC[0]. The function Norm(μ,σ) generates random numbers based on a normal distribution with mean μ and standard deviation σ to simulate noise. FWC is the maximum well-saturation capacity of the pixel tap, expressed in electrons. V PIX[n] represents the normalized pixel voltage seen by the comparator input.

[0113] In the above equations, the variables ENOBRO, ENOBVREF, and ENOBVRST establish the standard deviation of the noisy signal, which corresponds to the noise power required to achieve the required effective number of bits (ENOB) resolution in the pixel voltage signal readout path V PIX, the reference voltage V REF, and the reset power supply V RST.

[0114] For a reasonable comparison between SS-ADC and FDC, both quantization methods were simulated at a unit exposure time using the same number of comparator clock cycles required for a 10-bit SS-ADC, N=1024. VREF is also the maximum voltage level required to realize a 10-bit SS-ADC, i.e., ENOB. VREF The pixel taps are assumed to have a maximum well-saturation capacity (FWC) of 10,000 electrons, which allows for a maximum photon shot noise limited SNR of 40 dB. The readout noise floor is set at 54 dB below the saturation level, which is equivalent to 9 bits, or ENOB. RO= 9. This ensures that the noise floor of the readout path exceeds the quantization noise in the SS-ADC, especially in low light conditions.

[0115] To highlight the effect of reset noise, a relatively high reset noise power, i.e., ENOB, is used, which is 45 dB lower than the full-scale signal power. VRST Simulations were performed with I = 7.5. The cumulative charge leakage through the pixel tap and dark current was set to I = 400 electrons per unit of exposure time, a reasonable assumption for the CEP used in this study. The dark current follows a Poisson distribution, resulting in a noise power approximately 54 dB below the pixel's full well capacity.

[0116] It turns out that conventional CIS ADCs typically employ correlated double sampling (CDS) to mitigate the effects of reset noise and charge leakage at pixel taps. Consequently, at low intensities, the readout noise of the signal path becomes the dominant contributor to the output noise power, and the quantization error of the SS-ADC is less than the readout noise. However, as the light intensity increases, the shot noise from the incident photons becomes the dominant noise source. This transition leads to an increased headroom between total noise and quantization error, which can be perceived as an inefficiency in terms of ADC performance parameters such as power, speed, and dynamic range.

[0117] The FDC method described in this example addresses the inefficiency of such data converters and enhances their inherent dynamic range through a non-uniform quantization process. Simulation results of the FDC confirmed its ability to digitize a significantly wider flux range while employing the same number of digital codes as the SS-ADC. In contrast to conventional CIS-ADCs, the FDC facilitates sampling of flux before pixel tap saturation and digitizes flux above the pixel's saturation limit, thereby increasing the dynamic range. At intensities 20 dB above the pixel's maximum saturation capacity, the pixel's SNR performance is limited by quantization error. However, at flux values ​​near the pixel's maximum saturation capacity, the FDC's SNR performance is limited by photon shot noise, similar to the performance of the SS-ADC.

[0118] Simulations further revealed that at lower luminous flux levels, FDC performance is hindered by reset noise. Due to reset noise, the pixel voltage, denoted V PIX , does not always start at the ideal V RST . However, conventional CDS techniques for mitigating reset noise are not directly compatible with FDC methods. As an alternative to CDS, a recursive FDC technique is presented herein designed to suppress reset noise and increase the SNR up to the limits imposed by leakage current and dark current. In fact, in single-cross FDC, leakage current and dark current through the pixel tap have the second-highest noise contribution to the total noise power.

[0119] During the simulation, the readout noise and reference noise were set to 54 dB and 60 dB below the full-scale signal power, respectively, yet their actual noise contributions were observed to be even lower. This phenomenon becomes more apparent when examining the temporal voltage crossover between VREF and VPIX in more detail.

[0120] Figures 24(a) and 24(b) show that a noisy pixel voltage can be sampled multiple times during a crossing. Figure 24(a) shows an example of a noisy readout signal from two pixel taps sampled with a noisy VREF signal. Figure 24(b) shows the average number of comparator samples at a single crossing FDC using a sinusoidal reference voltage for a practical case of a noisy signal and a noisy reference voltage.

[0121] At low light intensities, pixel tap voltages, which are heavily affected by readout noise, are sampled multiple times, producing multiple measurements of the same luminous flux within a given exposure time. These FDC outputs are then averaged, thereby reducing the readout noise contribution to the total noise power. An example of this process is illustrated in Figure 24(a), which shows the output voltages of two pixels, each receiving a luminous flux that varies with a sinusoidal reference voltage. The output voltage of PIXEL1 varies at a slower pace than the output voltage of PIXEL2, resulting in three FDC samples from PIXEL1 but only a single FDC sample from PIXEL2. The multiple FDC samples are averaged, thereby suppressing the readout noise contribution in PIXEL1's output.

[0122] Figure 24(b) demonstrates the average number of samples (and their standard deviation) generated across the entire flux range in the presented simulation. The amount of samples steadily decreases as the flux increases, until at high flux values, only a single FDC sample is available within a frame. Therefore, we can conclude that averaging multiple noisy FDC samples significantly reduces the contribution of readout noise and reference voltage noise at low flux values. Such averaging is not feasible with conventional CIS-ADCs, which digitize a single sampled value at the end of the exposure.

[0123] Overall, the single-cross FDC method of digitizing CMOS pixel voltage output offers several advantages over traditional ADC methods, including the ability to capture a wider dynamic range and reduced frame latency. The use of nonuniform quantization and multiple sample averaging reduces readout noise at low light flux levels. This was possible due to the inherent signal constraints of pixel voltages, such as linearly increasing voltages and the signal-to-noise ratio (SNR) limited by inherent photon shot noise. Simulation results demonstrate the effectiveness of the FDC method in addressing the limitations of traditional ADC designs for CIS. In these simulations, the single-cross FDC uses a half-cycle of a sinusoidal signal as the reference voltage.

[0124] The inventors investigated the effect of various reference voltage waveforms VREF on non-uniform quantization in the FDC and its overall performance, with the aim of identifying the most suitable waveform. SNR analysis is performed for various reference voltage waveforms expressed by the following equation:

number

[0125] where n represents the number of comparisons at the end of the sub-exposure and N represents the total number of sub-exposures. During the entire exposure duration, the reference voltage waveform VREF CONST remains constant. V REF RAMP V REF transitions linearly from saturation to the reset value. SINE V REF varies sinusoidally from saturation to the reset value, with half the period equal to the total exposure time. ANGLE V REF samples the flux uniformly over a radial (angular) region. QUAD samples the flux so that the quantization error is always proportional to the photon shot noise.

[0126] The simplest reference voltage waveform for an FDC is V REF CONST, which is a constant voltage throughout the exposure period. Although this waveform is easy to generate, the FDC only digitizes it when the flux exceeds a certain threshold. As a result, additional techniques are required to quantize lower flux levels, such as coded exposure pixels or dual-gain pixels, with additional assistance from the conventional ADC. Furthermore, at higher flux values, the signal-to-noise ratio (SNR) is significantly lower than V REF SINE It is inferior to FDC using

[0127] Another waveform V REF RAMP V REF transitions linearly from the saturation level V SAT to the reset level V RST over the entire exposure period. As a result, every flux value is sampled only once during the exposure period. RAMP The SNR performance of the FDC using V REF SINE Therefore, the performance of the FDC using a ramp generator circuit is nearly identical to that of the FDC using a RAMP It may be assumed that no additional power needs to be expended to generate a low-noise sinusoidal voltage. Unlike ramp waveforms or other time-varying waveforms employed in SS-ADCs, a low-noise sinusoidal voltage can be generated using minimal power. For example, it can be achieved by a resonant clocked LC tank with small resistive losses.

[0128] Unlike a conventional ADC, which divides the pixel intensity into N uniform quantization steps, the reference voltage V REF ANGLE The FDC using V REF can divide the luminous flux into uniform radial steps of π / 2N radians each. ANGLE The SNR performance of the FDC using VREF at low luminous flux levels is SINE However, at higher luminous flux levels, the performance is comparable to that of V REF SINE is consistently V REF ANGLEAlthough the immediate application for such quantization may not be obvious, it highlights the flexibility of the FDC for reference voltages, allowing users to quantize the light flux according to their specific requirements. For example, to capture HDR scenes, a quantization strategy that exploits the inherent photon shot noise can be used, ensuring that the quantization error is always proportional to the photon shot noise. This is achieved by discarding the least significant bits, which correspond to the shot noise in the uniformly quantized digital number, without any loss of information. In an FDC, V REF QUAD The same quantization can be achieved using a reference voltage waveform, but this quantization strategy comes at the cost of higher waveform generation complexity and lower dynamic range compared to other non-stationary waveforms.

[0129] These simulation studies suggest that a sinusoidal waveform is most appropriate for the FDC, providing a wide dynamic range while conserving power. Interestingly, the SNR performance of the FDC was only slightly affected by small variations in the reference voltage waveform, as demonstrated by VREFRAMP and VREFSINE. This also suggests that a clean, undistorted, single-tone sinusoidal waveform is not a strict requirement for the reference voltage waveform. As long as the reference voltage remains periodic between frames, the single-cross FDC and multi-cross RFDC continue to perform well. Therefore, a sinusoidal waveform can be used.

[0130] According to this embodiment, a multi-cross recursive FDC (RFDC) with a sinusoidal reference voltage can be used. Such an approach overcomes the limitations of a single-cross FDC while still maintaining its advantages over conventional CIS-ADCs. RFDC improves the SNR of the measured flux using multiple samples (other than those due to noise) and reduces the contribution of reset noise. Multiple samples from RFDC also enable estimation of motion within a single exposure, which can be used to further reduce motion artifacts.

[0131] The RFDC method of this embodiment can utilize a multi-period oscillating sinusoidal reference voltage. This option offers advantages in terms of both area and SNR performance without a significant power penalty. Generating a higher frequency sinusoidal signal for more robust and rapid flux estimation requires only a small additional energy cost. For example, it simply requires smaller values ​​of inductors or capacitors in the resonator, thereby reducing the area.

[0132] Figures 25(a) through 25(d) show simulated performance of recursive flux-to-digital conversion (RFDC) using a sinusoidal reference voltage. Figure 25(a) shows a sinusoidal reference voltage with up to 4 and up to 20 crosses during one exposure. Figure 25(b) shows an SNR comparison between RFDC with increasing numbers of sinusoidal crosses ranging from 1 to 32 during exposure over a wide flux range. The SNR plots compare single-cross FDC and RFDC with up to 4 crosses in Figure 25(c) and up to 20 crosses in Figure 25(d).

[0133] Figure 25(a) shows two examples of sinusoidal reference voltages used in RFDC, allowing up to four and up to 20 crossings of the reference voltage and pixel tap voltage. This approach provides multiple comparator output transients spread over a single exposure period, which can be used to improve flux estimation. Depending on the flux, different numbers of voltage crossings can occur. In this case, low-flux pixels cross the sinusoidal reference voltage the maximum number of times (four in the example shown), while high-flux pixels cross it at least once. Linear regression is used for the line fitting, but linear regression also mitigates the effects of reset noise (corresponding to an offset in the fitted line) and serves as a viable alternative to traditional correlated double sampling (CDS) techniques.

[0134] Figure 25(b) maps the SNR of the RFDC method using a sinusoidal reference voltage and varying the number of crosses from 1 to 32 within a single exposure period. For a more thorough analysis, Figures 25(c) and 25(d) display the SNR plots for RFDC using a sinusoidal reference voltage, allowing up to 4 and up to 20 crosses, respectively. Both of these figures also plot the SNR of single-cross FDC using a sinusoidal reference voltage, which is limited by reset noise. From the figures, it is clear that the RFDC method consistently outperforms single-cross FDC, especially at lower flux values. RFDC mitigates the effects of reset noise by fitting a straight line through multiple crosses at the same flux value. However, a degradation in SNR is observed at higher flux values, and this degradation is more pronounced at 20 crosses than at 4 crosses. This degradation is due to the increased quantization error at higher flux values ​​as the reference voltage frequency increases. Because the number of partial exposures is fixed, the larger the reference voltage step VREF[n]-VREF[n+1] between two adjacent comparisons, i.e., the larger the slope, the larger the quantization error. Therefore, in this study, we adopted a sinusoidal waveform with a maximum of four crosses per flux value in a single exposure. This is an acceptable tradeoff between the benefit of the number of crosses at lower flux values ​​and the reduced SNR at higher flux values. For applications where motion estimation is more important than SNR, a higher-frequency sinusoidal reference voltage waveform may be used.

[0135] Linear regression was chosen for RFDC because it is best suited for application-specific integrated circuit (ASIC) design. Compared to other types of regression, such as polynomial regression and logistic regression, linear regression requires fewer computational resources and is more power-efficient. Linear regression is implemented using simple arithmetic operations, such as multiply and accumulate (MAC), which can be efficiently implemented using hardware circuits. In contrast, other types of regression, such as polynomial regression, require more complex mathematical operations, such as exponential calculations, which are computationally expensive and require more power. Logistic regression requires additional operations, such as sigmoid functions, which increase computational cost. Furthermore, linear regression is more easily scalable than other types of regression, making it suitable for large-scale image sensors. Its simplicity of implementation allows for more parallel processing, which significantly increases processing speed and reduces power consumption. The simplicity, scalability, and low computational requirements of linear regression make it a suitable choice for many applications requiring high processing speed and low power consumption.

[0136] The performance of RFDC using other reference voltage waveforms, specifically triangular and sawtooth waveforms, was also investigated. The results showed that both of these waveforms performed comparable to a sinusoidal waveform when the number of crossovers was low. However, as the number of crossovers increased, the performance deteriorated. The choice of reference voltage waveform is highly dependent on the specific application and design constraints, but a sinusoidal reference waveform provides a good choice.

[0137] Figure 26(a) shows a CIS block diagram of a VLSI implementation, and Figure 26(b) shows a schematic of a dual-tap coded exposure pixel. Figure 26(a) presents a block diagram of the image sensor. The sensor in this example contains a front-illuminated array of 640 x 480 pixels. Along the periphery of the pixel array is a bank of 340 comparators designated for RFDC. The outputs from these comparators are serialized and then sent to the FPGA by a bank of 17 20:1 serializers capable of operating at a frequency of 200 MHz.

[0138] The sensor's pixel array consists of a dual-tap coded exposure pixel (CEP). A recessed photodiode (PPD) within the pixel generates electrons from incoming photons. These photogenerated electrons are then transferred to an intermediate storage diode, which acts as a global buffer when the TG_GLOB signal is triggered simultaneously for all pixels, thereby enabling global shutter operation. When a row is selected using the ROW SEL signal, charge is transferred from the storage diode to one of the taps in response to the digital CODE signal. An in-pixel source follower buffers this charge onto a column-parallel readout line that extends to comparators at the periphery of the pixel array. Its NMOS-only structure enables the pixel to achieve a small coded exposure pixel pitch of 7 μm and a 38.5% aperture ratio. The signals DRAIN and RST are used to reset the photodiode and tap, respectively, at the beginning of each frame.

[0139] RFDC can be utilized in any CMOS pixel, not necessarily a CEP. Therefore, for simplicity, the pixel is configured as a conventional single-tap global shutter pixel by setting signals CODE=1 and TAP SEL=0. The resulting effective single-tap pixel structure is shown in Figure 27A. This configuration ensures that photo-generated charge is always collected in TAP1, while disabling the other tap, TAP2.

[0140] FIG. 27A shows the FDC / RFDC readout path, FIG. 27B shows a schematic of the strong-arm latch comparator, and FIG. 27C shows a timing diagram for exposure and readout using the FDC.

[0141] Figure 27A shows a signal path implemented for pixel readout using FDC / RFDC. This signal path includes a comparator, as shown in Figure 27B, with one terminal connected to a pair of adjacent readout lines from odd and even columns via an analog multiplexer. The other terminal is connected to a reference voltage. The sensor has a bank of 340 comparators around its periphery, with 320 comparators connected to the 640 columns of the pixel array and an additional 20 comparators used for general test and debug. The reference voltage terminals of all comparators are connected to one of the chip's analog IO pads, giving the user flexibility to supply reference voltages according to the needs of their application.

[0142] Figure 27C presents a timing diagram of sensor operation. Both exposure and readout occur simultaneously within a frame. The exposure time is divided into N sub-exposures, numbered 0 through N-1. RFDC comparison of the flux captured in sub-exposure i is performed during sub-exposure i+1. Assertion at the end of the sub-exposure. When the photodiode collects the flux for sub-exposure i+1, RFDC operates on the photo-generated charge accumulated in the pixel taps by the end of sub-exposure i. The COL_SEL signal sequentially links the comparator bank to the odd and even columns of the array within a given sub-exposure. For each state of COL_SEL, the row selector cycles through all row addresses. This method of scanning the array is preferred over switching between odd and even columns row by row because it allows equal settling time for both odd and even columns, thereby reducing column-locked pattern noise.

[0143] This comparator uses a strong-arm latch architecture and can function at clock frequencies exceeding 32 MHz, sufficient to perform up to 1000 full-frame 1-bit comparisons at 30 FPS for RFDC readout, consuming only 8.56 mW. Energy efficiency is expected to be even higher when pixels and comparators are 3D stacked and interconnect capacitance is low. A 1-bit binary frame buffer can be used to detect and send only the comparator's transient component timestamps, reducing the output data rate below that of most conventional image sensors. Linear regression adds very little power overhead and requires the calculation of the dot product of two low-dimensional vectors.

[0144] Figure 28 shows a simplified, basic implementation of the sinusoidal reference voltage generation concept for the FDC using a digitally pulsed PWM input controlled by an FPGA. Simulations confirmed that a sinusoidal signal serves as the optimal choice for the RFDC reference voltage waveform. This reference voltage signal can be easily generated using digital control signals and passive components. As demonstrated in Figure 28, the reference voltage signals for the FDC are bundled together and connected directly to the preamplifier gate of a strong-arm latch. This signal operates at a very low frequency and consumes negligible current. For a frame rate of 30 frames per second (FPS) and a sinusoidal reference voltage with a maximum of four crossings, the FDC requires a sine wave with a frequency of 60 Hz. To generate this waveform, an LC oscillator tuned to achieve a resonant frequency of 60 Hz and driven by a digital signal of the same frequency can be used, as depicted in Figure 28. The amplitude and offset of the reference signal VREF can be modified by controlling the duty cycle of the digital signal and the potentiometer RPOT, respectively. Doing so ensures that the generation of the reference voltage requires negligible power.

[0145] The generated reference voltage waveform will not be a single-tone sine wave due to non-idealities and component mismatches, but V REFRAMP and VREF SINE As demonstrated by the comparison, the FDC performance remains largely unaffected. However, accurate measurement of VREF[n] for each comparison is generally critical for FDC calculations. Therefore, after finalizing the parameters for the LC tank, RPOT, and duty cycle, the generated reference voltage can be accurately measured once using an oscilloscope and then utilized for all subsequent FDC calculations. The on-chip bandwidth of the reference voltage IO pin is significantly higher than the reference voltage frequency because it is constrained by the interconnect metal resistance and parasitic / gate capacitance, which have relatively low values. As a result, degradation of the reference voltage signal due to on-chip parasitic values ​​can be minimized. Therefore, a low-power reference voltage generation method is suitable for RFDC.

[0146] This sensor was compared with other CIS and SPAD imagers, all of which offer some form of light-to-digital conversion. The sensor uses embedded photodiodes and can be fabricated using a standard 110nm CMOS image sensor process. This sensor achieves the smallest pixel pitch among non-stacked sensors, enabling global shutter exposure while offering the highest pixel resolution compared to all other CIS.

[0147] The FDC method described herein can be utilized with any conventional global shutter or rolling shutter pixel architecture that provides the smallest pixel pitch. Further scaling can be achieved by integrating the FDC architecture with stacking technology. Per-pixel stacking reduces the capacitance of the readout lines, thereby simultaneously increasing the power efficiency of the sensor. Additionally, the stacked architecture eliminates the need to scan every row for every FDC comparison. This reduction in partial exposure time helps further reduce quantization noise, increase dynamic range, and improve SNR at higher flux values.

[0148] The performance of the FDC is demonstrated at 30 FPS, achieving a dynamic range of 95 dB while consuming 8.58 mW of power. The low power consumption of the FDC in this example is due to the power-efficient generation of a sinusoidal reference voltage, the reduced sampling rate, and the negligible static power consumption from the strong-arm latch comparator. In some cases, the linear regression for the RFDC can be implemented on-chip because it relies on a simple low-dimensional vector MAC operation.

[0149] FDCs have demonstrated the ability to digitize very high luminous flux values. The FDC architecture relies on continuous integration of photo-generated charge within the pixel tap, i.e., floating diffusion, which is subject to charge leakage and dark current over time. The inclusion of an in-pixel metal-insulator-metal (MIM) capacitor for charge collection, as opposed to a MOS capacitor or any other pixel leakage optimization, reduces this leakage and increases low-light sensitivity.

[0150] Thus, while multi-cross RFDC improves SNR by capturing multiple samples of the same flux value, this technique can be further extended to enable motion estimation. Changes in scene motion result in a shift in flux at a given pixel. If a conventional ADC were to digitize the intensity captured by this pixel, the measurement would contain motion blur. However, to estimate this flux change, multiple piecewise linear fits can be performed on subgroups of samples.

[0151] The recursive flux-to-digital converter (RFDC) in this example provides ADC-free HDR quantization in CIS. Using a detailed simulation model, the impact of various noise sources on SNR was analyzed for both single-cross FDC and multi-cross RFDC methods. These FDC methods were experimentally verified using a 640 x 480 CMOS image sensor. The FDC achieved a total dynamic range of 95 dB while consuming 8.56 mW while operating at 30 FPS. The multi-cross sinusoidal reference voltage waveform in the RFDC samples the same flux values, on which a linear regression is performed to mitigate reset noise.

[0152] While the present invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the appended claims.

Claims

1. 1. A method for luminous flux-to-digital conversion for an image sensor, comprising: receiving light for a pixel within the image sensor while the pixel is exposed to light for an exposure period, the pixel providing an output that is a function of the received light; performing a straight line fit to a transient component of a pixel output, the transient component being determined using a periodic reference voltage; outputting the straight line fitting value as a digital representation of the received light flux; A method comprising:

2. The method of claim 1 , wherein the pixel output is determined by comparing the transient component to the periodic reference voltage using a binary comparator.

3. The method of claim 1 , wherein the line fitting comprises linear regression.

4. 3. The method of claim 2, wherein a timestamp of the transient component of the binary comparator output is used to estimate an instantaneous incident flux, said instantaneous incident flux being equal to the slope of the fitted line.

5. The method of claim 2 , wherein the binary comparator outputs a comparison over multiple periods of the periodic reference voltage.

6. The method of claim 1 , wherein the periodic reference voltage is a sinusoidal reference voltage.

7. 7. The method of claim 6, wherein the sinusoidal voltage waveform has no more than one sinusoidal period for the duration of the exposure time.

8. The method of claim 6 , wherein the sinusoidal reference voltage is generated using a resonance-based circuit.

9. The method of claim 1 , wherein multiple transient components are generated within a single exposure period, and the multiple transient components within the single exposure period are used to determine a line in a line fitting.

10. The method of claim 1 , wherein the number of cycles of the periodic reference voltage is selected to detect multiple flux values ​​during the exposure period.

11. 1. A coded exposure pixel image sensor comprising an array of coded exposure pixels, each of which receives light, and a set of readout circuits that convert the light received at said coded exposure pixels into a digital representation that is a function of said received light, a first port for receiving an exposure code that configures the exposure of the coded exposure pixels in the array; a second port for reading out the digital representation value of the coded exposure pixel, wherein a linear fit is performed on a transient component of the readout of the coded exposure pixel, the transient component being determined using a periodic reference voltage, and the linear fit value is the digital representation value; 2. The coded exposure pixel image sensor, further comprising:

12. The image sensor of claim 11 , wherein the exposure code is received contemporaneously with a partial exposure readout of the coded exposure pixels.

13. The image sensor of claim 12 , wherein during each sub-exposure period, the digital representations are generated for only a subset of pixels requiring an exposure code update.

14. The image sensor of claim 11 , comprising a stacked wafer fabrication, wherein the digital representation values ​​are generated on a lower wafer for each pixel or group of pixels.

15. The image sensor of claim 11 , further comprising a pixel exposure circuit for generating different pixel exposure codes or for performing decompression of received pixel exposure codes.

16. 1. A method for high speed imaging over an exposure period using a coded exposure pixel image sensor, wherein the exposure period is subdivided into a plurality of sub-exposures, the coded exposure pixel image sensor comprising an array of coded exposure pixels, each coded exposure pixel comprising one or more taps, the method comprising: receiving an exposure code for each of said sub-exposures at each coded exposure pixel; receiving light at each coded exposure pixel and converting said light into photo-generated charges; For each partial exposure, selectively integrating the photo-generated charge onto one of the taps of the coded exposure pixel based on the exposure code of the each coded exposure pixel; exposing the photo-generated charges from one of the taps of the coding exposure pixel during successive partial exposures for each of the coding exposure pixels in turn; performing a parallel readout of the collectively output photo-generated charges after the exposure period; A method comprising:

17. 17. The method of claim 16, wherein each coding exposure pixel includes two taps, and the photo-generated charges are exposed from a first of the two taps.

18. 17. The method of claim 16, further comprising generating frames of high-speed video, the frames being generated from the parallel readout of the collective output photo-generated charges after the exposure period by spatially demultiplexing the frames into multiple images, the number of images equal to the number of partial exposures.

19. 17. The method of claim 16, wherein the array of coded exposure pixels is arranged into subsets of coded exposure pixels of programmable size, the size of the subsets varying based on local motion or speed of illumination.

20. 17. The method of claim 16, wherein the speed of the readout increases or decreases based on motion or changing lighting present in the scene being captured.

Images