Method and system for extending image dynamic range using pixel-by-pixel encoding of pixel parameters - Patents.com

Abstract

A method and system for extending the dynamic range of an image sensor using coded pixels is provided, the method including providing pixel codes to pixels in a photodetector array for each subframe of a frame, receiving sensor readouts for each of the pixels in the photodetector array for each subframe, routing each sensor readout to one or more taps for collection or to a drain based on the pixel code for each subframe, combining the collected sensor readouts at each of the taps to determine a single pixel value for the frame, and outputting the single pixel value for each pixel for the frame.

Description

[Technical field]

[0001] The following relates generally to imaging, and more particularly to methods and systems for extending image dynamic range using pixel-by-pixel encoding of pixel parameters. [Background technology]

[0002] Imaging sensors, such as those present in still and video cameras, have multiple photoconductors, or sensors, or photodetectors, such as pinned photodiodes. Typically, each photoconductor is a device fabricated using semiconductor technology, such as complementary metal-oxide semiconductor (CMOS). Photons of light are collected in photosites of the photoconductor, such as pn junctions. A pixel is the basic cell of an image sensor array. Typically, there are one or more photosites for each pixel. Typically, photons are directed through one or more lenses to the photoconductor of the imaging sensor. For each photosite, a charge is generated in the semiconductor of the photoconductor, such as silicon, and this photo-generated charge is typically proportional to the intensity of the received light. The value of each charge is turned into a digital value by an analog-to-digital converter (ADC) and used to generate images as still photographic images and as frames in videos. Summary of the Invention

[0003] In one aspect, a method for extending the dynamic range of an image sensor using coded pixels is provided, the image sensor comprising a photodetector array for converting incoming optical signals into photo-generated electronic signals, the method comprising providing a pixel code to pixels in the photodetector array for each subframe of a frame, receiving a sensor readout of each of the pixels in the photodetector array for each subframe, routing each sensor readout to one or more taps for collection or to a drain based on the pixel code for each subframe, combining the collected sensor readouts at each of the taps to determine a single pixel value for the frame, and outputting the single pixel value for each pixel for the frame.

[0004] In a particular example of the method, the pixel code comprises a code corresponding to the exposure time of the respective pixel, and the resulting photo-generated charge comprises the respective sensor readout.

[0005] In another particular example of the method, the pixel code comprises a code corresponding to a signal gain of the signal received at the photodetector.

[0006] In another particular example of the method, each pixel code is applied reciprocally to a group of pixels.

[0007] In yet another example of the method, the pixel codes are predetermined.

[0008] In yet another example of the method, the pixel code is adaptively determined based on previously received pixel values.

[0009] In yet another example of the method, pixel codes are preloaded into a code memory store based on previously received pixel values.

[0010] In yet another example of the method, the method further comprises sorting the collected charge at each tap based on the pixel signature for the previous subframe at the same time that the sensor readout for the current subframe is collected.

[0011] In yet another example of the method, each pixel comprises one tap and the pixel code for each pixel comprises a 1-bit binary number.

[0012] In yet another example of the method, a pixel code for one or more of the pixels defines one or more taps each having an individually programmable integration time interval.

[0013] In yet another example of the method, the average of the integration time interval is equalized over a number of frames.

[0014] In another aspect, a system for extending the dynamic range of an image sensor using coded pixels is provided, the image sensor comprising a photodetector array for converting incoming optical signals into photo-generated electronic signals, the system comprising circuit logic configured to execute: an encoding module for providing pixel codes to pixels in the photodetector array for each subframe of a frame and for each subframe routing a photo-generated electronic signal value of each one of the pixels to one or more taps for collection or to a drain based on the pixel code; a high dynamic range (HDR) decoding module for combining the collected signal values ​​available at each of the taps to determine a single pixel value for the frame; and an output module for outputting the single pixel value for each pixel for the frame.

[0015] In a particular example of a system, the pixel code comprises a code corresponding to the exposure time of the respective pixel, and the resulting photo-generated charge comprises the respective sensor readout.

[0016] In another example of the system, the pixel code comprises a code corresponding to a signal gain of the signal received at the photodetector.

[0017] In yet another embodiment of the system, each pixel code is applied to a group of pixels in turn.

[0018] In yet another example of the system, pixel codes are adaptively determined based on previously received pixel values.

[0019] In yet another example of the system, the encoding module uses pixel codes loaded into the code memory store based on previously received pixel values.

[0020] In yet another example of the system, the encoding module classifies the collected charge at each tap based on the sensor reading and pixel code for one or more previous subframes.

[0021] In yet another example of the system, each pixel comprises one tap and the pixel code for each pixel comprises a 1-bit binary number.

[0022] In yet another example of the system, each pixel in the photodetector array includes one or more charge collection nodes, and the charge collection nodes from one or more pixels are combined to modify the signal gain based on the pixel code.

[0023] In yet another example of the system, the pixel code defines a pixel exposure time for a pixel that senses the time of flight of an incoming optical signal.

[0024] In yet another example of the system, the pixel code defines a signal gain for a pixel that senses the time of flight of an incoming optical signal.

[0025] In yet another example of the system, the HDR decoding module comprises an analog-to-digital converter (ADC), and the resolution of the ADC is different for frame digitization compared to sub-frame digitization.

[0026] In yet another example of the system, the HDR decoding module includes an analog-to-digital converter (ADC), and one or more parameters of the ADC are dynamically adjusted with respect to one or more subframes or one or more frames.

[0027] In another aspect, a method is provided for extending an image dynamic range using a pixel-by-pixel coded exposure and a previous readout of a sensor using such pixel-by-pixel coded exposure, the method comprising: generating a current code matrix for a current coded exposure derived from the previous readout; transmitting the current coding matrix to the pixels; receiving a current raw sensor readout for each of the pixels; reconstructing a high dynamic range (HDR) image by processing the current raw sensor readout based on a pixel-by-pixel exposure time; and outputting at least one of the HDR image and the current raw sensor readout.

[0028] These and other embodiments are contemplated and described herein. It is understood that the above summary presents various aspects of the systems and methods to aid those skilled in the art in understanding the following detailed description.

[0029] The features of the present invention will become more apparent in the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a schematic diagram of a system for extending the dynamic range of an image sensor using encoded pixels, according to an embodiment. [Diagram 2] FIG. 1 is an example of a single-frame high dynamic range (HDR) approach. [Diagram 3] FIG. 1 is an example of a spatial light modulator (SLM) approach. [Figure 4A] 1 is a flowchart of an example of an open-loop high dynamic range (HDR) imaging method, according to an embodiment. [Figure 4B] 1 is a flowchart of an example of a closed-loop HDR imaging method, according to an embodiment. [Figure 5A] FIG. 1 illustrates a conventional image sensor exposure technique. [Figure 5B] FIG. 1 illustrates an exposure technique for a coded exposure pixel (CEP) image sensor. [Figure 6A] FIG. 1 illustrates an example exposure function of a two-tap CEP image sensor in which the two taps are controlled by an arbitrary binary code. [Figure 6B] FIG. 13 shows an example exposure function of a two-tap CEP image sensor in which the two buckets are controlled by a fully arbitrary binary code. [Figure 6C] FIG. 1 illustrates an example exposure function of an indirect time-of-flight (iToF) pixel controlled by two complementary 1-bit binary codes that demodulate or drain the photo-generated charge. [Figure 7A] FIG. 1 is a circuit diagram of an example of a coded pixel (CP) pixel architecture showing multiple taps and including both coded exposure pixel (CEP) and coded gain pixel (CGP) functionality. [Figure 7B] FIG. 1 is a circuit diagram of an example of a coding pixel (CP) pixel architecture in which an additional intermediate charge storage node is added. [Figure 7C] FIG. 1 is a circuit diagram of an example of a coded pixel (CP) pixel architecture showing multiple taps and including both coded exposure pixel (CEP) and coded gain pixel (CGP) functionality to sense the time of flight of photons coming from an illumination source to a photodetector. [Figure 7D] FIG. 13 is a circuit diagram of an example of a coding pixel (CP) pixel architecture that senses the time of flight of incoming photons from an illumination source to a photodetector, in which an additional intermediate charge storage node is added. [Figure 8A] 1 is a spatiotemporal visualization of an example pixel exposure code for one tap pixel. [Figure 8B] 1 is a spatiotemporal visualization of an example pixel exposure code for a 2-tap pixel. [Figure 8C] 13 is a spatiotemporal visualization of an example pixel exposure code for a 1-tap pixel, where both the subframe period and the number of subframes are dynamically adjusted per frame, and there are 5 subframes per frame in the nth frame. [Figure 8D]FIG. 13 is a spatiotemporal visualization of an example pixel exposure code for a 1-tap pixel, where both the subframe period and the number of subframes are dynamically adjusted per frame, and there are four subframes per frame in the (n+1)th frame. [Figure 9A] FIG. 13 is a circuit diagram for two examples of a two-tap pixel for HDR imaging by using two exposure codes for two intra-pixel taps. [Figure 9B] FIG. 13 shows an example where the exposure periods for both taps are continuous and start from the beginning of the frame. [Figure 9C] FIG. 13 shows an example in which tap exposure is set by an arbitrary time code. [Figure 9D] 1 is a plot of simulated per-tap signal-to-noise ratio (SNR) versus incident light intensity for a single two-tap pixel. [Figure 10A] FIG. 13 is a circuit diagram for two examples of a 4-tap pixel for HDR imaging by using four exposure codes for four intra-pixel taps. [Figure 10B] FIG. 13 shows an example in which the exposure periods for all taps are consecutive and start from the beginning of the frame. [Figure 10C] FIG. 13 shows an example in which tap exposure is set by an arbitrary time code. [Figure 10D] 1 is a plot of simulated per-tap signal-to-noise ratio (SNR) versus incident light intensity for a single 4-tap pixel. [Figure 11A] 1 is a diagram of a neighborhood of one tap pixels of size TILEx pixels by TILEy pixels that is repeated across the pixel array. [Figure 11B] FIG. 1 shows an example in which the exposure periods for all pixels are consecutive and centered halfway through the frame time. [Figure 11C] FIG. 13 shows an example of an arbitrary time code programmed for each of the pixels in a tile. [Figure 11D] 1 is an example plot of simulated per-tap signal-to-noise ratio (SNR) versus incident light intensity for a neighborhood of four 1-tap pixels. [Figure 12A] 1 is a diagram of a 2-tap pixel neighborhood of size TILEx pixels by TILEy pixels that is repeated across the pixel array. [Figure 12B] FIG. 13 shows another example in which the exposure periods for all pixels are consecutive and start from the beginning of the frame. [Figure 12C] FIG. 13 shows another example of an arbitrary time code programmed for each of the pixels in a tile. [Figure 12D] 13 is another example plot of simulated per-tap signal-to-noise ratio (SNR) versus incident light intensity for a neighborhood of four 1-tap pixels. [Figure 13] An example of a 2x2 1-tap pixel tile where each exposure code is rotated and repeated for four frames. [Figure 14] FIG. 2 is an example of a top-level system block diagram of a CP image sensor. [Figure 15A] 1 is a flow chart of a 2-tap coded exposure pixel (CEP) architecture for a coded memory pixel (CMP) pixel architecture. [Figure 15B] 1 is a flowchart of a generalized 2-tap coded exposure pixel (CEP) architecture. [Figure 15C] 1 is a flow chart of a 2-tap coded exposure pixel (CEP) architecture for a data memory pixel (DMP) architecture. [Figure 16A] FIG. 1 is a circuit diagram of an example of a CMP pixel architecture using a global drain. [Figure 16B] FIG. 1 is a circuit diagram of an example of a DMP pixel architecture using a global drain. [Figure 16C] FIG. 1 is a circuit diagram of an example CMP pixel architecture with per-pixel coded drain. [Figure 16D] FIG. 1 is a circuit diagram of an example of a DMP pixel architecture using per-pixel coded drain. [Figure 17A] FIG. 13 is a circuit diagram of another example of a CMP pixel architecture using a global drain. [Figure 17B]FIG. 13 is a circuit diagram of another example of a DMP pixel architecture using a global drain. [Figure 17C] FIG. 13 is a circuit diagram of another example of a CMP pixel architecture using per-pixel coded drain. [Figure 17D] FIG. 13 is a circuit diagram of another example of a DMP pixel architecture using per-pixel coded drain. [Figure 17E] 1 is an example timing diagram for CMP and DMP pixel architectures. [Figure 18A] 1 shows an example layout and potential diagram for an example 2-tap CMP image sensor pixel. [Figure 18B] 1 shows an example layout and electrogram diagram for an example 2-tap DMP image sensor pixel. [Figure 19A] FIG. 13 is a circuit diagram of an example of a 2-tap pixel with per-pixel coded exposure and global gain control for a CMP architecture. [Figure 19B] FIG. 1 is a circuit diagram of an example of a 2-tap pixel with per-pixel coded exposure and global gain control for a DMP architecture. [Figure 20A] FIG. 13 is a circuit diagram of an example of a 2-tap pixel with per-pixel coded exposure and per-pixel gain control for a CMP architecture. [Figure 20B] FIG. 1 is a circuit diagram of an example of a 2-tap pixel with per-pixel coded exposure and per-pixel gain control for a DMP architecture. [Figure 20C] FIG. 1 is a circuit diagram of an example 4-tap pixel with per-pixel coded exposure and per-pixel gain control for an indirect time-of-flight (iToF) CMP architecture. [Figure 20D] FIG. 13 is a circuit diagram of an example of a 4-tap pixel with per-pixel coded exposure and per-pixel gain control for the iToFDMP architecture. [Figure 21A] FIG. 1 is a diagram of an example of a digital readout circuit having a programmable resolution analog-to-digital converter (ADC). [Figure 21B]FIG. 13 illustrates an example mode of operation when the ADC resolution is coarse, for example as implemented by a comparator with a fixed or variable VREF. [Figure 21C] FIG. 13 is a diagram showing an example of an operation mode when the ADC resolution is set to fine resolution. [Figure 22A] 1 is a chart showing examples of possible pixel tap values ​​when the coarse ADC reference voltage VREF is held constant and charge accumulation at a given tap stops with a latency of one subframe after the tap value reaches VREF. [Figure 22B] 1 is a plot of SNR versus incident light intensity illustrating the variation in signal quality with code changes from subframe to subframe for a constant coarse ADC reference voltage VREF. [Figure 22C] 1 is a chart showing examples of possible pixel tap values ​​when the coarse ADC reference voltage VREF changes every subframe and charge accumulation at a given tap stops with a latency of one subframe after the tap value reaches VREF. [Figure 22D] 1 is a plot of SNR versus incident light intensity illustrating the variation in signal quality with sign changes from subframe to subframe when the coarse ADC reference voltage VREF is time-varying. [Figure 23] 1 shows an example of frame-wise dynamic spatio-temporal encoding of pixel exposure times. [Figure 24A] 4 illustrates another example of spatiotemporal coding of pixel exposure times within a single frame. [Figure 24B] FIG. 11 is a circuit schematic diagram of an example one-tap embodiment for each pixel. [Figure 25A] 1 is an example of a unary weighted pixel exposure time setting and code sequence. [Figure 25B] 1 is an example of binary weighted pixel exposure time settings and code sequences. [Figure 26A] 25B is a plot of an example of raw pixel output over a full range of illumination for the unary weighting scheme of FIG. 25A. [Figure 26B] 25C is a plot of an example of pixel raw output over the full range of illumination for the binary weighting scheme of FIG. 25B. [Figure 27A] FIG. 26B is a plot of an example of a normalized version of FIG. 26A. [Figure 27B] FIG. 26C is an example plot of the normalized version of FIG. 26B. [Figure 28] 1 is a flowchart illustrating an example of a pipeline implementation. [Figure 29] An example using pixel-wise coded exposure is shown in the figure. [Diagram 30] 30 shows example plots of raw output and HDR reconstruction for the example of FIG. 29 . [Diagram 31] 1 is an image of a scene used in an experimental example. [Figure 32A] 32 shows images captured at low exposure for the example experiment of FIG. 31. [Figure 32B] 32 shows images captured at high exposure settings for the example experiment of FIG. 31. [Diagram 33] A comparison of the results of the example experiment of FIG. 31 for three different encoding schemes is shown. [Diagram 34] A typical diagram of the encoding scheme is shown. [Diagram 35] This is a typical very large scale integrated circuit (VLSI) architecture. [Figure 36A] 1 is an image of an example camera having an image sensor. [Figure 36B] 1 is a typical chip micrograph for an image sensor. [Figure 36C] 3C shows the specifications and dynamic range of the example image sensor of FIG. 36B. [Figure 37] 4 is a flowchart of a method for extending an image dynamic range using pixel-wise coded exposure, according to an embodiment. [Figure 38A] 1 is an example of temporal pixel coding. [Figure 38B] 1 is an example of partial space-time coding. [Figure 38C] This is an example of pixel-based coding exposure coding. [Figure 39] 4 is a flow chart of a method for extending the dynamic range of an image sensor using encoded pixels, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The embodiments are now described with reference to the drawings. For simplicity and clarity of illustration, reference numerals may be repeated among the drawings where appropriate to indicate corresponding or similar elements. In addition, numerous specific details are described to provide a thorough understanding of the embodiments described herein. However, it will be understood by those skilled in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Additionally, this description should not be considered as limiting the scope of the embodiments described herein.

[0032] Various terms used throughout this description may be read and understood as follows, unless the context dictates otherwise: "or" as used throughout is inclusive, as is "and / or." Singular articles and pronouns used throughout include their plurals and vice versa. Similarly, gender pronouns include their counterparts, and should not be understood as limiting anything described herein to a single gender use, implementation, performance, or the like. "Exemplary" should be understood as "exemplary" or "exemplary," and not necessarily "preferred" over other embodiments. Further definitions of terms may be presented herein. These may also apply to previous and subsequent examples of these terms, as will be understood upon reading this description.

[0033] Any module, unit, component, server, computer, terminal, engine, or device that executes instructions as illustrated herein may include or have access to a computer-readable medium, such as, for example, a storage medium, computer storage media, or data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tapes. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storing information, such as, for example, computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store a desired method and that can be accessed by an application, a module, or both. Any such computer storage media may be part of a device or may be accessible or connectable to a device. Furthermore, unless the context clearly indicates otherwise, any processor or controller presented herein may be implemented as a single processor or multiple processors. The multiple processors may be arrayed or distributed, and any processing function referred to herein may be performed by multiple processors or one of them, even when a single processor is illustrated. Any method, application, or module described herein may be implemented using computer readable / executable instructions that may be stored or carried by such computer readable medium and executed by one or more processors.

[0034] The following relates generally to imaging, and more particularly to methods and systems for extending image dynamic range using pixel-wise coded exposure.

[0035] As used herein, "pixel-by-pixel" or "pixel-wise" refers to operations or functions on a pixel-by-pixel basis, but may in some cases include operations or functions on smaller groups of pixels.

[0036] The limited dynamic range of cameras using active pixels allows the capture of bright or dark scenes, but not both at the same time. To address this tradeoff, several high dynamic range (HDR) techniques have been introduced. Native resolution full frame rate HDR approaches add significant auxiliary circuit overhead to the pixel, such as multi-gain pixel readout, event-based readout, or range compression transfer functions (e.g., logarithmic), but often suffer from low fill factor and non-idealities of in-pixel analog circuitry. If frame rate can be traded for dynamic range, exposure often varies over many frames. This approach in many cameras generally requires a proportionately higher frame rate or may result in motion blur and burn-in. A single-frame HDR approach is used that uses, for example, 2×2 tiles of programmable pixels, each with a single storage node programmed with different successive exposure time intervals starting from the beginning of the frame, such as short, medium, long, and very long. This approach is illustrated in FIG. 2. This approach generally suffers from a quarter-fold reduction in image resolution due to its structure and does not allow for arbitrarily programmable non-continuous exposure time intervals. Other single-frame HDR approaches use a controllable 2D optical attenuator, such as a spatial light modulator (SLM) or digital micromirror device (DMD), in front of the camera, but this approach is generally not scalable to high image sensor spatial resolution, has high camera module complexity and high cost, and is prone to optical distortion. This approach is illustrated in FIG. 3.

[0037] Exposure or exposure time is the time during which a photon of light reaches a photosite and can be sensed by the corresponding photoreceptor, read out, transferred to a storage node, drained, or otherwise used within the pixel. The limited dynamic range of many image sensors in cameras allows the capture of either bright or dark scenes, but not both for a given frame.

[0038] Pixel gain or signal gain is the factor by which a pixel input signal, for example a light or light generated signal, is scaled as the signal passes to the output of the pixel.

[0039] Image sensor pixels can be used to estimate incident light intensity or other light properties, such as the time it takes for light to travel to a sensor. Time of Flight (ToF) is the time it takes for light to travel from a light source to a light receiver, such as a photodetector. Indirect Time of Flight (iToF) methods measure this time of flight without an explicit timer. For example, they use a modulation signal, such as a periodic pulse waveform, to modulate the transmitted light, and a demodulation signal, such as another periodic pulse waveform of the same frequency. The time can be determined by measuring the phase difference between the received modulated light signal and the demodulation signal.

[0040] A pixel code or signature is a numerical value or signal that conveys information used by a pixel to set, for example, pixel exposure time, pixel gain, or other parameters of the pixel. It may be represented by an electronic signal, for example voltage, current, charge, or time, and may be represented as a digital or analog signal, and may be represented in continuous time, discrete time, or other representation schemes.

[0041] Any-value pixel code or arbitrary pixel code refers to a pixel code representation that is not limited to any subclass of all possible code values, and the code may have any value at any time as desired or set by the system or user. For example, any sequence of discrete-time 1-bit binary pixel codes may take on a value of 0 or 1 for any code sample in the sequence as desired.

[0042] Some general-purpose coded exposure imaging systems are bulky and expensive because they use spatial light modulators (SLMs), digital micromirror devices (DMDs), or liquid crystal on silicon (LCoS) to pass or block the light coming into each 1-tap pixel of the camera. According to embodiments of the present disclosure, a multi-tap coded exposure pixel (CEP) image sensor may provide various advantages, such as (1) potentially small form factor and low cost due to pixel programmability directly in CMOS-IS technology, (2) good optical signal fidelity because external optical devices are often not required, (3) good optical efficiency because photo-generated charge can be sorted among multiple taps rather than discarding light when the 1-tap pixel is "off", and (4) low noise (compared to constant speed high frame rate cameras that perform a noise-inducing readout for every exposure) because photo-generated charge can be selectively accumulated over multiple time intervals and read out only once per frame.

[0043] Embodiments of the present disclosure provide methods that may be used to extend the dynamic range of an image sensor using coded pixels (CPs), such as (1) coded exposure pixels (CEPs), where each pixel or group of pixels may be programmed or controlled to have its own per-pixel / per-group (herein referred to as per-pixel) exposure time, (2) coded gain pixels (CGPs), where each pixel or group of pixels may be programmed or controlled to have its own per-pixel / per-group signal gain, and (3) other types of pixels, where each pixel or group of pixels may be programmed or controlled to have its own operating parameters, which may include exposure, gain, and / or any other parameters. Exposure time (also referred to herein as exposure) refers to the time interval(s) during which one or more photodetectors are exposed to light, or the time interval(s) during which one or more photodetectors are effectively exposed to light, to generate an electronic signal that is a function of an input optical signal. Signal gain (also referred to herein as gain) refers to any type of gain starting from the incoming optical signal all the way to the output in the signal path, such as, for example, the gain of an optical and / or electro-optical modulator, photo-generated charge conversion gain, voltage gain, current gain, digital gain, or any other type of gain. A photodetector (or photosite) is a device that converts an incoming optical signal into a photo-generated electronic signal, such as, for example, a photo-generated charge, voltage, or current. Examples of photodetectors include photodiodes, pinned photodiodes, photogates, phototransistors, or any other photosensitive electronic device.

[0044] 1, a system 100 for extending the dynamic range of an image sensor using encoded pixels is shown, according to an embodiment. As will be appreciated by those skilled in the art, in some cases, some components of system 100 may be implemented in separate hardware implementations. In other cases, some components of system 100 may be implemented in one or more general-purpose processors, which may be distributed locally or remotely.

[0045] FIG. 1 illustrates various physical and logical components of an embodiment of a system 100. As illustrated, the system 100 has multiple physical and logical components, including one or more processors 102, data storage 104, output interface 106, CP image sensor 110, encoding module 112, HDR decoding module 114, output module 116, and a local bus 118 that allows the components to communicate with each other. In an embodiment, the modules may execute on one or more processors 102. In other embodiments, the modules may be implemented in hardware, via a dedicated processor, or on a remote processor over a network. In some cases, the functionality of the modules may be combined or further separated, as appropriate. In some cases, the output interface 106 may be in communication with a display device or other computing device for outputting the output of the system 100. In other embodiments, non-essential elements of the system 100 may be omitted or combined with other elements.

[0046] The dynamic range of a CP image sensor can be extended by several different approaches, as illustrated in the flow charts shown in Figures 4A and 4B. Figure 4A illustrates an open-loop high dynamic range (HDR) imaging method according to an embodiment. To extend the dynamic range of the sensor, pixel-by-pixel operating parameters, such as exposure and pixel-by-pixel gain codes, are presented to the photodetector array without information from the scene currently being imaged. By way of example, these codes can be learned online or offline by one or more artificial intelligence machine learning algorithms based on available image data and other information, such as other current or past data from other sensors.

[0047] 4B illustrates a closed-loop HDR imaging method according to an embodiment. In addition to the functionality described in FIG. 4A, the pixel signature can be dynamically adapted based on the current and / or past scene(s) being imaged, such as the intensity information of each pixel and any other information about the scene available in the current or past frame or sub-frame.

[0048] It is an advantageous property of coded pixel (CP) image sensors that the operating parameters of individual pixels, e.g., exposure time and signal gain, can be programmed using pixel codes. Pixel codes in CP image sensors may have various types, formats, and representations. For example, FIG. 5B shows an example diagram of the operation of coded exposure pixel (CEP) image sensors, a subclass of CP image sensors in which the exposure time of each pixel is programmable. A conventional image sensor exposure technique is shown in FIG. 5A, which can be compared to the CEP image sensor exposure technique shown in FIG. 5B as an example of a one-tap CEP image sensor. The taps can be implemented as capacitors, MOS capacitors, trench capacitors, light-shielded floating diffusion nodes, light-shielded pinned diodes, or by any other means. In FIG. 5B, the total exposure interval of one frame is divided into multiple (N) programmable subintervals, known as "subframes" for time control of fine pixel-by-pixel exposure as controlled by digital coefficients, or "codes" transmitted to all or a portion of the pixels in each subframe. In this example, since the pixel has only one tap that can collect charge or not, a one-bit binary code can be used to select one of these two functions in a given subframe. As a result, in each subframe, the photo-generated signal is either integrated into the in-pixel tap or not, depending on whether the pixel code value is 1 or 0, respectively. If the photo-generated charge is not integrated into the tap in a given subframe, it is drained in the example of a one-tap pixel and may be shifted to another tap if multiple taps are available in the pixel as described herein.

[0049] 6A-6C show an example of the exposure function of a CEP image sensor for pixels with two or more taps per pixel. This example shows only two taps for ease of explanation. FIG. 6A shows an example of the exposure function of a two-tap CEP image sensor in which the two taps are controlled by an arbitrary binary code, which in this example is a complementary 1-bit binary number for the two taps. The 1-bit binary code (taking values ​​0 or 1) can represent either code 1 or code 2, corresponding to charge accumulation at taps 1 and 2, respectively. Since there is no representation for code 0, this does not generally allow for pixel-by-pixel programmable photodetector charge draining in individual subframes, but allows for simultaneous global charge draining for all pixels. In this example, only a 1-bit binary code per pixel is required, resulting in a compact pixel implementation. FIG. 6B shows an example of the exposure function of a two-tap CEP image sensor in which the two buckets are controlled by a completely arbitrary (i.e., not necessarily complementary) binary code. This allows for programmable photodetector draining on a pixel-by-pixel basis in individual subframes, but requires multiple bits to represent the pixel code. In this example, three states are required, codes 0, 1, and 2, as shown in FIG. 6B. This requires a two-bit binary code to encode up to four states. The remaining unused fourth state can be used for programmable gain control within the pixel, as described below with respect to the Coded Gain Pixel (CGP) architecture. FIG. 6C shows an example of pixel-by-pixel encoding for an indirect time-of-flight (iToF) pixel. In iToF sensing or imaging, two taps are required to demodulate the light-generated signal as needed to estimate the distance to the scene. The example of FIG. 6C shows how such a pixel can be selectively turned on and off with codes 1 and 0, respectively. This means that when the pixel receives code 1, light-generated signal demodulation occurs, and when the pixel receives code 0, no demodulation occurs. Light demodulation in various challenging conditions, such as when strong background illumination is present, presents a challenge that is addressed in the present invention by increasing the dynamic range of the pixel, including, for example, the iToF pixel architecture.

[0050] Each pixel or pixel group in the CP imager pixel may have its own operating parameters programmed with an arbitrary time code for each pixel or pixel group. For example, the operating parameters may include exposure time in the example CEP architecture and signal gain in the example CGP architecture. FIG. 7A shows an example of a coded pixel (CP) pixel architecture that combines both coded exposure pixel (CEP) and coded gain pixel (CGP) architectures per pixel. In a subframe, the photo-generated charge is transferred from the photodetector to one or more of the taps or drained to another node, e.g., VDD, as controlled by the pixel's exposure code. Thus, within a frame, the total photo-generated charge may be divided into a fraction, one per subframe, and sorted among multiple taps. Each tap may be symbolically represented, for example, as a capacitor that stores the photo-generated charge transferred from the photodetector. The charge transfer timing and duration may be controlled by the per-pixel exposure code provided to some or all pixels in some or all subframes. The signal gain may be symbolically represented by a variable value property of the capacitor, as shown in FIG. 7A. This symbolic representation may be explained by the following example. The same value of photo-generated charge shifted to two different value capacitors results in different voltages on those capacitors, and therefore different signal gains. For example, the size of each tap of each capacitor, and therefore the size of the signal gain of each tap, may be controlled by a per-pixel gain code provided to some or all pixels in some or all subframes. The tap capacitance may be modulated, for example, by combining or binning several floating diffusion nodes into one (one of these may be the drain node if not connected to a low impedance node), by connecting the tap to additional capacitance, or by other approaches.

[0051] FIG. 7B shows another example of a CP pixel in which an additional intermediate charge storage node is connected to a diode SD. Such an additional storage node provides, for example, a simple overall pixel circuit and a reduced overall pixel size. This node is symbolically represented as being connected to a diode symbol, and in fact a storage diode is an example for implementing such a storage node, but is not limited to such. In this example, the photo-generated charge is first shifted globally to node SD synchronously for all pixels, and then sorted among one or more taps based on the exposure code provided. The example architecture shown in FIG. 7A is referred to as a code memory pixel (CMP), and the example architecture shown in FIG. 7B is referred to as a data memory pixel (DMP). Both architectures may include various additional circuits within and outside the pixel as necessary to implement correlated double sampling and / or double sampling to reduce reset noise, flicker noise, electronic circuit offsets, and other low-frequency noise and offsets.

[0052] FIG. 7C shows an example of an indirect time-of-flight (iToF) CP pixel architecture with demodulation of the photo-generated charge onto a preselected tap pair as controlled by an arbitrary exposure code. Since this architecture combines both CEP and CGP functions, the pixel gain is also programmable by a variable charge storage capacitance value. During a subframe, the photo-generated charge is modulated by the signal MOD and its complement,

number

number

[0053] FIG. 7D shows another example of an iToF CP pixel architecture in which two additional intermediate charge storage nodes are introduced at the top terminals of the two storage diodes L-SD and R-SD. These storage nodes are connected to the signals MOD and

number

number

[0054] FIG. 8A shows an example of a spatiotemporal visualization of an arbitrary exposure code for a pixel having one tap (i.e., a one-tap pixel) for N subframes (here N=4) in a frame. x Pixels × FRAME y Each of the spatial dimensions of the pixel is shown (this example is for FRAME x = 5 pixels and FRAME y (The kth subframe duration T is 1.) The t axis corresponds to time. In this example, there is one charge transfer per subframe. A code 0 means that the charge is drained and a code 1 means that the charge is shifted to the storage node. k and the code c for a given pixel in that subframe k Both of these can be chosen arbitrarily. Thus, the programmable exposure time for that pixel is

number

[0055] The variable optional subframe duration and optional subframe count are further illustrated in Figures 8C and 8D for the example of a one-tap pixel. A spatiotemporal map visualization of the exposure signature for pixels of frames n and n+1 is shown, where the kth subframe duration T k may be dynamically adjusted for each subframe. The number of subframes for each frame may also be dynamically adjusted. In this example, there are five subframes per frame in the nth frame in FIG. 8C, but four subframes per frame in the (n+1)th frame in FIG. 8D. Thus, the kth subframe duration T k and the code c for a given pixel in that subframe k Both can be selected arbitrarily.

[0056] An example of a particular pixel exposure code may be a code where the total time a pixel is exposed to light may be expressed as a function, such as a ramp function, an exponential function, a binary weighting function, a logarithmic function, or any other analytical or non-analytic time function or any other parameter. In addition, the arbitrary nature of the pixel code is advantageous in other ways. For example, for a two-tap pixel exposure code, the exposure control may be defined more generally than just a continuous exposure time. In particular, if the tap code is 1 for k of the N subframes, then the exposure time for that tap is k. However, there are k-choose-N different ways to implement that exposure time (e.g., various ways of permuting the 1's and 2's in the code sequence). These ways may not be equivalent if the scene is non-static. For example, for compressed HDR video, the exact subframes where the pixel code is 1 or 2 must be controlled, not simply their total number. Also, the arbitrary nature of the pixel code may be the most flexible representation of the code, making it optimal for both open-loop and closed-loop HDR imaging. In many cases, the embodiments of the present disclosure may be utilized in both open-loop and closed-loop HDR imaging.

[0057] The examples shown in FIGS. 8A-8D show examples of programmable exposure per pixel. Similarly, system 100 may implement programmable gain per pixel. For example, photo-generated charge may be transferred to multiple capacitive nodes in a single subframe. Such capacitive nodes may be, for example, floating diffusion nodes corresponding to one or more taps, and floating diffusions of the drain nodes if not connected to a low impedance node. The overall tap well capacitance may be increased and pixel conversion gain reduced. In other examples, additional gain control elements may be used, including those described with respect to the example pixel circuits described herein.

[0058] Coded Exposure Pixel (CEP) image sensors are a subclass of CP image sensors. CEP image sensor pixels with two or more taps can be used to perform HDR imaging by programming different exposure codes for each tap. FIG. 9A shows a simplified example of two two-tap pixels that can be used for such purposes: a code memory pixel (CMP) at the top and a data memory pixel (DMP) with an additional charge storage node at the bottom. Each of the two taps in the pixel has an arbitrary programmable exposure time that ranges from zero subframe to the full subframe period. In this example, two complementary binary codes are used to collect all photo-generated charge (i.e., for maximum signal). FIG. 9B shows an example of two codes where the exposure times for both taps are consecutive and start from the beginning of the frame. FIG. 9C shows another example where the tap exposure intervals are not necessarily consecutive. An arbitrary time code can be programmed for each of the two taps in the pixel. The completely arbitrary nature of the codes allows additional degrees of freedom over non-arbitrary codes, for example, for improved dynamic range and / or other performance improvements. FIG. 9D shows an example of a plot of simulated signal-to-noise ratio (SNR) versus incident light intensity for each of the taps in a two-tap pixel. To illustrate intuitively, the only noise included in the simulation example shown in FIG. 9D is photon shot noise. In other examples, further improvement of SNR can be obtained by combining signals from multiple taps. When the light signal level is near the boundary between two light intensity subranges corresponding to two different taps, the SNR plot reaches a local minimum. Although FIGS. 9A-9D use a CEP architecture as an example, the same concept can be applied to all CP pixel architecture types, such as a coded gain pixel (CGP) pixel architecture or an indirect time-of-flight (iToF) coding pixel architecture.

[0059] FIG. 10A shows an example of two 4-tap pixel diagrams that may be utilized for HDR imaging: a CMP pixel architecture at the top and a DMP pixel architecture at the bottom. Each of the four taps has an arbitrary programmable exposure time that ranges from zero subframe to the full subframe period. FIG. 10B shows an example of such four codes, where the exposure intervals for all taps are consecutive and start from the beginning of the frame. FIG. 10C shows another example where the tap exposure period is controlled by an arbitrary time code. In this example, two complementary binary codes are used to collect all the photo-generated charge (i.e., for maximum signal). FIG. 10D shows an example of a plot of simulated signal-to-noise ratio (SNR) versus incident light intensity for each of the taps in a 4-tap pixel. For clarity of illustration, the only noise included in this simulation example is photon shot noise, and the individual tap signals are not combined to increase the SNR. 10D shows that the increased number of subranges (four versus two in the previous example) results in less noticeable SNR degradation at the boundaries between light intensity subranges corresponding to different taps. Although FIGS. 10A-10D show a CEP architecture as an example, in additional examples this may be applied to other CP pixel architecture types, such as a coded gain pixel (CGP) pixel architecture or an indirect time-of-flight (iToF) coded pixel architecture.

[0060] The codes of the CEP image sensor pixels can be used to group pixels into neighborhoods, such as spatial tiles, to trade dynamic range for improved spatial resolution. The system can advantageously do this using the code of any value sent to each pixel, without having to hard-code or hard-wire this grouping function in the image sensor diagram or layout. FIG. 11A shows a TILE, which is repeated throughout the pixel array. x =2 and TILE y TILE of 1 tap pixel for =2 x ×TILE yFIG. 11B shows an example of a pixel code that results in a spatial configuration. Each of the four pixels has an arbitrary programmable exposure time that ranges from zero subframe to a full subframe period. FIG. 11B shows an example of four such codes. In this example, the exposure intervals for all pixels are continuous and centered on the half-frame time point to reduce motion blur non-uniformity between the variously exposed pixels. An example is to use four levels of exposure (1, N / 3, 2N / 3, and N, where N is the number of subframes) distributed over a 2×2 Bayer RGB-style tile block that can then be demosaiced to obtain a high dynamic range. FIG. 11C shows another example where the pixel exposure intervals are not necessarily continuous. Since an arbitrary time code can be programmed for each of the four pixels in the tile, additional degrees of freedom are possible, for example, for improved dynamic range and / or other performance improvements. FIG. 11D shows an example of a plot of simulated signal-to-noise ratio (SNR) versus incident light intensity for each of the four taps. For clarity of illustration, the only noise included in this stimulus is photon shot noise, and individual tap signals are not combined to increase the SNR. FIG. 11D shows that the SNR reaches a local minimum for light signal levels near the boundary between light intensity subranges corresponding to different pixels. Although FIGS. 11A-11D show a CEP architecture as an example, in additional examples, this may be applied to other CP pixel architecture types, such as a coded gain pixel (CGP) pixel architecture or an indirect time-of-flight (iToF) coded pixel architecture.

[0061] FIG. 12A shows a TILE x =2 and TILE y TILE of 2 tap pixels for =2 x ×TILE yFIG. 12B shows an example of pixel codes that result in spatial configurations. Each of the two taps in each of the four pixels has an arbitrary programmable exposure time that ranges from zero subframe to the full subframe period. For example, t1-t4 are the exposure time intervals of the first taps of the four pixels in a 2×2 pixel tile. FIG. 12B shows an example of eight such codes, where the exposure intervals for all pixels are consecutive and start from the beginning of the frame. FIG. 12C shows another example where the pixel exposure intervals for each of the eight taps of the four pixels in the tile can be controlled by arbitrary time codes. FIG. 12D shows that with an increased number of subranges (eight subranges versus four in the example of FIGS. 11A-11D), the SNR drop at the border between the light intensity subranges corresponding to different taps is less noticeable. Although FIGS. 12A-12D show a CEP architecture as an example, in additional examples, this can be applied to other CP pixel architecture types, such as, for example, a coded gain pixel (CGP) pixel architecture or an indirect time of flight (iToF) coded pixel architecture.

[0062] In some examples, other criteria may be employed to enable snapshot compressed acquisition of HDR video. In some low dynamic range snapshot compressed video acquisition approaches, individual 1-tap pixels are assigned different exposure intervals to sample the space-time cube shown in FIG. 8A. These intervals are selected in such a way that the reconstruction of the entire space-time cube, i.e., the entire low dynamic range video sequence, can be computed from just one captured image, utilizing an appropriate decoding algorithm. To enable acquisition of HDR video from a single snapshot, the codes assigned to the CEP pixels may be selected such that both the exposure interval and the total exposure time vary between pixels and taps within a pixel, as shown in FIG. 8B. These codes may be assigned individually to each pixel in the pixel array to maximize the quality of the reconstructed HDR video frame. Alternatively, the pixel array may be assigned a TILE x ×TILE y8B, with each neighborhood receiving the same code, for example the code of FIG. 8B. The HDR video is then obtained using an appropriate decoding algorithm. The same concept can be applied to other CP pixel architecture types, for example the coded gain pixel (CGP) pixel architecture or the indirect time-of-flight (iToF) coded pixel architecture.

[0063] The embodiments of the present disclosure may be applied to both open-loop and closed-loop HDR imaging, as shown in FIG. 4. In open-loop operation, pixel codes and decoding algorithms may be manually input or calculated from previous training data to maximize HDR performance in a given application. In closed-loop operation, pixel codes may be adaptively calculated to reduce saturation and / or increase SNR in the scene being imaged, or more generally to improve application-specific imaging performance. Application examples include high-speed imaging, object detection, tracking, recognition, robot navigation, autonomous driving, 3D reconstruction, and remote monitoring and analysis of electrical grids (e.g., by imaging AC light bulbs, including but not limited to measuring grid phase and / or frequency, recognizing bulb type, etc.). The adaptive calculation of the pixel codes may be performed in any suitable manner, such as, for example, (1) a manually input function that maps a previous video frame or sequence of video frames to a code for the entire pixel array, (2) a manually input control algorithm, such as, for example, a PID controller, and (3) a learning control algorithm based on previous training data. Similar considerations may be applied to the decoding algorithm. An example of a manual input code generation feature is described herein.

[0064] In some examples, when multiple CP image sensor pixels are used to compose a single HDR pixel value in a video frame or HDR image obtained by burst mode acquisition, additional criteria may be employed to reduce artifacts such as motion blur non-uniformity or frame-to-frame variability. For example, in a CEP image sensor, this may be done by periodically swapping the exposure code between multiple pixels in the same spatial neighborhood, such as a pixel tile of size 2×2 pixels or any other size. FIG. 13 shows an example of a 2×2 1-tap pixel array, where the exposure code for four subsequent frames is rotated 90 degrees four times. This allows each pixel to be exposed for the same average time over the period of four frames, and motion blur in the imaged scene occurring during these frames can be equalized. Although FIG. 13 shows a CEP architecture as an example, the same concept may be applied to other CP pixel architecture types, such as a coded gain pixel (CGP) pixel architecture. The exposure code is swapped between multiple pixels in the neighborhood of a 1-tap or multi-tap pixel tile. x ×TILE y It may be used to programmably set up spatial configurations that are repeated across the pixel array. Each pixel in the neighborhood has an optionally programmable exposure interval(s). The exposure code is TILE n A pixel neighborhood in each subsequent frame can be recognized such that each pixel is exposed for the same or approximately the same time interval(s) across the frame. This allows for a burst mode demosaicing scheme, where the TILE in the video samples the exposure across space and time. x ×TILE y ×TILE n It can be expressed as a small spatiotemporal volume of the size of a pixel. In the example of a CGP architecture, the average pixel gain is generally n After the frame it is the same for all pixels.

[0065] The embodiments of the present disclosure may be used to programmably adjust the dynamic range, operating range, and / or color gamut of a CP pixel sensor with RGB color filter arrays, multispectral filter arrays, polarizing filter arrays, neutral density filter arrays, and / or various other filter arrays. For example, the color gamut of a CEP sensor may be shifted and / or expanded by selecting pixel codes that individually adjust the total exposure time of the R channel, G channel, and / or B channel pixels, respectively. These codes may be specified a priori or may be scene- and tile-dependent, for open-loop or closed-loop operation, respectively, as shown in FIG. 4. The same concept may be applied to other CP pixel architecture types, such as, for example, a coded gain pixel (CGP) pixel architecture. The same concept may be applied to sensors with attenuating, diffusing, and / or diffractive sensor masks, and / or metal lenses.

[0066] FIG. 14 shows an example of a block diagram of a CP image sensor 1400. Any suitable combination of the functional blocks shown may be used. A "Pixel Array" 1402 in the image sensor 1400 includes an array of photodetectors. The image sensor 1400 also includes a "Code Loading Circuit" block 1404 and a "Row Logic and Drivers" block 1406 for generating and sending per-pixel exposure codes and per-pixel gain codes. The pixel outputs are read out using an "Analog Readout Circuit" 1408 and an "ADC Readout Circuit" 1410. A "CONFIG.REG." block 1412 is used to generate on-chip control signals. The “Signal Processor” 1414 performs various computational functions such as, for example, open-loop and closed-loop pixel code generation, receiving and reorganizing pixel codes, decompressing pixel codes, receiving an output image or video from an image sensor, for example, one or more frames of a sub-frame, or one or more regions of interest within one or more frames of a sub-frame, generating pixel codes based on this information and other information provided to the signal processor, HDR image reconstruction, and / or any other type of image or video processing performed on the output of the image sensor readout circuitry. The additional blocks may include devices, structures, and circuits related to front and back illumination such as microlenses, color filters, deep or shallow trenches for electrical and optical isolation, attenuating, diffusive, and / or diffractive sensor masks, metal lenses, pixel blooming prevention control circuits, pixel binning circuits, correlated double sampling circuits, double sampling circuits, averaging circuits, circuits for calibrating fixed pattern noise, color correction and enhancement circuits, programmable gain control circuits, active illumination control circuits, time-of-flight imaging control circuits, or any other pixel control circuitry or module implemented on the same integrated circuit (referred to herein as a chip) or on the same wafer, or on separate integrated circuits or wafers, or on stacked wafers or stacked dies or in a single package or single module or multi-package or multi-module configuration.

[0067] Other functional blocks may be included either within the same image sensor chip or outside the image sensor chip, such as other standalone chips, or other chips attached to the image sensor chip in a stacked die or stacked wafer configuration, or implemented in any other hardware or software. These additional functional blocks may include, for example, digital or analog signal processors (or any other computing device that may be used for output of video signal processing), pixel code generators, code compression, code sending, and / or code decompression, temperature sensors, temperature-based control circuits, temperature control circuits, ambient light sensors, artificial light sensors including sensors that measure the amplitude and / or phase of indoor and outdoor artificial lighting, active light sources such as light projectors, laser light sources, LED light sources, color light sources, multispectral light sources, hyperspectral light sources, or any other light sources used to illuminate a scene or surrounding environment, or any other sensor and control blocks, optical elements or modules such as lenses, light modulators, mirrors, digital micromirror devices (DMDs), or color filters.

[0068] A specific circuit level implementation of a CP image sensor pixel is described herein for an example of a coded exposure pixel (CEP) image sensor, which is a subclass of a CP image sensor. The pixel level coded exposure pixel (CEP) function can be implemented with any suitable approach. FIG. 15B shows a flow chart illustrating an example operation of a multi-tap pixel in a CEP image sensor, for a two-tap example. To perform sign-dependent sorting of photo-generated charges among multiple taps, the photo-generated charges are collected into tap 1 or 2 based on the sign of that individual pixel, which is code 1 and 2, respectively. FIG. 15A and FIG. 15C show two example operations of a CEP pixel architecture following the principle of operation shown in FIG. 15B. FIG. 15A shows an example code memory pixel (CMP) and FIG. 15C shows an example data memory pixel (DMP).

[0069] In the CMP pixel architecture of Figure 15A, code values ​​are preloaded into an in-pixel memory, called "code memory," during the previous subframe in a pipelined fashion and applied for photo-generated charge classification at the beginning of the current subframe exposure time. Thus, while the code for the next subframe is preloaded into the pixel code memory, the photo-generated charges collected in the current subframe are transferred to the respective tap(s) based on the current subframe code. For this purpose, an in-pixel pipeline code memory may be used.

[0070] In the DMP pixel architecture of FIG. 15C, an intermediate storage node, called “data memory”, is used. It can be implemented as a charge domain memory, for example, storage diodes, storage pinned diodes, or different types of storage elements. Charge “sorting” is then performed on the storage diodes based on the code applied to the pixels in a row-wise fashion for each subframe. Thus, the storage diodes are capable of sorting the charge of the current subframe during the collection of photocharge for the next subframe. In detail, after global exposure in a given subframe, the photo-generated charge is first transferred to the intermediate charge storage node. In a pipelined fashion, during the exposure of the next subframe, the pixel code of the current subframe is applied row-wise from outside the pixel memory to sort the accumulated photo-generated charge by shifting to tap 1 or 2 for codes 1 and 2, respectively. This is repeated N times per frame. As a result, the photo-generated charge over all subframes of a frame is selectively integrated over two taps according to the code sequence per pixel and read out at once as two images at the end of the frame.

[0071] An example of a two-tap CEP pixel is shown in Figures 16A-16D for an example of a CMP pixel architecture (Figures 16A and 16C) and a DMP pixel architecture (Figures 16B and 16D). Figures 16A and 16B correspond to an example where the codes for the two taps are complementary, and thus can be represented by a one-bit binary number per pixel.

[0072] FIG. 16A shows a circuit diagram of an example CMP pixel architecture that includes one or more memory elements, e.g., flip-flops, used to load a binary subframe, one bit per row. In this example, two flip-flops and ROW_LOAD latch the code. If the codes are complementary, only one memory element with a complementary output may be used. When GLOBAL_EN is asserted, the memory output is enabled and applies the code globally (to the entire pixel array). When GLOBAL_EN is deasserted, all flip-flop outputs are reset to zero. The one or more memories are referred to as the "code memory." Based on the code in each pixel, one or both of the two transfer gates connect a photodetector, which in this example is a pinned photodiode (PPD), to the corresponding floating diffusion node C. FD1 and / or C. FD2 When both transfer gates are on, this effectively provides programmable per-pixel gain control and is an example of a CGP pixel architecture.

[0073] 16B shows a circuit diagram of an example of a DMP pixel architecture that includes an additional intermediate storage node called "data memory." In this example, the data memory is represented as a node at the top terminal of the storage diode SD. Charges photo-generated in the photodetector PPD are first shifted into the data memory and then sorted between two taps based on pixel coefficients in a row-by-row fashion.

[0074] Two other examples of pixel circuit diagrams for two 2-tap CEP pixel architectures, CMP and DMP pixel architectures, are shown in Figures 16C and 16D, respectively. Figures 16C and 16D correspond to examples where the codes for the two taps are not necessarily complementary, and the photo-generated charge accumulated during any given subframe may be drained if it is not transferred to any tap. Compared to the examples of Figures 16A and 16B, an additional code, Code0, controls when the photo-generated charge is drained in any given subframe at a given pixel. This requires a total of three codes: CODE0, CODE1, and CODE2.

[0075] FIG. 16C shows an example circuit diagram of a 2-tap CMP pixel architecture with a PPD as a photodetector and three in-pixel 1-bit binary code memories (using flip-flops in this example). The three binary codes may be stored in two 1-bit binary memories, but an example with three 1-bit binary memories is shown for simplicity. The codes, labeled CODE[0:2], are made available at the input of the memory by a peripheral code loading circuit and latched by ROW_LOAD in a row-by-row fashion. At the beginning of every subframe, the codes are made available at the memory output node when the GLOBAL_EN signal is asserted. At the end of the subframe, the GLOBAL_EN signal is deasserted. Based on the value of the code, the photo-generated charge is either advanced to the intended tap or drained to VDD during the exposure time of the subframe.

[0076] FIG. 16D shows an example schematic of a DMP pixel architecture where the codes for the two pixels are not necessarily complementary (including an additional intermediate storage node), and the photo-generated charge accumulated during any given subframe can be drained if not transferred to any tap. The photo-generated charge in the PPD can be transferred to SD by asserting the GLOBAL_EN signal at the end of every subframe. The charge at node SD is then sorted into the intended tap(s) based on the code value (CODE0, CODE1, and CODE2 in this example) or drained to VDD during the exposure time interval of the subsequent subframe. This is done in a row-by-row fashion. The photo-generated charge collected in the photodetector PPD during a subframe can also be flushed all at once by asserting the GLOBAL DRAIN signal.

[0077] Additional implementation examples of a two-tap CEP pixel are shown in Figures 17A-17D for examples of CMP pixel architectures (Figures 17A and 17C) and DMP pixel architectures (Figures 17B and 17D). Figures 17A and 17B correspond to examples where the two taps are complementary, and thus can be represented by one binary bit per pixel. Figure 17A shows an example of a CMP pixel architecture that includes two latches, one used to preload the subframe code per row, and the other used to apply this code globally (to the entire pixel array). Based on the code in each pixel, one of two transfer gates connects a pinned photodiode (PPD) to the corresponding floating diffusion node. The code in each pixel can be gated by an exposure signal to stop integration during the readout phase. This signal can also be held low while the global code updates to ensure that supply disturbances caused by this signal and digital switching in the array do not affect the PPD charge transfer. The charge generated during this time may accumulate in the PPD until the exposure signal is asserted again.

[0078] FIG. 17B shows an example of a DMP pixel architecture, which includes a "data memory" in the data storage diode. Charges photo-generated in the photodetector PPD are first shifted into the data memory and then sorted between two taps based on their complementary signs in a row-by-row fashion. The photodetector charges can only be globally drained, for example at the end of a subframe. This pixel topology can be similar to a global shutter pixel, but instead of a simple readout, the data memory (floating diffusion accumulation) node is used for charge sorting.

[0079] Figures 17C and 17D differ from Figures 17A and 17B, respectively, in that the codes for the two taps are not necessarily complementary, so that photo-generated charge accumulated during any given subframe may be drained if not transferred to any tap. An additional code, code 0, may be used to allow photo-generated charge to be drained in any given subframe in a programmable manner on a pixel-by-pixel basis.

[0080] As shown in the timing diagram of FIG. 17E, during each subframe, charge is generated and stored in the PPD, and at the end of the subframe, the GLOB_LOAD signal transfers the generated charge for all pixels from the PPD to the storage diode (charge mode memory). The pattern for that subframe is then sent to the pixels row by row to sort the charge from the storage node through the transfer gate into taps 1 and 2 based on sign. At the same time, the photo-generated charge for the next subframe is collected in the PPD, where CODE-UPLOAD(i) occurs during SUBFRAME(i-1) for the CMP pixel architecture, versus SUBFRAME(i+1) for the DMP pixel architecture. For simplicity, a per-pixel gain control function is not shown in FIGS. 17A-17D, but may be used, for example, by shifting charge to multiple taps at once, shifting charge simultaneously to drain nodes and taps configured as floating diffusions, or by including additional gain control elements as described above.

[0081] The layout and electrostatic potentials for the CMP pixel architecture in Figure 17A and the DMP pixel architecture in Figure 17B are shown in Figures 18A and 18B, respectively. These correspond to examples where the two taps are complementary, such that a one-bit binary number can be represented per pixel. In the CMP pixel example of Figure 18A, photo-generated charge is transferred to one of the taps based on the code within the individual pixel.

[0082] In the example DMP pixel of FIG. 18B, the transfer gate TGG is controlled by a GLOB_LOAD signal (as illustrated in FIG. 17) that transfers charge from the PPD to the storage diode globally in all pixels. The transfer gates TG1 and TG2 are responsible for the storage diode charge classification based on the corresponding code. After the global exposure in a given subframe, the photo-generated charge is first transferred to an intermediate charge storage node, for example, a data memory. In a pipelined fashion, the pixel code of the current subframe is applied row-by-row from the outer memory to classify the accumulated photo-generated charge during the exposure of the next subframe by shifting it to tap 1 or 2 for code 1 and 2, respectively. This is repeated N times per frame. As a result, the photo-generated charge across all subframes of a frame is selectively integrated across two taps according to the pixel-by-pixel code sequence and read out once at the end of the frame as two images. In some examples, the P-type metal oxide semiconductor (PMOS) device in the CMP structure may require a large amount of overhead space, thus reducing the pixel fill factor. On the other hand, DMP pixels can only use N-type metal-oxide-semiconductor (NMOS) devices, which have a small total area.

[0083] The dynamic range of the CEP pixel architecture can be increased by increasing the overall well capacitance of the individual taps. Figures 19A and 19B show two examples of 2-tap CP pixel architectures, with per-pixel coded exposure function and globally programmable pixel gain. For both CMP and DMP, the overall well capacitance of the taps within the pixel can be adjusted by controlling the global signal GAIN in each subframe. Both the CMP and DMP pixel architectures can be used for per-pixel coded exposure, but with additional dynamic gain control function.

[0084] When the GAIN signal is low, the photo-generated charge is i Based on tap i A capacitor C representing the floating diffusion node associated with FDi C FDi Since the capacitance value of is low, the conversion gain is high. This operating state of CP is generally optimal when the incident optical power is low.

[0085] When the GAIN signal is high, the capacitor C FDi and C HDRi are connected in a parallel configuration to increase the overall well capacitance at each tap of the CP. The photogenerated charge from the PPD is i Based on the value of , it is either sorted into its intended tap or drained to VDD. The conversion gain is low because the coupled capacitance at the tap is high. This operating state of the CP is generally optimal when the incident light intensity is high.

[0086] In some cases, the floating diffusion capacitor C FDi and HDR capacitor C HDRi In order to sample a wide dynamic range scene multiple times in, the GAIN signal can be dynamically changed by subframe.

[0087] FIG. 20A and FIG. 20B show two examples of a two-tap CP pixel architecture with pixel-wise coded exposure and pixel-wise coded gain functions. FIG. 20A shows a CMP and FIG. 20B shows a DMP with pixel-wise coded exposure and pixel-wise coded gain functions. FIG. 20A shows an example implementation of a CP pixel architecture with a photodetector PPD and in-pixel code memory (D flip-flop in this example) for both pixel-wise exposure and gain codes. This is an example of a CMP pixel architecture with two taps. The code EXPOSURE CODE[0:2] is used to determine which tap, if any, is used to collect the photo-generated charge in a given subframe, and the code GAIN CODE[0:1] is used to determine the conversion gain of each individual tap. Based on the value of CODEi (CODE0, CODE1, and CODE2 in this example), the photo-generated charge is advanced to the intended tap or drained to VDD during the exposure time of the subframe. The conversion gain of the taps can be individually selected based on the control signals GAINi (GAIN1, GAIN2 in this example). FIG. 20B shows a photodetector PPD and a floating diffusion node C (also called a tap). FDi 1 shows an implementation of a 2-tap DMP CP pixel architecture with a storage diode (SD) that is an intermediate node between CODE and the control signal CODE during the exposure time of the subsequent subframe. i Based on the value of (CODE0, CODE1, and CODE2 in this example), it is sorted into the intended tap or drained to VDD. When sorting the charges into the taps, the conversion gain of the individual taps is controlled by the control signal GAIN i (GAIN1, GAIN2 in this example).

[0088] Figures 20C and 20D show two examples of 4-tap iToF CP pixel architectures for sensing the time of flight of photons coming from an illumination source to a photodetector, with pixel-wise coded exposure and gain functions. Figure 20C shows an example of an iToF CMP, and Figure 20D shows an example of an iToF DMP, with pixel-wise coded exposure and gain functions.

[0089] FIG. 20C shows an example implementation of an iToF CMP pixel architecture with a photodetector PPD and an in-pixel code memory (D flip-flop in this example). The code EXPOSURE CODE[0:2] indicates the tap pair (C FD1 and C FD3 TAP1, or C FD2 TAP2 and C FD4 The codes EXPOSURE CODE[0:2] and GAIN CODE[0:3] are loaded into an in-pixel memory, such as a D flip-flop or latch. The values ​​of CODEi (in this example CODE0, CODE1, and CODE2) and the signals MOD and

number

[0090] FIG. 20D shows an example implementation of an iToF DMP pixel architecture with two additional intermediate charge storage nodes in the storage diodes R-SD and L-SD. These are the photodetector PPD and the floating diffusion node C (also called tap) as shown. FDiIn this example, photo-generated charges are collected in the right storage node R-SD when the signal MOD is asserted, and a signal

number

[0091] Both on the same chip as the image sensor array and on a different chip, an analog-to-digital converter (ADC) may be used to digitize the pixel tap output. The parameters of such an ADC, such as the reference voltage, may be dynamically configured for one or more subframes or one or more frames. FIG. 21A shows an example of a readout architecture in an image sensor with an ADC with programmable resolution. The image sensor includes a CP array. The ADC resolution may be dynamically configured using on-chip configuration registers and / or external control signals. In FIG. 21B, the ADC is configured in a coarse resolution mode (1-bit resolution in this example, assuming a constant VREF). In this mode, the ADC may provide coarse intensity information about the pixel output at a very high speed as limited by the short subframe exposure period. A vertical scanner selects the rows of the pixel array one by one. Once a row is selected, data from the pixel taps is loaded onto one or more readout lines READOUT[N-1:0]. Data from the taps may be available for readout at the same time, or the readout line may be shared for multiple taps, and data may be time and / or spatial multiplexed on the readout line. The readout line is connected to an analog readout circuit. The output of the analog readout circuit is connected to a low-resolution ADC (in this example a comparator is used) input that quantizes the input analog signal to generate a digital output. In this example, one bit of output is generated for each tap based on the analog signal on the readout bus and a reference voltage VREF that may be optionally programmed. In this example, the digital output from the ADC is transferred off-chip using a high-speed digital serializer. Such an operation may be performed during an exposure while the taps are still collecting photo-generated charge, for example for either the previous or current subframe. Since the coarse ADC readout generates a smaller number of bits compared to the high-resolution ADC, this digital information may be transferred at a higher speed than for the high-resolution ADC frame readout. In FIG. 21C, the ADC is configured in a high-resolution mode. The high-resolution ADC may or may not reuse the low-resolution ADC circuitry, for example the same comparator.In this mode, the ADC can provide a more accurate digital representation of each pixel tap output. In this example, the ADC is often used in high resolution mode, using high power or high output data rates or long readout times to convert the analog data from the taps to the most accurate digital values.

[0092] For example, the subframe-by-subframe high-speed analog-to-digital conversion and readout capability of an image sensor with a coarse ADC resolution is used to increase the dynamic range of the CP. The following approach is an example of how such an image sensor can be used when the resolution is varied. 1. Reset all pixels. · Resetting all pixels sets all pixel outputs to zero (in practice this is typically VDD or a high voltage close to VDD). 2. Set the ADC to coarse high-speed mode. 3. Set VREF = VREF_PER_SUBFRAME[0]. · VREF_PER_SUBFRAME[k] is the reference voltage level for subframe k. 4. Begin exposure. As the pixels collect the photo-generated charge onto the taps, their output goes up (in practice, the effective tap voltage signal typically goes down). 5. Set the code for all pixels to collect photo-generated charge onto TAP1. ·TAP_CURRENT=TAP1 6. After the first subframe, compare the readout voltages of all pixels for each row with VREF. For every pixel ■ When READOUT[TAP_CURRENT]>VREF · Change the sign of the pixel to collect (or drain, if there are no more taps) the photo-generated charge of the next tap. ·TAP_CURRENT=TAP_CURRENT+1 ■Others · Keep the sign the same as the previous sign. ·TAP_CURRENT=TAP_CURRENT 7. Read After the exposure is complete, set the ADC to high resolution mode. Read out sensor output frames with high ADC resolution.

[0093] For example, the reference voltage VREF for a coarse resolution ADC may be dynamically adjusted every subframe, and the code for the next subframe may be selected based on the ADC output in the current subframe. Figure 22A shows possible pixel tap values ​​when the subframe code is selected to ensure that a minimum number of pixels are saturated, where the reference voltage VREF is kept constant at a predefined value. With the above example, charge accumulation at a given tap stops with a latency of one subframe after the tap value reaches VREF.

[0094] By adjusting VREF every subframe, the SNR per subframe can be further improved, and thus an improvement in the overall SNR can be realized. FIG. 22B shows an example plot of SNR per subframe versus incident light intensity showing the variation in signal quality with a change in code on a subframe-by-subframe basis, with respect to a constant reference voltage VREF. FIG. 22C shows an example of possible pixel tap values ​​when the subframe code is selected to ensure that a minimum number of pixels are saturated. Here, the reference voltage VREF is adjusted every subframe k to further increase the dynamic range. VREF for subframe k is:

number

[0095] FIG. 39 shows a flow chart of a method for extending an image sensor dynamic range using the encoded pixels 200, according to an embodiment. In block 202, the encoding module 112 provides pixel codes to the pixels in the photodetector array of the image sensor 110 for each subframe. The pixel code for each pixel may include a code for exposure time or signal gain, or both, or other programmable pixel parameters. In block 204, the image sensor 110 receives a sensor readout for each of the pixels in the photodetector array for each subframe. In block 206, the encoding module 112 routes the sensor readout for each of the pixels to be collected or drained to one of a number of taps based on the pixel code. In block 208, the HDR decoding module 114 combines the values ​​collected at each of the taps to determine a single pixel value for the frame. In block 210, the output module 116 outputs the single pixel value for each pixel for the frame.

[0096] An implementation example of the closed-loop HDR imaging method is provided below. In this example, a one-tap CEP image sensor is used with a temperature planning pixel-by-pixel exposure coding dynamically updated based on the previous frame. The method is verified by example experimental results obtained from a manufactured prototype. For illustration purposes in this example, no other past or current frames (other than the previous frame) or current or past sub-frames are used, and no pixel-by-pixel gain coding is used.

[0097] Advantageously, in this example, the technique for dynamic range extension at the native resolution of the image sensor uses a coded exposure pixel (CEP) pixel architecture. The CEP image sensor allows arbitrary programmable control over the exposure time of individual pixels. For example, an embodiment of the present disclosure analyzes the light flux at each pixel in a captured frame (frame [n]) and makes dynamic adjustments to the respective exposure time in the next frame (frame [n+1]) as shown. An example of such an approach is illustrated in FIG. 23. In the illustrated example, white fill represents the time intervals during which the pixel is on, and black fill represents the time intervals during which the pixel is off. For example, pixels that sense low brightness in the captured frame, such as the top left pixel of the array in FIG. 23, are dynamically programmed to have more exposure time in the next frame. After normalization of the different brightness in the captured image, the HDR image is optimally reconstructed and visualized.

[0098] In an embodiment, the CEP image sensor 110 performs imaging by dividing the exposure time of one frame into multiple subframes, an example of which is shown in FIG. 24A. In each subframe, each pixel receives a single bit of exposure code provided by the encoding module 112. For example, as shown in FIG. 24A, a code of 0 turns the pixel off for the duration of the respective subframe, and a code of 1 turns the pixel on for the duration of the respective subframe. As shown in FIG. 24B, the exposure code is loaded into the pixel's latch such that a code of 1 (latch output Q is asserted) causes a signal to be read from a photosite, e.g., a pinned photodiode (PPD), to collection node C. FD1 or a zero (latch output)

number

[0099] In an embodiment, the exposure code sent to a pixel in a current frame depends on the brightness of the scene at or near that pixel in the previous frame. The exposure code sequence is presented to the pixels in order as predetermined by the photo-generated charge collected by that pixel or neighboring pixels in the previous frame. The code sequence is a series of digital bits of 1s and 0s. During the exposure phase, each pixel receives a predetermined code sequence. The number of 1s in the code sequence determines the percentage of the exposure period during which the pixel is exposed, and the number of 0s determines the percentage of the exposure period during which the pixel is not exposed. More 1s in the code sequence for a given pixel means the pixel is "on" for a longer period, which translates to a longer exposure time, allowing more light to be captured by that pixel. More 0s in the code sequence for a given pixel means the pixel is "off" for a longer period, which translates to a shorter exposure time, allowing less light to be captured by that pixel, e.g., to avoid saturation by bright elements in the imaging scene. In an embodiment of the present disclosure, the selection of 1s vs. 0s in the code sequence for a given pixel is based on the value of the pixel or its neighbors from the previous frame. In this way, the number of 1's and 0's in the code sequence can ensure that the pixel is exposed as long as possible to capture as much light while ensuring that it does not saturate during that time. In some cases, in some or all subframes of an initial frame, all pixels receive the same initial code, e.g., code 1, to collect initial photo-generated charge. In the subframes of subsequent frames, pixels in dark areas of the image receive many 1's and integrate many of the incoming photo-generated charge, while pixels in light areas receive many 0's and drain many of the incoming photo-generated charge.

[0100] As described herein, the received sequence of 1's and 0's as defined by the encoding module 112 is an exposure code sequence. In an embodiment, the code sequence is loaded into each pixel before each subframe. If the code is 0, the pixel is masked and does not sense any light during that subframe. If the code is 1, the pixel senses light during that subframe. Figures 6A and 6B show an example of exposure time settings and code sequences for a 4 subframe setting (i.e., N=4) for a two subframe exposure encoding scheme. Each of the four subframes is active at times T1, T2, T3, and T4, respectively. In this example, pixel P is loaded with code sequence D P1 , D P2 , D P3 , and D. P4 Receive D Pi is a digital bit of 0 or 1 applied to pixel P during subframe i. For example, for code sequence C2, the bit sequence is 1100. In this example, the pixel receives code 1 during subframe 1 and subframe 2, and zero elsewhere. This means that the pixel accumulates photo-generated charge during subframe 1 and subframe 2, and is masked (does not collect photo-generated charge) elsewhere. Thus, the effective exposure time for a pixel is T1+T2. Thus, for example, code sequence C4 is most useful in low light intensity conditions, and code sequence C1 is most useful in bright light conditions. Table 1 shows effective exposure times for various examples of four subframe implementations. [Table 1]

[0101] As shown in FIG. 25A, for the unitary weighted subframe exposure times, EXP The exposure time is divided equally into four subframes, T1=T2=T3=T4=T EXPIn contrast, as shown in FIG. 25B, for binary weighted subframe exposure times, the exposure time of each subframe is twice as long as the previous subframe, so 8T1=4T2=2T3=T4=8T EXP The effective exposure time of a pixel is determined by the cumulative exposure time of all the subframes. For example, the effective exposure time using C3 is 3T for unity weighted subframe exposure. EXP / 4, and for binary weighted subframe exposure, 7T EXP / 15.

[0102] 7A and 7B show examples of raw pixel output over the full range of illumination for the unary weighting scheme of FIG. 25A and the binary weighting scheme of FIG. 25B, respectively. Both plots comprise four line segments. Each segment corresponds to a different code sequence, C4 through C1, as exposure increases. When the scene intensity at the pixel starts out low, code sequence C4 is used because a high effective exposure time allows capture of scene detail. As the intensity increases, the code sequence changes to code sequence C3, C2, or C1 to prevent saturation of the pixel.

[0103] 27A and 27B show examples of normalized versions of FIG. 26A and FIG. 7B, respectively. In an embodiment, normalization by HDR decoding module 114 converts the piecewise linear output to a full-range linear response using the corresponding effective exposure time of each piece. After normalization, the maximum effective dynamic range of a pixel is generally improved. In this example, using only four subframes, the dynamic range is four times larger for unary weighted subframe exposure times and 15 times larger for binary weighted subframe exposure times. Thus, the normalized output for unary weighted subframes is N·I / C, and the normalized output for binary weighted subframes is (2 N -1) × I / (2 C −1), I is the sensor output, N is the total number of subframes, and C is the number of 1s (e.g., in C1, C2, C3, or C4) in the code sequence used to capture that output.

[0104] A flow chart showing an example pipeline implementation of an embodiment of the present disclosure is shown in FIG. 28. In this method, when a current frame is read out for display, it is also used to generate an exposure code for the next frame. This technique can be used for both unary and binary weighted subframe exposure encoding. In frame n, one or more subframes accumulate light at a given pixel. Based on the output, a new exposure code is generated and uploaded for frame n+1. Now, an HDR image is obtained by the HDR decoding module 114 by normalizing the output of frame n. In this 4 subframe implementation example, the normalization uses 8-bit sensor output and 2-bit exposure code to generate a 10-bit HDR image. The exposure fusion technique enables the visualization of a 10-bit HDR image on an LDR display. Although the implementation of the embodiment described herein may show a 4 subframe implementation, it is understood that any suitable number of subframes may be used.

[0105] In an embodiment, the exposure code

number

number

number

number

number

number

[0106] Normalization Matrix

number

number

number

number

[0107] The system 100 calculates a new exposure code matrix for frame n+1.

number

number

number

number

[0108]

number

number

[0109] Pixel Light Flux

number

number

[0110] The new normalization matrix is ​​obtained by dividing the desired pixel output J of the sensor by the flux matrix,

number

[0111] In some cases, the desired output J may be set to about half the saturation level of the pixel to ensure a maximum margin for overexposure or underexposure in the next frame. For example, if the saturation level of the pixel is about 180DN, a J of 90DN may be selected for operation (DN stands for digital number).

[0112] New Exposure Code Matrix

number

[0113] In an embodiment, the approximation may involve starting with a non-approximate code for pixel (i,j) in frame n+1 of Cn+1(i,j). For unary weighting, From equation (4),

number

number

number

number

number

number

number

[0114] In the case of binary weighting, similar steps may be applied to the binary weighting, resulting in the following two equivalent equations:

number

number

number

number

[0115] In some cases, HDR images

number

number

number

number

[0116] In some cases, for frame n, the output module 116 may output an HDR image for visualization on a low dynamic range (LDR) monitor.

number

[0117] In some cases, the reconstructed HDR image may be split into several 8-bit images to cover the full bit depth. In some cases, exposure fusion may combine this set of images into a single 8-bit image. The advantage of compositing is that it selectively renders images with the desired brightness and texture from the set of images. In some cases, gamma correction may be used to fine-tune the composite image. Generally, conventional displays are 24-bit RGB color displays. These displays are 24-bit RGB color displays. 8The images may represent different grayscale levels of different colors. By using exposure fusion, the system 100 may selectively enhance different portions from different images that carry the maximum amount of information. Exposure fusion uses as input a set of similar images with different exposure settings. A final image is generated that is created by a pixel-by-pixel weighted combination of the input images. The weights for different portions of the input images are determined based on qualities such as contrast, saturation, and good exposure.

[0118] FIG. 29 illustrates another example using pixel-wise coded exposure, according to an embodiment of the present disclosure. In this example, a frame is sub-divided into four sub-frames. For this example, code sequences C1-C4 are used. The codes are updated at the start of each sub-frame. The space-time volume on the right shows an example of how different pixels can have different exposure times within any given frame based on the code sequence applied to that pixel. When code sequence C1 is applied, the pixel is exposed for a short time, and when code sequence C4 is applied, the pixel is exposed for the entire exposure period within that frame. The ability to capture sub-frame exposures is used to capture HDR scenes, as shown in the plots of FIG. 30. In these plots, the x-axis shows increasing light intensity, and the y-axis shows the raw sensor output. For example, if code sequence C4 is applied to a pixel capturing a dark area of ​​the scene, it ensures that the pixel is exposed for the entire frame period, but at the same time, if there is a bright area in the scene, code sequence C1 is applied to the pixels in that area. This coded exposure ensures that most of the light from the dark areas of the scene is captured while not saturating the pixels in the bright areas of the scene. As shown in the right plot of Figure 30, the sensor output can be normalized based on the exposure time of each individual pixel. After normalization, the effective dynamic range of the image sensor can be increased.

[0119] FIG. 37 shows a flow chart for a method of extending image dynamic range using pixel-by-pixel coded exposure 300, according to an embodiment. In block 302, an initial raw sensor readout of each of the pixels of the CEP image sensor is received. In this initial readout, the CEP image sensor is coded with a uniform code matrix. In block 304, a code matrix for a current coding exposure is derived from the initial readout. In some cases, the current coding matrix is ​​for a subsequent frame after the initial readout. In other examples, the initial readout may be obtained from any previously acquired pixel readout. In block 306, the current coding matrix is ​​sent to a latch associated with the respective pixel in the CEP image sensor. In block 308, a current raw sensor readout of each of the pixels of the CEP image sensor is received. In block 310, the current raw sensor readout is normalized based on the pixel-by-pixel exposure time to reconstruct an HDR image. In block 312, the HDR image and / or the normalized value is output. In some cases, blocks 304-312 may be performed repeatedly for each subsequent image frame, except that the code matrix is ​​derived from a previous frame rather than from an initial readout.

[0120] The inventors of the present disclosure conducted example experiments to illustrate the advantages of the embodiments of the present disclosure. The example experiments were conducted for both unary and binary weighted subframe exposure times. The example experiments used an image sensor fabricated in 0.11 μm CMOS image sensor technology with pinned photodiode (PPD) pixels and an in-pixel latch to store the exposure code. The image sensor was configured in 1-tap mode (one signal storage node) and the number of subframes N was set to 4.

[0121] The scene used in the experimental examples, captured with an RGB camera, is shown in Figure 31. Figures 32A and 32B show two images captured with the CEP image sensor 110 at low and high exposure settings, respectively. Figures 32A and 32B show the difficulty of capturing both bright and dark areas simultaneously.

[0122] FIG. 33 shows a comparison of example experimental results for three different encoding schemes. The exposure code matrix is ​​shown on top and the reconstructed HDR output of the sensor is shown on the bottom. For unary weighted subframe exposure, two different timing setups are demonstrated. The first column shows the unary weighted subframe exposure times T1 = T2 = T3 = T4 = T EXP The second column shows the results of scheme 1 where the unary weighted subframe exposure times are set as T1 = T2 = T3 = T4 = T EXP The results of Scheme 2, which is set as 4T / 15 per frame, are shown. EXP The third column shows the results of scheme 3 with binary weighted subframe exposure times. The exposure time of the first subframe is T1=T EXP / 15, with the same total exposure time T as in Scheme 1 EXP 33. A dark scene, highlighted with a landscape rectangle in FIG. 33, is well reconstructed using schemes 1 and 3. A bright scene, highlighted with a portrait rectangle in FIG. 33, is well reconstructed using schemes 2 and 3. In experimental examples, it was determined that using binary weighted subframe exposure times produced optimal results, reducing the number of subframes required and thereby reducing the data rate required to load the code.

[0123] As shown in the experimental examples, the embodiments of the present disclosure provide an advantageous approach to extend the image sensor dynamic range. In the embodiments described herein, an approach is provided to extend the dynamic range of an image sensor at the native sensor resolution using scene-aware pixel-wise coded exposure.

[0124] Advantageously, with unity-weighted subframe exposure times, the effective dynamic range of the sensor is 20log 10 NdB higher, 20log for binary weighted subframe exposure times. 10 (2 N−1) dB higher, where N is the number of subframes in one full-rate video frame.

[0125] Although two encoding schemes are described in the embodiments of the present disclosure, any suitable encoding scheme may be used.

[0126] The embodiments described herein take advantage of a class of optical image sensors (or imagers or cameras) that have a CEP pixel architecture and have programmable exposure for pixels during one frame time using an exposure code (referred to as masking) provided by the encoding module 112. Using the encoding module 112, the system 100 can individually select which pixels are masked. In an embodiment, if a pixel is "masked" (code 0), the signal collection node associated with that pixel, referred to herein as a "bucket", does not accumulate (collect) a signal for the respective pixel, and the signal is drained. Conversely, if the pixel is "exposed" and no mask is applied (code 1), the bucket collects a signal for the respective pixel. Figure 34 shows a typical example of this approach. If the masking bit associated with a particular pixel provided by the encoding module 112 is 0, photons received at that pixel are ignored. Conversely, if the masking bit is 1, photons received at that pixel are accumulated. The masking bit can be changed every subframe. In some cases, the memory for storing mask bits may be split into two separate blocks, a first memory block for preloading the masking bits of the next pattern and a second memory block for applying the masking bits of the current pattern. When implementing mask loading, storage, and use, it may be useful to keep the mask handling independent of pixel exposure so as not to limit the time available for signal acquisition. Thus, a pipeline operation of pre-storing masking bits may be used via the two memory blocks in the pixel or elsewhere.

[0127] Although this disclosure refers to image sensor 110 as a photodiode, any suitable photodetector may be used, such as, for example, a pinned photodiode, a photogate, a charge-coupled device, a charge-injection device, a single-photon avalanche diode, etc. Although embodiments of the present disclosure refer to light in the visible spectrum, it is understood that light referred to herein can include any portion of the electromagnetic spectrum, such as, for example, ultraviolet, near infrared, short-wave infrared, and / or long-wave infrared, along with or in place of light from the visible spectrum.

[0128] FIG. 35 is a typical Very Large Scale Integrated Circuit (VLSI) architecture of the image sensor 110 of the system 100. FIG. 36A is an example of a camera that may be used with the image sensor 110 of the system 100. FIG. 36B is a typical chip micrograph (4 mm×3 mm in 0.11 μm technology) of the pixel-wise coding image sensor 110 of the system 100 with a resolution of 244×162. FIG. 36C shows the specifications and characteristics of the example image sensor of FIG. 36B. FIG. 36C also shows the dynamic range improvement values ​​calculated for different coding strategies and different numbers of subframes, e.g., N=4 and N=10, for the image sensor of FIG. 36B.

[0129] FIG. 38A illustrates temporal pixel encoding where a code sequence is applied to the entire image sensor pixel array to collect light at various time intervals in a given exposure period. FIG. 38B illustrates partial space-time encoding that provides improved programmability in the spatial domain by allowing different pixel sub-arrays of the image sensor to receive different code sequences. FIG. 38C illustrates pixel-by-pixel encoding exposure encoding of an embodiment of the present disclosure. This encoding scheme allows for code sequences for each pixel independently. Multiple codes may be updated within a single frame, and photo-generated charge received within a frame may be accumulated based on the code sequence.

[0130] Advantageously, embodiments of the present disclosure allow for pixel codes having arbitrary values ​​for increased flexibility and applicability. Additionally, embodiments of the present disclosure allow for closed-loop or scene-dependent pixel code generation for increased adaptability. Additionally, embodiments of the present disclosure allow for pixel codes to be updated simultaneously or near simultaneously.

[0131] Although the present invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the appended claims. The entire disclosures of all references cited above are hereby incorporated by reference.

Claims

1. 1. A method for extending a dynamic range of an image sensor using coded pixels, the image sensor comprising a photodetector array for converting incoming optical signals into photo-generated electronic signals, the method comprising: providing pixel codes for pixels in the photodetector array for each subframe of a frame; receiving a sensor readout of each of the pixels in the photodetector array for each subframe; for each subframe, routing each sensor reading to one or more taps for collection or to a drain based on the pixel code, the pixel code being adaptively determined based on previously received pixel values ​​by sorting the collected charge at each tap based on the pixel code for the previous subframe as the sensor reading for the current subframe is collected; combining the collected sensor readings at each of the taps to determine a single pixel value for the frame; outputting the single pixel value for each pixel for the frame; A method comprising:

2. The method of claim 1 , wherein the pixel code comprises a code corresponding to an exposure time of the respective pixel, and the resulting photo-generated charge comprises the respective sensor readout.

3. The method of claim 1 , wherein the pixel code comprises a code corresponding to a signal gain of a signal received at the photodetector.

4. The method of claim 1 , wherein each pixel code is applied to a group of pixels in turn.

5. The method of claim 1 , wherein the pixel codes are preloaded into a code memory store based on the previously received pixel values.

6. The method of claim 1 , wherein each pixel comprises one tap and the pixel code for each pixel comprises a 1-bit binary number.

7. The method of claim 1 , wherein a pixel code for one or more of the pixels defines that each of the one or more taps has an individually programmable integration time interval.

8. The method of claim 7 , wherein the average of the integration time interval is equalized over a number of frames.

9. 1. A system for extending the dynamic range of an image sensor using coded pixels, the image sensor comprising a photodetector array for converting incoming optical signals into photo-generated electronic signals, the system comprising a logic circuit, the logic circuit comprising: an encoding module for providing pixel codes to pixels in the photodetector array for each subframe of a frame and for routing, for each subframe, a photo-generated electronic signal value of a respective one of the pixels to one or more taps for collection or to a drain based on the pixel codes, the pixel codes being adaptively determined based on previously received pixel values ​​by sorting the photo-generated electronic signal values ​​at each tap based on collected signal values ​​and pixel codes for one or more previous subframes; a high dynamic range (HDR) decoding module for combining collected signal values ​​available at each of the taps to determine a single pixel value for the frame; an output module for outputting the single pixel value for each pixel for the frame; A system configured to run

10. 10. The system of claim 9, wherein the pixel code comprises a code corresponding to an exposure time of the respective pixel, and the resulting photo-generated charge comprises the respective sensor readout.

11. The system of claim 9 , wherein the pixel code comprises a code corresponding to a signal gain of a signal received at the photodetector.

12. The system of claim 9 , wherein each pixel code is applied to a group of pixels in turn.

13. 10. The system of claim 9, wherein the encoding module uses pixel codes loaded into a code memory store based on the previously received pixel values.

14. 10. The system of claim 9, wherein each pixel comprises one tap and the pixel code for each pixel comprises a 1-bit binary number.

15. 12. The system of claim 11, wherein each pixel in the photodetector array comprises one or more charge collection nodes, and charge collection nodes from one or more pixels are combined to modify the signal gain based on the pixel code.

16. The system of claim 10 , wherein the pixel code defines a pixel exposure time for a pixel that is sensitive to a time-of-flight of the incoming light signal.

17. The system of claim 11 , wherein the pixel code defines a signal gain for a pixel that is sensitive to the time of flight of the incoming optical signal.

18. 10. The system of claim 9, wherein the HDR decoding module comprises an analog-to-digital converter (ADC), and wherein a resolution of the ADC is different for the digitization of the frame compared to the digitization of the sub-frame.

19. 10. The system of claim 9, wherein the HDR decoding module comprises an analog-to-digital converter (ADC), and one or more parameters of the ADC are dynamically adjusted for one or more subframes or one or more frames.

Images