Image sensor with high dynamic range lateral overflow integrated capacitors and split pixels with light emitting diode flicker mitigation

Abstract

Disclosed herein is an image sensor having a lateral overflow integrated capacitor (LOFIC) for high dynamic range (HDR) and split pixels for light emitting diode (LED) flicker mitigation. The pixel array is composed of a plurality of pixel cells, each pixel cell including a pixel photodiode region having at least a large light sensitive element (LPD) and a small light sensitive element (SPD), where the LPD includes a lateral overflow integrated capacitor (LOFIC). A method of generating an HDR image includes exposing a large light sensitive element (LPD) of a pixel cell for a first duration; exposing a small photosensitive element SPD of the pixel unit for a second duration; and combining the resulting readout into a combined pixel readout (CPR). This CPR is then used in combination with the generated LED flicker map (LFM) bits and the resulting readout from the SPD to produce a final corrected pixel readout that mitigates LED flicker in the resulting HDR image.

Description

Technical Field

[0001] This disclosure generally relates to image sensors, and more specifically, but not exclusively, to image sensors that mitigate the flickering effect of light-emitting diodes (LEDs) in images, such as high dynamic range (HDR) image sensors. Background Technology

[0002] CMOS image sensors (CIS) have become ubiquitous. They are widely used in digital still cameras, cellular phones, security cameras, and in medical, automotive, and other applications. A typical image sensor operates in response to image light reflected from an external scene incident on it. An image sensor comprises a pixel array having photosensitive elements (e.g., photodiodes) that absorb a portion of the incident image light and immediately generate an image charge after absorption. The image charge of each pixel can be measured as the output voltage of each photosensitive element, which varies depending on the incident image light. In other words, the amount of image charge generated is proportional to the intensity of the image light, and this is used to generate a digital image (i.e., image data) representing the external scene.

[0003] A typical image sensor operates as follows: Image light from an external scene is incident on the image sensor. The image sensor contains multiple photosensitive elements, each of which absorbs a portion of the incident image light. Each photosensitive element (e.g., a photodiode) within the image sensor generates an image charge immediately after absorbing the image light. The amount of image charge generated is proportional to the intensity of the image light. The generated image charge can be used to produce an image representing the external scene.

[0004] Integrated circuit (IC) technology for image sensors is constantly improving, especially in response to the growing demand for higher resolution and lower power consumption. Such improvements typically involve scaling down device geometry to achieve lower manufacturing costs, higher device integration density, higher speed, and better performance.

[0005] However, with the miniaturization of image sensors, defects within the image sensor architecture become more apparent and can degrade image quality. For example, excessive current leakage in specific areas of an image sensor can lead to high dark current, sensor noise, white pixel defects, and so on. These defects can significantly degrade the image quality from the image sensor, resulting in lower yield rates and higher production costs.

[0006] High dynamic range (HDR) image sensors may present other challenges. For example, some HDR image sensor layouts are not spatially efficient and are difficult to miniaturize to smaller pitches to achieve higher resolutions. Therefore, systems and methods for improving HDR are still needed. Summary of the Invention

[0007] In one aspect, this disclosure relates to a pixel array for a CMOS image sensor, comprising: a plurality of pixel units formed in a semiconductor substrate, each pixel unit including: a pixel photodiode region having at least a large photodiode (LPD) and a small photodiode (SPD), wherein the LPD is larger than the SPD and wherein the LPD includes a lateral overflow integrated capacitor (LOFIC); and a pixel transistor region disposed adjacent to the pixel photodiode region; wherein the LPD is exposed for a first duration during each exposure period, and the SPD is exposed for a second duration during each exposure period; wherein the first duration and the second duration are at least partially simultaneous; wherein for each exposure: the LPD generates a high conversion gain (HCG) readout, a low conversion gain (LCG) readout, and a LOFIC readout; and the SPD generates a short exposure (S) readout; and wherein the HCG readout, the LCG readout, the LOFIC readout, and the S readout are combined into a combined pixel readout (CPR).

[0008] In another aspect, this disclosure relates to a method for generating a high dynamic range (HDR) image, comprising: exposing a large photosensitive element (LPD) of a pixel unit for a first duration, wherein the LPD includes a lateral overflow integrated capacitor (LOFIC), and wherein the LPD generates a high conversion gain (HCG) readout, a low conversion gain (LCG) readout, and a LOFIC readout; exposing a small photosensitive element (SPD) of the pixel unit for a second duration, wherein the LPD is larger than the SPD, and wherein the SPD generates a short exposure (S) readout, and wherein the first duration and the second duration are at least partially simultaneous; and combining the HCG readout, the LCG readout, the LOFIC readout, and the S readout into a combined pixel readout (CPR). Attached Figure Description

[0009] Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein, unless otherwise specified, reference numerals refer to similar parts.

[0010] Figure 1 This is a block diagram illustrating an exemplary image sensor according to an embodiment of the present technology.

[0011] Figure 2 This is an illustrative schematic diagram of an example of a pixel unit according to an embodiment of the present disclosure.

[0012] Figures 3A to 3B These are examples of pixel arrays and their associated timing diagrams according to embodiments of the present disclosure.

[0013] Figures 4A to 4C This is an embodiment of the pixel array according to the present disclosure.

[0014] Figure 5A This is a timing diagram based on an embodiment of the present disclosure.

[0015] Figure 5B Is with Figure 5A The time-series graph shows the signal-to-noise ratio (SNR) curve.

[0016] Figure 5C This is a timing diagram based on an embodiment of the present disclosure.

[0017] Figure 5D Is with Figure 5C The time-series graph shows the signal-to-noise ratio (SNR) curve.

[0018] Figure 6 This is a schematic diagram of signal processing according to an embodiment of the present disclosure.

[0019] Several views throughout the figures are accompanied by reference characters indicating corresponding components. Those skilled in the art will understand that the elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the figures may be exaggerated relative to other elements to aid in understanding the various embodiments of this disclosure. Furthermore, common and well-understood elements that are typically useful or necessary in commercially viable embodiments are often omitted to facilitate unobstructed viewing of these various embodiments of this disclosure. Detailed Implementation

[0020] This document describes examples of apparatuses and methods for generating HDR images using CMOS image sensors, employing lateral overflow integrated capacitors (LOFICs) and LED flicker mitigation. Therefore, numerous specific details are set forth in the following description to provide a thorough understanding of the examples. However, those skilled in the art will recognize that the techniques described herein can be practiced without one or more of the specific details stated herein, or can be practiced using other methods, components, materials, etc. In other instances, well-known structures, materials, or operations have not been shown or described in detail to avoid obscuring certain aspects.

[0021] Throughout this specification, references to "an example" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the example is included in at least one example of the invention. Therefore, the appearance of the phrase "in an example" or "in an embodiment" in various places throughout this specification does not necessarily refer to the same example. Furthermore, the particular features, structures, or characteristics described in one or more examples can be combined in any suitable manner.

[0022] For ease of explanation, spatial relative terms such as “below,” “under,” “lower,” “below,” “above,” and “upper” may be used herein to describe the relationship of one element or feature relative to another element(s) illustrated in the figures. It should be understood that, in addition to the orientation depicted in the figures, these spatial relative terms are also intended to cover different orientations of the device during use or operation. For example, if the device in the figures is rotated, an element described as “below,” “under,” or “below” other elements or features will be oriented “above” other elements or features. Therefore, the exemplary terms “below” and “below” can encompass both the orientations above and below. The device may be oriented in other ways (rotated 90 degrees or in other orientations) and the spatial relative descriptions used herein shall be interpreted accordingly.

[0023] Furthermore, it should be understood that when a layer is described as being “between” two layers, it can be the only layer between the two layers, or there may be one or more intermediate layers. Similarly, it should be understood that when an element is described as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. In contrast, when an element is described as being “directly connected” or “directly coupled” to another element, there are no intermediate elements. Other terms used to describe relationships between elements or layers should be interpreted in a similar manner (e.g., “between” vs. “directly between”, “adjacent” vs. “directly adjacent”, “on” vs. “directly on”).

[0024] Based on the foregoing, it should be understood that although specific embodiments of the present invention have been described herein for illustrative purposes, various modifications can be made without departing from this disclosure. Furthermore, while various advantages and features associated with specific embodiments have been described above in the context of those embodiments, other embodiments may also exhibit such advantages and / or features, and not all embodiments must exhibit such advantages and / or features to fall within the scope of the present invention. In the case of described methods, the methods may include more, fewer, or other steps. Additionally, the steps may be performed in any suitable order. Therefore, this disclosure may cover other embodiments not explicitly shown or described herein. In the context of this disclosure, the terms "about," "approximately," etc., mean + / - 5% of the stated value.

[0025] Several technical terms are used throughout this specification. These terms will be given their common meaning in the field of their respective domains, unless otherwise specifically defined herein or the context in which they are used will clearly imply otherwise. It should be noted that in this document, component names and symbols are used interchangeably (e.g., Si and silicon); however, they have the same meaning.

[0026] Examples of pixel units for an image sensor are disclosed as shown. One or more of these examples may be arranged in a pixel array and used for, for example, high dynamic range imaging. In some examples, the pixel units of the pixel array may each employ a 4T or 5T pixel architecture. In some examples, a shared pixel unit architecture is employed, in which two or more photoelectric conversion regions (e.g., photodiode regions) are coupled to a common floating diffuser via a first transfer gate and a second transfer gate. This shared pixel unit architecture may include pixel transistors, such as reset transistors, source followers, row select transistors, dual floating diffuser transistors, etc.

[0027] For example, these CMOS image sensors are frequently used in the automotive industry, requiring not only sharp images of vehicles and other obstacles, but also sharp images of LED headlights, taillights, and road signs. While these LED indicators may appear to be constantly lit to the naked eye, many are designed to blink at a high frequency to extend the lifespan of these diodes and conserve power. In some examples, the LEDs blink at a rate of 90 Hz, meaning the emitted light is visible for 11.1 ms intervals. Using conventional techniques, CMOS image sensors typically expose for only a short time because prolonged exposure generally results in image blurring when the image sensor or the object in the photograph moves. Therefore, if these image sensors are exposed for the correct duration at the correct time, they can only capture LED indicators. However, extended exposure times also result in better spatial alignment across multiple readouts because it allows the photodiodes, including those of the image sensor, to capture a larger amount of image data, which can then be processed into a single, calibrated image. Thus, the exposure times of the various photodiodes must be manipulated and balanced to ensure the final image is sufficiently sharp.

[0028] To mitigate LED flicker while still capturing high-quality images, existing image sensors utilize both small photodiodes (SPDs) and large photodiodes (LPDs), the results of which are combined into a single HDR image. In some examples, the SPD exposure is approximately 11.1 ms, capturing a single “short” readout (S). The longer exposure time of the S readout allows the image sensor to mitigate LED flicker. In some embodiments, the SPD is exposed for up to 12 ms. In contrast, the LPD can be exposed for 1 to 3 ms to capture three distinct readouts: a “high conversion gain” readout (HCG), a “low conversion gain” readout (LCG), and a “very short” readout (VS). In some instances, the HCG and LCG are captured during the first exposure of the LPD, while the VS readout is captured during a subsequent exposure. Because the LPD requires two exposures to obtain all three readouts, there is still a risk of spatial misalignment in the final image, caused by the time interval between the HCG / LCG signal acquired during the first exposure to the LED and the VS signal acquired during the second exposure to the LED.

[0029] In some examples, each pixel cell is configured according to a LOFIC architecture. In a pixel cell with a LOFIC architecture, or a LOFIC pixel cell, a lateral overflow integrated capacitor (LOFIC) and an associated selection transistor, sometimes referred to as a dual floating diffuser (DFD) transistor, are provided. For example, when the first floating diffuser (FD1) reaches saturation, excess charge is routed to the second floating diffuser (FD2) and can be stored in the LOFIC. As a result, the photodiode has an increased full-well capacitance (FWC). Furthermore, the capacitance of the floating diffuser (FD) of the pixel cell can be selectively increased / decreased by modulating its capacitance, for example, by changing the voltage on one plate of the LOFIC capacitor. As a result, the dynamic range (e.g., HDR) of the pixel cell is increased.

[0030] In some examples, each LPD for each pixel unit is configured according to the LOFIC architecture. In this configuration, each LPD only needs to be exposed once to capture HCG readout, LCG readout, and LOFIC readout. Thus, a second LPD exposure is unnecessary because LOFIC captures the light intensity range previously captured in the VS readout. In some embodiments, the LPD and SPD are exposed simultaneously. In some other embodiments, the LPD is exposed before the SPD.

[0031] The technology of this invention seeks to solve the problems associated with rapidly acquiring clear and complete HDR images. For example, examples of the disclosed subject matter aim to reduce blur and mitigate LED flicker. Specifically, examples of the disclosed subject matter reduce the number of photodiode exposures required for HDR imaging. While illustrative embodiments have been illustrated and described, it should be understood that various changes can be made thereto without departing from the spirit and scope of the invention.

[0032] Figure 1 This is a block diagram illustrating an exemplary image sensor according to an embodiment of the present technology. Image sensor 100 may be implemented as a complementary metal-oxide-semiconductor (“CMOS”) image sensor. Figure 1 As illustrated in the example, the image sensor 100 includes a pixel array 102 coupled to a control circuit system 108 and a readout circuit system 104 coupled to functional logic 106.

[0033] The illustrated embodiment of pixel array 102 is an imaging sensor or pixel unit 110 (e.g., pixel units P1, P2, ..., Pn) of a two-dimensional (“2D”) array. In one example, each pixel unit includes one or more subpixels or pixel regions that can be used for HDR imaging according to the techniques and methods of this disclosure. As illustrated, each pixel unit 110 is arranged in rows (e.g., rows R1 to Ry) and columns (e.g., columns C1 to Cx) to acquire image data of people, places, or objects, which can then be used to render images of people, places, or objects. As will be described in more detail below, each pixel unit 110 (e.g., pixel units P1, P2, ..., Pn) may include, for example, LOFIC and associated structures to provide HDR imaging, for example, according to the techniques and methods of this disclosure.

[0034] In one example, after each pixel unit 110 has acquired its image data or image charge, the image data is read out by the readout circuitry system 104 via the readout column line 112 and then transferred to the functional logic 106. In various embodiments, the readout circuitry system 104 may include an amplification circuitry system (not illustrated), a column readout circuitry including analog-to-digital converter (ADC) circuitry, or other circuitry. The functional logic 106 may simply store the image data or even manipulate the image data by applying image post-processing effects (e.g., cropping, rotation, red-eye removal, brightness adjustment, contrast adjustment, or others). In one example, the readout circuitry system 104 may read out one row of image data at a time along the readout column line (illustrated), or a variety of other techniques may be used to read out the image data (not illustrated), such as serial readout or simultaneous full parallel readout of all pixels.

[0035] In one example, control circuitry 108 is coupled to pixel array 102 to control the operational characteristics of pixel array 102. For example, in one example, control circuitry 108 generates a transfer gate signal and other control signals to control the transfer and readout of image data from subpixels or pixel regions of shared pixel units 110 of pixel array 102. Additionally, control circuitry 108 may generate a shutter signal for controlling image acquisition. In one example, the shutter signal is a global shutter signal used to simultaneously enable all pixels within pixel array 102 to simultaneously capture their respective image data during a single acquisition window. In another example, the shutter signal is a rolling shutter signal, such that each row, each column, or each group of pixels is sequentially enabled during successive acquisition windows. The shutter signal may also establish an exposure time, which is the length of time the shutter remains open. In one embodiment, the exposure time is set to be the same for each frame.

[0036] In one example, control circuitry 108 can control the timing of various control signals provided to pixel unit 110 to reduce dark current associated with the floating diffusion portion of each of pixel units 110. In some non-limiting embodiments, pixel unit 110 may be a so-called 4T pixel unit, such as a four-transistor pixel unit. In other non-limiting embodiments, pixel unit 110 may be a so-called 5T pixel unit, such as a five-transistor pixel unit, which includes a 5T pixel unit with a LOFIC architecture. For example, in some non-limiting embodiments, pixel unit 110 may further include dual floating diffusion (DFD) transistors and associated capacitors (e.g., LOFIC). The associated capacitors may be selectively coupled via the dual floating diffusion transistors to increase / decrease the capacitance of the floating diffusion portion, which can modulate the conversion gain.

[0037] In one example, image sensor 100 may be included in a digital camera, mobile phone, laptop computer, etc. Additionally, image sensor 100 may be coupled to other hardware components, such as a processor (general purpose or other), memory elements, outputs (USB port, wireless transmitter, HDMI port, etc.), lighting devices / flash, electrical inputs (keyboard, touch display, tracking pad, mouse, microphone, etc.), and / or a display. Other hardware components may send instructions to image sensor 100, retrieve image data from image sensor 100, or manipulate image data supplied by image sensor 100.

[0038] Figure 2 This is an illustrative schematic diagram of an example of a pixel unit 210 according to the teachings of this disclosure. It should be understood that... Figure 2 The pixel unit 210 can be Figure 1The example of pixel unit 110, and similarly named and numbered elements referenced below, can be coupled and function similarly to those described above. For instance, pixel unit 210 may be coupled to bit lines (e.g., readout columns) that provide image data to a readout circuitry system (e.g., readout circuitry system 106), and pixel unit 210 may receive control signals from a control circuitry system (e.g., control circuitry system 108) to control the operation of various transistors of pixel unit 210. The control circuitry system can use relative timing to control the operation of the transistors in a desired order to, for example, reset the pixel to a dark state and read out image data, for example, after integration.

[0039] The illustrated example of pixel unit 210 includes a first photosensitive or photoelectric conversion element, such as a first photodiode 214, and a second photosensitive or photoelectric conversion element, such as a second photodiode 216. In operation, the first photodiode 214 and the second photodiode 216 are coupled to generate image charge in response to incident light. In an embodiment, for example, the first photodiode 214 and the second photodiode 216 can be used to provide image data for high dynamic range (HDR) images. In an embodiment, the first photodiode 214 is a large photodiode (LPD), and the second photodiode is a small photodiode (SPD).

[0040] Pixel unit 210 further includes a first transfer gate 218, a second transfer gate 220, and a first floating diffusion section (FD1) 222 disposed between the first transfer gate 218 and the second transfer gate 220. The first transfer gate 218 is coupled to transfer image charge from the first photodiode 214 to the first floating diffusion section 222 in response to a first transfer gate signal TX1. The second transfer gate 220 is coupled to transfer image charge from the second photodiode 216 to the first floating diffusion section 222 in response to a second transfer gate signal TX2. In the depicted arrangement, the first floating diffusion section 222 is common to both the first photodiode 214 and the second photodiode 216, and may be referred to as the common floating diffusion section 222.

[0041] A reset transistor 228 is coupled to a common floating diffuser 222 to reset pixel unit 210 in response to a reset signal RST (e.g., discharging or charging first photodiode 214, second photodiode 216, and floating diffuser 222 to a preset voltage). The gate terminal of amplifier transistor 224 is also coupled to the first floating diffuser 222 to generate an image data signal in response to image charge in the first floating diffuser 222. In the illustrated example, amplifier transistor 224 is a source follower (SF) transistor. A row select transistor 226 is coupled to the source follower (SF) 224 to output an image data signal to output bit line 212 in response to a row select signal RS. The output bit line 212 is coupled to a readout circuit system, for example... Figure 1 The readout circuit system 104.

[0042] In another exemplary embodiment, a dual floating diffusion transistor (DFD) 230 may optionally be coupled between the floating diffusion portion 222 and the reset transistor 228. A capacitor (CAP) 232 (e.g., a LOFIC) may also optionally be included and coupled to the dual floating diffusion transistor 230 to form a LOFIC pixel cell. When included, a second floating diffusion portion (FD2) 242 is formed between the reset transistor 228 and the dual floating diffusion transistor 230. In operation, the dual floating diffusion transistor 230 is adapted to couple the capacitor 232 to the floating diffusion portion 222 in response to the dual floating diffusion signal DFD when needed to provide additional dynamic range capability to the pixel cell 210. In the depicted arrangement, the capacitor 232 is also coupled to a voltage, such as the voltage VDD of adjusting the capacitance of the capacitor 232, to store charge overflowing from the pixel cell 210.

[0043] Control signals TX1 and TX2 enable transfer gates 216 and 218 to transfer charge from photodiodes 214 and 216 to the first floating diffuser 222. The amount of charge transferred from the photodiodes to the floating diffuser 222 may depend on the current operation of the pixel unit 210. For example, during a reset operation, the charge may be the charge generated during the dark state of the photodiodes, but during integration, the charge may be photogenerated image charge. At the end of integration, the image charge can be read twice, with one or more dark readings occurring, to perform correlated double sampling (CDS).

[0044] Figures 3A to 3B This is an example of a pixel array 302 and its associated timing diagram according to an embodiment of the present disclosure. Figure 3AAn embodiment comprising a plurality of first photodiodes 314 and a plurality of second photodiodes 316 configured in a split pixel array is depicted. In the illustrated embodiment, the first photodiodes 314 are large photodiodes (LPDs), and the second photodiodes 316 are small photodiodes (SPDs). In the split pixel array, the plurality of LPDs and the plurality of SPDs are each arranged in rows and columns such that adjacent LPDs share a common boundary, and the SPDs are embedded between the LPDs to create a tessellation pattern. However, other spatial arrangements of the SPDs and LPDs are also available in different embodiments. According to this embodiment of the prior art, no LOFIC is connected to any photodiode.

[0045] Figure 3B The timing and duration of exposures of the first photodiode 314 and the second photodiode 316 according to this embodiment are depicted to generate a timing diagram of the corresponding readouts. The timing diagram depicts a first duration T1. In some embodiments, T1 is 16.6 ms, depicting a frame rate of 60 frames per second (FPS). In this embodiment, because there is no LOFIC, LPD 314 is exposed twice. The first exposure has a duration T2. ​​In some embodiments, T2 is approximately 1 to 3 ms. In this embodiment, during the first exposure, LPD 314 captures HCG and LCG readouts. In this embodiment, the second exposure immediately follows the first exposure and has a duration T4. In some embodiments, the duration of T4 is 1 ms. During the second exposure, LPD 314 captures a very short (VS) readout. Furthermore, in this embodiment, SPD 316 is exposed for the same duration T3, thereby capturing a short (S) readout. In some embodiments, the duration of T3 is approximately 11.1 ms. Because the second exposure occurs after the first exposure, it will inevitably capture a slightly different scene than the first exposure. This difference could be due to changes in the scene surrounding the LPD or movement of the LPD itself (e.g., when the pixel array described in this embodiment is applied to a car). Thus, the resulting image may be blurry, which could negatively impact automated processing based on the resulting image (e.g., automatic emergency braking) or the broader user experience.

[0046] Figures 4A to 4C This is an embodiment of pixel array 402 according to the present disclosure. Figure 4AThis embodiment depicts a plurality of first photodiodes 414 and a plurality of second photodiodes 416 disposed in a split pixel array. In the illustrated embodiment, the first photodiodes 414 are large photodiodes (LPDs), and the second photodiodes 416 are small photodiodes (SPDs). In the illustrated pixel array 402, the plurality of LPDs and the plurality of SPDs are each arranged in rows and columns such that adjacent LPDs share a common boundary, and the SPDs are embedded between the LPDs to create a tessellation pattern. According to this embodiment of the present disclosure, each LPD 414 is electrically coupled to a LOFIC 432, which achieves a higher dynamic range for the LPDs.

[0047] Figure 4B The diagram depicts a split diode tile arrangement according to an embodiment of this disclosure, illustrating a plurality of LPDs 414 (e.g., LPD1, LPD2, ..., LPD) forming a pixel array 402. n ) and SPD 416 (e.g., SPD1, SPD2, ..., SPD) n As illustrated in the diagram, each LPD 414 is arranged in rows (e.g., row R). L1 To R Ly ) and columns (e.g., column C) L1 To C Lx Similarly, each SPD 416 is arranged in rows (e.g., row R). S1 To R Sy ) and columns (e.g., column C) S1 To C Sx In this configuration, each LPD 414 is arranged adjacent to each other, such that it shares a common boundary with neighboring LPD 414. Then, each SPD 416 is embedded between neighboring LPDs 414, such that the two photodiodes (LPD and SPD) of the same pixel share a common boundary. This arrangement has the visual appearance of a tessellated tile pattern of photodiodes 414, 416, without gaps or overlaps between neighboring photodiodes 414, 416. This arrangement allows for simultaneous and spatially close exposure of LPDs 414 and SPDs 416, resulting in simultaneous capture of HCG readout, LCG readout, LOFIC readout, and S readout according to this embodiment. However, other spatial arrangements of LPDs and SPDs of the same pixel are also possible in other embodiments.

[0048] Figure 4CThe illustration shows an exemplary arrangement of photodiodes 414 and 416 according to their color sensitivity. In the illustrated embodiment, the pixel array 402 consists of a plurality of Bayer units 405, each of which contains a plurality of pixels 402. In the illustrated embodiment, each of the plurality of LPDs 414 and SPDs 416 has an arrangement of two green (G) photodiodes, one blue (B) photodiode, and one red (R) photodiode of a corresponding size in each Bayer unit 405. These color sensitivities of the individual photodiodes alternate according to their respective rows and columns. For example, row R L1 LPD 414 in the diagram will be arranged by alternating colors (R and G in the illustrated example), while column C... S1 SPD 416 in the diagram will also alternate colors (B and G in the illustrated example). Subsequent rows and columns can alternate according to different color patterns. For example, row R... L2 LPD 414 in the column can alternate between B and G, while column C S2 The SPD 416 in the image can alternate between R and G, as illustrated in the diagram. This allows for the capture of a wide range of colored light, which can then be processed to provide an image.

[0049] Figures 5A to 5D This is a graphical illustration of timing diagrams and resulting signal-to-noise ratios (SNR) according to different embodiments of the present disclosure. Figure 5A This is a timing diagram according to an embodiment of the present disclosure. In this embodiment, the timing diagram depicts a first duration T1 of exposure of photodiodes 414, 416 to light. In some embodiments, T1 is 16.6 ms, depicting a frame rate of 60 frames per second (FPS), thus ensuring capture of at least one active cycle of an LED flickering at 90 Hz. Because the illustrated LPD 414 includes a LOFIC 432, the LPD only needs to be exposed once to capture the entire HDR of the scene. This exposure of LPD 414 has a duration T2. ​​In some embodiments, T2 is less than 5 ms. In the illustrated example, T2 is approximately 1 to 3 ms. In this embodiment, during its exposure, LPD 414 captures HCG readout, LCG readout, and LOFIC readout. In this embodiment, SPD 416 and LPD 414 are simultaneously exposed for a duration T3. In some embodiments, T3 is greater than or equal to 11 ms. In the illustrated example, the duration of T3 is approximately 11.1 ms. The readout obtained from the SPD 416 is called a short (S) readout.

[0050] Figure 5B Is for Figure 5AThe graph shows the SNR (vertical axis, measured in decibels) as a function of illumination (horizontal axis, measured in lux) for each of the four collected readouts. The end of the monotonically increasing cycle for each readout represents its saturation point, beyond which the SNR drops rapidly. The distribution of these monotonically increasing cycles for different readouts ensures image capture under varying illumination conditions. For example, the LOFIC 432, electrically coupled to the LPD 414, will be best able to capture high illumination, while responding least to low illumination. Conversely, the high conversion gain (HCG) and low conversion gain (LCG) readouts of the LPD 414 will have a relatively better response to low illumination, but will also saturate faster than the LOFIC (at approximately 80 and 300 lux, respectively). In the illustrated embodiment, it can be said that the SPD exposure of the LCG (indicated by the dashed line) covers the “middle range” of illumination. Thus, the illustrated embodiment relies on the extended exposure time of the SPD 416 to ensure capture of LED flicker; however, doing so limits the range of illuminance that can capture LED flicker to those captured by the SPD 416 (in the illustrated example, a range of approximately 0.316 lux to 1000 lux).

[0051] Figure 5C This is a timing diagram according to another embodiment of the present disclosure. In this embodiment, the timing diagram depicts a first duration T1. In some embodiments, T1 is 16.6 ms, which is related to a frame rate of 90 FPS. Because the LPD 414 contains the LOFIC 432, the LPD only needs to be exposed once. The LPD 414 exposure has a duration T2. ​​In some embodiments, T2 is approximately or equal to 11 ms. In the illustrated example, T2 is approximately 11.1 ms. In this embodiment, during its exposure, the LPD 414 captures the HCG readout, LCG readout, and LOFIC readout over a longer period of time, thereby allowing the LPD 414 to capture any LED flicker that may occur across these three readouts, as referenced below. Figure 5D As further explained. Therefore, the illustrated embodiment has the benefit of increasing the illumination range within which the pixel array 402 can capture the LED image despite LED flickering. In this embodiment, the exposure time T3 of the SPD 416 is delayed, and the exposure of the SPD begins after the LPD 414 has started. In some embodiments, the duration of T3 is less than 5 ms. In the illustrated example, the duration of T3 is 3 ms.

[0052] Figure 5D Is for Figure 5CThe graph shows the SNR (vertical axis, measured in decibels) as a function of illuminance (horizontal axis, measured in lux) for each of the four collected reads. Figure 5D Somewhat similar Figure 5B For example, regarding Figure 5B As described, the end of each monotonically increasing cycle of a readout indicates its saturation point, beyond which the SNR drops rapidly. The distribution of these monotonically increasing cycles for different readouts ensures image capture under varying illumination conditions. However, in the illustrated embodiment, the LPD 414 is exposed for a longer duration than the SPD 416. Given the extended exposure time of the LPD 414, a higher SNR is achieved under low-light conditions compared to previous embodiments. Additionally, the illumination range for detecting LED flicker is increased compared to other embodiments. In the illustrated embodiment, the range spans from approximately 0.007 lux to 31,622 lux. This captures a higher SNR than... Figure 5B The embodiments described herein offer a significantly wider illumination range, with the lower end of the captured range decreasing from 0.316 lux to 0.007 lux, and the upper end increasing from 1,000 lux to 31,622 lux. The illustrated embodiments are advantageous in instances where the task of the pixel array is to capture LED flicker while exposed to a wider range of illumination—for example, when a car emerges from a dark tunnel into daylight. However, the illustrated embodiments are not without drawbacks. As described above, the likelihood of a blurred image increases with the duration of photodiode exposure. Therefore, while the longer exposure of the LPD 414 across three of the four collected readouts (HCG, LCG, and LOFIC) produces better spatial alignment, the longer exposure also increases the likelihood of a blurred image on those same readouts.

[0053] Figure 6This is an illustration of an exemplary HDR and LED Flicker Reduction (LFM) engine (HALE) processing chain 600 according to embodiments of the present disclosure. Processing chain 600 illustrates the process of converting photodiode readouts (e.g., HCG 602, LCG 604, S 606, and LOFIC 608) into a corrected output (C) 646. As described above, after photodiode exposure according to any embodiment, LPD 414 generates HCG readout 602, LCG readout 604, and LOFIC readout 608; while SPD 416 generates S readout 606. The corrected output 646 requires three inputs: Combined Pixel Readout (CPR) 622, S readout 606, and LED Flicker Map (LFM) bit 642. CPR 622 is first generated by the operation of the classification module 610, which generates an output 616 fed to the combination module 620. The combination module 620 combines HCG readout 602, LCG readout 604, S readout 606, and LOFIC readout 608 to generate Combined Pixel Readout (CPR) 622, which is the HDR image routed to the correction module 644. Classified HCG readout 612, classified LCG readout 614, and classified LOFIC readout 618 will also be further used to determine LFM bit 642.

[0054] Photodiodes of any size can capture some amount of LED flicker, but LED flicker is most likely to be captured by photodiodes with longer exposure times. Thus, an LFM bit 642 is generated to ensure that the LED image or information is conveyed in the calibrated output 646 of the calibration module 642. This is achieved by first processing the S-readout 606 through spatial shift interpolation 624, which corrects for spatial misalignment of the SPD, and merging the S-readout 606 into a shifted S-signal 626, which is more directly comparable to the classified HCG readout 612, classified LCG readout 614, and classified LOFIC readout 618. In some embodiments, spatial shift interpolation 624 is performed in both the horizontal and vertical directions to correct for cross-rows of the SPD (e.g., Figure 4B The line R S1 To R Sy ) and columns (e.g., Figure 4B Column C S1 To C SxThe two are spatially misaligned. Then, four readouts can be used in the difference module 628 to determine an absolute difference (Abs.Diff.) 630 in groups of three: a first absolute difference between the classified HCG readout 612 and the shifted S readout 626; a second absolute difference between the classified LCG readout 614 and the shifted S readout 626; and a third absolute difference between the classified LOFIC readout 618 and the shifted S readout 626. These absolute differences 630 can be adjusted based on the exposure ratio between the compared readouts, which can vary based on factors such as exposure time, gain, and pixel sensitivity. The absolute difference 630 interprets the different brightness captured by each readout, such that LED flashing that is not fully captured in short-exposure LPD readouts 602, 604, 608 but fully captured in long-exposure SPD readout 606 is used to generate the LFM bit 642.

[0055] The absolute difference 630 is then processed by module 632 to assign an equivalent set of three LED flicker reduction (LFR) weights 634—one weight for each of the determined absolute differences described above. These weights are first assigned by establishing a threshold for each difference to account for noise and those incorrect due to, for example, saturation. Next, each weight 634 is proportionally assigned to its corresponding difference 630 such that the weight 634 is a monotonically increasing function of its corresponding difference 630. The weights 634 can then be scaled to produce values ​​between 0 and 1, such that a larger weight 634 signifies a greater likelihood that LED flicker is captured by the shifted S-readout 626, because the difference between the S-readout 626 and the readouts it is compared to (e.g., HCG 612, LCG 614, or LOFIC 618) is larger. Once the weight 634 is assigned to each absolute difference 630, the maximum weight (W) 638 can be selected by module 636 and used by LED blink mapping (LFM) module 640 to generate LFM bit 642, which can then be processed by LED blink mapping module 640 to detect and map the LED blink that needs to be corrected in the final corrected output (C) 628.

[0056] Ultimately, CPR 622, LFM bit, and S readout 606 are used to output 644 corrected pixel readouts (C) 646. The corrected pixel readout (C) 646 can be determined according to the equation C = (CPR × W) + S × (1 - W).

[0057] In the context of this specification, the term "module" refers to electronic circuitry or software running on a computer or controller. When such modules are hardware-based, those skilled in the art will know how to design and apply them as a combination of active components (controllers, operational amplifiers, transistors, etc.) and / or passive components (resistors, capacitors, etc.), and / or how to design them to suit software when they are software-based. Furthermore, some or all modules may be implemented as a combination of hardware and software. In various embodiments, the described modules may be external to pixel array 102 (e.g., executed by readout circuitry system 104 and / or functional logic 106) or internal to pixel array 102.

[0058] The advantages of the disclosed invention over conventional methods allow users to generate HDR images while mitigating LED flicker across a wide range of illumination. By electrically coupling the LOFIC 432 to each LPD 414, it is no longer necessary to expose the LPD twice, as the illumination range captured by the LOFIC readout is the same as that previously captured by the spatially misaligned VS readout. Because the LOFIC 432 is electrically coupled to each LPD 414, three of the four readouts (HCG, LCG, and LOFIC) generated by the exposure of pixel 402 only require a single short exposure duration, resulting in a sharp image with high dynamic range. This also allows the included SPD 416 to be exposed simultaneously for a longer duration. In contrast, examples of conventional techniques that specifically utilize the LPD produce blurry HCG / LCG / LOFIC readouts because the LPD is exposed for a relatively long duration (e.g., 11.1 ms) to mitigate LED flicker, resulting in a blurred image. Efforts to improve these image qualities still carry the risk of blurring because the image requires a second LPD exposure. As discussed above, this second exposure increases the risk of spatial misalignment in the resulting image. With the extended exposure time of the SPD 416, the disclosed invention does not eliminate the possibility of blurring in the resulting image; however, the disclosed invention allows for economical use of S-readout. Thus, the disclosed invention, by simultaneously exposing the SPD and LPD with LOFIC in a split pixel arrangement, allows users to produce HDR images while still mitigating LED flicker as needed.

[0059] Many embodiments of the techniques described above may take the form of computer- or controller-executable instructions comprising routines executable by a programmable computer or controller. Those skilled in the art will appreciate that the techniques can be practiced on computer / controller systems other than those shown and described above. The techniques may be embodied in special-purpose computers, application-specific integrated circuits (ASICs), controllers, or data processors specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described above. Of course, any logic or algorithm described herein may be implemented in software or hardware, or a combination of both.

Claims

1. A pixel array for a complementary metal-oxide-semiconductor image sensor, comprising: Multiple pixel units are formed in a semiconductor substrate, each pixel unit comprising: A pixel photodiode region having at least a large photosensitive element and a small photosensitive element, wherein the large photosensitive element is larger than the small photosensitive element, and wherein the large photosensitive element includes a lateral overflow integrated capacitor; and A pixel transistor region is disposed adjacent to the pixel photodiode region; The large photosensitive element is exposed for a first duration during each exposure period, and the small photosensitive element is exposed for a second duration during each exposure period; Wherein the first duration and the second duration are at least partially simultaneous; For each exposure: The large photosensitive element generates high conversion gain readout, low conversion gain readout, and lateral overflow integrated capacitor readout; and The small photosensitive element generates short-exposure readout; Furthermore, the high conversion gain readout, the low conversion gain readout, the lateral overflow integrated capacitor readout, and the short exposure readout are combined into a combined pixel readout.

2. The pixel array according to claim 1, wherein the first duration is less than the second duration, and wherein the large photosensitive element and the small photosensitive element simultaneously begin their respective exposures.

3. The pixel array according to claim 2, wherein the first duration is less than 5 ms and the second duration is greater than or equal to 11 ms.

4. The pixel array of claim 1, wherein the first duration is greater than the second duration, and wherein the large photosensitive element begins its exposure before the small photosensitive element begins its exposure.

5. The pixel array according to claim 4, wherein the first duration is greater than or equal to 11 ms and the second duration is less than 5 ms.

6. The pixel array of claim 1, wherein the pixel units are arranged in a split diode tiled arrangement, wherein the arrangement includes: Multiple large photosensitive elements; and A plurality of small photosensitive elements, wherein the plurality of large photosensitive elements and the plurality of small photosensitive elements are each arranged in rows and columns, wherein the plurality of large photosensitive elements are arranged adjacent to each other, and wherein the plurality of small photosensitive elements are embedded between adjacent large photosensitive elements, such that the large photosensitive elements and the small photosensitive elements are embedded to share a common boundary.

7. The pixel array according to claim 6, wherein each of the plurality of large photosensitive elements and the plurality of small photosensitive elements comprises two green photodiodes, one blue photodiode and one red photodiode; and wherein the plurality of large photosensitive elements and the plurality of small photosensitive elements are alternately colored through their respective rows and columns.

8. The pixel array of claim 1, wherein the combined pixel readout is a high dynamic range image.

9. The pixel array of claim 1, wherein the high conversion gain readout, the low conversion gain readout, the lateral overflow integrated capacitor readout, and the short exposure readout are combined into a combined pixel readout by the following steps: This causes the short exposure readout to shift; Determine the first absolute difference between the high conversion gain readout and the short exposure readout; Determine the second absolute difference between the low conversion gain readout and the short exposure readout; Determine the third absolute difference between the lateral overflow integrated capacitor readout and the short exposure readout; Assign a weight to each calculated absolute difference; Choose the maximum weight to calculate for each absolute difference; Generate a blinking pixel map; and Based on the combined pixel readout, the flashing pixel mapping, and the small photosensitive element readout output, the corrected pixel readout is obtained.

10. A method for generating a high dynamic range image, comprising: The large photosensitive element of the pixel unit is exposed for a first duration, wherein the large photosensitive element includes a lateral overflow integrated capacitor, and wherein the large photosensitive element generates a high conversion gain readout, a low conversion gain readout, and a lateral overflow integrated capacitor readout. Expose the small photosensitive element of the pixel unit to a second duration, wherein the large photosensitive element is larger than the small photosensitive element, and wherein the small photosensitive element produces a short exposure readout, and wherein the first duration and the second duration are at least partially simultaneous; and The high conversion gain readout, the low conversion gain readout, the lateral overflow integrated capacitor readout, and the short exposure readout are combined into a combined pixel readout.

11. The method of claim 10, further comprising: This causes the short exposure readout to shift; Determine the first absolute difference between the high conversion gain readout and the short exposure readout; Determine the second absolute difference between the low conversion gain readout and the short exposure readout; Determine the third absolute difference between the lateral overflow integrated capacitor readout and the short exposure readout; Assign a weight to each calculated absolute difference; Choose the maximum weight to calculate for each absolute difference; Generate a blinking pixel map; and Based on the combined pixel readout, the flashing pixel mapping, and the small photosensitive element readout output, the corrected pixel readout is obtained.

12. The method of claim 11, wherein the corrected pixel readout is calculated by the equation C = (CPR × W) + S × (1 - W).

13. The method of claim 10, wherein the first duration is less than the second duration, and wherein the large photosensitive element and the small photosensitive element simultaneously begin their respective exposures.

14. The method of claim 13, wherein the first duration is less than 5 ms and the second duration is greater than or equal to 11 ms.

15. The method of claim 10, wherein the first duration is greater than the second duration, and wherein the large photosensitive element begins its exposure before the small photosensitive element begins its exposure.

16. The method of claim 15, wherein the first duration is greater than or equal to 11 ms and the second duration is less than 5 ms.

17. The method of claim 10, wherein the pixel units are arranged in a split diode tiled arrangement, wherein the arrangement comprises: Multiple large photosensitive elements; and A plurality of small photosensitive elements, wherein the plurality of large photosensitive elements and the plurality of small photosensitive elements are each arranged in rows and columns, wherein the plurality of large photosensitive elements are arranged adjacent to each other, and wherein the plurality of small photosensitive elements are embedded between adjacent large photosensitive elements, such that the large photosensitive elements and the small photosensitive elements are embedded to share a common boundary.

18. The method of claim 17, wherein each of the plurality of large photosensitive elements and the plurality of small photosensitive elements comprises two green photodiodes, one blue photodiode and one red photodiode; and wherein the plurality of large photosensitive elements and the plurality of small photosensitive elements are alternately colored by their respective rows and columns.

19. The method of claim 10, wherein the combined pixel readout is a high dynamic range image.

Images