High Dynamic Range Image Sensor

Abstract

To provide an image sensor which is improved for slower external image frame transmission speed and has faster internal readout and processing speed of pixel row data.SOLUTION: In an image sensor 100 for laminating a first substrate 101 and a second substrate 103, the first substrate 101 includes an array 102 of pixels, and the pixel array is organized into a plurality of pixel sub-arrays 102-1 to 102-4. The second substrate 103 includes control and readout circuits 104-1 to 104-4 for selecting, controlling, and reading out pixel rows of the pixel array and for processing pixel data read out from the pixel rows of the array. The control circuit includes a plurality of row drivers 105-1 to 105-4 and control logic, and operates to trigger partial or complete transmission of two rolling shutter control sequences and to time-interleave at least two rolling shutter control sequences.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to the field of image sensors and digital cameras with improved dynamic range. In particular, it relates to high dynamic range CMOS image sensors using a stacked chip architecture.

Background Art

[0002] In this technical field, image sensors and digital cameras having high dynamic range (HDR) imaging capabilities based on multi-subframe or multi-frame exposure techniques are known. For example, US858,2010B2 discloses an image sensor that operates to repeatedly read out incomplete pixel signals during a single exposure procedure. The incomplete pixel signals are caused by the respective partial transmission of the photoelectric charges continuously generated within the photosensitive element of the pixel. However, in an image sensor with a large number of pixels and limited bandwidth for data transmission from the sensor chip, the number of readout steps and the exposure time for each readout step are very limited. This is because the generation speed of the pixel data to be transmitted and processed increases rapidly with the number of readout steps and the shortness of the readout steps, resulting in a bottleneck in data transmission. Increasing the I / O bandwidth of the image sensor makes the sensor design more complex, costly, and energy-consuming.

[0003] US2012 / 127354A1 (Cieslinski [DE]) (May 24, 2012) discloses a CMOS image sensor for electronic cameras in which a column readout circuit associated with each column line of the sensor includes two parallel amplifiers capable of operating at different gain coefficients. In high dynamic range mode, the two amplifiers are associated with different readout channels. Of the different readout channels, the first readout channel is used to perform multiple partial intermediate readouts, and the second channel is used to perform the final complete readout. The pixel signals obtained from the multiple intermediate readouts can be stored in a corresponding number of signal storage units connected before the amplifiers. A drawback of this image sensor is related to the different readout paths used for high-signal gain and low-signal gain, which are more susceptible to artifacts such as crosstalk and banding. Furthermore, the use of multiple amplifiers per column readout circuit increases cost and area requirements. As a result, when the sensor is operating in HDR mode, the effective frame rate for a given I / O bandwidth is reduced, and the saturation capacitance (FWC) of the sensor pixels is still limited by the FWC of the photodiodes of the pixels.

[0004] US10103193B1 (Manabe [US], et al.) (October 16, 2018) discloses a CMOS image sensor and imaging method for achieving low dark current. The method involves repeated sampling of image charge on a pixel photodiode, the sampled charge being transmitted to a stray diffuse. While the image charge is repeatedly sampled, an additional capacitance is coupled to the stray diffuse, and a capacitance voltage is applied to the additional capacitance. By modulating the additional capacitance during image charge accumulation, the dark current associated with the stray diffuse can be reduced. The additional capacitance reduces the conversion gain, which is beneficial at high light intensity. However, this method cannot achieve a further increase in dynamic range beyond what is provided by dual conversion gain.

[0005] WO2015084991A1 discloses an integrated circuit image sensor that is oversampled a first number of times within a first frame interval. A first output image can be constructed from a first number of image data frames corresponding to the first number of oversamplings. Based on the first number of image data frames, the oversampling coefficient can be adjusted to oversample the pixel array a second number of times within a second frame interval. Readout operations for shorter oversampled subframes may be conditional, thereby increasing the cumulative exposure time and the resulting dynamic range. However, the verification circuits and associated delays required to confirm that the conditions are met increase the complexity of the image sensor and limit the speed at which the image sensor can operate.

[0006] It is desirable to improve currently available HDR image sensors so that HDR imaging can be performed at a faster speed and with less power consumption. [Overview of the project]

[0007] One object of embodiments of the present invention is to provide an image sensor capable of acquiring image frames in a high dynamic range, and an associated method for operating the image sensor. A further object of the present invention is to provide an image sensor having faster internal readout and processing speed of pixel row data, improved for slower external image frame transmission speeds.

[0008] The above objectives are achieved by the methods and devices according to the present invention.

[0009] In one embodiment, the present invention relates to a vertically stacked image sensor device provided as an integrated circuit. The stacked image sensor device comprises a first substrate having a pixel array organized into a plurality of pixel subarrays. Each pixel in the pixel array comprises a photoelectric element, a transmission gate, and a buffered charge-voltage converter. Preferably, the photoelectric element, implemented as a fixed photodiode, is adapted to accumulate photocharge while exposed to light during each subframe exposure of a plurality of subframe exposures included in a frame interval. The transmission gate is configured to transmit at least a portion of the accumulated photocharge present in the photoelectric element at the end of each subframe exposure to a buffered charge-voltage converter, which is configured to receive the transmitted photocharge and convert it into a voltage signal. This voltage signal, and optionally a reference voltage signal after resetting the pixel, constitutes pixel data to be read out. Furthermore, the first charge storage element of the charge-voltage converter is operably connectable to at least one second charge storage element via a gain switch, allowing the pixel's conversion gain to be switched between a high conversion gain and a low conversion gain. The image sensor device includes a control circuit configured to trigger partial or complete transmission of the integrated photocharge within the photoelectric elements in each pixel row. The amplitude of the transmission pulse that can be applied to the transmission gate is adjustable through the control circuit so that partial transmission of the integrated photocharge is triggered for all but the last of a plurality of subframe exposures, and complete transmission is triggered only for the last of the plurality of subframe exposures. The control circuit is also configured to interleave at least two rolling shutter control sequences associated with at least two time-overlapping subframe exposures among the plurality of subframe exposures with respect to each pixel subarray. Furthermore, the control circuit is configured to control the gain switch for each pixel row so that the pixels operate at a low conversion gain for all but the last of the plurality of subframe exposures, and for the last of the plurality of subframe exposures, the pixels operate first at a high conversion gain and then at a low conversion gain.The second substrate of the image sensor device is stacked vertically and electrically interconnected with the first substrate, and includes a readout circuit. The readout circuit includes a separate readout block for each pixel subarray. Each readout block comprises, within a pipelining architecture, an A / D conversion unit for sampling and digitizing pixel row data of the corresponding pixel subarray; pixel memory logic for processing the digital pixel row data and conditionally combining the processed digital pixel row data with previously processed digital pixel row data buffered in a pixel memory unit; and a pixel memory unit for buffering the processed digital pixel row data output by the pixel memory logic. Multiple readout blocks are configured for parallel operation.

[0010] In embodiments of the present invention, the rolling shutter control sequence preferably controls the timing of reading out pixel data at the end of each subframe exposure by applying appropriate reset signals, photocharge transmission pulses, and row selection signals to the pixels of the row of addressed pixels. By time-interleaving two or more rolling shutter control sequences, it is ensured that control signals (e.g., reset signals, photocharge transmission pulses, and row selection signals) are transmitted to only one pixel row of each pixel subarray at a time, and that two or more corresponding subframes are read out simultaneously. Here, simultaneous reading of two or more subframes means that the reading of one subframe is not yet complete and is continuing when the reading of at least one next subframe has started.

[0011] Preferably, the control circuit is provided on a second substrate, for example, formed on or within the second substrate. However, in less preferred but equally functional embodiments, the control circuit may be provided entirely or partially on the first substrate. Providing the control circuit on the second substrate has the advantage, with respect to the first substrate, of allowing the selection of a less complex and lower-cost manufacturing process, e.g., a manufacturing process requiring fewer lithography steps / masks (the first substrate, for example, contains only one doping type (e.g., p or n doping) active areas and transistors). Additionally, if the control circuit is provided on the second substrate, for example, such that the pixel subarray overlaps with the row drivers of the control circuit when viewed from above in the substrate stacking direction, the time required to access the pixel rows of the pixel subarray for readout can be reduced due to shorter interconnect and wiring lengths and corresponding capacitances.

[0012] In embodiments of the present invention, the second charge storage element may be part of a charge-voltage converter provided directly on the first substrate, or it may be provided as a separate gain circuit on the second substrate. Furthermore, although embodiments of the present invention are described in preference to pixels having dual conversion gains, those skilled in the art will understand that the present invention is also applicable to pixels having multiple (e.g., two or more) conversion gain coefficients, including pixels or pixel circuits comprising two or more switchable charge storage elements, or pixels or pixel circuits comprising at least one variable capacitance such as a voltage-controlled capacitance.

[0013] The parallel, pipelining chip architecture of stacked image sensors enables the integration of in-chip pixel memory and pixel memory logic for in-chip processing of pixel data read at high internal speeds without affecting the external speed at which ready frames are transmitted from the chip. This leads to a significant reduction in the amount of data that needs to be communicated via the image sensor's I / O interface, as well as a significant reduction in the amount of post-processing that needs to be performed off-chip. This results in lower power consumption of the image sensor chip. Furthermore, multiple subframes can be pre-combined within the image sensor device to generate a final image frame with improved dynamic range. The dual conversion gain characteristics of the image sensor pixels allow for subframe exposure with a good signal-to-noise ratio under both low-light and high-light illumination conditions. By partially transmitting the photocharge generated under low-light conditions and applying a low conversion gain during all subframe exposures except the last subframe exposure, the noisy high-gain channel can be applied only once during the last subframe exposure. This significantly reduces the total readout noise present in the summed subframe exposures.

[0014] In addition to high dynamic range, other application modes can be executed internally, allowing for the benefit of higher internal processing speeds while maintaining a lower external frame rate defined by the conventional I / O circuitry of the image sensor. Such other application modes may include multiple subframe exposure modes in which all subframes are obtained using a single identical conversion gain, and / or multiple subframe exposure modes in which all subframes are acquired under conditions of complete photocharge transmission. The image sensor device according to the present invention can operate in different application modes, and the control circuitry of the image sensor can be reconfigured accordingly.

[0015] According to some embodiments of the present invention, the control circuit may comprise a plurality of row drivers for driving pixel rows in each pixel subarray, with at least two of the plurality of row drivers associated with each pixel row to drive different portions of the pixel row. In certain embodiments, the plurality of row drivers may be associated with each pixel subarray, and each row driver of the plurality of row drivers is configured to correspond to each pixel subarray of the pixel subarray and to drive only a subset of pixels in each pixel row corresponding to that subarray. Preferably, the plurality of row drivers correspond to the subdivision of a single row driver, for example, subdivision along the direction in which the pixel column number in the subarray increases (column-by-column grouping of pixels in each row such that each group is driven by one of the plurality of drivers). Subdividing the row drivers has the advantage of reducing read access time compared to conventional row drivers that are located only to the left / right of the pixel row.

[0016] According to some embodiments of the present invention, the ratio between the subframe exposure rate and the output frame rate is at least 2, for example, 2 or more, for example, 4 or more.

[0017] An advantage of the embodiments of the present invention is that the intermediate reading of a pixel only needs to occur in a low conversion gain configuration of the pixel, which makes it possible to use only a single data memory location per pixel, even with multiple (e.g., two or more) subframe exposures per full image frame.

[0018] A further advantage of embodiments of the present invention is that by operating the image sensor device in partial transmission mode, i.e., by performing partial transmission of accumulated photocharge at the end of each subframe exposure except for the last subframe exposure, the requirement for uniformity of potential barrier height across pixels in the array, which is particularly affected by amplitude noise on the partial transmission pulses, is relaxed. Non-uniformity of the potential barrier across pixels in the array is no longer a concern, as long as it is ensured that the amount of accumulated photocharge remaining in each pixel (i.e., not transmitted) after the sequence of partial photocharge transmissions is sufficient to cover the signal range in the pixel's high-conversion-gain channel at the end of the last subframe exposure (which satisfies a much more lenient requirement). This is because if the amount of accumulated photocharge fully transmitted at the end of the final subframe exposure exceeds the signal range in the pixel's high-conversion-gain channel, pixel reading is performed at a low-conversion-gain. Pixel reading at a low-conversion-gain channel allows for the favorable summation of all accumulated photocharge partially and completely transmitted over the set of subframe exposures and with respect to the same pixel gain configuration (e.g., a combination of interconnected first and second charge storage elements). Therefore, non-uniformity resulting from partial transmission operation does not affect the final output obtained for low-gain channels.

[0019] An advantage of the embodiments of the present invention is that there is no increase in readout noise for multiple subframe exposures, thereby achieving a higher dynamic range (DR).

[0020] An advantage of the embodiments of the present invention is that a DR of 80 dB or more can be obtained. By using two or more subframe exposures, a DR of, for example, 100 dB or more, for example, 120 dB or more, 80 dB to 120 dB, for example, 80 dB to 150 dB can be obtained.

[0021] An advantage of the embodiments of the present invention is that HDR imaging can be performed at high speed, for example, faster than current methods, and with limited power consumption, for example, with less power than current methods. In conventional image sensors, as the imaging speed increases, the required bandwidth increases, and therefore power consumption increases. Higher power consumption generally results in larger cameras due to the need for cooling, or leads to higher operating temperatures, which often negatively impacts the performance of the image sensor (e.g., higher dark current). Therefore, embodiments of the present invention have the additional advantage that image frame data is transmitted over a slower I / O bandwidth compared to the higher internal bandwidth available for reading out pixel data and internal processing of pixel data, which keeps power consumption related to off-chip transmission of image frame data low. This enables the design of small and compact camera modules and a reduction in dark current levels.

[0022] In another aspect, the present invention relates to a method for operating an integrated circuit image sensor. The image sensor comprises a pixel array disposed within a pixel substrate and organized into a plurality of pixel subarrays. The method includes accumulating photocharges in the pixels of the pixel array during each subframe exposure of a plurality of subframe exposures included in a frame interval, and partially transmitting the accumulated photocharges of the pixels to the respective charge-voltage converters of the pixels at the end of all subframe exposures except the last subframe exposure. The charge-voltage converters are configured to apply a low conversion gain when reading out the partially transmitted photocharges at the end of all subframe exposures except the last subframe exposure. At the end of the last subframe exposure, the accumulated photocharges of the pixels are fully transmitted to the respective charge-voltage converters of the pixels. The charge-voltage converters are configured to provide a high-gain channel during readout by first applying a high conversion gain when reading out the fully transmitted photocharges, and then immediately provide a low-gain channel during readout by applying a low conversion gain. The pixel row data of each subarray is sequentially read out at the end of each subframe exposure, and the process includes the steps of sampling and digitizing the pixel row data, conditionally combining the digitized pixel row data with previously digitized pixel row data buffered in the pixel memory unit of the readout block, and buffering the digitized pixel row data in the pixel memory unit of the readout block. According to the present invention, pixel row data from different pixel subarrays are read out in parallel, and for each pixel subarray, at least two of the multiple subframe exposures overlap in time, and the rolling shutter control sequences associated with at least two subframe exposures are time-interleaved.

[0023] In some embodiments of the present invention, two or more image frames each composed of a plurality of sub-frames may be fused or combined into one output frame, and the method steps may be applied separately (e.g., sequentially) to each of the two or more image frames. The fusion or combination may be performed externally, i.e., outside the chip.

[0024] Certain preferred aspects of the present invention are described in the appended independent and dependent claims. Features from the dependent claims may, where appropriate, be combined with the features of the independent claims and the features of other dependent claims, and are not necessarily limited to being as explicitly described in the claims.

[0025] To summarize the advantages achieved over the present invention and the prior art, certain objects and advantages of the present invention are described above in this specification. It should be understood, of course, that not necessarily all such objects or advantages can be achieved according to any particular embodiment of the present invention. Thus, for example, those skilled in the art will recognize that the present invention may be embodied or implemented so as to achieve or optimize one advantage or group of advantages taught in this specification without necessarily achieving other objects or advantages that may be taught or suggested in this specification.

[0026] The above and other aspects of the present invention will become apparent from the embodiments described below and will be described in detail by reference to them.

Brief Description of the Drawings

[0027] The present invention will be further described, by way of example, with reference to the following appended drawings.

[0028] [Figure 1] It is a schematic layer-by-layer diagram of a stacked image sensor having an extended dynamic range function according to an embodiment of the present invention. [Figure 2] It is a circuit diagram of a pixel having a dual conversion gain that may be used in an embodiment of the present invention. [Figure 3]Shows the circuit components that make up the read block used in an embodiment of the present invention. [Figure 4A] It is a timing diagram showing the parallel and pipelined read architecture of an image sensor according to the present invention. [Figure 4B] It is a timing diagram showing the parallel and pipelined read architecture of an image sensor according to the present invention. [Figure 5] It is a timing diagram showing the parallel and pipelined read architecture of an image sensor according to the present invention. [Figure 6A] It is a timing diagram showing the time-interleaved operation of two electronic rolling shutters on a pixel subarray of an image sensor according to the present invention, and a timing diagram showing the combination and intermediate storage of pixel row data obtained from two sub-frame exposures. [Figure 6B] It is a timing diagram showing the time-interleaved operation of two electronic rolling shutters on a pixel subarray of an image sensor according to the present invention, and a timing diagram showing the combination and intermediate storage of pixel row data obtained from two sub-frame exposures. [Figure 7A] It is a timing diagram showing the time-interleaved operation of two electronic rolling shutters on a pixel subarray of an image sensor according to the present invention, and a timing diagram showing the combination and intermediate storage of pixel row data obtained from two sub-frame exposures. [Figure 7B] It is a timing diagram showing the time-interleaved operation of two electronic rolling shutters on a pixel subarray of an image sensor according to the present invention, and a timing diagram showing the combination and intermediate storage of pixel row data obtained from two sub-frame exposures. [Figure 8] Describes a method of operating an image sensor under different illumination conditions according to an embodiment of the present invention, and the sub-frame exposure is combined into a final image frame having an extended dynamic range. [Figure 9]This invention describes a method for operating an image sensor under different lighting conditions, according to one embodiment of the present invention, in which subframe exposure is combined with a final image frame having an extended dynamic range. [Figure 10] This invention describes a method for operating an image sensor under different lighting conditions, according to one embodiment of the present invention, in which subframe exposure is combined with a final image frame having an extended dynamic range. [Figure 11] Each image shows the image sensor output before and after applying linearization to the sensor output signal, and the sensor output signal was acquired for image frames with four different subframe exposure timings. [Figure 12] Each image shows the image sensor output before and after applying linearization to the sensor output signal, and the sensor output signal was acquired for image frames with four different subframe exposure timings. [Figure 13A] This diagram shows the time-interleaved operation of four electronic rolling shutters on a pixel subarray of an image sensor according to the present invention, and also shows the combination and intermediate storage of pixel row data obtained from four subframe exposures. [Figure 13B] This diagram shows the time-interleaved operation of four electronic rolling shutters on a pixel subarray of an image sensor according to the present invention, and also shows the combination and intermediate storage of pixel row data obtained from four subframe exposures.

[0029] The drawings are schematic and non-limiting. Some element sizes in the drawings may be exaggerated for illustrative purposes and may not be drawn to scale. Dimensions and relative dimensions do not necessarily correspond to actual reductions in the implementation of the invention.

[0030] None of the reference numerals in the claims should be construed as limiting the scope.

[0031] The same reference numeral in different drawings refers to the same or similar element. [Modes for carrying out the invention]

[0032] The present invention will be described in relation to specific embodiments and with reference to certain drawings, but the present invention is not limited thereto and is limited only by the claims.

[0033] The terms relating to direction, such as up, down, front, back, front, rear, downward, and upward, used herein and in the claims are for illustrative purposes only, with reference to the orientation of the drawings described, and are not necessarily used to describe relative positions. Since the components of the embodiments of the present invention can be arranged in multiple different orientations, the terms relating to direction are used for illustrative purposes only and are not intended to limit in any way unless otherwise indicated. Accordingly, the terms used in this manner are interchangeable under appropriate circumstances, and it should be understood that the embodiments of the present invention described herein may operate in orientations other than those described or illustrated herein.

[0034] It should be noted that the terms “equipped with / including” as used in the claims should not be construed as being limited to the means enumerated thereafter, nor should they exclude other elements or steps. Therefore, they should be interpreted as identifying the presence of the features, integers, steps, or components described as mentioned, but not as excluding the presence or addition of one or more other features, integers, steps, or components, or groups thereof. Accordingly, the expression “a device comprising means A and B” should not be limited to a device consisting solely of components A and B, but rather, with respect to the present invention, means that A and B are the only reasonable components of the device.

[0035] Throughout this specification, any reference to “one embodiment” or “embodiment” means that the specific features, structures, or characteristics described in relation to that embodiment are included in at least one embodiment of the present invention. Therefore, the occurrence of the phrase “in one embodiment” or “in one embodiment” in various parts of this specification does not necessarily refer to the same embodiment, although it may. Furthermore, as will be apparent to those skilled in the art from this disclosure, certain features, structures, or characteristics may be combined in any preferred manner in one or more embodiments.

[0036] Similarly, in describing exemplary embodiments of the Invention, it should be understood that various features of the Invention may be grouped together in a single embodiment, drawing, or description thereof in order to streamline the disclosure and aid in understanding one or more of the various embodiments of the Invention. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed Invention requires more features than are explicitly enumerated in each claim. Rather, as reflected in the claims described later, embodiments of the Invention exist in fewer features than all the features of a single, previously described embodiment. Thus, the claims following a mode for carrying out the Invention are explicitly incorporated into the mode for carrying out the Invention, and each claim stands on its own as a distinct embodiment of the Invention. Furthermore, some embodiments described herein include some features included in other embodiments, but not others, while combinations of features from multiple different embodiments are intended to be included within the scope of the Invention and form different embodiments, as will be understood by those skilled in the art.

[0037] Numerous specific details are provided in this specification. However, it should be understood that embodiments of the present invention may be carried out without using these specific details. In other instances, well-known methods, structures, and techniques are not described in detail so as not to obscure the understanding of this description.

[0038] In the context of the present invention, the photocharge accumulated by a photosensitive element such as a photodiode when irradiated with light is understood as a fixed amount of charge, and can be expressed in units of fundamental charge, for example, as a plurality of electrons generated by light.

[0039] In a first aspect, the present invention relates to a stacked image sensor, for example, a vertically stacked CMOS image sensor. The image sensor provides the acquisition of image or video frames with a higher dynamic range and further provides a higher internal data readout and processing rate compared to the device's external readout speed, i.e., the rate at which units of data (rows or full frames) are transmitted from the image sensor chip. When a conventional image sensor device operates at its maximum external frame rate, it also operates at the lowest internal row time (the time required to address a row of pixels, read out the row of addressed pixels, perform analog-to-digital (A / D) conversion of the readout pixel data, and transmit a complete row of pixel data from the chip (external readout)). In the stacked image sensor according to the present invention, operating at the maximum external frame rate does not hinder the implementation of the pixel row readout operation, as well as the further data processing operations using the readout pixel data, to be performed internally at a much faster rate. In embodiments of the present invention, an exemplary method for utilizing faster internal pixel row readout and pixel data processing operations is to operate the image sensor device in a dual or multi-subframe exposure mode, in which pixel row data from multiple subframe exposures are combined into a high dynamic range (HDR) image frame.

[0040] Figure 1 shows a stacked image sensor 100 as a stacked structure in which an upper first substrate 101 is stacked vertically on a lower second substrate 103. The first substrate 101 of the image sensor 100 comprises a pixel array 102 and may be configured to operate under forward-illuminated or back-illuminated conditions. The pixel array 102 may be organized into a plurality of pixel subarrays 102-1 to 102-4 by, for example, vertically dividing the pixel array into blocks of independently addressable pixel rows, each block containing a plurality of consecutive pixel rows of the pixel array. A block of consecutive pixel rows is just one specific example of dividing the pixel array 102 into a set of pixel subarrays. Other methods exist for assigning individual pixel rows of a pixel array to one of several pixel subarrays, such as random assignment of individual pixel rows to one of several subarrays, or assignment of consecutive pixel rows of a pixel array to different pixel subarrays, such as an interleaved assignment (n=(i mod N)) where the i-th pixel row is assigned to the n-th pixel subarray out of N pixel subarrays. Interleaved assignment of pixel rows to a set of pixel subarrays, and the resulting interleaved connections of pixel rows to different readout blocks, may have the additional advantage of maintaining a uniform rolling shutter effect across the entire pixel array.

[0041] The second substrate 103 includes control and readout circuits for selecting, controlling, and reading out pixel rows of a pixel array, and for processing pixel data read from the pixel rows of the array. The readout circuits are organized into a plurality of blocks 104-1 to 104-4, referred to as readout blocks, which in number correspond to pixel subarrays. More specifically, each readout block is matched to and electrically connected to exactly one of the pixel subarrays. The electrical connection between the pixel subarrays and the readout blocks may be a fixed connection for easier implementation, or a reconfigurable connection to provide greater flexibility. For example, if vertical and / or horizontal windowing is applied to the pixel array of an image sensor, the pixel subarrays may be reconfigured to connect to the readout blocks to optimize the imaging speed. This association between the pixel subarrays and the readout blocks also enables parallel control of each pixel subarray, simultaneous readout of k different pixel rows associated with k different pixel subarrays, and parallel processing of the readout pixel row data. Typical numbers k for the pixel subarray and corresponding readout blocks can be k = 2, ..., 8, 16, ..., 128. When reading two or more pixel rows simultaneously, the image sensor is configured to operate in parallel. As will be described in more detail below, the readout operation and the data processing operations performed on the readout pixel row data are pipelined with respect to each pixel subarray-readout block pair.

[0042] For example, an image sensor may include 16 readout blocks and a pixel array having thousands of pixel rows, for example 6,000 pixel rows, where two vertically adjacent pixels share a common pixel (i.e., 2 x 1 vertical sharing). During the first row time, the first readout block reads the pixel data from row 0, the second readout block reads the pixel data from row 2, the third readout block reads the pixel data from row 4, and so on. Next, during the second row time, the first readout block reads the pixel data from row 1, the second readout block reads the pixel data from row 3, the third readout block reads the pixel data from row 5, and so on. During the third row time, the first readout block reads the pixel data from row 32, the second readout block reads the pixel data from row 34, the third readout block reads the pixel data from row 36, and so on. During the fourth row time, the first read block reads the pixel data for row 33, the second read block reads the pixel data for row 35, the third read block reads the pixel data for row 37, and so on. This process continues until the last row of pixels in the pixel array is read.

[0043] The control circuit preferably comprises a plurality of row drivers 105-1 to 105-4 that correspond in number to a plurality of readout blocks 104-1 to 104-4, and control logic (not shown) for controlling the plurality of row drivers, for example, for controlling the sequencing of row control signals driven by the row drivers (e.g., row selection control signals, reset control signals, charge transmission gate control signals). Due to the vertical stacking of the two substrates 101 and 103, as shown in Figure 1, the row drivers 105-1 to 105-4 are located on the second substrate 103 such that, when viewed from above the image sensor (e.g., the light-emitting side), they are below the pixel array and overlap the pixel rows of the pixel array, and may extend mainly in the (pixel) row parallel direction x, i.e., in the direction in which the number of pixel columns increases.

[0044] Alternatively, row drivers may be located on a second substrate 103 such that, when viewed from above the image sensor, they are to the left and / or right of the pixel rows and do not substantially overlap with the pixel rows of the pixel array, and may extend mainly in the direction in which the (pixel) row number y increases. Furthermore, in an alternative embodiment of the present invention, row drivers may be located on the first substrate 101, for example, to the left and / or right of the pixel rows. Providing row drivers on a second substrate has the advantage that row address noise that may interfere with the pixel signal can be reduced, and furthermore, a larger surface area of ​​the first substrate is available for imaging.

[0045] Although shown as individual blocks in Figure 1, one or more (e.g., all) row drivers 105-1 to 105-4 in the x-direction may be further subdivided, meaning that multiple row drivers are used to drive a single pixel row. Such subdivision has the advantage of reducing read access time (reducing the RC time constant) compared to conventional row drivers located only to the left / right of the pixel row. The second substrate 103 may also include additional circuits such as serialization circuits and I / O drivers configured to generate and output data streams relating to the image frame.

[0046] The vertical stacking of the first and second substrates 101 and 102 can be achieved by die-to-die bonding, or, with subsequent wafer dicing, by die-to-wafer or wafer-to-wafer bonding. The first and second substrates are electrically interconnected, for example, via silicon vias (TSVs) or direct or hybrid bonding techniques (e.g., copper-to-copper interconnects), such that sets of pixel row data signals associated with multiple pixel rows located within each pixel subarray of the first substrate are simultaneously read by corresponding sets of readout blocks in the second substrate. Each pixel row data signal is transmitted from the first substrate to the second substrate on a column bit line. The column bit line is understood to extend through the interconnection layer between the first and second substrates.

[0047] Figure 2 shows a possible circuit architecture for an active pixel configured to have two different charge-to-voltage conversion gains, which will be referred to below as a dual-gain pixel or dual-conversion-gain pixel. The pixel 200 comprises a photoelectric element 201 (preferably an embedded photodiode), a buffered charge-to-voltage converter 202, and a transmission gate 203 connected between the photoelectric element and the charge-to-voltage converter. The buffered charge-to-voltage converter includes a floating diffusion node 202-1 as a first charge storage element of the charge-to-voltage converter, a source follower transistor 202-2 having a gate terminal electrically connected to the floating diffusion node 202-1, and a combination of a gain switch 202-3 and a gain capacitance 202-4 as a second charge storage element of the charge-to-voltage converter.

[0048] Pixel 200 further includes a reset transistor 204 connected between a positive voltage source VDD and a floating-spread node 202-1. The reset transistor 204 erases previously stored pixel data in the floating-spread node by resetting the floating-spread node to a predetermined voltage level close to VDD each time the reset transistor is turned on by the corresponding reset control signal VRST. Furthermore, the source follower transistor 202-2 is controllably connectable to the bit line 206 via a row selection control signal VRS that can be applied to the gate of the row selection transistor 205 of the pixel. In Figure 2, the row selection transistor 205 is connected between the source follower transistor and the voltage source VDD, but those skilled in the art will understand that different arrangements of the row selection transistor are possible, for example, an arrangement in which the row selection transistor is connected between the source follower transistor 202-2 and VOUT on the bit line 206.

[0049] When light is shone on the pixel 200, the photoelectric element 201 begins accumulating photocharges generated in proportion to the amount of light received. When the transmission gate 203 is activated by a suitable charge transmission control signal VTX, i.e., a transmission pulse, the accumulated photocharges, or at least a portion thereof, are transmitted to the stray-diffusion node 202-1 of the buffered charge-voltage converter 202. The control circuit of the image sensor (not part of the pixel circuit) sets the amplitude of the transmission pulse, e.g., the voltage amplitude, so as to trigger partial transmission of the generated photocharges to the stray-diffusion node or to the stray-diffusion node and connected gain capacitance, or so as to complete transmission of the generated photocharges to the stray-diffusion node or to the stray-diffusion node and connected gain capacitance. Generally, a higher amplitude of the transmission pulse reduces the potential barrier separating the charge well associated with the photoelectric element from the charge well associated with the stray-diffusion node (with or without connected gain capacitance), and therefore more photocharge carriers are transmitted away from the photoelectric element. The capacitance associated with the stray diffusion node allows for the temporary storage of transmitted photocharge, converting the deposited photocharge into a voltage signal sensed by the source follower transistor 202-2. When the row selection transistor 205 is turned on, i.e., when a pixel is selected for readout, a current set by an external current source begins to flow through the row selection and source follower transistors on the corresponding bit line 206. The voltage VOUT at the source terminal of the source follower transistor follows directly the voltage signal applied to its gate terminal. When the gain switch 202-3 is open (e.g., low voltage in the VCG), the first charge-voltage conversion gain is determined by the capacitance value of the stray diffusion node. When the gain switch 202-3 is switched to the closed state (e.g., high voltage in the VCG), some of the photocharge originally stored in the stray diffusion node flows into the additional gain capacitance 202-4. Since the additional gain capacitance and the stray diffusion node capacitance are connected in parallel here, a larger overall capacitance becomes available for storing the transmitted photocharge.This reduces the voltage signal sensed at the gate terminal of the source follower transistor, which is then directly converted into a lower second charge-voltage conversion gain. In an alternative embodiment, the additional gain capacitance may also be connected to a positive voltage source instead of ground, or the additional gain capacitance may correspond to the stray diffusion node of an adjacent pixel. Thus, in the latter alternative example, the pixel is configured to dynamically share its stray diffusion node with at least one adjacent pixel of a different row, and the shared stray diffusion node of the pixel is temporarily connected to the adjacent pixel and functions as additional gain capacitance. This has the advantage of allowing for a more compact design that does not require a separate pixel component for gain capacitance.

[0050] The row control circuit of the image sensor, of which pixel 200 forms part, is configured to control gain switches 202-3 via a dedicated conversion gain control signal VCG that can be applied to the control terminal of its gain switch. Thus, pixel 200 can be controlled to apply a first or second charge-to-voltage conversion gain to the photocharges accumulated during readout. Since the readout is non-destructive, i.e., the transmitted and stored photocharges are not destroyed or modified by the readout action, the charge-to-voltage converter is buffered.

[0051] Figure 3 shows the different circuit components that make up each read block 104 of the second board when multiple read blocks are configured for parallel operation. Each read block 104 comprises at least one analog-to-digital conversion unit 106 for sampling and digitizing pixel row data of the corresponding pixel subarray, a pixel memory logic 107 for processing the samples of the digitized pixel row data, and a pixel memory unit 108 for buffering the processed samples of the digital pixel row data output by the pixel memory logic 107. The data flow between components, or between components and peripheral I / O circuits, is indicated by arrows. In particular, intermediate pixel row data stored in the pixel memory unit 108 may be accessed by the pixel memory logic (PML) 107 in order to conditionally combine the currently processed sample of digital pixel row data with a previously processed sample of digital pixel row data that is buffered in the pixel memory unit 108. The combination of two such processed samples of digital pixel row data may then be written back to the pixel memory unit 108. From there, for example, if the buffered combination of processed samples of digital pixel row data is the final sample transmitted from the image sensor chip, it is accessed by peripheral I / O circuits, or for example, if the buffered combination of processed samples of digital pixel row data is an intermediate result that is not yet ready for output, it is accessed again by the PML. In the latter case, the PML may use the retrieved processed sample of digital pixel row data and the intermediate combination of the currently processed sample of digital pixel row data to calculate an updated or final combination of processed samples of digital pixel row data. In embodiments of the present invention, the PML may combine two or more samples of digital pixel row data by adding or subtracting samples.Furthermore, the processing operations performed by PML on a sample of digital pixel row data may include scaling the sample, for example, comparing the sample to a threshold pixel by pixel before combining it with another sample, calculating changes in the representation of the sample (e.g., from Gray code to binary), or a combination thereof. The processing capabilities of PML are not limited to the given examples and can be extended to perform additional processing operations on acquired digital pixel row data samples, which may depend on the specific application or operating mode for the image sensor.

[0052] Embodiments of the present invention are not limited to readout blocks that do not have an analog column bit line signal amplification stage, which may be part of an A / D conversion unit. However, it is preferable to directly acquire digital samples of pixel data within a selected pixel row by directly connecting the A / D converter to the column bit line without using an intermediate amplifier. This has the advantage of reducing conversion time, allowing for shorter unit time slots, which in turn increases the number of subframe exposures within a single full-frame period. It is also possible to further reduce the time associated with conversion by lowering the bit resolution of the A / D converter (ADC) within the conversion unit, at the expense of reading out less accurate pixel data. A typical embodiment of the present invention implements an ADC having 12-bit resolution. With respect to readout speed, given the goal of achieving multiple subframe exposures within a single full-frame period, a high-speed ADC is preferred, for example, a SAR-ADC when efficient use of silicon area is not critical. However, embodiments may also use other ADC types, for example, a single-slope ADC may be used when a more energy-efficient and / or area-efficient implementation is desired in embodiments where an ADC is implemented per pixel. Ramp-based ADC architectures (e.g., including single-slope ADCs) have a simple layout and use a smaller design area per pixel row compared to other ADC architectures. They are preferred in embodiments of the present invention where bit resolution can be traded for operating speed, for example, by the number of clock cycles of the ADC counter. This allows for flexible and dynamic adjustment of the bit resolution per subframe (e.g., 12.7 bits, not necessarily an integer), and therefore the number of subframe exposures included within the full frame duration of the image sensor.

[0053] The pixel memory units of all read blocks function as a global frame buffer. In certain embodiments of the present invention, this global buffer has a data buffer capacity smaller than the size of a full image frame and can only hold, for example, a portion of a full image frame generated by the image sensor. This is possible, for example, if pixel row data is retrieved from the global buffer quickly enough for transmission from the chip so as to prevent buffer overflow. In this case, different pixel rows can be mapped to the same location in the pixel memory unit without losing pixel row data. Intermediate storage of pixel row data in the pixel memory unit is typically performed at time intervals longer than the total time to select a row of pixels and convert the pixel data of the selected row in the A / D conversion unit, but shorter than the full frame duration. Preferably, the pixel memory units 108 on the second substrate of the image sensor chip are provided, for example, as blocks of SRAM unit cells used as memory banks of banked SRAM memory. The memory units on the second substrate, for example, SRAM memory units, are preferably managed independently of each other at the read block level. The memory unit corresponding to each read block may be further subdivided into smaller memory subunits, for example, as well as row driver subdivisions. Pixel memory units managed at the read-block level, or subdivided versions thereof, are advantageous due to their small physical size and address space, and they speed up read / write operations of the pixel memory units. They may also prove useful for yield reasons; for example, each memory unit corresponding to a read block can be constructed with some redundancy to allow for independent handling of memory defects.

[0054] Figure 4 is a flowchart showing a parallelized, pipelining architecture of stacked image sensors during image or video acquisition. Pipelining includes the following steps for each pixel row of the image sensor: pixel row reset, pixel row exposure, pixel row data readout and analog-to-digital conversion, a complete fetch-process-write (FPW) cycle performed on the pixel row data digitized by the Pixel Memory Logic (PML), writing the digitized pixel row data back to the pixel memory unit for intermediate storage, and accessing the pixel memory to generate a global I / O stream of the pixel row data when transmitting one frame (or consecutive frames) of the processed image data from the sensor chip. As described above, the digital processing applied by the PML may include conditionally combining two samples of the digital pixel row data, for example, if the current sample of the digital pixel row data supplied by the A / D conversion unit satisfies a predetermined or programmable condition, for example, if it exceeds a certain threshold. In such cases, the current sample of the digital pixel row data supplied by the A / D conversion unit and previously acquired samples of the digital pixel row data buffered in the pixel memory unit are loaded into the PML during the prefetch cycle, and a combination of the two samples (e.g., sample addition, and optionally, subsequent compression of the sum) is performed during the PML processing cycle. Then, during the PML write cycle, the processing result is written back into the pixel memory unit.

[0055] For example, in an operating mode that uses partial transmission of photocharges accumulated for all subframe exposures except the last subframe exposure (partial transmission mode), pixel readings from low-gain channels are combined (e.g., cumulative sum) under the condition that the added pixel reading is not related to the last subframe exposure. The intermediate partial sum is stored in the pixel memory unit. Low-gain pixel readings from the last subframe exposure are added to the partial sum only if the corresponding high-gain pixel reading exceeds a threshold. The updated partial sum then becomes the final sum and is used as the output of the image sensor device. If the corresponding high-gain pixel reading does not exceed a threshold, only the high-gain pixel reading is used as the output. Alternatively, if compression is possible (e.g., using compression to operate the image sensor device in partial transmission mode), a compressed output is obtained as a combination of the high-gain pixel reading and the preceding partial sum of all low-gain pixel readings (belonging to all subframe exposures except the last subframe exposure) (e.g., applied compression algorithm). The compressed output data can be transmitted immediately from the chip and does not need to be written back to the pixel memory unit.

[0056] As a further example, when an image sensor device operates in full transmission mode, i.e., when all of the accumulated photocharge is transmitted at the end of each subframe exposure, the pixel readings in the low-gain and high-gain channels are summed separately across multiple subframe exposures (e.g., by updating the independent partial sums of the low-gain and high-gain channels, respectively). If compression is possible in this operating mode (e.g., full transmission mode with compression), the partial sums of the low-gain and high-gain channels can be input to a compression algorithm at the end of each subframe exposure, and only the compressed partial sums need to be written back to the pixel memory unit. This has the advantage of reducing storage requirements, but requires additional calculations for decompression during readback.

[0057] Figure 4 further illustrates that pipelined operation is applied to the sequence of pixel rows within a single pixel subarray, with different pipelines operating in parallel for different pixel subarrays. In other words, a separate pipeline is implemented for each pixel subarray and its corresponding readout block, resulting in K different ways of acquiring and processing pixel row data for a total of K independent pixel subarray / readout block combinations. To stream the processed frame data from the image sensor chip, access to the pixel memory is time-interleaved to prevent duplication of pixel row data belonging to different pixel subarrays. Because (partial) exposure and (partial) readout of pixel row data occur sequentially, the image sensor's electronic rolling shutter works well in conjunction with the fully pipelined architecture.

[0058] For clarity, the flowchart in Figure 4 includes only two pixel subarrays, each having three pixel rows. In a typical embodiment of the present invention, there may be three or more readout blocks and pixel subarrays, for example, 2 to 16, each pixel subarray typically containing several hundred pixel rows.

[0059] Figure 5 is a flowchart illustrating the pipeline of pixel row data in the case of multiple subframe exposures, in this example showing two subframe exposures SF1 and SF2 that have equal and temporally consecutive subframe exposure times. Temporal continuity is beneficial in reducing rolling shutter distortion in the final image. For clarity, Figure 5 shows only the data pipeline for a single pixel subarray and its corresponding readout block. As described above in this application, embodiments of the present invention provide multiple parallel operation pipelines with respect to multiple pixel subarrays and their corresponding readout blocks. Embodiments of the present invention are also not limited to two subframe exposures (e.g., three, four, or five or more subframe exposures may constitute a full frame), and the duration of the subframe exposures does not need to be equal. If the image sensor operates in a mode that does not use partially transmitted photocharge, the subframe exposures do not need to be temporally consecutive.

[0060] After the completion of the first subframe exposure SF1, in step 502-1, the pixel data for each row K, K+1, ..., K+4 is read out, converted to digital pixel row data, processed by PML in step 503-1, and written to the pixel memory unit in step 504-1. At this point, the processed pixel row data is stored in the pixel memory unit and is not yet used as output, for example, as part of the final image frame transmitted from the image sensor chip. Next, the following steps are repeated for the second subframe exposure SF2: after the completion of the subframe exposure for the second subframe exposure SF2, the pixel data for each row K, K+1, ..., K+4 is read out, converted to digital pixel row data in step 502-2, processed by PML in step 503-2, and written to the pixel memory unit in step 504-2. However, the processing by PML in step 503-2 here includes conditionally using a sample of the previously acquired digital pixel row data as an additional input operand. During the time elapsed between the end of step 504-1 and the start of step 503-2, previously acquired samples are buffered in the pixel memory unit. In step 503-2, after the PML processing of two samples of digital pixel row data associated with subframe exposures SF1 and SF2 is completed, the processing result (e.g., combination of the two samples, e.g., sum or difference) is written back to the pixel memory unit in step 504-2, and then read out from the pixel memory unit in step 505 to transmit the processed pixel data row as part of the final full image frame from the image sensor chip. From Figure 5 (e.g., dotted vertical lines for guidance), it can be seen that multiple pixel rows are processed in parallel, but the different pipeline stages are time-balanced so that each pipeline stage operates only on the pixel data of a single pixel row. In particular, the data path for the pixel row data is organized so that two rows are not addressed simultaneously for reading and transforming the pixel row data.

[0061] In embodiments of the present invention, the overall exposure period of a full image frame of an image sensor, i.e., the sum of all subframe exposure periods, is controllable via a reset control signal, for example, by controlling the time when the photoelectric elements of a pixel row are reset and subsequently exposed to incident light relative to the full frame period. The ratio of the first subframe exposure period to the second subframe exposure period is controllable via a readout control signal for the first subframe exposure. More specifically, the second subframe exposure period begins immediately as soon as the first subframe exposure period ends, a pixel row is selected for readout, and a transmission pulse is applied to the transmission gate of the pixels in that row, causing partial transmission of the photocharge within each pixel that has already been generated. In contrast, the second subframe exposure period ends immediately as soon as a pixel row is selected a second time within the same frame interval for readout, and a larger transmission pulse is applied to the transmission gate of the pixels in that row compared to the transmission pulse for the first subframe exposure, thereby initiating the complete transmission of all remaining photocharge within each pixel. In embodiments of the present invention, when three or more subframe exposures are performed, a row selection signal and a transmission pulse applied at the end of each subframe exposure except the last subframe exposure determine the duration of that subframe exposure. Furthermore, the amplitude of the transmission pulse applied to each subframe exposure except the last subframe exposure is adapted to induce only partial transmission of photocharge present in the photoelectric element of the pixel, while for the last subframe exposure, the amplitude is increased to trigger the complete transmission of the remaining photocharge. Preferably, the amplitude of the transmission pulse is constant for each subframe exposure except the last subframe exposure.Furthermore, the embodiments are not limited to reading out the buffered photocharge signal of a pixel, referred to as the pixel signal level, i.e., the voltage signal generated by a buffered charge-voltage converter in response to the transmitted photocharge present on one or both charge storage elements, but preferably include further reading out of the buffered reset signal of a pixel at both high and low conversion gains, referred to as the high-gain reset level and low-gain reset level of a pixel, i.e., the voltage signal generated by a buffered charge-voltage converter in response to the residual charge still present on the first charge storage element, or the first and second charge storage elements, after the pixel has been reset. This has the advantage that correlated double sampling (CDS) can be performed by the image sensor readout block.

[0062] Figure 6 is a timing diagram illustrating in more detail the timing and time resources required at each pipeline stage. For better understanding of this figure, this exemplary timing diagram considers only 16 pixel rows (rows 0 to F) per pixel subarray. Embodiments of the present invention may include many more pixel rows per pixel subarray, e.g., several hundred pixel rows, or even more than 1,000 pixel rows.

[0063] According to the timing diagram in Figure 6, each full frame period (e.g., frame 0, frame 1, etc.) is divided into multiple unit time slots (e.g., time slots labeled "0" or "1" in the first line of the diagram). Consecutive unit time slots are assigned to either even positions labeled "0" or odd positions labeled "1". The even and odd time slot positions are associated with the first and second rolling shutter operations, respectively. Importantly, the control sequences for the first and second rolling shutters, namely reset and readout selection, are time-interleaved, and the row control signals for the operation of the first and second rolling shutters are supplied only during the even and odd time slots, respectively. The unit time slot indicating the start of each subframe exposure for a particular pixel row of the pixel subarray is labeled with the letter "S", and the unit time slot indicating the end of that subframe exposure for that row is labeled with the letter "E". The start of the first subframe exposure may correspond to the falling edge of a reset control signal applied row by row, which passes through each row of pixels constituting the subarray and resets the photoelectric element of the pixels in that row to a predetermined voltage level. In contrast, a second subframe exposure or subsequent subframe exposures, if temporally continuous with the preceding subframe exposure, do not require sending an extra reset signal to the photoelectric element of the pixels for initiation, but begin seamlessly after the preceding subframe exposure has finished with the partial transmission of the photocharge generated by the photoelectric element. This does not eliminate the transmission of reset signals only to the first and second charge storage elements of each pixel, in which case the previously transmitted photocharge is removed, thus creating a margin for another subframe exposure readout. Unlike the full or complete transmission of photocharge generated during a conventional exposure interval, partial transmission scoops up only a portion of the photocharge present in the potential well associated with the photoelectric element that exceeds a programmable threshold potential (e.g., threshold voltage level). The programmable threshold is determined by the magnitude of the transmission pulse supplied to the transmission gate of the pixel.As can be seen in the timing diagram, the first rolling shutter sequence starts from row 0 and progresses incrementally up to row F, selecting the next row for each second unit time slot. As a result, the first rolling shutter control sequence and its associated first subframe exposure are always timed to align with even unit time slots (i.e., group "0"). Similarly, the second rolling shutter control sequence and its associated second subframe exposure always occupy odd unit time slots (i.e., group "1"), thereby avoiding interference between two simultaneously operating electronic rolling shutters. Here, interference between two or more rolling shutters operating in parallel for a pixel row of a pixel subarray is understood as attempting to simultaneously select and read pixel row data (signal level or reset level) from two different rows of the same subarray.

[0064] In this timing diagram, two subframe exposures SF0 and SF1 are selected for each frame period, but more subframe exposures may be included within a full frame period. For example, four subframe exposures may constitute the total exposure time within one full frame acquired by the image sensor, in which case the unit time slots are assigned position numbers "0" to "3" (e.g., position modulus 4), and each position group "0" to "3" is associated with a row control signal that corresponds to only one of the four time-interleaved rolling shutter sequences.

[0065] The duration of a unit time slot is typically determined by the pipeline stage with the highest latency. In this embodiment, for example, a unit time slot corresponds to the duration of the settling time of the signal present on the bit line, the sample-and-hold time, and the time required for the A / D conversion unit to perform analog-to-digital conversion for fast sequential measurement of pixel signal levels at both high and low conversion gains. When CDS is applied, the signal present on the bit line includes both the pixel reset level and the pixel signal level, meaning that the unit time slot is the sum of the settling time, the sample-and-hold time, and the time for A / D conversion for fast sequential CDS measurement in high-gain and low-gain readout channels. Nevertheless, to implement the image sensor at even higher speeds, more pipelines may be added to the readout path, and the unit time slot may be subdivided or redefined to realistically reflect the presence of the added pipeline stages. Fast sequential measurement of high-gain and low-gain pixel signal levels may be performed by reducing the resolution of the ADC components in the A / D conversion unit, for example, two 12-bit conversions may be performed in the same time as a single 14-bit conversion. Alternatively, the A / D conversion unit may include two parallel-operating ADCs instead of assigning a single ADC to a pixel row, as in Figure 6. Thus, within each unit time slot, the A / D conversion unit can convert the pixel data of just one row of pixels in the pixel subarray into the digital domain. The row of pixels undergoing A / D conversion during a particular unit time slot is indicated by the row number in the pixel subarray (e.g., the numbers "0" through "F" in the ADC line in Figure 6). The converted pixel data is available for just one time slot after the corresponding subframe exposure period has ended (e.g., indicated by the letter "E"). There may be moments when the A / D conversion unit is idle and does not perform pixel row data conversion (e.g., the blank unit time slot in the ADC line in Figure 6).

[0066] Thus, each subframe exposure period SF0, SF1, and full frame period can be expressed in terms of the number of equivalent unit time slots. For example, in the embodiment shown in Figure 6, the first subframe exposure SF0 lasts for only 15 time slots, the second subframe exposure lasts for only 13 time slots, and each full frame (frame 0, frame 1) consists of 32 time slots. The shorter the unit time slots are relative to the full frame period, and the shallower the pipeline depth, the more subframe exposures can be included within a single full frame period, which is considered fixed over time. For example, the shortest possible single frame period is fixed by the maximum achievable external I / O rate at which preprocessed HDR frame data can be transmitted from the image sensor chip to an external device, such as an external storage device like an external RAM or hard drive. In embodiments of the present invention, a typical value for the maximum achievable external I / O rate may be 120 fps in double subframe exposure operation mode, but it also depends on other factors, such as the number of pixel rows and readout blocks, and the ADC bit resolution. In the example in Figure 6, this means that a pixel row corresponding to frame data is transmitted every 520.8 μs, corresponding to two consecutive time slots of readout rows (the last row in Figure 6) that contain the same numerical value and therefore refer to data acquired from the same pixel row, while the duration of a single unit time slot is only 260.4 μs. These values ​​are given for illustrative purposes and do not necessarily reflect the frame data rate and unit time slot duration of an actually manufactured image sensor, which may contain more pixel rows than the example in Figure 6.

[0067] For example, an image sensor with megapixel resolution may have a unit time slot duration of approximately 15 μs, which enables a maximum internal subframe rate of 240 fps with a 14-bit ADC resolution (ADC resolution can be traded for speed or a larger number of subframes). Thus, this exemplary image sensor supports outputting pre-processed HDR frame data to an external device (i.e., off-chip) with an I / O rate of 120 fps in double subframe exposure mode and 60 fps in quad subframe exposure mode, each with a 14-bit ADC resolution.

[0068] As shown in Figure 6, the final frame data read from the off-chip pixel memory may be transmitted in a compressed format. Here, compression relates to the fact that the combined pixel signal levels for high-conversion-gain and low-conversion-gain settings of pixels, which result in improved dynamic range of the image sensor, may exceed the available bit depth of the pixel memory. In such cases, a scaling operation is performed before writing the combination of high-gain and low-gain pixel signal levels back to the pixel memory, thereby ensuring that the scaled signal combination fits back into the available bit depth of the pixel memory. For example, a combination of a 12-bit wide high-gain pixel signal level and a 12-bit wide low-gain pixel signal level may result in a 13-bit wide combined HDR value, which will not fit into the 12-bit wide pixel memory without causing an overflow. In these cases, the combined HDR value is rescaled (scaling factor < 1) so that it fits back into the 12-bit wide pixel memory. Instead of a simple scaling operation, a more sophisticated compression algorithm may be used, which combines pixel readings from high-gain and low-gain channels in different ways across different signal subranges. For example, it may use varying noise characteristics across different signal subranges to change the accuracy of high-gain and / or low-gain pixel readings. The compressed output data may be transmitted out of the chip at twice the rate compared to the separately transmitted high-gain and low-gain data (e.g., one time slot for every two time slots per pixel row in the last line of Figure 6).

[0069] As can be further seen from the timing diagram in Figure 6, the processed high-gain and low-gain pixel row data are written to different addresses in the pixel memory, thereby enabling separate readout of pixel data acquired for either high or low conversion gain in image sensor operating modes that do not use the pixel's dual-gain function, such as simple low-gain or high-gain operation without extended DR. Such operating modes may also include different HDR modes of the image sensor that do not rely on the partial transmission of photocharge generated within a single full-frame period. For example, multiple subframe exposures or multiple frame exposures with different exposure times and / or conversion gains can be merged offline or online into a single HDR image frame (single-frame or multi-frame exposure bracketing), which can be done on the image sensor chip or on external data processing means. The case of combining multiple subframe exposures or multiple full-frame exposures with at least two exposure time settings into an HDR image frame is also known as multiple exposure operation and can be performed on the image sensor chip according to the present invention additionally or alternatively to pixel-based dual-gain conversion in order to obtain an HDR image frame. Different exposure time settings yield different but perfectly deterministic slopes and knee points between slopes in the linearity plot of the image sensor (digital number as a function of light illumination). Therefore, in image frames with a large intra-scene dynamic range, pixel output signals obtained using unequal exposure time settings can be easily relinearized internally or externally to obtain a linear HDR signal without calibration. The ratio of subframe exposure periods in pairs of consecutive subframes, along with the current conversion gain, controls the change in response slope.

[0070] Figure 11 shows different response slopes and knee points for a single conversion gain (e.g., low gain only) and a total of four subframe exposures per full image frame. In this example, the exposure duration of subsequent subframes is increased, for example, Texp1=128Trow, Texp2=32Trow, Texp3=8Trow, and Texp4=2Trow for a full exposure time of 10.2 ms, where the row time (Trow) is approximately 60 μs. The corresponding sensor output after linearization is shown in Figure 12. The position of the knee point on the vertical axis (raw response) can be fully programmable, for example, by introducing a programmable clipping block into the signal path to clip the pixel signal in the digital domain.

[0071] As a specific example of combining multi-frame bracketing and multiple exposure, an operating mode can be described in which a first full frame with increased DR is acquired via two subframe exposures with a low conversion gain and a large exposure time ratio, and a second full frame with increased DR is acquired via two subframe exposures with a high conversion gain and a large exposure time ratio. Finally, the first and second full frames can be combined offline into a final image frame with an even larger DR.

[0072] In contrast to separate storage locations for high-conversion-gain and low-conversion-gain pixel data, processed digital pixel data acquired for two different pixel rows within the same subarray (e.g., rows 0 and 8, rows 1 and 9, etc.) is mapped to the same address in pixel memory to reduce memory capacity requirements and associated chip area. Furthermore, in embodiments of the present invention, as will be described in more detail below, where multiple subframe exposures are combined with partial photocharge readout and different conversion-gain settings to generate HDR image frames, only one storage location (e.g., address or line) in pixel memory is required for high-conversion-gain and low-conversion-gain pixel row data. Thus, the allocation of two pixel memory lines per pixel row in the subarray of Figure 6 is considered optional, and a more area and energy-efficient solution may allocate only a single pixel memory line per pixel row and subarray. In the latter case, the overall storage capacity of the pixel memory can be considered smaller than the storage size associated with a full frame of image data.

[0073] As shown in Figure 6, immediately after the A / D conversion is completed for the pixel row selected for reading, that is, after exactly one unit time slot has elapsed, the processed pixel row data for high and low conversion gains is stored in the pixel memory, while the A / D conversion unit proceeds to the next selected pixel row. Here, it is assumed that since these two steps have individual latencies shorter than one unit time slot, the processing of the converted digital pixel row data and the writing operation of the processed pixel row data can be performed within one unit time slot.

[0074] More specifically, immediately after the A / D conversion of the pixel row is completed at the end of the first subframe exposure SF0 within each frame, the pixel row data acquired for the low-conversion gain channel of the pixel after the first partial transmission (e.g., in the case of CDS, the pixel signal level minus the pixel reset level) may be unconditionally written to the pixel memory. If three or more subframe exposures are programmed, during the intermediate subframe exposures occurring between the first and last subframe exposures, the PML may process each subsequent low-gain pixel row data acquired for each further partial transmission (e.g., in the case of CDS, the pixel signal level minus the pixel reset level), and the result of this processing is written back to the pixel memory. In addition to managing the data flow from pixel memory to pixel memory, processing by the PML may include performing basic clipping and / or scaling operations on the digitized pixel data (e.g., after CDS).

[0075] In a preferred embodiment, processing by PML also includes conditionally combining, for example, conditionally adding, the processed or unprocessed (raw) pixel row data of the current subframe to buffered pixel row data in pixel memory. In such a case, the buffered previous data is fetched from pixel memory by PML, and the fetch may be performed while the pixel row data of the current subframe is still undergoing A / D conversion. The conditions that must be satisfied for PML to perform a combination of pixel row data relating to different subframes may include the step of comparing the pixel data to a first programmable threshold level TLG. Depending on the result of this comparison, the pixel data of the current subframe is combined with the pixel data of the previous subframe buffered in pixel memory, for example, if the pixel data is lower than the threshold TLG, or discarded, for example, if the pixel data is greater than the threshold TLG. For example, pixel data discarding may occur if the image sensor device operates in a short intermediate subframe exposure mode in which only the pixel data of the shortest subframe exposure is retained. To select only the pixel data of the shortest intermediate subframe for output, events that discard pixel data during each full-frame exposure period may be monitored. If the pixel memory does not yet contain valid data for the current image frame (i.e., including multiple subframes), the combination step can be replaced by a direct write to the pixel memory. Alternatively, the pixel memory can be initialized to a default value, such as zero, at the start of each new image frame.

[0076] Finally, processed or raw pixel row data for high-gain and low-gain channels, e.g., the pixel CDS signals for high-gain and low-gain channels, are either sent directly to the sensor output interface at the end of the last subframe exposure (e.g., after full photocharge transmission in subframe SF1 in Figure 6), depending on the selected output format, or conditionally processed by PML. Conditional processing by PML for the last subframe may include the following steps: the high-gain CDS signal of each pixel is compared to a second programmable threshold THG, and if it is lower than the threshold THG, the high-gain CDS signal of the pixels in one row is written to pixel memory. In this case, the pixel data stored before that row is overwritten. Alternatively, the pixel data stored before that row may be fetched by PML and combined with the currently processed high-gain pixel data, for example, by compressing the high-gain and low-gain pixel row data into a single row of data, and the result of this PML operation is stored in pixel memory as the final result of the pixel data for that row in the current frame. Instead, if the threshold THG is exceeded, the high-gain data of a pixel row may be discarded, and the low-gain data of that pixel row may be used by PML instead, for example, by fetching the low-gain data of the pixel row from pixel memory via PML and then combining it with previously stored pixel row data (e.g., adding, and optionally compressing).

[0077] The second full frame in the timing diagram of Figure 6 also includes two subframe exposures, just like the first full frame. In this embodiment, the time interval between rolling shutter exposures, from the end of the last subframe exposure SF1 of the first full frame to the start of the first subframe exposure SF0 of the second full frame, is selected to be as short as possible compared to the full frame period, resulting in the longest possible full frame exposure. In practice, the two subframe exposures span almost the entire full frame period (e.g., 87.5%) and cannot be further extended without avoiding interframe interference of the electronic rolling shutter, for example, without avoiding the second rolling shutter of the first frame and the first rolling shutter of the second frame attempting to read different rows of the subarray simultaneously, i.e., without avoiding the two unit time slots labeled "E" being perfectly time-coordinated. The points tA and tB, when the delayed second electronic rolling shutter begins to traverse the rows of the subarray before the preceding first electronic shutter finishes, are located near the end and midpoint of the frame, respectively. This is a further indicator of the long overall frame exposure time for the two subframe exposures, and the nearly balanced exposure time.

[0078] However, embodiments of the present invention are not limited to long frame exposures and can equally be adjusted to perform shorter frame exposures, for example, to better suit the image sensor for distortion-free capture of fast-moving objects. A detailed timing diagram of the shortest possible overall frame exposure, consisting of two consecutive subframe exposures SF0 and SF1 (each with three unit time slots) having the same subframe exposure period, is shown in Figure 7 for a pixel subarray with the same number of rows, unit time slot duration, and full frame period as in Figure 6.

[0079] Furthermore, in embodiments of the present invention, full frames may be acquired continuously, or a predetermined number of consecutive frames may be acquired, as shown in Figures 6 and 7. In embodiments of the present invention, generally, each new full frame begins with a reset operation on the photoelectric elements of a pixel row when the pixel row is first selected in the new frame. In contrast to Figure 6, the points tA and tB at which the delayed second electron rolling shutter begins to move through the subarray row before the preceding first electron shutter has finished are both located near the end of the frame and separated by only a very short subframe exposure period.

[0080] Figure 13 shows the timing diagram of the quad subframe exposure operation (partial transmission mode or full transmission mode) of the image sensor device according to the present invention. Subframe exposures SF0 to SF3 are temporally continuous, and the subframe exposure periods are equal. In partial transmission mode, only the fourth subframe exposure SF3 is read out in a high-gain configuration for the pixels. The readout pipeline is similar to that described above, for example, in Figures 6 and 7.

[0081] In the above embodiments, the first and second thresholds (TLG and THG) are preferably programmable, for example, by software or directly by the user, as a function of the total exposure time and the number of subframes of the acquired image frame, but in other embodiments they may be fixed values. The first and second thresholds (TLG and THG) are generally constant across the pixel array, but they do not need to be the same value; for example, TLG and THG may have different magnitudes and are typically independent of the potential barrier set by the partial charge transmission pulses and full charge transmission pulses applied to the pixel transmission gate.

[0082] In the above embodiment, the readout, A / D conversion, digital data processing, and pixel memory write stages are performed for the entire row of image sensor pixels. However, depending on the trade-offs between speed, area, and power consumption of the image sensor according to the present invention, blocks of pixel-related data (e.g., reset level and signal level) do not necessarily have to correspond to pixel rows, but can be performed for pixel groups (e.g., row segments or groups of columns within a row, e.g., even / odd channels), or in some cases, on a pixel-by-pixel basis.

[0083] In embodiments of the present invention, a state machine may be used to trigger the processing of digitized pixel data in a parallel PML, not just the A / D conversion of pixel data, on a per-unit time slot basis. When triggered by the state machine, the A / D conversion unit may use its own clock, which is typically the fastest clock among the components of the control and readout circuitry. The components of the control and readout circuitry may use other dedicated clocks, each with a specific clock speed (e.g., a sequencer clock, a clock for coordinating memory access, etc.), which generally operate at a lower speed than the A / D conversion unit's clock, for example, 5 to 10 times slower.

[0084] In addition to the multi-subframe and partial readout operations described above, the image sensor can also be operated in LOFIC-like mode (lateral overflow integrated capacitor). In this LOFIC-like mode, the photocharge of the oversaturated photodiode flows into a second charge storage element, or a combination of the first and second charge storage elements. The bias voltages of the transmission gate and gain switch are set accordingly. In the LOFIC-like mode, which is particularly suited to high illumination conditions, the low-gain channel of each pixel is read out twice, i.e., the overflow charge on the sense node is read out first (defining the overflow level), followed by a sense node reset operation and reset readout (e.g., voltage reset for the connected first and second charge storage elements), and a further readout of the sense node for the photocharge transmitted completely or partially from the photosensitive element (defining the photodiode / signal level in the low-gain channel). Preferably, correlated double sampling (CDS) is performed on the photodiode level in the low-gain channel using the reset readout for pixel noise correction. Digital double sampling may be performed on the overflow level using the reset readout again for correction. Alternatively, a reset operation and corresponding reset read may be performed at the start of each new image frame and used to implement a true CDS for the overflow level.

[0085] In LOFIC-like mode, before writing pixel data to the pixel memory unit at the end of each subframe exposure except the final subframe exposure, the low-gain pixel data may first be compared to a threshold for the low-gain channel (TLG). If the low-gain pixel data is lower than the threshold TLG, the overflow signal level is ignored, for example, by setting it to zero to avoid noise or dark current contributions. If the low-gain pixel data is greater than the threshold TLG, the overflow signal and the pixel signal for the low-gain channel are directly added together, and the partial sum of each subframe exposure is stored in the pixel memory unit. Since both the overflow signal and the low-gain pixel signal are obtained with respect to the same low-conversion gain setting of the pixel, they can be directly added together. Only in the case of the final subframe exposure, the pixels are read out in the high-gain channel during the readout operation of the overflow signal and the low-gain pixel signal. High-gain pixel data is retained for sensor output operation only if the high-gain pixel data is below the threshold for the high-gain channel (THG), and is transmitted separately, for example, outside the chip. If the threshold is exceeded, high-gain pixel data is ignored or transmitted in a compressed format along with low-gain pixel data for the purpose of image sensor data output. In the case of a non-final subframe exposure, the low-gain pixel data is first compared with the threshold of the low-gain channel (TLG), which is also used for the final subframe exposure, to determine whether an overflow signal should be added to the low-gain pixel data before adding the result to the partial sum read back from the pixel memory unit.

[0086] In a second aspect, the present invention relates to a method for operating an image sensor according to the first aspect, which provides an image frame having an increased dynamic range. This method utilizes the fact that all but the last subframe exposure of a plurality of subframe exposures having exposure times shorter than the full frame period are partially readable, and by combining all subframe exposures, the saturation capacitance of the photoelectric element is effectively increased, limiting the increase in readout noise associated with the plurality of exposures. Furthermore, the image sensor does not saturate even under high illumination conditions. In addition, the pixel conversion gain of the image sensor is switched between a high conversion gain and a low conversion gain in order to obtain an optimal signal-to-noise ratio for each pixel under low or high illumination exposure conditions.

[0087] Conventional image sensors that use multiple exposures with full transmission of photocharge generated within the pixel's photoelectric element require that the floating diffusion node has the same associated saturation capacitance (FWC) as the photoelectric element. This limits the FWC of the photoelectric element, such as a PPD, when pixels with good charge-voltage conversion gain are required. In embodiments of the present invention, this limitation is overcome by causing only partial transmission of the generated photocharge from the photoelectric element to the charge storage element, allowing the FWC of the photoelectric element to be greater than the FWC associated with the first charge storage element, and in some cases greater than the combined FWC associated with the first and second charge storage elements. In addition, embodiments of the present invention limit noise related to multiple readouts of each pixel by transmitting only a portion of the generated photocharge from the photoelectric element to the charge storage element at the end of each subframe exposure except for the last subframe exposure, and initiating full transmission of the remaining photocharge only in the case of the last subframe exposure. In conventional methods based on the summation of N subframe exposure readings (full transmission of the accumulated photocharge), the resulting readout noise increases by a factor of sqrt(N). However, in embodiments of the present invention, the readout noise (e.g., dark noise) occurs only once at the end of the last subframe exposure when a high conversion gain is applied to read out the pixel signal level under low light conditions. Under such conditions, the intermediately generated photocharge is not affected by the partial transmission operation. Under higher illumination conditions, some of the intermediately generated photocharge is transmitted, converted at the low conversion gain setting of the pixel, and then sequential readouts at the low conversion gain are added to obtain the final result. In such cases, the high-gain path is not used, and therefore, the benefits from a larger FWC associated with the low conversion gain setting of the pixel are obtained.

[0088] Figures 8-10 illustrate the method of the present invention under three different illumination conditions: low light, high light, and supersaturation. The image sensor was operated to generate HDR image frames from multiple subframe exposures for each of the three illumination conditions. For illustrative purposes, only two subframe exposures with approximately equal exposure periods are assumed, but embodiments of the present invention can use three or more subframe exposures. Similarly, it is not essential to the present invention that the end of the first subframe exposure is programmed (via a rolling shutter sequence) to occur at or near the midpoint of the combined full-frame exposure time (i.e., the sum of all subframe exposure times). For example, even if the subframe exposures are not temporally continuous, the ratio of the first subframe exposure time to the second subframe exposure time may be 1:9. Also, depending on the exposure settings of the image sensor, the (combined) full-frame exposure time "Texp" can be made smaller than the full-frame period. As shown in the preceding drawings, a delayed reset signal can be used to extinguish the photocharge of the photoelectric element at the start of the first subframe exposure, which can be delayed relative to the start of the corresponding full-frame time interval.

[0089] Refer to the low-luminosity illumination conditions in Figure 8. After the initial reset of the pixel photoelectric element (first solid vertical line at 0% of the frame exposure time Texp), the photocharge does not accumulate fast enough to be affected by the partial transmission pulse applied to the pixel transmission gate during the middle of the exposure (second solid vertical line at approximately 50% of Texp). As a result, when the pixel switches to a low-gain configuration during readout, none of the generated photocharge is transmitted to the first and connected second charge storage elements (e.g., floating diffusion nodes and additional gain capacitance). The high-gain channel is not used for the first subframe. Therefore, after correlated double sampling (i.e., subtracting the pixel reset level from the pixel signal level before A / D conversion), the converted pixel data sent at the output of the A / D conversion unit is zero. This value is unconditionally written to the pixel memory without further processing by PML.

[0090] At the end of the second subframe exposure, the reset levels for the pixel's low-gain and high-gain settings are read out. Subsequently, after the complete transfer of photocharge from the photoelectric element to the first charge storage element of the charge-voltage converter, the signal level is read out in the high-gain configuration (the third solid vertical line at 100% of Texp). Then, the pixel's conversion gain is switched to the low-gain channel, and the signal levels of the same photocharge present on the first and second charge storage elements are determined. Correlated double sampling is applied to both the high-gain and low-gain channels to cancel the kTC noise in each gain channel, and the A / D conversion unit converts the reset noise-corrected signal levels (pixel data) for high-gain and low-gain into the digital domain. All of these readouts are performed in the same unit time slot and are executed in the order of reset level for low-gain, reset level for high-gain, signal level for high-gain, and signal level for low-gain.

[0091] Next, PML compares the pixel data for the high-gain setting with the high-gain threshold THG. If the pixel data is lower than the THG threshold, corresponding to the low-light illumination conditions depicted in Figure 8, the high-gain pixel data is stored in the pixel memory, replacing the currently stored data. However, if the high-gain pixel data exceeds the high-gain threshold THG, the high-gain pixel data is discarded, and only the low-gain data is considered. Therefore, under the low-light illumination conditions shown in Figure 8, only the high-gain pixel data is retained and stored as the final data output by the image sensor chip.

[0092] In an alternative embodiment of the present invention, instead of discarding high-gain pixel data acquired for the last subframe when a second threshold THG is exceeded, it is possible to combine high-gain pixel data and low-gain data into a single output word. For example, a 13-bit wide data word for low gain may be combined with a 14-bit wide data word for high gain to obtain a single compressed output word of, for example, 16 bits wide. A level-dependent compression algorithm may internally amplify the low-gain signal to approximate the high-conversion gain. At very low illuminance, only high-gain data is included in the compressed output word, while at very high illuminance, only low-gain data is included. In the intermediate region, for example, the transition between very low and very high illuminance, the number of bits of high data retained in the compressed output data word is progressively reduced, while the number of effective bits of low-gain data in the output word is progressively increased. Furthermore, the inevitable inter-pixel and inter-sensor variations in the ratio between high-conversion gain and low-conversion gain, which result in differences between high-gain and low-gain samples for each pixel, can be taken into account by replacing the shot noise-dominant bits in the high-gain data word with this difference information.

[0093] The generation of compressed output words has the advantage of reducing the total I / O bandwidth (i.e., minimizing the amount of data sent off-chip) while the respective low-gain and high-gain image data remain available for a wide range of illumination levels. It is also advantageous for smooth signal transitions from high-gain to low-gain image data in image frames reconstructed externally (e.g., off-chip) (e.g., by applying the corresponding decompression operation to the compressed output words).

[0094] The threshold level THG can be relatively low compared to the saturation capacitance associated with the first charge storage element. For the noise in the pixel signal level to be dominated by the shot noise limit, only a few tens or hundreds of electrons may remain in the photoelectric element after partial transmission. This justifies readout in a low-gain configuration of the pixel for stronger signal levels, but encourages readout in a high-gain configuration of the pixel for weaker signal levels at the end of the final subframe exposure. The threshold THG can be externally provided by the user, programmed into the image sensor device by the user, or set to a default value, which can determine when to use high-gain or low-gain pixel data as output. Generally, this reflects a measured or expected amount of shot noise, and in some cases, the noise contribution forms a variation in the transmission pulse amplitude, and applying a higher readout gain above this point will not result in a significantly more favorable signal-to-noise ratio compared to a lower readout gain. Finding a good value for the amplitude of the partial transmission pulse is typically a result of balancing two conflicting requirements. On the one hand, it is preferable that the accumulated photocharge remaining in the photosensitive element after all partial transmissions is large enough to be dominated by intrinsic shot noise when the readout channel is switched to low gain; on the other hand, too much accumulated photocharge should not remain in the photosensitive element after each non-final subframe exposure in order to avoid excessively restricting the pixel's FWC during subsequent subframe exposures.

[0095] Continuing with the low-light illumination conditions depicted in Figure 8, an alternative method for providing pixel data at the image sensor output may include transmitting one full image frame of pixel data belonging to a high-gain readout channel, and independently transmitting another full image frame of pixel data belonging to a low-gain readout channel. The two image frames (high-gain and low-gain) can be combined off-chip, for example, within an image blending unit (HDR synthesizer) of a camera equipped with an image sensor, or in another device. This provides the user with greater flexibility when combining low-gain and high-gain images into a single HDR image. In this alternative output format, the high-gain pixel data is transmitted directly to the sensor's I / O circuitry and is therefore not stored in pixel memory. The low-gain pixel data (previous subframe exposures) currently stored in pixel memory is not replaced and is available for further processing, for example, for combination with the low-gain pixel data of the current (last) subframe exposure.

[0096] Looking at the bright light illumination conditions considered in Figure 9, we notice that the photocharge generated by the end of the intermediate exposure, i.e., the first subframe exposure, is affected by the partial transmission pulse delivered to the pixel transmission gate. As a result, when the pixel is configured to have a low conversion gain, after reading the reset levels for the first and second charge storage elements, some of the photocharge generated so far is transmitted to the charge storage elements, and the resulting signal voltage is read out. Here again, correlated double sampling is applied, and the reset noise-corrected signal level is converted by the A / D converter. The digital value of the acquired low-gain pixel data is written to the pixel memory.

[0097] At the end of the second subframe exposure, the remaining photocharge in the photoelectric element is fully transmitted, and the pixels are reset and the signal levels are read out for the low-light conditions shown in Figure 8. Furthermore, correlated double sampling is applied to obtain high-gain and low-gain pixel data after A / D conversion. In the case of bright illumination conditions as depicted in Figure 9, when the high-gain pixel data by PML is compared with the high-gain threshold THG, the THG threshold is exceeded, and the high-gain pixel data is discarded. Therefore, PML considers only the low-gain pixel data. More specifically, PML fetches the low-gain pixel data for the preceding first subframe from the pixel memory, adds the fetched previous low-gain pixel data and the currently supplied low-gain pixel data, and writes the result of the addition back to the pixel memory.

[0098] In embodiments of the present invention, the threshold for the high-gain channel THG and the threshold for the low-gain channel TLG are preferably programmable values ​​that can be modified by the user or are determined as a function of the number of subframes in each full frame, the pulse amplitude of the partial charge transmission pulse applied to the transmission gate, and the ratio of the exposure times of the individual subframes.

[0099] In exemplary embodiments of the present invention, the FWC of a pixel associated with a high-gain channel may be about 10 ke-, and the FWC associated with a low-gain channel may be about 40 ke-. A partial transmission pulse (TX) may be selected to leave a photocharge of 500 e- to 1000 e- on each of the photodiodes of the pixel.

[0100] Considering the supersaturated illumination conditions, Figure 10 shows that the generated photocharge exceeds the well capacitance (upper horizontal line) associated with the photoelectric element, e.g., PPD, which causes the excess photocharge to flow from the photoelectric element to the charge associated with at least the first charge storage element of the pixel's buffered charge-voltage converter, e.g., the connected first and second charge storage elements. The presence of overflow charge on the first charge storage element, or on the interconnected first and second charge storage elements, can be detected for each pixel before a reset pulse is transmitted at the end of each subframe exposure. Thus, a supersaturated regime in one or more pixels of the image sensor can be detected. Under the illumination conditions shown in Figure 10, since the complete photocharge transfer from the saturated photoelectric element to the first charge storage element always exceeds the THG value, only pixel row data (reset noise corrected) relating to the pixel's low conversion gain setting is used for all subframe exposures. As a result, for all subframe exposures contained within the full image frame, only the pixel data related to the low-conversion-gain configuration of the pixels is added by PML, and the resulting partial or final sum is optionally compressed before the intermediate or final data values ​​are written to the pixel memory.

[0101] Optionally, in the method described above, the overflow level of the charge storage element receiving the photocharge is determined after each subframe exposure, before the transmission of a reset pulse marking the start of sampling of a new signal and / or reset level. Additional, unoccupied, and / or pixel memory locations marked as invalid for external readout may store the determined pixel overflow level in addition to the pixel signal level and / or pixel reset level. The overflow level of each pixel thus acquired and stored may also be processed by the PML. For example, the PML may fetch the cumulative overflow level associated with the preceding subframe exposure from the pixel memory and add it to the overflow determined with respect to the current subframe exposure, e.g., a further intermediate subframe exposure or the final subframe exposure in the image frame. The cumulative overflow level after the final subframe exposure may then be added to the final low-gain pixel data (e.g., the cumulative sum of low-gain pixel data across all subframe exposures in the image frame). Alternatively, if an overflow of photocharge is detected, the photoelectric element and charge storage element may simply be reset without processing and / or storing the specific overflow level. Detecting photocharge overflow events can still be useful in informing the user or internal components of the image sensor that an image frame was acquired under overexposure conditions.

[0102] While embodiments of the present invention prefer correlated double sampling to reduce reset noise, the method of the present invention does not depend on correlated double sampling, and the signal level can be sampled directly without sampling the preceding reset level. If the pixel overflow level is also determined as described above, digital double sampling can be performed to correct the overflow level of residual reset noise present on the charge storage element after the initial reset of a pixel in each image frame or after each subsequent reset.

[0103] Furthermore, two or more subframe exposures may occur during the full frame interval of the image sensor. For example, at least three subframe exposures may occur during the full frame interval, with the first two consecutive subframe exposures positioned in the middle of the cumulative exposure time across all subframes within a single full frame. In such embodiments, the partial exposure time allocated to the second subframe exposure may be short enough to allow the image sensor to capture a very bright signal. In the exemplary embodiments described above, if the PPD as a photoelectric element can store 40 ke- (saturation capacitance-FWC), the FWC can be effectively increased to about 80 ke- (neglecting to determine the overflow level, otherwise about 160 ke-) over the period of cumulative (i.e., total or full) exposure time of an image frame consisting of two substantially equal subframe exposures, provided that the proportion of accumulated photocharge remaining in the PPD after partial transmission is very small. Under very bright illumination conditions, even an extended FWC of about 80 ke- is insufficient to prevent the pixel output from saturating. In contrast, if the second subframe exposure among three or more subframe exposures is selected to be short relative to the full exposure time, assuming bright illumination conditions similar to the previous double exposure (i.e., exceeding the saturation level), for example, if it is about one-eighth of the full exposure time of the image frame, the pixel photocharge that can be collected during this relatively short second subframe exposure will be greater than about 80 / 8 = 10 ke- and have an associated SNR greater than 40 dB. Since the short second subframe exposure does not reach the full exposure limit (FWC), the DR of the image sensor can be further extended by selecting the pixel data derived from the short subframe exposure as the only pixel data valid for output. More specifically, the critical level of photocharge accumulated during the first subframe exposure, for example about 80*7 / 8 / 2 = 35 ke- in this example, defines a low-gain threshold level (TLG) for the digitized pixel data, above which a saturation condition is detected with respect to the first or last subframe exposure. Therefore, above the low-gain threshold level (TLG), the pixel data related to the shortest subframe exposure period (in this case, the second) is selected as the corresponding output data.In all embodiments of the present invention, the ratio of subframe exposure periods can be accurately calculated, and therefore the acquired pixel data can be linearized. Embodiments of the present invention that result in short intermediate subframe exposures benefit from even higher DR values ​​(for example, a short exposure of about one-eighth of the full exposure time adds about 18 dB of DR). Other modifications of the disclosed embodiments can be understood and realized by those skilled in the art practicing the claimed invention from the study of the drawings, disclosures, and appended claims. In the claims, the word “equipped with” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude plurals. The mere fact that certain means are enumerated in mutually different dependent claims does not indicate that combinations of these means cannot be used to their advantage. No reference numeral in the claims should be construed as limiting the scope.

Claims

1. An image sensor provided as an integrated circuit, the image sensor comprising: a first substrate; a second substrate stacked perpendicularly to the first substrate and electrically interconnected with the first substrate; and a pixel array disposed within the first substrate, wherein the pixel array comprises a plurality of pixels logically organized into pixel rows and pixel columns, the pixel array is organized into a plurality of pixel subarrays, and each pixel of the pixel array is A photoelectric element for accumulating photocharge while exposed to light during each subframe exposure of multiple subframe exposures included in the frame interval, A transmission gate for transmitting at least a portion of the accumulated photocharge present in the photoelectric element at the end of each subframe exposure to a buffered charge-voltage converter, A buffered charge-voltage converter for receiving the transmitted photocharge and converting it into a signal voltage, wherein the buffered charge-voltage converter comprises a first charge storage element operably connected via a gain switch to at least one second charge storage element for switching the conversion gain of the pixel between a high conversion gain and a low conversion gain, the at least one second charge storage element provided within the first or second substrate, and the buffered charge-voltage converter. A control circuit adapted to trigger partial or complete transmission of the integrated photocharge within the photoelectric element of each pixel row by adjusting the amplitude of the transmission pulse applied to the transmission gate, A readout circuit disposed on the second substrate, the readout circuit comprising a plurality of parallel-operable readout blocks, each readout block associated with one of the plurality of pixel subarrays, and configured to sequentially read out the pixel row data of each of the pixel subarrays at the end of each subframe exposure, The control circuit, in at least one operating mode of the image sensor, The partial transmission is triggered for all of the multiple subframe exposures except the last subframe exposure, and the complete transmission is triggered for the last subframe exposure among the multiple subframe exposures. With respect to each pixel subarray, at least two rolling shutter control sequences associated with at least two time-overlapping subframe exposures among the plurality of subframe exposures are time-interleaved. The system is configured to control the gain switch within each pixel row so that, for all of the multiple subframe exposures except the last subframe exposure, the pixel operates with the low conversion gain, and for the last subframe exposure, the pixel operates first with the high conversion gain and then with the low conversion gain. Each read block of the read circuit is located within a pipelining architecture. An A / D conversion unit for sampling and digitizing the pixel row data of the corresponding pixel subarray, A pixel memory logic that conditionally combines digitized pixel row data with previously digitized pixel row data buffered in a pixel memory unit, An image sensor comprising: a pixel memory unit for buffering digitized pixel row data output by the pixel memory logic;

2. The pixel memory logic of each readout block considers when previously digitized pixel row data relates to a preceding subframe exposure within the same frame interval, and - If the digitized pixel row data belonging to the low conversion gain in one intermediate subframe among the multiple subframes between the first subframe and the last subframe of the multiple subframes is lower than the first threshold level (TLG), or - If the digitized pixel row data belonging to the low conversion gain in the last subframe is greater than the first threshold level (TLG), while the corresponding digitized pixel row data belonging to the high conversion gain in the last subframe exceeds the second threshold level (THG), The image sensor according to claim 1, configured to combine the digitized pixel row data with the previously digitized pixel row data buffered in the pixel memory unit.

3. The image sensor according to claim 1, wherein the pixel memory unit of the read block is provided as an SRAM bank.

4. The image sensor according to claim 1, wherein the total size of the pixel memory units in the read block is such that it is only permitted to store a portion of the full frame acquired by the pixel array.

5. The pixel memory logic is further configured to process digitized pixel row data, and the processing of the digitized pixel row data by the pixel memory logic includes converting Gray code digital pixel row data to binary code digital pixel row data, and / or scaling the digital pixel row data by a scaling factor, and / or The image sensor according to claim 1, wherein combining digitized pixel row data with digitized pixel row data stored in the pixel memory unit under the conditions includes adding the currently acquired digital pixel row data to previously acquired digital pixel row data stored in the pixel memory unit, or subtracting the currently acquired digital pixel row data from the previously acquired digital pixel row data.

6. The image sensor according to claim 1, wherein the pipeline architecture of the read block is configured to input and output pixel row data at the row level, at the pixel group level, or at the pixel level.

7. The image sensor according to claim 1, wherein the control circuit is configured to select a pixel row of each pixel subarray for readout before triggering partial or complete transmission of the integrated photocharge, so that each readout block can detect and read out overflow photocharge exceeding the saturation capacitance associated with each photoelectric element.

8. The control circuit is configured to reset the charge storage element of the charge-voltage converter in the pixel row before triggering partial or complete transmission of the integrated photocharge, thereby causing each readout block to detect the reset voltage of the pixel in the pixel row and, with respect to the signal voltage of the pixel in the pixel row, The image sensor according to claim 1, wherein correlated double sampling can be applied and the signal voltage represents partially or completely transmitted photocharge.

9. The image sensor according to claim 1, wherein the pixel memory logic of each readout block is configured to combine the digital pixel row data belonging to the high conversion gain and the low conversion gain and compress them into a single output data row.

10. The image sensor according to claim 1, wherein the subframe exposure is continuous within the frame interval and / or the subframe exposure is centrally located with respect to the frame interval.

11. The image sensor according to claim 1, wherein the plurality of subframe exposures, for example, two, three, or four subframe exposures, have substantially equal durations.

12. The image sensor according to claim 1, wherein the total frame exposure period, which is the sum of the exposure periods of all subframes, is programmable by adjusting the exposure periods of one or more subframes.

13. The image sensor according to claim 1, wherein the control circuit comprises a plurality of row drivers for driving the pixel rows of each subarray, and the plurality of row drivers are configured to drive different pixel subsets within each row.

14. A method for operating an integrated circuit image sensor comprising a pixel array, wherein the pixels of the pixel array are logically organized into pixel rows and pixel columns, the pixel array is organized into a plurality of pixel subarrays, and the method is - During each subframe exposure of the multiple subframe exposures included in the frame interval, the photocharge in the pixels of the pixel array is accumulated, - At the end of all subframe exposures except the last subframe exposure among the plurality of subframe exposures, the accumulated photocharge of the pixel is partially transmitted to the charge-voltage converter of each pixel, wherein the charge-voltage converter is configured to apply a low conversion gain when reading out the partially transmitted photocharge. - At the end of the last subframe exposure among the plurality of subframe exposures, the integrated photocharge of the pixel is completely transmitted to the respective charge-voltage converter of the pixel, wherein the charge-voltage converter is configured to apply a high conversion gain, followed by a low conversion gain, when reading out the completely transmitted photocharge. - Reading sequentially the pixel row data of each subarray at the end of each subframe exposure, wherein the sequential reading of the pixel row data further includes a pipelined step of sampling and digitizing the pixel row data, conditionally combining the digitized pixel row data with previously digitized pixel row data buffered in the pixel memory unit of the read block, and buffering the digitized pixel row data in the pixel memory unit of the read block, Pixel row data from different pixel subarrays are read in parallel. A method wherein, for each pixel subarray, at least two of the plurality of subframe exposures overlap in time, and the rolling shutter control sequences associated with the at least two subframe exposures are time-interleaved.

15. Combining the digitized pixel row data with the previously digitized pixel row data buffered in the pixel memory unit of the read block, under the conditions described above, - If the previously digitized pixel row data relates to the exposure of a preceding subframe within the same frame interval, the low-conversion gain readout of the digitized pixel row data is compared with a first threshold level (TLG), The method according to claim 14, comprising: if the low-conversion-gain readout of the digitized pixel row data of the last subframe is less than the first threshold level (TLG), comparing the corresponding high-conversion-gain readout of the digitized pixel row data with a second threshold level (THG).

Images