High dynamic range image sensor

Abstract

A vertically stacked image sensor with HDR imaging functionality and a method of operating the same are disclosed. The image sensor includes a first substrate; a pixel array organized into a plurality of pixel sub-arrays, each pixel in the pixel sub-arrays including a photoelement for integrating photocharge during each of a plurality of subframe exposures; a transfer gate and a buffer charge-to-voltage converter. A first charge accumulation element of the charge-to-voltage converter is operably connected to at least a second charge accumulation element through a gain switch. The image sensor includes control circuitry configured to trigger partial or full transfer of at least two rolling shutter control sequences and time interleave them. A separate readout block is provided on a second substrate for each pixel sub-array, each readout block including an A / D conversion unit, a pixel memory logic, and a pixel memory unit in a pipelined architecture.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of image sensors and digital cameras with improved dynamic range. In particular, to high dynamic range CMOS image sensors using a stacked chip architecture. BACKGROUND

[0002] Image sensors and digital cameras with high dynamic range (HDR) imaging capabilities based on multi-subframe or multi-frame exposure techniques are known in the art. For example, US8582010B2 discloses an image sensor operated to repeatedly read out incomplete pixel signals during a single exposure process. The incomplete pixel signals result from a respective partial transfer of photocharges successively generated in the light sensitive element of the pixel. However, for image sensors with a large number of pixels and a limited bandwidth for transferring data from the sensor chip, the number of readout steps and the exposure duration in respect of each readout step is very limited, as the rate of pixel data to be transferred and processed increases rapidly with the number of readout steps and the exposure duration, leading to a data transfer bottleneck. Increasing the I / O bandwidth of the image sensor leads to a more complex, cost and energy intensive sensor design.

[0003] US2012127354A1 (Cieslinski [DE]), published May 24, 2012, discloses a CMOS image sensor for an electronic camera, wherein the column readout circuit associated with each column line of the sensor comprises two parallel amplifiers that can be operated with different gain factors. In a high dynamic range mode, the two amplifiers are associated with different readout channels, wherein a first readout channel is used for a plurality of partial intermediate readouts and a second channel is used for a final complete readout. The pixel signals obtained from the plurality of intermediate readouts can be stored on a corresponding number of signal storages connected before the amplifiers. The disadvantages of the image sensor are related to the different readout paths for high and low signal gain, which are more sensitive to artifacts such as cross talk and banding. Furthermore, the use of several amplifiers per column readout circuit is cost and area intensive. Finally, when the sensor is operated in HDR mode, the effective frame rate is reduced for a given I / O bandwidth and the full well capacity (FWC) of the sensor pixels is still limited by the FWC of the photodiodes of the pixels. US10103193B1 (Manabe [US] et al.), published October 16, 2018, discloses a CMOS image sensor and imaging method according to which low dark current is achieved. The method comprises repeated sampling of image charge on the pixel photodiode, the sampled charge being transferred to a floating diffusion. While repeatedly sampling the image charge, an additional capacitance is coupled to the floating diffusion and a capacitor voltage is applied to the additional capacitance. During image charge integration, the additional capacitance can be modulated, thereby reducing the dark current associated with the floating diffusion. The additional capacitance reduces the conversion gain, which is beneficial in high light intensity scenarios. However, further increase of the dynamic range beyond that provided by the dual conversion gain cannot be achieved with this method.

[0004] WO2015084991 A1 discloses an integrated circuit image sensor that is oversampled a first number of times within a first frame interval. A first output image can be constructed from a first number of image data frames corresponding to the first number of oversamplings. Based on the first number of image data frames, the oversampling factor can be adjusted to oversample the pixel array a second number of times within a second frame interval. The readout operations related to the shorter oversampled subframes can be conditional, thereby increasing the cumulative exposure time and the resulting dynamic range. However, the verification circuitry necessary to check whether the conditions have been met and the associated delay increase the complexity of the image sensor and limit the speed at which the image sensor can operate.

[0005] It is desirable to improve the currently available HDR image sensors so that HDR imaging can be performed at higher speed and with less power consumption. SUMMARY

[0006] It is an object of embodiments of the present invention to provide an image sensor capable of acquiring image frames having a high dynamic range and a related method of operating the image sensor. It is a further object of the present invention to provide an image sensor having an improved fast internal readout and processing speed of pixel row data relative to a slower external image frame transfer speed.

[0007] The above objects are achieved by the method and apparatus according to the present invention.

[0008] In one aspect, the invention relates to a vertically stacked image sensor device provided as an integrated circuit. The stacked image sensor device comprises a first substrate having a pixel array organized into a plurality of pixel sub-arrays. Each pixel of the pixel array comprises a photoelement, a transfer gate and a buffer charge-to-voltage converter. The photoelement, preferably implemented as a pinned photodiode, is adapted to integrate photocharge while being exposed to light during each of a plurality of subframe exposures comprised in a frame interval. The transfer gate is configured to transfer at least a portion of the integrated photocharge present in the photoelement to the buffer charge-to-voltage converter at the end of each subframe exposure, and the buffer charge-to-voltage converter is configured to receive and convert the transferred photocharge into a voltage signal. This voltage signal (and optionally a reference voltage signal after resetting the pixel) constitutes the pixel data to be read out. Furthermore, a first charge accumulation element of the charge-to-voltage converter is operably connected to at least one second charge accumulation element through a gain switch, allowing the conversion gain of the pixel to be switched between a high conversion gain and a low conversion gain. The image sensor device comprises control circuitry configured to trigger a partial or full transfer of the integrated photocharge in the photoelement of each pixel row. The amplitude of a transfer pulse applicable to the transfer gate is adjustable by the control circuitry, such that a partial transfer of the integrated photocharge is triggered for all but the last of the plurality of subframe exposures, and a full transfer is triggered only for the last of the plurality of subframe exposures. The control circuitry is further configured to interleave, with respect to each pixel sub-array, at least two rolling shutter control sequences associated with at least two time-overlapping subframe exposures of the plurality of subframe exposures. Further, the control circuitry is configured to control the gain switch of each pixel row to operate the pixel with a low conversion gain for all but the last of the plurality of subframe exposures, and to operate the pixel first with a high conversion gain and subsequently with a low conversion gain for the last of the plurality of subframe exposures. A second substrate of the image sensor device is vertically stacked and electrically interconnected with the first substrate, and comprises readout circuitry. The readout circuitry comprises a separate readout block for each pixel sub-array. In a pipelined architecture, each readout block comprises an A / D conversion unit for sampling and digitizing pixel row data of the respective pixel array, a pixel memory logic for processing the digital pixel row data and conditionally combining the processed digital pixel row data with previously processed digital pixel row data buffered in a pixel memory unit, and a pixel memory unit for buffering the processed digital pixel row data output by the pixel memory logic. The plurality of readout blocks is configured to operate in parallel.In embodiments of the application, the rolling shutter control sequence preferably controls the timing of the readout of the pixel data at the end of each subframe exposure by applying appropriate reset signals, photocharge transfer pulses and row select signals to the pixels of the addressed pixel row. Time interleaving of two or more rolling shutter control sequences ensures that the control signals (e.g. reset signals, photocharge transfer pulses and row select signals) are only delivered to one row of pixels of each pixel subarray at the same time while reading out the corresponding two or more subframes. Here, the simultaneous readout of two or more subframes refers to the case where the readout of one subframe has not yet been completed and is still ongoing at the start of the readout of at least one next subframe.

[0009] Preferably, the control circuitry is provided on the second substrate, e.g. formed on or in the second substrate. However, in less preferred but equally well operating embodiments, the control circuitry can be provided in whole or in part on the first substrate. Providing the control circuitry on the second substrate has the advantage that, for example, a less complex and less expensive manufacturing process can be chosen for the first substrate, e.g. requiring less lithography steps / masks, e.g. including active regions and transistors comprising only one type of doping (e.g. p- or n-doped). Additionally, if the control circuitry is provided on the second substrate, the time required to access the pixel rows of the pixel subarray for readout can be reduced, e.g. such that the pixel subarray overlaps with the row drivers of the control circuitry when viewed from the top and in the direction of the substrate stack due to shorter interconnect and wiring lengths and corresponding capacitances.

[0010] In embodiments of the application, the second charge accumulation element can be part of a charge-to-voltage converter provided directly on the first substrate or can be provided as a separate gain circuit on the second substrate. Further, while embodiments of the application are described with preference to pixels having a dual conversion gain, it will be appreciated by the skilled person that the application is also applicable to pixels having a plurality (e.g. two or more) of conversion gain factors, including pixels or pixel circuits comprising two or more switchable charge accumulation elements or pixels or pixel circuits comprising at least one variable capacitance (such as a voltage controlled capacitance).

[0011] The parallel pipelined chip architecture of stacked image sensors allows for the integration of on-chip pixel memory and pixel memory logic for on-chip processing of read-out pixel data at high internal speed without impacting the external speed of ready frames out of the chip substrate. This results in a significant saving of the amount of data that has to be communicated via the I / O interface of the image sensor and the amount of data post-processing that has to be performed off-chip. As a result, less power is consumed on the image sensor chip. Furthermore, multiple sub-frames can already be combined within the image sensor device to generate a final image frame with improved dynamic range. Due to the dual conversion gain characteristic of the image sensor pixels, sub-frame exposures can be obtained with good signal-to-noise ratio both in low light and bright light illumination conditions. By only partially transferring the photo charges generated in low light conditions, and by applying a low conversion gain during all sub-frame exposures except the last one, the high gain channel with higher noise can be applied only once during the last sub-frame exposure. This results in a significant reduction of the total read-out noise present in the summed sub-frame exposures.

[0012] Other application modes (not limited to high dynamic range only) can also be performed internally and benefit from the increased internal processing speed while maintaining the lower external frame rate specified by the regular I / O circuitry of the image sensor. Such other application modes can include a multi-sub-frame exposure mode in which all sub-frames are obtained with one and the same conversion gain and / or a multi-sub-frame exposure mode in which all sub-frames are obtained under full photo charge transfer conditions. The image sensor device according to the present invention can operate in different application modes and reconfigure the control circuitry of the image sensor accordingly.

[0013] According to some embodiments of the present invention, the control circuitry can comprise a plurality of row drivers for driving the pixel rows of each pixel sub-array, wherein at least two of the plurality of row drivers are associated with each pixel row to drive different portions thereof. In particular embodiments, a plurality of row drivers can be associated with each pixel sub-array, and each of the plurality of row drivers corresponding to a respective one of the pixel sub-arrays is configured to drive only a subset of the pixels in each pixel row corresponding to that sub-array. Preferably, the plurality of row drivers corresponds to a subdivision of a single row driver, e.g. a subdivision along a direction of increasing number of pixel columns in the sub-array (column-wise grouping of pixels in each row, such that each group is driven by one of the plurality of drivers). The subdivided row driver(s) have the advantage that the read-out access time can be reduced compared to regular row drivers that are exclusively arranged at the left / right side of the pixel row.

[0014] According to some embodiments of the present invention, the ratio between the sub-frame exposure rate and the output frame rate is at least 2, e.g. equal to or larger than 2, e.g. equal to or larger than 4.

[0015] An advantage of embodiments of the invention is that the mid-read of a pixel only needs to occur in the low conversion gain configuration of the pixel, which enables the use of only a single data storage location per pixel even for multiple (e.g. more than two) subframe exposures per full image frame.

[0016] A further advantage of embodiments of the invention is that operating the image sensor device in partial transfer mode, i.e. performing a partial transfer of integrated photocharge at the end of each but the last subframe exposure, relaxes the requirement of uniformity of the barrier height across the pixels of the array, which is especially affected by amplitude noise on the partial transfer pulse. As long as it is ensured that the amount of integrated photocharge that is left behind (i.e. not transferred) in each pixel after the partial photocharge transfer sequence is sufficient to cover the signal range in the high conversion gain channel of the pixel at the end of the last subframe exposure - which is a less stringent requirement - non-uniformity of the barriers across the pixels of the array is no longer an issue. The reason for this is that as soon as the amount of integrated photocharge that is fully transferred at the end of the final subframe exposure causes the signal range in the high conversion gain channel of the pixel to be exceeded, the pixel read is performed in low conversion gain. In the pixel read in the low conversion gain channel, all integrated photocharge that was partially and fully transferred can advantageously be summed up on subframe exposure grant and with respect to the same pixel gain configuration (e.g. the combination of the first and second charge accumulation elements being interconnected). Thus, any non-uniformity resulting from the partial transfer operation does not affect the final output obtained with respect to the low gain channel either.

[0017] An advantage of embodiments of the invention is that the read noise for multi-subframe exposure is not increased, thereby enabling a higher dynamic range (DR).

[0018] An advantage of embodiments of the invention is that a DR of 80 dB or more can be obtained. For example, a DR of 100 dB or more (e.g. 120 dB or more, e.g. between 80 dB and 150 dB, such as between 80 dB and 120 dB) can be obtained using two or more subframe exposures. An advantage of embodiments of the invention is that HDR imaging can be performed at high speed, e.g. higher than in the current case, and with limited power consumption, e.g. less than in the current case. In conventional image sensors, an increase in imaging speed comes with a higher bandwidth requirement and thus an increase in power consumption. Higher power consumption typically means that more cooling is required, leading to a larger camera or an increase in operating temperature, which typically has a negative impact on image sensor performance (e.g. higher dark current). Therefore, embodiments of the invention have the additional advantage that the image frame data is transferred with a slower I / O bandwidth compared to the higher internal bandwidth available for pixel data readout and internal processing of the pixel data, which keeps the power consumption off-chip with respect to the image frame data transmission low. This allows for a small and compact camera module design as well as a reduction in dark current levels.

[0019] In another aspect, the application relates to a method of operating an integrated circuit image sensor. The image sensor comprises a pixel array arranged in a pixel substrate and organized into a plurality of pixel sub-arrays. The method comprises integrating photo charges in the pixels of the pixel array during each of a plurality of sub-frame exposures comprised in a frame interval, and partially transferring the integrated photo charges of the pixels to respective charge-to-voltage converters of the pixels at the end of all sub-frame exposures except the last sub-frame exposure. The charge-to-voltage converters are configured to apply a low conversion gain when reading out the partially transferred photo charges at the end of all sub-frame exposures except the last sub-frame exposure. At the end of the last sub-frame exposure, the integrated photo charges of the pixels are fully transferred to the respective charge-to-voltage converters of the pixels, and the charge-to-voltage converters are configured to first apply a high conversion gain when reading out the fully transferred photo charges, thereby providing a high gain channel during readout, and to apply a low conversion gain immediately thereafter, thereby providing a low gain channel during readout. Pixel row data of each sub-array is sequentially read out at the end of each sub-frame exposure, and comprises a pipeline of steps of sampling and digitizing pixel row data, conditionally combining digitized pixel row data with previously digitized pixel row data buffered in pixel memory cells of a readout block, and buffering digitized pixel row data in pixel memory cells of a readout block. According to the application, pixel row data originating from different pixel sub-arrays is read out in parallel, and for each pixel sub-array, at least two of the plurality of sub-frame exposures overlap in time, and the rolling shutter control sequence associated with said at least two sub-frame exposures is time interleaved. In some embodiments of the application, two or more image frames (each image frame consisting of a plurality of sub-frames) can be merged or combined into a single output frame, wherein these method steps are applied to each of the two or more image frames, e.g. in sequence. The merging or combining can be performed externally, i.e. off-chip.

[0020] Particular and preferred aspects of the present application are set out in the appended dependent and independent claims. Features from the dependent claims can be combined with those of the independent claims and of other dependent claims, as appropriate and explicitly recited, without departing from the scope of the application as defined by the appended claims.

[0021] For purposes of summarizing the application and the advantages achieved over the prior art, certain objects and advantages of the application have been described herein above. Of course, it is to be understood that not necessarily all such objects or advantages can be achieved in accordance with any particular embodiment of the application. Thus, for example, those skilled in the art will recognize that the application can be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other objects or advantages as can be taught or suggested herein.

[0022] The above and other aspects of the application will become apparent from the following description of embodiments thereof.

[0023] BRIEF DESCRIPTION OF DRAWINGS

[0024] The application will now be further described by way of example with reference to the drawings in which:

[0025] Figure 1 is a schematic per-layer view of a stacked image sensor with extended dynamic range functionality according to embodiments of the application.

[0026] 0 Figure 2 is a circuit diagram of a pixel with dual conversion gain usable in embodiments of the application.

[0027] Figure 3 Circuit components making up a readout block as used in embodiments of the application are illustrated.

[0028] Figures 4A-4B and Figure 5 is a timing diagram illustrating a parallel and pipelined readout architecture of an image sensor according to the application.

[0029] Figure 6A -B and Figure 7A -B is a timing diagram illustrating the time-interleaved operation of five two-electron rolling shutters on a pixel subarray of an image sensor according to the application and the combination and intermediate storage of pixel row data obtained from two subframe exposures.

[0030] Figures 8 to 10 Methods of operating an image sensor under different lighting conditions according to embodiments of the application are explained in which subframe exposures are combined into a final image frame with extended dynamic range.

[0031] Figure 11 and Figure 12 illustrate image sensor output before and after applying linearization to the sensor output signal, respectively, where the sensor output signal is obtained for image frames with four different subframe exposure timings.

[0032] Figure 13A -B is a timing diagram illustrating the time-interleaved operation of four electronic rolling shutters on a pixel subarray of an image sensor according to the application and the combination and intermediate storage of pixel row data obtained from four subframe exposures.

[0033] The drawings are merely schematic and are non-limiting. In the drawings, the size of some of the elements can be exaggerated and not drawn on scale for illustrative purposes. The dimensions and the relative dimensions do not necessarily correspond to actual reductions to practice of the application.

[0034] Any reference signs in the claims should not be construed as limiting the scope.

[0035] 0In the various drawings, like reference numerals refer to like or similar elements throughout.

[0036] DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0037] The present application will be described with reference to specific embodiments and to certain drawings but the application is not limited thereto but only by the claims. Directional terms such as top, bottom, front, back, leading, trailing, lower, upper, left, right, etc. in the description and claims are used for descriptive purposes only and are not necessarily the only orientation for use of the application. Since components of embodiments of the application can be positioned in a number of orientations, the directional terms are used for descriptive purposes only and are in no way limiting unless otherwise stated. Thus, it is to be understood that such terms are merely used for convenience and are in no way limiting, unless otherwise made clear by context. Thus, it is to be understood that such terms are merely used for convenience and are in no way limiting, unless otherwise made clear by context.

[0038] The scope of the application is not intended to be limited to the embodiments described herein but is intended to be limited only by the claims.

[0039] It is to be noted that the term "comprising" as used in the claims should not be interpreted as being restricted to the means listed thereafter; it does not exclude other elements or steps. It is thus to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression "a device comprising means A and B" should not be limited to devices consisting only of components A and B. It means that with respect to the present application, the only relevant components of the device are A and B.

[0040] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearance of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but can refer to different embodiments. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0041] Similarly, it is to be appreciated that, in the description of exemplary embodiments of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description of related features. This method of disclosure, however, is not to be interpreted as reflecting an intention that the application requires more features than are explicitly recited in each claim. Rather, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Thus, the claims following, in this application are hereby expressly incorporated into this detailed description, with each claim acting as a separate embodiment of the application. Further, the features of the different embodiments can be mixed and matched even if not explicitly described in a particular combination, by one of ordinary skill in the art. Also, it is to be understood that not necessarily all objects or advantages can be achieved in accordance with any particular embodiment described herein. And, it is not necessary for a particular embodiment to achieve each object or advantage described herein. Thus, for example, a specific embodiment can not address a particular object or advantage, and that is, in whole or in part, desirable, and yet be within the scope of the application.

[0042] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0043] In the context of the present application, photo charges integrated by a photosensitive element, such as a photodiode, at illumination are understood to be an amount of charge, which can be expressed as elementary charge, e.g. as a number of photo-generated electrons. In a first aspect, the present application relates to a stacked image sensor, e.g. a vertical stacked CMOS image sensor, which provides for image or video frame acquisition at increased dynamic range with respect to the external readout speed of the device and also increases the internal data readout and processing rate, i.e. the speed at which data (row or full frame) units are transferred off the image sensor chip. If a conventional image sensor device is operated at its maximum external frame rate, it is also operated at its lowest internal row time - the time necessary to address a row of pixels, read out the addressed row of pixels, analog-to-digital convert (A / D convert) the read out pixel data and transfer the complete row of pixel data out of the chip (external readout). In a stacked image sensor according to the present application, there is an internal higher speed in a way that the operation at maximum external frame rate does not enable pixel row readout operations and further data processing operations using the read out pixel data. In embodiments of the present application, an exemplary way of making use of the faster running internal pixel row readout and pixel data processing operations is to operate the image sensor device in a double or multiple subframe exposure mode, wherein the pixel row data related to the multiple subframe exposure are combined into a high dynamic range (HDR) image frame. Figure 1A stacked image sensor 100 is shown as a layered structure with an upper first substrate 101 vertically stacked onto a lower second substrate 103. The first substrate 101 of the image sensor 100 comprises a pixel array 102 and can be configured for operation in front- or back-illumination conditions. For example, the pixel array 102 is organized into a plurality of pixel arrays 102-1 to 102-4 by vertically partitioning the pixel array into independently addressable blocks of pixel rows, with each block containing a plurality of consecutive pixel rows of the pixel array. The blocks of contiguous pixel rows are just one specific example of partitioning the pixel array 102 into a set of pixel sub-arrays. There are other ways of assigning individual pixel rows of the pixel array to one of the pixel sub-arrays, for example, randomly assigning individual pixel rows to one of the plurality of sub-arrays, or according to an interleaved assignment of which consecutive pixel rows of the pixel array are assigned to different pixel sub-arrays, for example, the ith pixel row is assigned to the nth pixel sub-array of N pixel sub-arrays, where n = (i mod N). The interleaved assignment of pixel rows to the set of pixel sub-arrays and the resulting interleaved connection of pixel rows to different readout blocks can have the additional advantage of maintaining a uniform rolling shutter effect across the pixel array. The second substrate 103 contains control and readout circuitry for selecting, controlling and reading out the pixel rows of the pixel array, and for processing the pixel data read out from the pixel rows of the array. The readout circuitry is organized into a plurality of blocks 104-1 to 104-4, referred to as readout blocks, which correspond in number to the pixel sub-arrays. More specifically, each readout block matches and is electrically connected to exactly one of the pixel sub-arrays. The electrical connection between the pixel sub-arrays and the readout blocks can be a fixed connection that is easier to implement or a reconfigurable connection that provides, for example, more flexibility, for example, in case of applying vertical and / or horizontal windowing to the pixel array of the image sensor, reconfiguring the way the pixel sub-arrays are connected to the readout blocks to optimize the imaging speed. This association between the pixel sub-arrays and the readout blocks enables the parallel control of each pixel sub-array while reading out k different pixel rows associated with k different pixel sub-arrays, and also the parallel processing of the readout pixel row data. The typical number k of pixel sub-arrays and corresponding readout blocks can be k = 2,..., 8, 16,..., 128. The image sensor is configured for parallel operation with the simultaneous readout of more than one pixel row. As explained in more detail below, the readout and data processing operations performed on the readout pixel row data are pipelined with respect to each pixel sub-array-readout block pair.

[0044] As an example, an image sensor may include 16 readout blocks and a pixel array with several thousand pixel rows (e.g., 6k pixel rows), where two vertically adjacent pixels are shared (i.e., 2×1 vertical sharing). During the first row time, the first readout block reads pixel data for row 0, the second readout block reads pixel data for row 2, the third readout block reads pixel data for row 4, and so on. Then, during the second row time, the first readout block reads pixel data for row 1, the second readout block reads pixel data for row 3, the third readout block reads pixel data for row 5, and so on. During the third row time, the first readout block reads pixel data for row 32, the second readout block reads pixel data for row 34, the third readout block reads pixel data for row 36, and so on. And during the fourth row time, the first readout block reads pixel data for row 33, the second readout block reads pixel data for row 35, the third readout block reads pixel data for row 37, and so on. This process continues until the last pixel row of the pixel array has been read. The control circuit system preferably includes a plurality of row drivers 105-1 to 105-4, which are matched in number with the plurality of readout blocks 104-1 to 104-4, and control logic (not shown) for controlling the plurality of row drivers, such as controlling the order of row control signals (e.g., row selection control signals, reset control signals, charge transfer gate control signals) driven by the row drivers. Due to the vertical stacking of the two substrates 101 and 103, the row drivers 105-1 to 105-4 may be located on the second substrate 103 and extend primarily along (pixel) rows parallel to the X direction, i.e., as shown in the figure. Figure 1 As shown, in the direction of increasing pixel column number, the pixel rows of the pixel array overlap below the pixel array when viewed from the top of the image sensor (e.g., the illumination side).

[0045] Alternatively, the row driver may be located on the second substrate 103 and extend primarily in the direction of increasing the number of (pixel) rows y, such that when viewed from the top of the image sensor, the row driver is to the left and / or right of the pixel rows and substantially does not overlap with the pixel rows of the pixel array. In yet another alternative embodiment of the invention, the row driver may be located on the first substrate 101, for example, to the left and / or right of the pixel rows. Providing a row driver on a second substrate has the advantages of reducing row addressing noise that can interfere with pixel signals, and further, the larger surface area of ​​the first substrate can be used for imaging.

[0046] Despite Figure 1The readout blocks are shown as separate blocks, but can be further subdivided in the x-direction one or more (e.g. all) row drivers 105-1 to 105-4, meaning that multiple row drivers are used to drive a single pixel row. This subdivision has the advantage that the readout access time can be reduced (reduction of RC time constant) compared to a conventional row driver that is exclusively arranged at the left / right side of a pixel row. The second substrate 103 can further comprise additional circuitry, such as serialization circuitry and I / O drivers, configured to generate and output a data stream related to an image frame.

[0047] The vertical stack of the first and second substrates 101, 102 can be realized by die-to-die bonding, or die-to-wafer or wafer-to-wafer bonding and subsequent wafer dicing. The first and second substrates are electrically interconnected, e.g. through- silicon vias (TSVs) or direct or hybrid bonding techniques (e.g. copper-to-copper interconnects), such that a set of pixel row data signals related to a plurality of pixel rows located in respective pixel sub-arrays of the first substrate are simultaneously read out by a corresponding set of readout blocks in the second substrate. Each pixel row data signal is transferred from the first substrate to the second substrate on a column bit line, wherein a column bit line is understood to extend through the interconnect layer between the first and second substrates. Figure 2 A possible circuit architecture of an active pixel is shown, which is configured to have two different charge-to-voltage conversion gains, in the following referred to as dual gain pixel or dual conversion gain pixel. The pixel 200 comprises a photoelement 201 (preferably a pinned photodiode), a buffered charge-to-voltage converter 202 and a transfer gate 203 connected between the photoelement and the charge-to-voltage converter. The combination of a first charge accumulation element, a floating diffusion node 202-1, a source follower transistor 202-2 having a gate terminal electrically connected to the floating diffusion node 202-1, and a second charge accumulation element, a gain switch 202-3 and a gain capacitor 202-4, as charge-to-voltage converter, are comprised via the buffered charge-to-voltage converter.

[0048] The pixel 200 further comprises a reset transistor 204 connected between a positive voltage source VDD and the floating diffusion node 202-1 for resetting the floating diffusion node to a predetermined voltage level close to VDD upon each time the reset transistor is turned on by a corresponding reset control signal VRST, thereby erasing previously stored pixel data on the floating diffusion node. Furthermore, the source follower transistor 202-2 is controllably connected to a bit line 206 via a row select control signal VRS, which can be applied to the gate of a row select transistor 205 of the pixel. Although in the shown embodiment the row select transistor 205 is arranged in the second substrate 102, it is also possible that the row select transistor 205 is arranged in the first substrate 101. Figure 2The column select transistor 205 is connected between the source follower transistor and the voltage source VDD, but it will be appreciated by those skilled in the art that different arrangements of column select transistors are possible, for example an arrangement in which the column select transistor is connected between the source follower transistor 202-2 and VOUT on the bit line 206.

[0049] When the pixel 200 is illuminated, the photovoltaic element 201 starts integrating photocharges generated in response to the received amount of illumination. When the transfer gate 203 is activated by a suitable charge transfer control signal VTX, i.e. a transfer pulse, the integrated photocharges, or at least a part thereof, are transferred to the floating diffusion node 202-1 of the buffered charge-to-voltage converter 202. The control circuitry of the image sensor, not being part of the pixel circuitry, sets the amplitude, e.g. the voltage amplitude of the transfer pulse, such that a partial transfer of the generated photocharges to the floating diffusion node or the floating diffusion node and the connected gain capacitance is triggered or a full transfer of the generated photocharges to the floating diffusion node or the floating diffusion node and the connected gain capacitance occurs. Generally speaking, a higher amplitude of the transfer pulse leads to a more pronounced reduction of the potential barrier separating the charge well associated with the photovoltaic element and the charge well associated with the floating diffusion node, with or without the connected gain capacitance, and thus to a larger number of photocharge carriers being transferred away from the photovoltaic element. The capacitance associated with the floating diffusion node allows for a temporary storage of the transferred photocharges and converts the deposited photocharges into a voltage signal sensed by the source follower transistor 202-2. When the row select transistor 205 is turned on, i.e. when the pixel is selected for readout, a current set by an external current source starts flowing on the respective bit line 206 and through the row select and source follower transistors. The voltage VOUT at the source terminal of the source follower transistor directly follows the voltage signal applied to its gate terminal. If the gain switch 202-3 is open, e.g. a low voltage at VCG, a first charge-to-voltage conversion gain is determined by the capacitance value of the floating diffusion node. If the gain switch 202-3 is switched to a closed state, e.g. a high voltage at VCG, a part of the photocharges initially stored at the floating diffusion node flows onto the additional gain capacitance 202-4. The additional gain capacitance and the floating diffusion node capacitance are now connected in parallel, leading to a larger total capacitance available for storing the transferred photocharges. This in turn leads to a drop of the voltage signal sensed at the gate terminal of the source follower transistor and directly translates into a lower second charge-to-voltage conversion gain. In alternative embodiments, the additional gain capacitance can also be connected to a positive voltage supply instead of ground, or the additional gain capacitance can correspond to the floating diffusion node of an adjacent pixel. In the latter alternative, the pixels are thus configured to dynamically share their floating diffusion nodes with at least one adjacent pixel of a different row, wherein the shared floating diffusion node of a pixel is temporarily connected to the adjacent pixel and acts as additional gain capacitance. This has the advantage that a more compact design can be obtained which does not require a separate pixel component for the gain capacitance. The row control circuitry of which the pixel 200 forms part is configured to control the gain switch 202-3 via a dedicated conversion gain control signal VCG applied to a control terminal of the gain switch.Thus, the pixel 200 can be controlled to apply a first or second charge-to-voltage conversion gain with respect to the integrated photocharge during readout. The charge-to-voltage converter is buffered because the readout is non-destructive, i.e. the photocharge that is transferred and stored is not destroyed or altered by the readout action.

[0050] Figure 3The different circuit components that make up each readout block 104 of the second substrate are illustrated while multiple readout blocks are configured to operate in parallel. Each readout block 104 includes at least one analog-to-digital conversion unit 106 for sampling and digitizing pixel row data of a respective pixel array, pixel memory logic 107 for processing samples of the digitized pixel row data, and a pixel memory unit 108 for buffering processed samples of the digital pixel row data output by the pixel memory logic 107. Data flow between components or components and peripheral I / O circuitry is indicated by arrows. In particular, the pixel memory logic (PML) 107 has access to intermediate pixel row data stored in the pixel memory unit 108 in order to conditionally combine a current processed sample of the digital pixel row data with a previously processed sample of the digital pixel row data that has been buffered in the pixel memory unit 108. This combination of the two processed samples of the digital pixel row data can then be written back to the pixel memory unit 108, for example, in the case that the buffered combination of processed samples of the digital pixel row data constitutes a final sample to be transferred off the imager sensor chip or accessed again by the PML, for example, in the case that the buffered combination of processed samples of the digital pixel row data constitutes an intermediate result that is not yet ready for output. In the latter case, the PML can use the retrieved processed sample of the digital pixel row data and the intermediate combination of the current processed sample of the digital pixel row data to compute an updated or final combination of the processed sample of the digital pixel row data. In embodiments of the invention, the PML can combine two or more samples of the digital pixel row data by adding or subtracting the samples. Furthermore, the processing operations performed by the PML on the samples of the digital pixel row data can include scaling the samples of the digital pixel row data, comparing the samples of the digital pixel row data to a threshold value pixel by pixel before combining them with another sample, computing a change in representation of the samples of the digital pixel row data (e.g., from Gray coding to binary), or combinations thereof. The processing functions of the PML are not limited to the given examples and can be extended to perform additional processing operations on the acquired samples of the digital pixel row data, which can depend on the particular application or operating mode for the image sensor. While embodiments of the invention are not limited to readout blocks that do not have an analog column bitline signal amplification stage, which can be part of the A / D conversion unit, it is preferred to obtain digital samples of the pixel data in the selected pixel row directly by connecting the A / D converter directly to the column bitline without any intervening amplifier. This has the advantage of reducing conversion time and allows to obtain shorter unit time slots, which in turn enables an increased number of subframe exposures within a single full frame period. It is also possible to further reduce the time due to conversion by reducing the bit resolution of the A / D converter (ADC) in the conversion unit at the expense of reading out less accurate pixel data.Typical embodiments of the present invention implement ADCs with 12-bit resolution. In terms of readout speed and given the goal of implementing multiple subframe exposures in a single full frame period, fast ADCs are preferred, e.g., SAR-ADCs where efficient use of silicon area is not a key concern. However, if striving for more energy efficient / or area efficient embodiment implementations, embodiments can also use other ADC types, e.g., single slope ADCs, e.g., in embodiments implementing pixel-wise ADCs. Slope-based ADC architectures, e.g., including single slope ADCs, have a simple layout and use less design area per pixel column compared to other ADC architectures. In embodiments of the present invention where speed of operation is traded for bit resolution, e.g., via the number of clock cycles for the ADC counter, flexibility and dynamic adjustment of bit resolution per subframe, e.g., 12.7 bits, not necessarily an integer, is preferred, and thus flexibility and dynamic adjustment of the number of subframe exposures accommodated in the full frame period of the image sensor is obtained.

[0051] The pixel memory cells of all readout blocks act as a global frame buffer. In specific embodiments of the invention, this global buffer has a data buffer capacity smaller than the size of a full image frame, e.g. can hold only a fraction of a full image frame as generated by the image sensor. It is possible, e.g. if the pixel row data is retrieved from the global buffer fast enough for off-chip transfer, such that buffer overflow is prevented. Then, it is allowed to map different pixel rows to the same location in the pixel memory cells without loss of pixel row data. The time interval of the intermediate storage of the pixel row data in the pixel storage cells is typically larger than the combined time for selecting a pixel row and converting the pixel data of the selected row in the A / D conversion unit, but smaller than the full frame period. Preferably, the pixel storage memory cells 108 on the second substrate of the image sensor chip are provided as a block of SRAM cells, e.g. as a memory bank of a banked SRAM memory. The storage cells, e.g. SRAM storage cells, on the second substrate are preferably managed independently from each other on a per readout block level. The storage cells corresponding to each readout block can be subdivided into even smaller storage sub-cells, e.g. similar to a row driver subdivision. The pixel storage memory cells or their subdivisions managed per readout block level are advantageous in terms of their smaller physical size and address space, which makes the read / write operations to the pixel storage memory cells faster. It can also prove useful for yield reasons, e.g. the storage cells corresponding to each readout block can be built with some redundancy in order to allow for independent handling of memory defects. Fig. 4 is a flow chart illustrating the pipelined architecture of the stacked image sensor during image or video capture. The pipelining includes the following stages related to each pixel row of the image sensor: pixel row reset, pixel row exposure, pixel row data readout and analog-to-digital conversion, a full fetch- process-write (F-P-W) cycle performed by the pixel memory logic (PML) on the digitized pixel row data, writing back of the digitized processed pixel row data to the pixel memory cells for intermediate storage, and accessing the pixel memory to produce a global I / O stream of pixel row data when transferring one frame (or consecutive frames) of processed image data out of the sensor chip. As mentioned above, the digital processing applied by the PML can include a conditional combination of two samples of the digital pixel row data, e.g. in case the current sample of the digital pixel row data provided by the A / D conversion unit meets a predetermined or programmable condition, e.g. exceeds a threshold value. In such a case, the current sample of the digital pixel row data provided by the A / D conversion unit and a previously obtained sample of the digital pixel column data buffered in the pixel memory cells are loaded into the PML during a pre-fetch period, and the combination of these two samples (e.g. sample addition and optionally subsequent sum compression) is performed during a processing period of the PML. The result of the processing operation is then written back to the pixel memory cells during a write period of the PML.

[0052] For example, in an operation mode where a partial transfer of integrated photocharge is used for all sub-frames except the last sub-frame exposure (partial transfer mode), the pixel readout from the low gain channel is combined (e.g., cumulatively summed) under the condition that the pixel readout to be added is independent of the last sub-frame exposure. The intermediate partial sum is stored in the pixel memory unit. If the condition that the corresponding high gain pixel read exceeds a threshold is met, only the low gain pixel read from the last sub-frame exposure is added to the partial sum. The updated partial sum then becomes the final sum and is used as the output of the image sensor device. If the corresponding high gain pixel read does not exceed the threshold, only the high gain pixel read is used as the output. Alternatively, if compression is enabled (e.g., the image sensor device is operated in a partial transfer mode with compression), the output in compressed format is obtained as a combination (e.g., applied compression algorithm) of the high gain pixel read and the previous partial sum of all low gain pixel reads (i.e., of all sub-frame exposures except the last one). The compressed output data can be sent off-chip immediately and thus does not need to be written back into the pixel memory unit.

[0053] As a further example, if the image sensor device is operated in a full transfer mode, i.e., the integrated photocharge is fully transferred at the end of each sub-frame exposure, the pixel readouts in the low gain channel and the high gain channel are summed over the number of sub-frame exposures, respectively (e.g., by updating independent partial sums for the low gain channel and the high gain channel, respectively). If compression is enabled in this operation mode (e.g., full transfer mode with compression), the partial sums of the low gain channel and the high gain channel can be input to the compression algorithm at the end of each sub-frame exposure, and only the compressed partial sums need to be written back into the pixel memory unit. This has the advantage of reducing the storage requirements, but requires additional computation for decompression during read-back. Fig. 4 further illustrates that the pipelines are applied to a sequence of pixel rows contained in one pixel sub-array, while different pipelines work in parallel with respect to different pixel sub-arrays. In other words, a separate pipeline is implemented for each pixel sub-array and corresponding readout block, resulting in K-way pixel row data acquisition and processing for a total of K independent pixel sub-array / readout block combinations. In order to stream the processed frame data off the image sensor chip, the access to the pixel memory is time-interleaved, such that the pixel row data pertaining to different pixel sub-arrays do not overlap. The electronic rolling shutter of the image sensor works well in combination with the full-pipelining architecture, since the (partial) exposure of a pixel row and the readout of the pixel row data after the (partial) exposure are performed sequentially.

[0054] For clarity, the flowchart of Fig. 4 includes only two pixel sub-arrays, each comprising three pixel rows. In typical embodiments of the invention, there can be more than two readout blocks and pixel sub-arrays, for example between 2 and 16, and each pixel sub-array typically contains hundreds of pixel rows.

[0055] Figure 5 is a flowchart illustrating the pipelining of pixel row data in the case of multiple sub-frame exposures, in this example two sub-frame exposures SF1 and SF2 having equal sub-exposure times and being consecutive in time. Consecutiveness in time is beneficial to reduce rolling shutter distortion in the final image. For clarity, Figure 5 Only the data pipeline of a single pixel sub-array and corresponding readout block is shown; as mentioned earlier in this application, embodiments of the invention provide for multiple parallel working pipelines with respect to multiple pixel sub-arrays and corresponding readout blocks. Embodiments of the invention are also not limited to two sub-frame exposures - for example, three, four or more sub-frame exposures can constitute a full frame - and the sub-frame exposures need not be equal in duration. The sub-frame exposures need not be consecutive in time, provided that the image sensor operates in a mode that does not use the photo charges of the partial transfer.

[0056] After termination of the first sub-frame exposure SF1, the pixel data of each row K, K+1,..., K+4 is read out and converted to digital pixel row data in step 502-1, processed by the PML in step 503-1 and written to the pixel memory unit in step 504-1. At this point, the processed pixel row data is stored in the pixel memory unit and has not yet been used as output, for example as part of a final image frame transferred from the image sensor chip. The following steps are then repeated for the second sub-frame exposure SF2: after termination of the sub-frame exposure, the pixel data of each row K, K+1,..., K+4 related to the second sub-frame exposure SF2 is read out, converted to digital pixel row data in step 502-2, processed by the PML in step 503-2 and written to the pixel memory unit in step 504-2. However, the processing by the PML in step 503-2 now includes the conditional use of previously obtained samples of the digital pixel row data as additional input operands, where the previously obtained samples have been buffered in the pixel memory unit during the time elapsed between the end of step 504-1 and the start of step 503-2. After the processing of two samples of the digital pixel row related to the sub-frame exposures SF1 and SF2 by the PML is completed in step 503-2, the processing result, for example a combination of the two samples (for example, sum or difference), is written back to the pixel memory unit in step 504-2 and subsequently read out therefrom in step 505 for the purpose of transferring the processed pixel data row as part of a final full image frame off the image sensor chip.Figure 5 As can be seen (e.g., the vertical dashed lines for guidance), while multiple rows of pixels are processed in parallel, the different pipeline stages are balanced in time such that each pipeline stage only operates on pixel data for a single row of pixels. In particular, the data path for pixel row data is organized such that no two rows are simultaneously addressed for their pixel data readout and conversion.

[0057] In embodiments of the application, the total exposure period of a full image frame of the image sensor, i.e. the sum of all sub-frame exposure periods, can be controlled via the reset control signal, e.g. by controlling the instant in time relative to the full frame period at which the photo-elements of a pixel row are reset and thereafter exposed to incident light. Via the readout control signal with respect to the first sub-frame exposure, the ratio between the first sub-frame exposure period and the second sub-frame exposure period can be controlled. More specifically, once a pixel row has been selected for readout and a transfer pulse has been applied to the transfer gates of the pixels of this row, the first sub-frame exposure period ends and the second sub-frame exposure period starts immediately, which causes a partial transfer of the photo-charge that has been generated in the respective pixels. In contrast thereto, once a pixel row has been selected for readout a second time within the same frame interval, the second sub-frame exposure period ends and a transfer pulse of larger amplitude than the transfer pulse related to the first sub-frame exposure is applied to the transfer gates of the pixels of this row, thereby starting a full transfer of all remaining photo-charge in the respective pixels. If more than two sub-frame exposures occur in embodiments of the application, the row selection signal and the transfer pulse applied at the end of each sub-frame exposure except the last one determines the duration of this sub-frame exposure. Furthermore, the amplitude of the transfer pulse applied for each sub-frame exposure except the last one is adapted to cause only a partial transfer of the photo-charge present in the photo-elements of the pixels, whereas for the last sub-frame exposure it is increased, thereby triggering a full transfer of the remaining photo-charge. Preferably, the amplitude of the transfer pulse remains constant for each sub-frame exposure except the last one. Furthermore, embodiments are not limited to read out the buffered photo-charge signal of a pixel, which is referred to as the signal level of the pixel, i.e. the voltage signal generated by the buffered charge-to-voltage converter in response to the transferred photo-charge on one or both of the charge accumulation elements, but preferably include further read out the buffered reset signal of a pixel, i.e. the voltage signal generated by the buffered charge-to-voltage converter in response to the residual charge still present on the first charge accumulation element or the first and second charge accumulation elements after the pixel has been reset, at both a high conversion gain and a low conversion gain, which are referred to as the high gain and low gain reset levels of the pixel. This has the advantage that the readout block of the image sensor can perform correlated double sampling (CDS). Fig. 6 is a timing diagram that describes in more detail the timing and time resources required for each pipeline stage. For a better understanding of this figure, the exemplary timing diagram only considers sixteen pixel rows per pixel sub-array (row 0 to row F). For example, embodiments of the application can contain more pixel rows per pixel sub-array, e.g. hundreds of pixel rows or even more than a thousand pixel rows.

[0058] According to the timing diagram of Fig. 6, each full frame period, e.g. frame 0, frame 1, etc., is divided into a plurality of unit slots, e.g. the slots labeled '0' or '1' in the first row of the figure. It is noted that consecutive unit slots are assigned to even positions labeled "0" or odd positions labeled "1". The even and odd slot positions are associated with the first and second rolling shutter operation, respectively. Importantly, the control sequences for the first and second rolling shutter, i.e. the reset and readout selection, are time interleaved with the row control signals involved in the first and second rolling shutter operation, which are provided only during even and odd slots, respectively. The unit slot marking the beginning of each subframe exposure with respect to a particular pixel row of the pixel subarray is labeled by the letter "S", while the unit slot marking the end of the subframe exposure in that row carries the letter "E". The beginning of the first subframe exposure can correspond to the falling edge of a reset control signal applied line by line, which steps through the pixel rows making up the subarray and resets the optoelectronic elements of the pixels in that row to a predetermined voltage level. In contrast thereto, the second or any further subframe exposure, if consecutive in time to the preceding subframe exposure, does not require the delivery of an additional reset signal to the optoelectronic elements of the pixels in order to begin, but seamlessly begins with the partial transfer of the photocharges generated in the optoelectronic elements after the preceding subframe exposure has ended. It is observed that this does not exclude the delivery of a reset signal to only the first and second charge accumulation elements of each pixel, which expunges the photocharges previously transferred and thus clears space for another subframe exposure readout. In contrast to a full or complete transfer of the photocharges generated during a regular exposure interval, the partial transfer only skips the portion of the photocharges present in the potential well associated with the optoelectronic element that exceeds a programmable threshold potential, e.g. a threshold voltage level. The programmable threshold is determined by the amplitude of the transfer pulse supplied to the transfer gate of the pixel. As can be seen in the timing diagram, the first rolling shutter sequence starts with row 0 and proceeds incrementally to row F, where the next row is selected at every second unit slot. Thus, the first rolling shutter control sequence and associated first subframe exposure are always timed to align with even unit slots, i.e. group "0". Likewise, the second rolling shutter control sequence and associated second subframe exposure always occupies odd unit slots, i.e. group "1", thereby avoiding any interference between the two simultaneously running electronic rolling shutters. Here, interference between two or more rolling shutters operating in parallel on the pixel rows of a pixel subarray is understood as an attempt to select and read out the pixel row data (signal level or reset level) of two different rows of the same subarray at the same time. In the present timing diagram, a double subframe exposure SF0, SF1 is selected per frame period, but a larger number of subframe exposures can be accommodated within a full frame period.For example, four sub-frame exposures can constitute the total exposure time within a single full frame acquired by the image sensor, in which case the unit time slots are assigned position numbers "0" to "3" (e.g. position modulus four) and each position group "0" to "3" is associated with row control signals involving only one of the four time-interleaved rolling shutter sequences.

[0059] The duration of a unit time slot is generally determined by the pipeline stage with the largest latency. In the present embodiment, for example, the unit time slot corresponds to the combined duration of the settling time of the signal present on the bit line, the sample and hold, and the time required for the fast sequential measurement of the pixel signal level at both high and low conversion gain by the A / D conversion unit to perform the analog-to-digital conversion. If CDS is applied, the signal present on the bit line includes both the pixel reset level and the pixel signal level, meaning that the unit time slot is the sum of the settling time, the sample and hold time, and the time of the A / D conversion for the fast sequential CDS measurement in the high gain readout channel and the low gain readout channel. Nonetheless, for the purpose of achieving the image sensor at higher speed, more pipeline can be added in the readout path, and the unit time slot can be subdivided or redefined so as to truly reflect the presence of the added pipeline stages. The fast sequential measurement of the high gain pixel signal level and the low gain pixel signal level can be performed by reducing the resolution of the ADC components in the A / D conversion unit, e.g. two 12-bit conversions can be performed simultaneously with a single 14-bit conversion. Alternatively, in the case of Fig. 6, the A / D conversion unit can comprise two ADCs working in parallel instead of a single ADC assigned to a pixel row. Within each unit time slot, the A / D conversion unit is thus able to convert the pixel data of exactly one row of pixels in the pixel sub-array into the digital domain. The pixel row that undergoes A / D conversion during a particular unit time slot is indicated by its row number in the pixel sub-array (e.g. the numbers "0" to "F" in the ADC line of Fig. 6). The converted pixel data is available exactly one time slot after the respective sub-frame exposure period has ended (e.g. indicated by the letter "E"). There can be time instants in which the A / D conversion unit is idle and does not perform any pixel row data conversion (e.g. the blank unit time slots in the ADC line of Fig. 6).

[0060] Each sub-frame exposure period SF0, SF1 and full-frame period cycle can thus be expressed as an equal number of unit slots. For example, in the embodiment referred to in Fig. 6, the first sub-frame exposure SF0 lasts for fifteen slots, the second sub-frame exposure lasts for thirteen slots, and each full-frame (Frame 0, Frame 1) consists of 32 slots. The shorter the unit slot is relative to the full-frame period, and the shallower the pipeline depth, the more sub-frame exposures can be accommodated in a single full-frame period, which is considered fixed over time. For example, the shortest possible single-frame period is fixed by the maximum achievable external I / O rate at which pre-processed HDR frame data can be transferred from the image sensor chip to an external device, e.g. an external storage device such as an external RAM or hard drive. A typical value for the maximum achievable external I / O rate in embodiments of the present application can be 120 fps for a dual sub-frame exposure operating mode, but also depends on other factors, e.g. the number of pixel rows and readout blocks and ADC bit resolution. For the example in Fig. 6, this means that every 520.8 μβ a pixel row worth of frame data is transferred, corresponding to two consecutive slots containing the same number in the readout row (last row in Fig. 6), and thus involving data obtained from the same pixel row, while one unit slot only lasts for 260.4 μβ. These values are given for illustrative purposes and do not necessarily reflect the frame data rate and unit slot duration of an actually manufactured image sensor, which can comprise more pixel rows than the example of Fig. 6, compared to the example of Fig. 6.

[0061] For example, an image sensor with a megapixel resolution can have a unit slot duration of about 15 μβ, which allows for a maximum internal sub-frame rate of 240 fps at 14-bit ADC resolution (the ADC resolution can be traded for speed or a higher number of sub-frames). Thus, this exemplary image sensor supports outputting pre-processed HDR frame data to an external device (i.e. off-chip) at an I / O rate of 120 fps (in case of a dual sub-frame exposure mode) and 60 fps (in case of a four sub-frame exposure mode), each with 14-bit ADC resolution.

[0062] As indicated in Fig. 6, the final frame data read from the on-chip pixel memory can be transferred in a compressed format. Here, compression involves the fact that the combined pixel signal level for high conversion gain settings and low conversion gain settings of a pixel (which leads to an improved dynamic range of the image sensor) can occasionally exceed the available bit depth of the pixel memory. In such cases, a scaling operation is performed before writing the combination of high gain and low gain pixel signal levels back to the pixel memory, whereby the scaled signal combination fits again into the available bit depth of the pixel memory. For example, a combination of a 12 bit wide high gain pixel signal level and a 12 bit wide low gain pixel signal level can lead to a 13 bit wide combined HDR value, which does not fit into the 12 bit wide pixel memory without causing an overflow. In these cases, the combined HDR value is rescaled (scaling factor < 1) such that it fits again into the 12 bit wide pixel memory. Instead of a simple scaling operation, a more detailed compression algorithm can be used, which combines the pixel readings of the high gain channel and the low gain channel differently in different signal sub-ranges, for example with varying noise characteristics in different signal sub-ranges to vary the precision of the high gain and / or low gain pixel readings. For example, the compressed output data can be sent off-chip at twice the high rate compared to the high gain data and the low gain data sent off-chip separately, for example one time slot per row of pixels compared to two time slots in the last row of Fig. 6.

[0063] As further visible from the timing diagram in Fig. 6, the processed high gain and low gain pixel row data is written to different addresses of the pixel memory, thereby allowing to separately read out pixel data obtained for high conversion gain or low conversion gain in an operating mode of the image sensor which does not use the dual gain functionality of the pixels, e.g. simple low gain or high gain operation in case of no extended DR. Such operating mode can also include different HDR modes of the image sensor which do not rely on partial transfer of generated photo charges within a single full frame period. For example, multiple subframe exposures or multiple frame exposures with different exposure times and / or offline or online mixing of the gain conversion into one HDR image frame (single frame or multiple frame exposure scaling) which can take place on the image sensor chip or on an external data processing device. The case of combining multiple subframe exposures or multiple full frame exposures with at least two exposure time settings into an HDR image frame is also referred to as multi-exposure operation and can be performed on the image sensor chip according to the present application in addition or instead of the dual gain conversion of the pixels to obtain an HDR image frame. Different exposure time settings result in different but fully determined slopes and inflection points between the slopes in the linear plot of the image sensor (digital numbers as a function of illumination). Thus, in an image frame with large in-scene dynamic range, the pixel output signals obtained by unequal exposure time settings can easily be re-linearized in-chip or off-chip to produce a linear HDR signal without calibration. The ratio of the subframe exposure periods of a pair of consecutive subframes together with the current conversion gain controls the change of the response slope.

[0064] Figure 11 Different response slopes and inflection points for a single conversion gain (e.g. low gain only) and a total of four subframe exposures per full image frame are illustrated. In this example, the exposure period of the subsequent subframes is increased, e.g. for a full exposure time of 10.2 ms, Texp1 = 128 Trow, Texp2 = 32 Trow, Texp3 = 8 Trow and Texp4 = 2 Trow, where the row time (Trow) is equal to about 60 ps. In Figure 12 The respective sensor output after linearization is shown in Fig. 6. The position of the inflection point on the vertical axis (raw response) can be fully programmable, e.g. by introducing a programmable clipping block in the signal path for clipping the pixel signal in the digital domain.

[0065] As a specific example for a combined multi-frame scaling and multi-exposure, the following operating mode can be listed: where a first full frame with increased DR is acquired via two subframe exposures with low conversion gain and with large exposure time ratio and a second full frame with increased DR is acquired via two subframe exposures with high conversion gain and also with large exposure time ratio. Finally, the first and second full frames can be combined offline into a final image frame with even further increased DR.

[0066] In contrast to separate storage locations for high and low conversion gain pixel data, the processed digital pixel data obtained with respect to two different pixel rows in the same subarray, such as row 0 and row 8, row 1 and row 9, etc., are mapped to the same address of the pixel memory in order to save memory capacity requirements and associated chip area. Furthermore, in embodiments of the present application described in more detail below in which an HDR image frame is generated due to the combination of multiple subframe exposures with partial photocharge readout and different conversion gain settings, only one storage location (e.g., by address or line) in the pixel memory needs to be available for high and low conversion gain pixel row data. Thus, the allocation of two rows of pixel memory per pixel row in the subarray in Fig. 6 is considered optional, and a more area and energy efficient solution can allocate only a single row of pixel memory per pixel row and subarray. In the latter case, it can be seen that the total storage capacity of the pixel memory is less than the storage size associated with a full frame of image data. According to Fig. 6, immediately after the A / D conversion is completed, the processed pixel row data for high and low conversion gain is stored in the pixel memory with respect to the selected pixel row for readout, i.e., exactly one unit time slot later, while the A / D conversion unit proceeds with the subsequent selected pixel row. Here, it is assumed that the processing of the converted digital pixel row data and the write operation of the processed pixel row data can be performed within one unit time slot, as both steps have individual delays shorter than one unit time slot. More specifically, once the A / D conversion of a pixel row is completed at the end of the first subframe exposure SF0 in each frame, the pixel row data obtained for the low conversion gain channel of the pixel after the first partial transfer (e.g., pixel signal level minus pixel reset level in case of CDS) can be unconditionally written into the pixel memory. During the intermediate subframe exposures, if more than two subframe exposures are programmed, each subsequent low gain pixel row data obtained with respect to each further partial transfer (e.g., pixel signal level minus pixel reset level in case of CDS) occurring between the first and the last subframe exposure can be subjected to the processing by the PML, and the result of this processing is subsequently written back to the pixel memory. In addition to managing the data flow from and to the pixel memory, the processing by the PML can include performing basic clipping and / or scaling operations on the digitized pixel data (e.g., after CDS).

[0067] In a preferred embodiment, the processing by the PML further comprises conditionally combining (e.g. conditionally adding) the processed or unprocessed (raw) pixel line data of the current subframe to buffered pixel line data in the pixel memory. In such a case, buffered previous data is extracted from the pixel memory by the PML, where this extraction can be performed while the pixel line data of the current subframe is still undergoing A / D conversion. The condition to be fulfilled by the PML to perform the combination of pixel line data related to different subframes can involve a step of comparing the pixel data to a first programmable threshold level TLG. Depending on the result of this comparison, the pixel data of the current subframe is combined with the data of the previous subframe buffered in the pixel memory, e.g. when the pixel data is below the threshold TLG, or discarded, e.g. the pixel data is greater than the threshold TLG. The discarding of pixel data can occur for example if the image sensor device operates in a short intermediate subframe exposure mode where only the pixel data of the shortest subframe exposure is kept. The event of discarding pixel data can be monitored during each full frame exposure period to select only the pixel data of the shortest intermediate subframe for output. If the pixel memory does not already contain valid data of the current image frame (i.e. comprising multiple subframes), the combining step can be replaced by a direct write to the pixel memory. Alternatively, the pixel memory can be initialized to a default value, e.g. zero, at the beginning of each new image frame.

[0068] Finally, the processed or raw pixel line data is related to the high gain channel and the low gain channel, e.g. the CDS signals of the pixels for the high gain channel and the low gain channel are directly sent to the sensor output interface at the end of the last subframe exposure (e.g. after full photo charge transfer in subframe SF1 in Fig. 6), or are conditionally processed by the PML depending on the output format that has been selected. The conditional processing of the last subframe by the PML can comprise the following steps: comparing the high gain CDS signal of each pixel to a second programmable threshold THG, and if it is below the threshold THG, writing the high gain CDS signal of the pixels of a line to the pixel memory. In this case, the previously stored pixel data of this line is overwritten. Alternatively, the previously stored pixel data for this line can be fetched by the PML and combined with the currently processed high gain pixel data, e.g. by compressing the high gain pixel line data and the low gain pixel line data into a single line of data, and the result of this PML operation is stored in the pixel memory as the final result for the pixel data of this line in the current frame. Conversely, if the threshold THG is exceeded, the high gain data of the pixel line can be discarded and the PML uses the low gain data of the pixel line, e.g. after extraction from the pixel memory by the PML, by combining it (e.g. adding and optionally also compressing) with the previously stored pixel line data.

[0069] The second full frame in the timing diagram of Fig. 6 also comprises two subframe exposures identical to the first full frame. In the present embodiment, the time interval between the end of the last subframe exposure SF1 of the first full frame and the beginning of the first subframe exposure SF0 of the second full frame (during which no rolling shutter exposure takes place) is chosen as short as possible compared to the full frame period, resulting in the longest possible total frame exposure. In fact, the two subframe exposures extend over almost the entire full frame period (e.g. 87.5%) and cannot be extended further without avoiding inter-frame interference of the electronic rolling shutters (e.g. avoiding that the second rolling shutter of the first frame and the first rolling shutter of the second frame try to read out different rows of the subarray at the same time, i.e. that the two unit time slots marked with "E" are fully coincident in time). The delayed second electronic rolling shutter starts stepping through the rows of the subarray at times tA and tB located at the end and around the midpoint of the frame, respectively, before the earlier first electronic shutter has ended. This is another indicator for a long total frame exposure and almost balanced exposure times of the two subframe exposures. However, embodiments of the present application are not limited to long frame exposures but can equally be adapted to perform shorter frame exposures, e.g. to better adapt the image sensor to distortion-free capture of fast moving objects. For a pixel subarray with the same number of rows, unit time slot duration and full frame period as shown in Fig. 6, a detailed timing diagram of the shortest possible total frame exposure consisting of two consecutive subframe exposures SF0, SF1 of equal subframe exposure period (three unit time slots each) is shown in Fig. 7.

[0070] Furthermore, in embodiments of the present application, full frames can be acquired consecutively or a predetermined number of consecutive frames can be acquired, as shown in Figs. 6 and 7. In embodiments of the present application, each new full frame typically starts with a reset operation of the photovoltaic elements of the pixel rows at the first time they are selected in the new frame. In contrast to Fig. 6, the times tA and tB at which the delayed second electronic rolling shutter starts stepping through the rows of the subarray before the earlier first electronic shutter has ended are both located around the end of the frame and separated only by a very short subframe exposure period.

[0071] Fig. 13 shows a timing diagram of a four subframe exposure operation (partial or full transfer mode) of an image sensor device according to the present application. The subframe exposures SF0 to SF3 are consecutive in time and have equal subframe exposure periods. In partial transfer mode, only the fourth subframe exposure SF3 with the high gain configuration of the pixels is read out. The readout pipeline is similar to the readout pipeline described above, e.g. as in Figs. 6 and 7.

[0072] In the above described embodiments, the first and second thresholds (TLG and THG) are preferably programmable, for example by software or directly by the user, as a function of the total exposure time and the number of sub-frames of the image frame to be acquired, but can be fixed values in other embodiments. The first and second thresholds (TLG and THG) are generally constant over the pixel array, but they do not necessarily have the same value, for example TLG and THG can differ in magnitude, and they are generally independent of the potential barriers set by the partial and full charge transfer pulses applied to the pixel transfer gates. In the above described embodiments, the readout, A / D conversion, digital data processing and pixel memory write phases are performed on entire rows of image sensor pixels. However, depending on the trade-off between speed, area, power consumption of the image sensor according to the present application, the multiple pixel related data blocks (e.g. reset level and signal level) do not necessarily have to correspond to pixel rows, but can be performed on pixel groups (e.g. row segments in a row or a group of columns, e.g. even / odd channels) or even pixel by pixel.

[0073] In embodiments of the present application, a state machine can be used that triggers the A / D conversion of the pixel data and the processing of the digitized pixel data in the PML running in parallel for each unit time slot. Once triggered by the state machine, the A / D conversion unit can use its own clock, which is typically the fastest running clock of the components comprising the control and readout circuitry. Other dedicated clocks with their specific clock speed can be used in the components of the control and readout circuitry, e.g. a sequencer clock, a clock for regulating memory access, etc., which typically run at a lower speed, e.g. 5 to 10 times slower, than the clock of the A / D conversion unit.

[0074] It is also possible that in addition to the multiple sub-frames and partial read-out operations already mentioned, the image sensor can also be operated in a LOFIC-like mode (Lateral Overflow Integrated Capacitor). In the LOFIC-like mode, the photo charge of an oversaturated photodiode overflows into a second charge accumulation element or a combination of the first and second charge accumulation elements. The bias voltages of the transfer gates and gain switches are set accordingly. In the LOFIC-like mode, which is particularly suitable for high illumination conditions, the low gain channel of each pixel is read out twice: the overflow charge on the sense node is first read out (defines the overflow level), followed by a reset operation of the sense node and a reset read (e.g. voltage reset with respect to the connected first and second charge accumulation elements) and a further readout of the sense node with respect to the fully or partially transferred photo charge from the photosensitive element (defines the photodiode / signal level in the low gain channel). Preferably, a reset read for pixel noise correction is used, a correlated double sampling (CDS) is performed with respect to the photodiode level in the low gain channel. A digital double sampling can be performed on the overflow level, again using a reset read for correction. Alternatively, the reset operation and the corresponding reset read can be performed at the beginning of each new image frame and used to implement a true CDS for the overflow level.

[0075] In the LOFIC-like mode, at the end of each sub-frame exposure, except the last sub-frame exposure, the pixel data is written to the pixel memory cell, the low gain pixel data can first be compared to a threshold of the low gain channel (TLG). If the low gain pixel data is below the threshold TLG, the overflow signal level is ignored, e.g. by setting it to zero to avoid noise or dark current contributions, or if the low gain pixel data is greater than the threshold TLG, the overflow signal and the pixel signal of the low gain channel are directly summed and the partial sum of the respective sub-frame exposure is stored in the pixel memory cell. The direct summation is possible because both the overflow signal and the low gain pixel signal are obtained with respect to the same low conversion gain setting of the pixel. Only for the last sub-frame exposure, between the readout operations for the overflow signal and the low gain pixel signal, respectively, the pixel is read out in the high gain channel. Only when the high gain pixel data is smaller than a threshold of the high gain channel (THG), the high gain pixel data is retained for the sensor output operation, e.g. separate off-chip transmission. Otherwise, the high gain pixel data is ignored for the purpose of the image sensor data output or transmitted together with the low gain pixel data in a compressed format. As for the non-final sub-frame exposures, the low gain pixel data can first be compared to a threshold of the low gain channel (TLG), which is also used for the final sub-frame exposure, before the result is added to the partial sum read back from the pixel memory cell to decide whether the overflow signal should be added to the low gain pixel data.

[0076] In a second aspect, the present invention relates to a method of operation of an image sensor according to the first aspect, which method produces an image frame with increased dynamic range. The method takes advantage of the fact that all but the last subframe exposure of a plurality of subframe exposures can be partially read out, each subframe exposure having an exposure time less than the full frame period, and that all subframe exposures are combined to effectively increase the full well capacity of the photoelement and limit the increase in readout noise associated with multiple exposures. In addition, the image sensor is not saturated even under high illumination conditions. In conjunction therewith, the conversion gain of the image sensor pixel is switched between a high conversion gain and a low conversion gain to obtain the best signal-to-noise ratio under low light or strong light exposure conditions relative to each pixel.

[0077] Conventional image sensors using multiple exposures with full transfer of photocharge generated in the photoelement of the pixel require a floating diffusion node to have the same associated full well capacity (FWC) as the photoelement. If a pixel with good charge-to-voltage conversion gain is sought, this limits the FWC of the photoelement, e.g. PPD. In embodiments of the present invention, the limitation is overcome by only inducing partial transfer of the generated photocharge from the photoelement to the charge accumulation element, and the FWC of the photoelement can be greater than the FWC associated with the first charge accumulation element, and possibly greater than the combined FWC associated with the first and second charge accumulation elements. In addition to this, embodiments of the present invention limit the noise associated with multiple readouts of each pixel by only transferring a portion of the generated photocharge from the photoelement to the charge accumulation element at the end of each subframe exposure except the last subframe exposure, and initiating full transfer of the remaining photocharge dedicated to the last subframe exposure. While conventional approaches relying on adding N subframe exposure readings (full transfer of integrated photocharge) increase the resulting readout noise by a factor of sqrt(N), in embodiments of the present invention the readout noise (e.g. dark noise) only occurs once at the end of the last subframe exposure when high conversion gain is applied to read out the pixel signal level under low light conditions. Under such conditions, the photocharge generated in between is not affected by the partial transfer operation. In the case of higher illumination conditions, a portion of the photocharge generated in between is transferred and converted in the low conversion gain setting of the pixel, and the continuous readout at low conversion gain is added to the final result. In this case, the high gain path is not used, and therefore benefits from the larger FWC associated with the low conversion gain setting of the pixel.

[0078] Figures 8 to 10The inventive method is illustrated for three different illumination conditions: low light, high light, and over-saturated conditions. For each of the three illumination conditions, the image sensor is operated accordingly to generate an HDR image frame from a plurality of sub-frame exposures. For illustration purposes, only two sub-frame exposures of approximately equal exposure duration are assumed, but embodiments of the invention can use more than two sub-frame exposures. Likewise, it is not essential to the invention that the end of the first sub-frame exposure (by a rolling shutter sequence) occurs at or near the midpoint of the composite full-frame exposure time (i.e., the sum of all sub-frame exposure times). For example, the ratio of the first and second sub-frame exposure times can very well be 1 :9, even for temporally non-contiguous sub-frame exposures. It is also noted that the (composite) full-frame exposure time 'Texp' can be less than the full-frame period, depending on the exposure settings of the image sensor. As shown in the preceding figures, a delayed reset signal can erase photocharge in the photoelement at the beginning of the first sub-frame exposure, which can be delayed relative to the beginning of the respective full-frame time interval.

[0079] Reference is now made to Figure 8 the low light illumination condition in FIG. 1 1. After the initial reset of the pixel photoelement (first vertical solid line at 0% of the frame exposure time Texp), the photocharge does not build up fast enough to be affected by the partial transfer pulse (second vertical solid line at about 50% of Texp) applied to the pixel transfer gate at the mid-exposure. As a result, when the pixel is switched to the low gain configuration during readout, none of the generated photocharge is transferred to the first and connected second charge accumulation elements (e.g., the floating diffusion node and the additional gain capacitance). The high gain channel is not used for the first sub-frame. After double sampling correlation (i.e., subtracting the pixel reset level from the pixel signal level prior to A / D conversion), the converted pixel data delivered at the output of the A / D conversion unit is thus zero. This value is unconditionally written to the pixel memory without any further processing by the PML.

[0080] At the end of the second subframe exposure, the reset levels of the low gain and high gain reset of the pixel are read out. Thereafter, after full transfer of photocharges from the photoelectric element to the first charge accumulation element of the charge-voltage converter has taken place (third vertical solid line at 100% of Texp), the signal level in the high gain configuration is read out. The conversion gain of the pixel is then switched to the low gain channel, and the signal level of the same photocharges present on the first and second charge accumulation elements at this time is determined. Relevant double sampling is applied to both the high gain channel and the low gain channel to eliminate kTC noise of the respective gain channel, and the A / D conversion unit converts the reset noise corrected signal levels (pixel data) for high gain and low gain to the digital domain. Note that all these readings take place in the same unit time slot and are performed in the following order: reset level for low gain, reset level for high gain, signal level for high gain, signal level for low gain.

[0081] Next, the PML compares the pixel data set relative to the high gain to the threshold value THG for high gain. If the pixel data is below the THG threshold, which is actually the case in the low light illumination conditions depicted in Figure 8 the high gain pixel data is stored in the pixel memory and replaces the data currently stored therein. However, if the high gain pixel data has exceeded the high gain threshold THG, the high gain pixel data can have been discarded and only the low gain data can have been taken into account. Thus, for low light conditions referred to in Figure 8 the near high gain pixel data is preserved and stored as final data to be output by the image sensor chip.

[0082] In an alternative embodiment of the present invention, instead of discarding the high gain pixel data obtained for the last subframe in case of exceeding the second threshold THG, it is possible to combine the high gain pixel data and the low gain data into a single output word. For example, a 13 bit wide data word for the low gain can be combined with a 14 bit wide data word for the high gain to obtain a single compressed output word, e.g. 16 bit wide. The level dependent compression algorithm internally amplifies the low gain signal to approximately match the high conversion gain. For very low illumination, only the high gain data is included in the compressed output word, while for very high illumination only the low gain data is included. In the intermediate region, e.g. the transition between very low and very high illumination, the number of bits of the high data retained in the compressed output data word decreases stepwise, while the number of significant bits of the low gain data in the output word increases stepwise. Furthermore, by replacing the shot noise dominated bits in the high gain data word with this difference information, the inevitable pixel-to-pixel and sensor-to-sensor variations between the ratio of high conversion gain and low conversion gain can be taken into account, resulting in a difference between the high gain sample and the low gain sample of each pixel.

[0083] The generation of the compressed output word has the advantage that the total I / O bandwidth (i.e. the amount of data to be transmitted by the chip) can be reduced, while for an enlarged illumination level range, corresponding low gain and high gain image data is still available. It is also advantageous for a smooth signal transition from high gain image data to low gain image data in the image frame reconstructed externally (e.g. off-chip), e.g. by applying a corresponding decompression operation to the compressed output word.

[0084] The threshold level THG can be relatively low compared to the full well capacity associated with the first charge accumulation element. After a noise contribution in the signal level of the pixel is partially transferred to be dominated by shot noise limitation, as few as tens or hundreds of electrons can be left in the photo element, the shot noise limitation makes the readout in the low gain configuration of the pixel reasonable for stronger signal levels, but promotes the readout in the high gain configuration of the pixel for weaker signal levels at the end of the final subframe exposure. The threshold THG can be externally provided by the user, programmed into the image sensor device by the user, or set to a default value and determines when high gain pixel data or low gain pixel data is used as output. It typically reflects a measured or expected amount of shot noise and possibly noise contributions forming variations of the transfer pulse amplitude, above which the application of a high readout gain compared to a low readout gain does not lead to a significantly more favorable signal-to-noise ratio. Finding a good value for the amplitude of the partial transfer pulse is typically a result of balancing two conflicting requirements: on the one hand, the integrated photo charge left in the photosensitive element after all partial transfers is preferably large enough to be dominated by inherent shot noise when the readout channel is switched to low gain, but on the other hand, not too much integrated photo charge should be left in the photosensitive element after each non-final subframe exposure in order not to unduly limit the FWC of the pixel during the subsequent subframe exposure.

[0085] Still referring to the low light illumination conditions depicted in Figure 8 In this alternative output format, the high gain pixel data is directly transmitted to the I / O circuitry of the sensor and thus not stored in the pixel memory. The low gain pixel data currently stored in the pixel memory (from the previous subframe exposure) is not replaced and can be used for its further processing, e.g. for combining with the low gain pixel data of the current (last) subframe exposure.

[0086] Turning now toFigure 9 In the case of the bright light illumination condition considered, it is noted that the photocharges generated up to the middle exposure, i.e. the end of the first subframe exposure, are affected by the partial transfer pulse delivered to the pixel transfer gate. As a result, when the pixel is configured to have a low conversion gain, after the reset level with respect to the first and second charge accumulation elements has been read out, a part of the photocharges generated so far is transferred to the charge accumulation elements and the induced signal voltage is read out. Again, the relevant double sampling is applied and the reset noise corrected signal level is converted by the A / D conversion unit. The digital values of the low gain pixel data obtained are written to the pixel memory.

[0087] At the end of the second subframe exposure, the complete transfer of the photocharges remaining in the optoelectronic element is achieved and the reset and signal levels of the pixel are read out, as in the low light condition shown in Figure 8 Moreover, the relevant double sampling is applied to obtain the high gain and low gain pixel data after A / D conversion. In the case of the bright light illumination condition depicted in Figure 9 When the high gain pixel data is compared to the high gain threshold THG by the PML, the THG threshold is exceeded, which leads to the decision to discard the high gain pixel data. Thus, the PML only takes the low gain pixel data into account. More specifically, the PML retrieves the low gain pixel data related to the preceding first subframe from the pixel memory, adds the retrieved preceding low gain pixel data to the currently provided low gain data and writes the addition result back to the pixel memory.

[0088] In an embodiment of the application, the threshold THG of the high gain channel and the threshold TLG of the low gain channel are preferably programmable values that can be changed by the user or determined according to the number of subframes in each full frame, the pulse amplitude of the partial charge transfer pulse applied to the transfer gate and the ratio between the individual subframe exposure times.

[0089] In an exemplary embodiment of the application, the FWC of the pixel associated with the high gain channel can be about 10 ke- and the FWC associated with the low gain channel can be about 40 ke-. The partial transfer pulse (TX) can be chosen to leave between 500 e- and 1000 e- photocharges in the photodiode of each pixel.

[0090] With regard to the over-saturated illumination condition, Figure 10The generated photocharge is illustrated to exceed the well capacity (upper horizontal line) associated with the photoelectric element, such as a PPD, which causes the excess photocharge to overflow from the photoelectric element into the charge well associated with at least the first charge accumulation element (e.g., the connected first and second charge accumulation elements) of the pixel's buffered charge-to-voltage converter. The presence of overflow charge on the first charge accumulation element, or the interconnected first and second charge accumulation elements, can be detected for each pixel prior to the delivery of a reset pulse at the end of each subframe exposure. Thus, the over-saturation regime of one or more pixels of the image sensor can be detected. For Figure 10 With reference to the illumination conditions referenced in the above, only the pixel row data associated with the low conversion gain setting of the pixel is used in all subframe exposures (corrected reset noise) because the full photocharge transfer from the saturated photoelectric element to the first charge accumulation element always results in the THG value being exceeded. Thus, for all subframe exposures comprising the full image frame, only the pixel data associated with the low conversion gain configuration of the pixel is added by the PML and the resulting partial or final sum can be optionally compressed prior to the intermediate or final data value being written to the pixel memory.

[0091] Optionally, in the above-described method, the overflow level of the photocharge receiving charge accumulation element is determined after each subframe exposure prior to the delivery of the reset pulse that marks the beginning of the sampling of the new signal and / or reset level. In addition, the determined pixel overflow level can be stored in an unoccupied and / or pixel memory location that is marked as invalid for external readout in addition to the pixel signal level and / or pixel reset level. The overflow level so obtained and stored for each pixel can also be processed by the PML. For example, the PML can acquire the accumulated overflow level from the pixel memory in relation to a previous subframe exposure and add it to the overflow determined in relation to a current subframe exposure (e.g., another intermediate subframe exposure or the final subframe exposure in the image frame). The accumulated overflow level after the final subframe exposure can then be added to the final low gain pixel data (e.g., the cumulative sum of the low gain pixel data over all subframe exposures in the image frame). Alternatively, in the event that overflow of photocharge is detected, the photoelectric element and charge accumulation element can simply be reset without processing and / or storing a specific overflow level. The detection of a photocharge overflow event can still be used to inform a user or internal components of the image sensor that the image frame was acquired in an overexposure condition.

[0092] While the embodiments of the present application preferably incorporate correlated double sampling to reduce reset noise, the methods of the present application do not rely on correlated double sampling and the signal level can be directly sampled without sampling a preceding reset level. In the event that the overflow level of the pixel is determined as described above, digital double sampling can be implemented to correct for the overflow level of residual reset noise present on the charge accumulation element(s) after the initial or each subsequent reset of the pixel in each image frame.

[0093] Furthermore, more than two subframe exposures can occur during a full frame interval of the image sensor. For example, at least three subframe exposures can occur during a full frame interval, with the first two consecutive subframe exposures being located in a middle portion relative to a cumulative exposure time across all subframes in one full frame. In such embodiments, the portion of the exposure time allocated to the second subframe exposure can be short enough to allow the image sensor to capture very bright signals. In the above exemplary embodiment, if the PPDs as photonic elements can store 40ke- (full well capacity - FWC), the FWC can effectively increase to about 80ke- (ignoring the determination of the overflow level, which is about 160ke- otherwise) over the duration of the cumulative (i.e. total or full) exposure time of an image frame comprising two substantially equal subframe exposures, provided that only a very small percentage of the integrated photocharge is retained in the PPDs after the partial transfer. In very bright lighting conditions, even an extended FWC of about 80ke- is not enough to prevent pixel output saturation. In contrast, assuming similar bright lighting conditions as in the case of the previous double exposure (i.e. exceeding the saturation level), if the second of three or more subframe exposures is chosen to be relatively short, for example about 1 / 8 of the full exposure time of the image frame, the pixel photocharge that can be collected during this relatively short second subframe exposure will be greater than about 80 / 8 = 10ke- and have an associated SNR greater than 40dB. Since the FWC does not reach the short second subframe exposure, it is possible to even further extend the DR of the image sensor by selecting the pixel data originating from the short subframe exposure as the only pixel data relevant for the output. More specifically, for a threshold level point of the photocharge integrated during the first subframe exposure, for example in the present example about 80*7 / 8 / 2 = 35ke- defines a low gain threshold level TLG for the digitized pixel data, above which a saturation condition is detected relative to the first or last subframe exposure. Thus, if the low gain threshold level TLG is exceeded, the pixel data belonging to the shortest subframe exposure period (i.e. in this case the second period) is selected as the relevant output data. The ratio of the subframe exposure periods can be precisely calculated in all embodiments of the present invention and thus the acquired pixel data can be linearized. Embodiments of the present invention in which a short intermediate subframe exposure is provided benefit from even higher DR values (e.g. a short exposure of about 1 / 8 of the full exposure time increases the DR by about 18dB).

[0094] Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. An image sensor provided as an integrated circuit, the image sensor comprising: First substrate; The second substrate is vertically stacked and electrically interconnected with the first substrate; and a pixel array disposed within the first substrate, the pixel array comprising a plurality of pixels logically organized in pixel rows and pixel columns, wherein the pixel array is organized into a plurality of pixel subarrays, and each pixel of the pixel array comprises: Optoelectronic components for integrating photocharge while being exposed to light during each of a plurality of subframe exposures included in a frame interval; A transfer gate for transferring at least a portion of the integrated photocharge present in the photoelectric element at the end of each subframe exposure to a buffered charge-to-voltage converter; The buffered charge-to-voltage converter is used to receive the transferred photocharge and convert it into a signal voltage. The buffered charge-to-voltage converter includes a first charge accumulation element, which is operatively connected to at least one second charge accumulation element via a gain switch to switch the conversion gain between a high conversion gain and a low conversion gain. The at least one second charge accumulation element is provided in the first substrate or the second substrate. A control circuit system adapted to trigger partial or complete transfer of the integrated photocharge in the photoelectric element of each pixel row by adjusting the amplitude of a transfer pulse applied to the transfer gate; and A readout circuit system is disposed on the second substrate, the readout circuit system comprising a plurality of readout blocks operating in parallel, each readout block being associated with a corresponding one of the plurality of pixel subarrays and configured to sequentially read out the pixel row data of the corresponding pixel subarray at the end of each subframe exposure; In at least one operating mode of the image sensor, the control circuit system is configured to: This triggers partial shifts in the exposure of all subframes except the last one, as well as a complete shift in the exposure of the last subframe of the multiple subframe exposures. Relative to each pixel subarray, temporal interleaving is associated with at least two rolling shutter control sequences for at least two temporally overlapping subframe exposures in the plurality of subframe exposures, and The gain switch (202-3) of each pixel row is controlled to operate the pixel with the low conversion gain for all sub-frame exposures except the last sub-frame exposure in the plurality of sub-frame exposures, and the pixel is first operated with the high conversion gain and then with the low conversion gain for the last sub-frame exposure in the plurality of sub-frame exposures. Each readout block of the readout circuitry system described herein includes, in the pipeline architecture: The A / D conversion unit is used to acquire and digitize the pixel row data of the corresponding pixel subframe. The pixel memory logic conditionally combines the digitized pixel row data with previously digitized pixel row data buffered in the pixel memory cell, and A pixel memory unit is used to buffer digitized pixel row data output by the pixel memory logic.

2. The image sensor as described in claim 1, characterized in that, The pixel memory logic of each readout block is configured such that if the previously digitized pixel row data relates to an earlier subframe exposure within the same frame interval, and -If the digitized pixel row data with respect to the low conversion gain in one of the intermediate subframes is below a first threshold level, or If the digitized pixel row data with respect to the low conversion gain in the last subframe is greater than the first threshold level, and the corresponding digitized pixel row data with respect to the high conversion gain in the last subframe exceeds the second threshold level, then the digitized pixel row data is combined with the previously digitized pixel row data buffered in the pixel memory unit.

3. The image sensor as described in claim 1, characterized in that, The pixel memory cells of the readout block are provided as an SRAM group.

4. The image sensor as described in claim 1, characterized in that, The total size of the pixel memory cells in the readout block is only allowed to store a portion of the full frame acquired using the pixel array.

5. The image sensor as described in claim 1, characterized in that, The pixel memory logic is further configured to process digitized pixel row data, wherein the processing of the digitized pixel row data through the pixel memory logic includes converting Gray-coded digitized pixel row data into binary-coded digitized pixel row data and / or scaling the digitized pixel row data by a scaling factor.

6. The image sensor as claimed in claim 1, characterized in that, The conditional combination of digitized pixel row data with digitized pixel row data buffered in the pixel memory unit includes adding or subtracting the currently obtained digitized pixel row data from the previously obtained digitized pixel row data stored in the pixel memory unit.

7. The image sensor as claimed in claim 1, characterized in that, The pipeline architecture of the readout block is configured to input and output pixel row data line by line, pixel by pixel, or pixel by pixel.

8. The image sensor as claimed in claim 1, characterized in that, The control circuitry is configured to select a row of pixels in each pixel subarray for readout before triggering a partial or complete transfer of the integrated photocharge, thereby enabling each readout block to detect and read out excess photocharge exceeding the full-well capacity associated with each photoelectro-element.

9. The image sensor as claimed in claim 1, characterized in that, The control circuitry is configured to reset the charge accumulation element of the charge-to-voltage converter in the pixel row before triggering a partial or complete transfer of the integrated photocharge, thereby enabling each readout block to detect the reset voltage of the pixel in the pixel row and apply a correlated double sample of the signal voltage of the pixel in the pixel row, wherein the signal voltage represents the partially or completely transferred photocharge.

10. The image sensor as claimed in claim 1, characterized in that, The pixel memory logic of each readout block is configured to combine and compress the digitized pixel row data associated with the high conversion gain and the low conversion gain into a single row of output data.

11. The image sensor as claimed in claim 1, characterized in that, The subframe exposure is continuous within the frame interval, and / or the subframe exposure is centered relative to the frame interval.

12. The image sensor as claimed in claim 1, characterized in that, The exposure times of the multiple subframes are substantially equal.

13. The image sensor as claimed in claim 1, characterized in that, The total frame exposure time can be programmed by adjusting the exposure time of one or more subframes, where the total frame exposure time is the sum of the exposure times of all subframes.

14. The image sensor as claimed in claim 1, characterized in that, The control circuit system includes multiple row drivers for driving pixel rows of each subarray, the multiple row drivers being configured to drive different subsets of pixels in each row.

15. A method of operating an integrated circuit image sensor comprising a pixel array, wherein the pixels of the pixel array are logically organized in pixel rows and pixel columns, and the pixel array is organized into a plurality of pixel subarrays, the method comprising: - Integrate photocharge into the pixels of the pixel array during each of the multiple sub-frame exposures included in the frame interval. - At the end of all sub-frame exposures except the last one, the integrated photocharge of the pixel is partially transferred to the corresponding charge-to-voltage converter of the pixel, wherein the charge-to-voltage converter is configured to apply a low conversion gain when reading out the partially transferred photocharge. - At the end of the last sub-frame exposure in the plurality of sub-frame exposures, the integrated photocharge of the pixel is completely transferred to the corresponding charge-to-voltage converter of the pixel, wherein the charge-to-voltage converter is configured to apply a high conversion gain when reading out the completely transferred photocharge and a subsequent low conversion gain. - At the end of each subframe exposure, the pixel row data of each subarray is read out sequentially. The sequential reading out of the pixel row data further includes the pipeline steps of sampling and digitizing the pixel row data, conditionally combining the digitized pixel row data with previously digitized pixel row data buffered in the pixel memory unit of the readout block, and buffering the digitized pixel row data in the pixel memory unit of the readout block. This involves parallel reading of pixel row data from different pixel subarrays, and For each pixel subarray, at least two of the plurality of subframe exposures overlap in time, and the rolling shutter control sequence associated with the at least two subframe exposures is temporally interleaved.

16. The method as described in claim 15, characterized in that, The conditional combination of digitized pixel row data with previously digitized pixel row data buffered in the pixel memory unit includes: If the previously digitized pixel row data is related to an earlier subframe exposure within the same frame interval, then the low conversion gain readout of the digitized pixel row data is compared with a first threshold level, and If the low conversion gain readout of the digitized pixel row data of the last subframe is lower than the first threshold level, then the corresponding high conversion gain readout of the digitized pixel row data is compared with the second threshold level.

Images