Imaging system with rolling shutter readout and electronic shutter

Abstract

The invention discloses an imaging system with a rolling shutter readout and an electronic shutter. The system may include an image sensor, an electronic shutter, a light source, and a lens module. The imaging system may include a control circuit configured to selectively control one or more of the image sensor, the electronic shutter, and the light source. In particular, the electronic shutter may be controlled in synchronization with the image sensor. The electronic shutter may have a maximum transparency during a common time interval for all rows when imaging pixels of all rows in the image sensor are integrating. The electronic shutter may have a minimum transparency when more than but less than all of the multiple rows of imaging pixels are integrating. In this way, the electronic shutter is synchronized with the image sensor, and a rolling shutter image sensor can be used to achieve global shutter-like performance.

Description

BACKGROUND OF THE INVENTION

[0001] Image sensors are often used in electronic devices such as mobile phones, cameras, and computers to capture images. In a typical arrangement, an image sensor includes an array of imaging pixels arranged in rows and columns of pixels. Circuits can be coupled to each column of pixels to read out image signals from the image pixels.

[0002] The embodiments described herein arise in this context. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A is a schematic diagram of an exemplary system having an image sensor in accordance with some embodiments.

[0004] Figure 1B is a schematic diagram of an exemplary vehicle having an imaging system in accordance with some embodiments.

[0005] Figure 2 is a schematic diagram of an exemplary pixel array and associated row and column control circuits for reading out image signals from an image sensor in accordance with some embodiments.

[0006] Figure 3 is a circuit diagram of an exemplary imaging pixel in accordance with some embodiments.

[0007] Figure 4A is a cross-sectional side view of an imaging system having an electronic shutter interposed between an image sensor and a lens module in accordance with some embodiments.

[0008] Figure 4B is a cross-sectional side view of an imaging system having a lens module interposed between an image sensor and an electronic shutter in accordance with some embodiments.

[0009] Figure 4C is a cross-sectional side view of an imaging system having an electronic shutter that forms a front cover of an image sensor package in accordance with some embodiments.

[0010] Figure 5 is a schematic diagram showing an exemplary rolling shutter image sensor timing sequence with staggered reset that operates in synchronization with an electronic shutter and a light source to achieve global shutter performance in accordance with some embodiments.

[0011] Figure 6 is a schematic diagram showing an exemplary rolling shutter image sensor timing sequence with global reset that operates in synchronization with an electronic shutter and a light source to achieve global shutter performance in accordance with some embodiments.

[0012] Figure 7FIG. 0 is a schematic diagram showing an exemplary rolling shutter image sensor timing sequence with a global reset and delay period that operates in synchronization with an electronic shutter and a light source to achieve global shutter performance in accordance with some embodiments.

[0013] Figure 8 FIG. 4 is a diagram showing an exemplary rolling shutter image sensor timing sequence with a common integration period that is greater than the integration time of one or more rows in accordance with some embodiments. DETAILED DESCRIPTION

[0014] Embodiments of the present inventive technique relate to image sensors. Those skilled in the art will appreciate that the exemplary embodiments of the present invention may be practiced without some or all of these specific details. In other instances, well-known operations have not been described in detail to avoid unnecessarily obscuring the embodiments of the present invention.

[0015] Electronic devices such as digital cameras, computers, cellular phones, and other electronic devices may include an image sensor that collects incoming light to capture an image. The image sensor may include a pixel array. The pixels in the image sensor may include photosensitive elements, such as photodiodes, that convert incoming light into an image signal. The image sensor may have any number (e.g., hundreds or thousands or more) of pixels. A typical image sensor may, for example, have hundreds or thousands or millions of pixels (e.g., megapixels). The image sensor may include control circuitry (such as circuitry for operating the pixels) and readout circuitry for reading out an image signal corresponding to the charge or current generated by the photosensitive elements.

[0016] Figure 1A FIG. 16 is a schematic diagram of an exemplary system that includes an imaging system that uses an image sensor to capture an image. Figure 1A The system 100 of FIG. 18 can be an electronic device such as a camera, cellular phone, video camera, or other electronic device that captures digital image data, can be a vehicle safety system (e.g., an active braking system or other vehicle safety system), or can be a surveillance system.

[0017] As Figure 1A shown in FIG. 23, the system 100 may include an imaging system (such as imaging system 10) and a host subsystem (such as host subsystem 20). The imaging system 10 may include a camera module 12. The camera module 12 may include one or more image sensors 14, such as in an image sensor array integrated circuit, and one or more lenses.

[0018] During an image capture operation, each lens may focus light onto an associated image sensor 14. The image sensor 14 may include photosensitive elements (i.e., image sensor pixels) that convert light into analog data. The image sensor may have any number (e.g., hundreds, thousands, millions, or more) of pixels. A typical image sensor may have, for example, millions of pixels (e.g., megapixels).

[0019] Each image sensor in the camera module 12 may be the same, or different types of image sensors may be present in a given image sensor array integrated circuit. In some examples, the image sensor 14 may also include a bias circuit (e.g., source follower load circuit), a sample and hold circuit, a correlated double sampling (CDS) circuit, an amplifier circuit, an analog-to-digital converter circuit, a data output circuit, a memory (e.g., buffer circuit), and / or an addressing circuit.

[0020] Static image data and video image data from the sensor 14 may be provided via path 28 to the image processing and data formatting circuit 16. The image processing and data formatting circuit 16 may be used to perform image processing functions such as data formatting, adjusting white balance and exposure, implementing video image stabilization, or face detection. The image processing and data formatting circuit 16 may additionally or alternatively be used to compress the raw camera image file (e.g., compressed into Joint Photographic Experts Group or JPEG format) when needed.

[0021] In one exemplary arrangement, such as a system-on-chip (SoC) arrangement, the sensor 14 and the image processing and data formatting circuit 16 are implemented on a common semiconductor substrate (e.g., a common silicon image sensor integrated circuit die). If desired, the sensor 14 and the image processing circuit 16 may be formed on separate semiconductor substrates. For example, the sensor 14 and the image processing circuit 16 may be formed on separate substrates that have been stacked.

[0022] The imaging system 10 may transfer the acquired image data to the host subsystem 20 via path 18. The host subsystem 20 may include an input-output device 22 and a storage and processing circuit 24. The host subsystem 20 may include processing software that is used to detect objects in the image, detect the movement of an object between image frames, determine the distance to an object in the image, or filter or otherwise process the image provided by the imaging system 10. For example, the image processing and data formatting circuit 16 of the imaging system 10 may transfer the acquired image data to the storage and processing circuit 24 of the host subsystem 20.

[0023] If desired, system 100 can provide a number of advanced functions for the user. For example, in a computer or mobile phone, the ability to run user applications can be provided for the user. For these functions, the input-output device 22 of host subsystem 20 can include a keyboard, input-output ports, buttons, light sources, and a display. The storage and processing circuitry 24 of host subsystem 20 can include volatile memory and / or non-volatile memory (e.g., random access memory, flash memory, hard disk drive, solid state drive, etc.). The storage and processing circuitry 24 can additionally or alternatively include a microprocessor, a microcontroller, a digital signal processor, and / or an application specific integrated circuit.

[0024] As shown in FIG. 1, the input-output device 22 can optionally include a light source 30. The light source 30 can be a visible light source, an infrared light source, or any other desired type of light source. The light source can be controlled to illuminate the scene captured by the image sensor 14. The control circuitry within the imaging system 10 can control the light source 30 synchronously with the image sensor 14. The light source 30 can include a laser, a light emitting diode, or other desired type of light source.

[0025] Figure 1B An illustrative example of a vehicle 8 such as an automobile is shown. As Figure 1B shown in the illustrative example, the automobile 8 can include one or more imaging systems 10. The imaging system can form part of a vehicle safety system 100, as described above. The imaging system 10 can be an imaging system with dedicated image capture and / or image processing functions. If desired, the imaging system 10 can perform some or all of the image processing functions associated with a given driver assistance operation. A dedicated driver assistance processor (e.g., Figure 1A the processing circuitry 24 in

[0026] In another suitable example, the imaging system 10 can perform only a part of the image processing operations associated with a given driver assistance function or no image processing operations at all. For example, the imaging system 10 can only capture an image of the environment around the vehicle 8 and transmit the image data to the processing circuitry 24 for further processing. Such an arrangement can be used for vehicle safety system functions that require a large amount of processing power and storage (e.g., full-frame buffering and processing of the captured images).

[0027] In Figure 1B the illustrative example, a first imaging system 10 is shown mounted on the front of the automobile 8 (e.g., to capture an image of the surrounding environment in front of the automobile), and a second imaging system 10 is shown mounted inside the automobile 8 (e.g., to capture an image of the driver of the vehicle). If desired, the imaging system 10 can be mounted at the rear end of the vehicle 8 (i.e., opposite to Figure 1Bopposite vehicle ends where the first imaging system 10 is installed). An imaging system at the rear end of the vehicle can capture images of the surrounding environment behind the vehicle. These examples are merely illustrative. One or more imaging systems 10 can be installed at any desired location on or within the vehicle 8.

[0028] Figure 2 is shown in Figure 1A an example of the arrangement of the image sensor 14 in the camera module 12. As Figure 2 shown, the camera module 12 can include control and processing circuitry 44. The control and processing circuitry 44 can sometimes be referred to as part of the image sensor 14 and / or part of the imaging system 100. The control and processing circuitry 44 (sometimes referred to as control and processing logic) can be part of the image processing and data formatting circuitry 16 in FIG. 1 or can be separate from the circuitry 16. The image sensor 14 can include a pixel array, such as an array 32 of pixels 34 (sometimes referred to herein as image sensor pixels, imaging pixels, or image pixels). The control and processing circuitry 44 can be coupled to the row control circuitry 40 via a control path 27 and can be coupled to the column control and readout circuitry 42 via a data path 26.

[0029] The row control circuitry 40 can receive row addresses from the control and processing circuitry 44 and can provide corresponding row control signals to the imaging pixels 34 via one or more control paths 36. The row control signals can include pixel reset control signals, charge transfer control signals, overflow control signals, row selection control signals, dual conversion gain control signals, or any other desired pixel control signals.

[0030] The column control and readout circuitry 42 can be coupled to one or more columns of the pixel array 32 via one or more conductors (such as column lines 38). A given column line 38 can be coupled to a column of imaging pixels 34 in the imaging pixel array 32 and can be used to read out image signals from the imaging pixels 34 and to provide bias signals (e.g., bias current or bias voltage) to the imaging pixels 34. In some examples, each pixel column can be coupled to a corresponding column line 38. For an imaging pixel readout operation, the row driver circuitry 40 can be used to select a row of pixels in the imaging pixel array 32, and the image data associated with the imaging pixels 34 of that pixel row can be read out by the column readout circuitry 42 on the column lines 38. The column readout circuitry 42 can include column circuitry, such as column amplifiers for amplifying the signals read out from the array 32, sample and hold circuitry for sampling and storing the signals read out from the array 32, analog-to-digital converter circuitry for converting the read out analog signals to corresponding digital signals, or column memories for storing the read out signals and any other desired data. The column control and readout circuitry 42 can output digital pixel readout values to the control and processing logic 44 via the line 26.

[0031] The array 32 can have any number of rows and columns. Generally, the size of the array 32 and the number of rows and columns in the array 32 will depend on the specific implementation of the image sensor 14. Although rows and columns are typically described herein as horizontal and vertical, respectively, rows and columns can refer to any grid-like structure. Features described herein as rows can be arranged vertically, and features described herein as columns can be arranged horizontally.

[0032] The pixel array 32 can include a color filter array having a plurality of color filter elements that allows a single image sensor to sample light of different colors. For example, image sensor pixels (such as the imaging pixels in the array 32) can include a color filter array that allows a single image sensor to sample red, green, and blue (RGB) light using corresponding red, green, and blue image sensor pixels. The red, green, and blue image sensor pixels can be arranged in a Bayer mosaic pattern. The Bayer mosaic pattern consists of repeating cells of 2×2 imaging pixels, where two green imaging pixels are opposite each other along a diagonal, and adjacent to a red imaging pixel that is opposite a blue imaging pixel along a diagonal. As another example, broadband imaging pixels having broadband filter elements (e.g., transparent filter elements) can be used in place of the green pixels in the Bayer pattern. These examples are merely illustrative, and generally, color filter elements of any desired color (e.g., cyan, yellow, red, green, blue, etc.) and any desired pattern can be formed on any desired number of imaging pixels 34.

[0033] The imaging pixels 34 in the pixel array 32 can include infrared imaging pixels. The infrared pixels are formed by a color filter that allows infrared radiation and light in other bands to pass through.

[0034] In some embodiments, the array 32 can be part of a stacked die arrangement, where the pixels 34 of the array 32 are divided between two or more stacked substrates. In such an arrangement, each pixel 34 in the array 32 can be divided between two dies (sometimes referred to as chips) at any desired node within the pixel. Generally, the array 32, the row control circuit 40, and the column control and readout circuit 42 can be divided between two or more stacked substrates.

[0035] Figure 3 is a circuit diagram of an imaging pixel having photosensitive elements. In the present application, each transistor is depicted as having three terminals: a source, a drain, and a gate. The source terminal and the drain terminal of each transistor can vary depending on the way the transistor is biased and the type of transistor used. For simplicity, the source and drain terminals are referred to herein as source-drain terminals or simply terminals.

[0036] As Figure 3As shown, the image pixel 34 includes a photosensitive element 102 (e.g., a photodiode) that generates charge in response to incident light. The photosensitive element 102 has a first terminal coupled to ground. The photosensitive element 102 can be a pinned photodiode having a corresponding pinned voltage. A second terminal of the photosensitive element 102 is coupled to a transfer transistor 104. The transfer transistor 104 is coupled to a floating diffusion (FD) region 118. The floating diffusion region 118 can be a doped semiconductor region (e.g., a region doped in a silicon substrate by ion implantation, impurity diffusion, or other doping processes). The floating diffusion 118 has an associated capacitance.

[0037] A reset transistor 106 can be coupled between the floating diffusion region 118 and a voltage source 124-1. Herein, the voltage source can sometimes alternatively be referred to as a voltage source terminal, a bias voltage source terminal, a bias voltage source, etc. The voltage source 124-1 can provide a supply voltage VDD. A source follower transistor 112 (SF) has a gate terminal that is coupled to the floating diffusion region 118 and a first terminal of the reset transistor 106. The source follower transistor 112 also has a first source-drain terminal coupled to a voltage source 124-2. The voltage source 124-2 can provide a supply voltage VDD. The voltage source 124-2 can provide the same supply voltage as the voltage source 124-1 or a different supply voltage from the voltage source 124-1.

[0038] A second source-drain terminal of the source follower transistor 112 is coupled to a transistor 132. The transistor 132 (sometimes referred to as a row select transistor) is interposed between the source follower transistor 112 and a column output line 38. When the row select transistor 132 is asserted, an output voltage (PIXOUT) is provided on the column output line 38. The source follower transistor 112, the row select transistor 132, and the column output line 38 can sometimes be collectively referred to as a readout circuit or a readout circuitry. Other readout circuits can be used if desired.

[0039] A gate terminal of the transfer transistor 104 receives a control signal TX. A gate terminal of the reset transistor 106 receives a control signal RST. A gate terminal of the row select transistor 132 receives a control signal RS. The control signals TX, RST, and RS can be provided by a row control circuit (e.g., Figure 2 the row control circuit 40 therein) via a control path (e.g., Figure 2 the control path 36 therein). If desired, the voltages at the power supplies 124-1 and 124-2 can also be provided by the row control circuit.

[0040] In Figure 3During operation of the imaging pixel 34, transistors 104 and 106 may become active to reset the photodiode 102 and the floating diffusion region 118 to a reset voltage. Resetting the photodiode 102 and the floating diffusion region 118 in this manner may be referred to as a reset period or a reset operation.

[0041] After the reset operation is completed, an integration period may begin, during which the photodiode 102 generates charge in response to incident light. During the integration period, charge accumulates in the photodiode. At the end of the integration period, transistor 104 may become active to then transfer the accumulated charge from the photodiode 102 to the floating diffusion region 118. Then, the row select transistor 132 may become active to read out (sometimes referred to as sample) the charge on the floating diffusion region 118. The process of transferring charge from the photodiode 102 to the floating diffusion region 118 and then sampling the charge by activating the row select transistor 132 may sometimes be collectively referred to as a readout operation or a readout period.

[0042] The example of the structure and operation of the imaging pixel 34 is merely illustrative. The imaging pixel may optionally include one or more additional transistors. For example, the imaging pixel may include an anti-blooming transistor configured to reset the photodiode 102 without resetting the floating diffusion region 118, a dual conversion gain transistor configured to adjust the conversion gain associated with the imaging pixel, and the like. The imaging pixel may also include one or more additional charge storage regions (e.g., storage capacitors, storage transistors, floating diffusion regions, etc.). Generally speaking, the imaging pixel 34 may have any desired structure.

[0043] The imaging system 10 may optionally include an electronic shutter. The electronic shutter may be used to achieve global shutter type performance in a rolling shutter image sensor.

[0044] Figures 4A to 4C is a side view of an illustrative imaging system having a light source and an electronic shutter. Generally speaking, the electronic shutter 200 may be incorporated at any desired location in the imaging system. As Figure 4A shown, the electronic shutter 200 may be incorporated between the image sensor 14 and the lens module 202 in the imaging system 10. The lens module 202 may include one or more lenses that focus light onto the image sensor 14. One or more lenses in the lens module 202 may be controlled by one or more corresponding actuators that physically move the lenses.

[0045] Figure 4A The example where the electronic shutter 200 in is located between the image sensor 14 and the lens module 202 is merely illustrative. In Figure 4B Another possible arrangement shown, the lens module 202 is interposed between the image sensor 14 and the electronic shutter 200.

[0046] The electronic shutter 200 may be separated from the image sensor 14 by an air gap (as shown in the example of Figure 4A ). In other words, the image sensor 14 may have a transparent encapsulation cover separated from the electronic shutter 200 by an air gap. Alternatively, as shown in Figure 4C , the electronic shutter may be directly integrated into the image sensor package. For example, the electronic shutter may serve as the encapsulation cover of the image sensor package. The electronic shutter 200 may be formed above the image sensor chip 122.

[0047] Figures 4A to 4C Each shows how the imaging system 10 may include a light source 30. The light source 30 may emit light (e.g., infrared light, visible light, etc.) in the direction 204 to illuminate the scene captured by the image sensor 14. Figures 4A to 4C Each also shows how a control circuit such as the control and processing circuit 44 may be used to control one or more of the light source 30, the image sensor 14, and the electronic shutter 200. The control and processing circuit 44 may synchronize the operations of the light source 30, the image sensor 14, and / or the electronic shutter 200.

[0048] The electronic shutter may be implemented with a liquid crystal shutter. This example is merely illustrative. Generally speaking, any material with controllable optical properties (e.g., responsive to an electric field) may be used to implement the electronic shutter. As an example, the electronic shutter 200 may have any arrangement in the U.S. Non - Provisional Patent Application No. 17 / 643,714, filed on December 10, 2021, which is hereby incorporated by reference in its entirety.

[0049] The electronic shutter may have two states: a first state with maximum transparency (sometimes called the open state) and a second state with minimum transparency (sometimes called the closed state). It is desirable that the maximum transparency be as high as possible (e.g., as close to 100% as possible) and the minimum transparency be as low as possible (e.g., as close to 0% as possible).

[0050] In some cases, it is desirable to have an image sensor that operates in the global shutter mode. In the global shutter mode, all imaging pixels are exposed to incident light during a simultaneous integration time.

[0051] Two possible techniques for implementing a global shutter image sensor include using pixels with charge storage and using pixels with voltage storage.

[0052] In pixels based on charge storage elements, all pixels start integrating simultaneously and stop integrating simultaneously. At the end of the integration time, the integrated charge is transferred to a charge storage area (also referred to as a "memory node") within the imaging pixel. Subsequently, the integrated charge can be read out row by row from the charge storage area (e.g., in a rolling shutter readout). Thus, global shutter operation based on charge storage requires an additional charge storage area in each imaging pixel 34. This can undesirably increase the cost and complexity of the imaging system 10 and / or decrease the pixel fill factor (which is the percentage of the pixel area dedicated to the conversion of light particles to charge). Additionally, the undesired sensitivity of the storage elements to light can result in stray light artifacts in the captured image.

[0053] Global shutter pixels based on voltage storage have a relatively high background noise because correlated double sampling is not possible for this configuration.

[0054] To mitigate the cost and complexity in the imaging system 10, the imaging system 10 can use an electronic shutter such as the electronic shutter 200 to achieve a global shutter-like operation with a rolling shutter image sensor. An example of this type is shown in Figures 5 to 7 in.

[0055] Figure 2 The image sensor 14 in can be a rolling shutter image sensor, and Figure 2 and Figure 3 the imaging pixels 34 in can be rolling shutter pixels. A given column output line can be shared among a column of imaging pixels in the pixel array 32. Thus, data from only one row can be read out using a given column line at a time. Therefore, the imaging pixels can be read out row by row in a rolling shutter readout operation.

[0056] Figure 5 shows a first timeline 402 for the operation of each row of imaging pixels 34 in the pixel array 32 (e.g., Figure 2 and Figure 3 the type of imaging pixel shown), a second timeline 404 for the operation associated with the image sensor 14 (e.g., Figure 1A and Figure 2 the type of image sensor shown), a third timeline 406 for the operation associated with the electronic shutter 200 (e.g., Figures 4A to 4C the type of electronic shutter shown), and a fourth timeline 408 for the operation associated with the light source 30 (e.g., Figure 1A and Figures 4A to 4C the type of light source shown).

[0057] Figure 5Two frames (sometimes referred to as imaging frames) of the image sensor 14 are shown (frame N and frame N+1). The timing of the operation of the image sensor can be repeated for each frame. The operation for a single frame will be discussed below, but it should be understood that these operations are then repeated for each subsequent frame.

[0058] As Figure 5 shown, each frame begins with an interleaved reset period 410. During the interleaved reset period, a reset operation 412 is performed on each row of image pixels in the pixel array. The reset operations can be interleaved (e.g., the reset operation for a given row begins after the reset operation for the previous row). During the reset operation 412, the control signals RST and TX can be pulsed simultaneously to activate the transistors 106 and 104, respectively. This causes the photodiode and the floating diffusion region to be reset. This is a preferred rolling shutter operation. This example is merely illustrative. For imaging pixels having a structure different from that of Figure 3 the structure shown, other transistors can be selectively activated to reset the photodiode and / or the floating diffusion region of the imaging pixel.

[0059] The interleaved reset period can end at t2. For each row of imaging pixels, an integration time 414 can begin after the reset operation 412 for that row ends. During the integration time, charge can accumulate in the photodiode 102 of each imaging pixel 34. Each row can have a corresponding integration time that ends during the readout operation for the corresponding readout period 420 of that pixel row. The readout period 420 for a given pixel row can include transferring charge from the photodiode 102 to the floating diffusion region 118 (e.g., by activating the transfer transistor 104), and then sampling the charge by activating the row selection transistor 132. This example is merely illustrative. For imaging pixels having a structure different from that of Figure 3 the structure shown, other transistors can be selectively activated to read out image data from the imaging pixel 34.

[0060] Figure 5 A rolling shutter (RS) readout period 418 between t3 and t4 of the image sensor 14 is shown. The readout period 420 for each row of pixels can begin at the end of the readout period for the previous row of pixels. As Figure 5 shown, row 1 can have a corresponding integration time T INT1 1, row 2 can have a corresponding integration time T INT2 2, etc. The reset period 412 and the readout period 420 can be interleaved in a synchronous manner such that the integration time for each row is equal (e.g., T INT1 1 = T INT2 2 = T INT3 3, etc.). At the end of the RS readout period 418 and the subsequent delay period 419, the reset period for the subsequent frame can begin at t5.

[0061] In Figure 5 , the integration time of each row of imaging pixels in the image sensor 14 is the same duration, but not concurrent. Without a shutter, if there are moving objects in the scene and / or if the camera moves during image capture, this can result in motion artifacts. To simulate global shutter operation, an electronic shutter 200 may be included in the system and controlled synchronously with the image sensor 14.

[0062] Timeline 404 shows how the timing of the image sensor 14 is designed to have a common integration period 416 (T_INT_COMMON), during which all pixel rows are integrated.

[0063] The electronic shutter 200 can be controlled to be open (e.g., with maximum transparency) at least during the integration period 416. The electronic shutter is open between t2 and t3 while all pixel rows are integrating. In Figure 5 example, the electronic shutter is opened at time t2' before T_INT_COMMON starts at t2. Similarly, the electronic shutter is closed at time t3' after T_INT_COMMON ends at t3. Opening the shutter before t2 and after t3 can result in darker rows at the top and bottom of the array, which may be acceptable in some applications. The advantage of opening the shutter before t2 and after t3 is to reduce artifacts caused by the transition times of the shutter and the light source.

[0064] When the electronic shutter is closed (e.g., before t2', between t3' and t6', etc.), the electronic shutter can block incident light from reaching the image sensor. Synchronously opening the electronic shutter 200 during the integration period 416 enables the image sensor 14 to have a performance similar to that of a global shutter. When all pixel rows are integrated concurrently, the imaging data sampled during the RS readout period 418 is controlled by the charge accumulated during the integration period 416. The sampled data for a given row may include a dark signal component caused by charge generated outside T_INT_COMMON (e.g., between the reset operation 412 and the start of T_INT_COMMON or between the end of T_INT_COMMON and the readout operation 420), as well as a weak light signal component in the case where the shutter transmittance in the closed state is not absolutely zero.

[0065] A light source 30 may also be included in the system and controlled synchronously with the image sensor and the electronic shutter, especially in cases where the light signal generated by the ambient light illumination level may not be high enough to achieve the target signal-to-noise ratio. As Figure 5 shown, the light source 30 can be controlled to turn on when the electronic shutter is open and turn off when the electronic shutter is closed. Turning off the light source can prevent unwanted light generation.

[0066] In summary, in Figure 5 the electronic shutter 200 and the light source 30 can be controlled synchronously with the image sensor 14 such that during the period (e.g., integration period 416) in which all pixel rows are integrally accumulated concurrently, the electronic shutter 200 is opened (e.g., with maximum transparency) and the light source 30 is turned on. The electronic shutter 200 and the light source 30 can be controlled synchronously with the image sensor 14 such that during at least some of the interleaved reset period 410 and the RS read period 418, the electronic shutter 200 is closed (e.g., with minimum transparency) and the light source 30 is turned off.

[0067] Some electronic shutters and / or light sources may have timing limitations, such as limitations attributable to rise and fall times, operating frequency ranges, and duty cycle range requirements. To accommodate the timing limitations of the electronic shutter, light source, image sensor, and / or application, a delay period may optionally be added to each frame. Figure 5 An example in which each frame includes a delay period between t4 and t5 is shown. The delay period can have any desired duration. In some cases, the duration of the delay period can be 0.

[0068] Figure 6 A first timeline 302 showing the operation for each row of imaging pixels 34 (e.g., Figure 2 and Figure 3 imaging pixels of the type shown) in the pixel array 32 is shown. This rolling shutter operation mode is sometimes referred to as global reset release (GRR) because, unlike standard rolling shutter operation, in this mode, the reset periods for all rows and thus for all imaging pixels 34 start and end simultaneously, i.e., the reset is “global”. Figure 6 A second timeline 304 showing the operation associated with the image sensor 14 (e.g., Figure 1A and Figure 2 image sensors of the type shown), a third timeline 306 showing the operation associated with the electronic shutter 200 (e.g., Figures 4A to 4C electronic shutters of the type shown), and a fourth timeline 308 showing the operation associated with the light source 30 (e.g., Figure 1A and Figures 4A to 4C light sources of the type shown) are also shown.

[0069] Figure 6 Two frames (sometimes referred to as imaging frames) (frame N and frame N+1) of the image sensor 14 are shown. The timing of the operation of the image sensor can be repeated for each frame. The operation for a single frame will be discussed below, but it should be understood that these operations are then repeated for each subsequent frame.

[0070] As Figure 6As shown, each row in the image sensor has a concurrent reset period 310 (sometimes referred to as a global reset period) that begins at t1. During the global reset period, the control signals RST and TX for all pixels can be pulsed simultaneously to activate transistors 106 and 104, respectively. This causes the photodiodes and floating diffusion regions of all imaging pixels to be reset. This example is merely illustrative. For imaging pixels having a structure different from that of Figure 3 other transistors can be selectively activated to reset the photodiodes and / or floating diffusion regions of the imaging pixels.

[0071] The reset period for each row of imaging pixels can occur between t1 and t2. For each row of imaging pixels, the integration time 314 can begin after the reset operation for that row is completed. In Figure 6 each row of pixels can begin the integration time that starts at t2. During the integration time, charge can accumulate in the photodiode 102 of each imaging pixel 34. Each row can have a corresponding integration time that ends when the readout operation is performed during the corresponding readout period 318 for that pixel row. The readout period 318 for a given pixel row can include transferring the charge from the photodiode 102 to the floating diffusion region 118 (e.g., by activating the transfer transistor 104), and then sampling the charge by activating the row selection transistor 132. This example is merely illustrative. For imaging pixels having a structure different from that of Figure 3 other transistors can be selectively activated to read out the image data from the imaging pixel 34.

[0072] Figure 6 Shown is the rolling shutter (RS) readout period 316 between t3 and t4 of the image sensor 14. During the RS readout period 316, the readout period 318 for each row of pixels can begin at the end of the readout period for the previous row of pixels. As Figure 6 shown, row 1 can have a corresponding integration time T INT1 , row 2 can have a corresponding integration time T INT2 and so on. Thus, the integration time for each row is longer than the integration time for the previous row (e.g., T INT1 < T INT2 < T INT3 and so on). At the end of the RS readout period 316, the reset period for the subsequent frame can begin at t4.

[0073] Due to the global reset period 310 and the row-by-row rolling shutter readout performed during the RS readout period 316, the integration time of each row of imaging pixels in the image sensor 14 is different. In the absence of an electronic shutter, this can manifest as image brightness variations as the row order in the image increases (which corresponds to an increase in integration time) and / or motion artifacts in the presence of moving objects in the scene. The electronic shutter 200, which is controlled synchronously with the image sensor 14, can be used with the rolling shutter image sensor to simulate global shutter operation.

[0074] Timeline 304 shows how the image sensor 14 can have a common integration period 312 (T_INT_COMMON) during which all pixel rows are integrated. The electronic shutter 200 can be controlled to be open (e.g., at maximum transparency) during the common integration period 312. Figure 6 Shows how the electronic shutter 200 is open between t2 and t3 while all pixel rows are integrating. Once the common integration period ends at t3, the electronic shutter can be closed (e.g., set to minimum transparency) to block incident light from reaching the image sensor while some rows are still integrating. Figure 6 Shows how the electronic shutter can be closed between t3 and t5 (when the shutter is opened for the next frame's common integration period). Blocking the incident light by closing the electronic shutter at t3 simultaneously ends the effective integration period for all rows, even though the integration period can nominally last different lengths of time (per row) when a rolling shutter readout period occurs between t3 and t4.

[0075] Synchronizing the electronic shutter 200 to be open during the integration period 312 enables the image sensor 14 to have performance similar to a global shutter without including any additional charge storage areas in the imaging pixels 34 or the circuitry required to implement voltage storage. When all pixel rows are integrating concurrently (and the electronic shutter is open), the imaging data sampled during the RS readout period 316 is controlled by the charge accumulated during the integration period 312.

[0076] A light source 30 can also be added to the system and controlled synchronously with the image sensor and the electronic shutter, especially in cases where the light signal generated by the ambient light illumination level may not be high enough to achieve the target signal-to-noise ratio. As Figure 6 shown, the light source 30 can be controlled to be turned on while all rows are integrating during the common integration time 312. Once the common integration period 312 ends at t3, the light source can be turned off to prevent unwanted light generation. Figure 6 Shows how the light source can be turned off between t3 and t5 (when the light source is turned on for the next frame's common integration period).

[0077] In summary, the electronic shutter 200 and the light source 30 can be controlled synchronously with the image sensor 14 such that the electronic shutter 200 is open (e.g., at maximum transparency) and the light source 30 is turned on during a common time period for integration of all pixel rows (e.g., integration period 312). The electronic shutter 200 and the light source 30 can be controlled synchronously with the image sensor 14 such that the electronic shutter 200 is closed (e.g., at minimum transparency) and the light source 30 is turned off during time periods outside of the common integration period 312 (e.g., during RS readout 316 where imaging pixels are sampled row by row).

[0078] Some electronic shutters, light sources, image sensors, and / or applications may have timing constraints such as those due to rise and fall times, operating frequency ranges, and duty cycle range requirements. To accommodate the timing constraints, a delay period can optionally be added to each frame. Figure 7 An example where each frame includes a delay period is shown.

[0079] Figure 7 An example is shown having a first timeline 302 for the operation of each row of imaging pixels 34 in the pixel array 32 (e.g., Figure 2 and Figure 3 imaging pixels of the type shown), a second timeline 304 for the operation associated with the image sensor 14 (e.g., Figure 1A and Figure 2 image sensors of the type shown), a third timeline 306 for the operation associated with the electronic shutter 200 (e.g., Figures 4A to 4C electronic shutters of the type shown), and a fourth timeline 308 for the operation associated with the light source 30 (e.g., Figure 1A and Figures 4A to 4C light sources of the type shown). Figure 7 Many of the timings and operations shown are the same as in Figure 6 and will not be repeated herein for simplicity.

[0080] Figure 7 An example of how each frame can include a delay period 320 is shown. The delay period 320 can occur after the RS period 316 in each frame (as one example). Figure 7 An example is shown of an electronic shutter that is closed between t3 and t9, open between t9 and t7, closed between t7 and t 10 etc. The light source is controlled synchronously with the electronic shutter. Thus, the light source is turned off between t3 and t9, turned on between t9 and t7, turned off between t7 and t 10 etc.

[0081] The state of the electronic shutter during the delay period 320 does not affect the image sensor performance because any charge integrated during the delay period 320 is discarded during the global readout period 310. Thus, the electronic shutter can be in the open state during at least some of the delay periods 320.

[0082] It should be understood that the timing shown herein is merely illustrative and, if desired, the timing of various transitions can be tuned. For example, the electronic shutter 200 can optionally be opened at the start of the reset period 310 or at another time point before t5 in Figure 7 , such as between t9 and t5. Similarly, the light source 30 can be turned on at the start of the reset period 310 or at another time point before t5 in Figure 7 , such as between t9 and t5. As another example, as Figure 8 shows, the time period during which the shutter is open and the light source is on while one or more rows are integrating (e.g., t2 to t3 and t6 to t7) can be slightly greater than the integration time of one or more rows at the top of the image sensor (e.g., T INT1 and T INT2 ) and thus greater than T_INT_COMMON (which is equal to Figure 8 's T INT1 ). In this type of arrangement, the first few rows of the image sensor can have a lower exposure relative to the remaining rows and thus can appear darker relative to the remaining rows. However, this is acceptable in some applications.

[0083] The above is merely illustrative and various modifications can be made to the described embodiments. The above embodiments can be implemented individually or in any combination.

Claims

1. A method of operating an image sensor, the image sensor being overlapped by an electronic shutter and including a plurality of rows of imaging pixels, the method comprising, during a frame: Resetting the plurality of rows of imaging pixels; Integrating charge during a respective integration period of each row of imaging pixels among the plurality of rows of imaging pixels, wherein the electronic shutter is in a first state having a first transparency during a first time period, the first time period including all of the plurality of rows of imaging pixels concurrently being in their integration periods; And Reading out the plurality of rows of imaging pixels row by row, wherein the electronic shutter is in a second state having a second transparency less than the first transparency during a second time period after the first time period, the second time period including reading out the plurality of rows of imaging pixels row by row.

2. The method according to claim 1, wherein resetting the multi-line imaging pixels comprises: Resetting the plurality of rows of imaging pixels concurrently.

3. The method according to claim 1, wherein resetting the multi-line imaging pixels comprises: Resetting the plurality of rows of imaging pixels row by row.

4. The method according to claim 3, wherein the electronic shutter is in the second state having the second transparency during a third time period before the first time period, the third time period including resetting the plurality of rows of imaging pixels row by row.

5. The method according to claim 1, wherein resetting the multi-line imaging pixels comprises: Resetting the imaging pixels of each row during a reset period of the imaging pixels of each row, and wherein the reset periods are staggered.

6. The method according to claim 1, wherein resetting the multi-line imaging pixels comprises: Resetting the imaging pixels of each row during a reset period of the imaging pixels of each row, and wherein the reset period of a given row of imaging pixels starts after the reset period of the previous row of imaging pixels.

7. The method according to claim 1, the method further comprising: After reading out the plurality of rows of imaging pixels row by row, delaying the start of a subsequent frame for a duration.

8. The method according to claim 7, wherein the electronic shutter is in the first state during at least a part of the duration.

9. The method according to claim 1, wherein the image sensor is configured to image a scene, and wherein a light source is configured to be turned on and illuminate the scene during the first time period.

10. The method according to claim 9, wherein the light source is configured to be turned off during the second time period.

11. The method according to claim 1, wherein the first transparency is the maximum transparency of the electronic shutter, and wherein the second transparency is the minimum transparency of the electronic shutter.

12. The method according to claim 1, wherein each imaging pixel of the imaging pixels includes a reset transistor, and wherein resetting the plurality of rows of imaging pixels includes: Activating the reset transistors in each of the imaging pixels.

13. A method of operating a system, the system including an image sensor having a plurality of rows of imaging pixels and an electronic shutter overlapping the image sensor, the method comprising: Integrating charge during a plurality of integration periods, wherein each row of imaging pixels among the plurality of rows of imaging pixels has a respective one of the plurality of integration periods; Placing the electronic shutter in a first mode when all of the plurality of rows of imaging pixels are in their respective integration periods; Sampling image data row by row from the plurality of rows of imaging pixels; And After all of the multi-line imaging pixels are in their respective integration periods and while sampling image data row-by-row from the multi-line imaging pixels, the electronic shutter is placed in a second mode having a lower transparency than the first mode.

14. The method according to claim 13, wherein the electronic shutter has maximum transparency in the first mode and wherein the electronic shutter has minimum transparency in the second mode.

15. The method according to claim 13, the method further comprising: Globally resetting all of the multi-line imaging pixels prior to integrating charge during the plurality of integration periods.

16. The method according to claim 13, the method further comprising: Resetting each row of imaging pixels in the multi-line imaging pixels during a plurality of respective reset periods, wherein the reset periods are staggered.

17. The method according to claim 16, wherein the integration period for each row begins at the end of the reset period for that row.

18. The method according to claim 13, wherein the system includes a light source, and wherein the method further comprises: Turning on the light source when all of the multi-line imaging pixels are in their respective integration periods; And Turning off the light source after all of the multi-line imaging pixels are in their respective integration periods and while sampling image data row-by-row from the multi-line imaging pixels.

19. A system, the system comprising: An image sensor including multi-line imaging pixels and configured to capture an image of a scene; An electronic shutter overlapping the image sensor, wherein the electronic shutter is capable of operating in a first state having a first transparency and a second state having a second transparency, the second transparency being lower than the first transparency; And A control circuit configured to control the electronic shutter to be in the first state when all of the multi-line imaging pixels are integrating charge and to control the electronic shutter to be in the second state when more than one but less than all of the multi-line imaging pixels are integrating charge.

20. The system according to claim 19, wherein the control circuit is configured to switch the electronic shutter from the first state to the second state once a readout period for a first row of the multi-line imaging pixels begins.

21. The system according to claim 19, the system further comprising: A light source configured to illuminate the scene, wherein the control circuit is configured to control the light source to be turned on when all of the multi-line imaging pixels are integrating charge and to control the light source to be turned off when more than one but less than all of the multi-line imaging pixels are integrating charge.

Images