Image sensor and method of reading out an image sensor

Abstract

An image sensor and a method of reading out the image sensor. The image sensor comprises: a plurality of high-sensitivity photoelectric conversion elements; a plurality of low-sensitivity photoelectric conversion elements; and a processor for processing signals read out from the plurality of low-sensitivity photoelectric conversion elements and the plurality of high-sensitivity photoelectric conversion elements, wherein the processor is configured to read out signals from the plurality of low-sensitivity photoelectric conversion elements multiple times in a single frame after multiple exposures, and to obtain a plurality of low-sensitivity images at different times in a single frame.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to image sensors, and in particular, image sensors with image blur correction and methods of reading out image sensors. BACKGROUND

[0002] Image sensors including a plurality of high-sensitivity photoelectric conversion elements and a plurality of low-sensitivity photoelectric conversion elements are known. These image sensors can produce a high dynamic range (HDR) image in both a bright scene and a dark scene. In a bright scene, for the high-sensitivity photoelectric conversion elements, the exposure time is short. If the exposure time is shorter than the light pulse and / or the interval between light pulses in pulse width modulation (PWM) light, such as light emitting diode (LED) light, the brightness of the captured image can be incorrect.

[0003] On the other hand, for the low-sensitivity photoelectric conversion elements, the exposure time can be longer, and thus the above problem of PWM light, e.g., LED light, can be resolved. However, the long exposure time can cause a blurred image of a moving object.

[0004] Furthermore, in a complementary metal-oxide-semiconductor (CMOS) image sensor with a lateral overflow integration capacitor (LOFIC) structure, a similar blurred image of a moving object can be produced.

[0005] Therefore, there is a need for an image sensor including a low-sensitivity photoelectric conversion element or a LOFIC structure with image blur correction. SUMMARY

[0006] The following summary is representative and non-limiting. The above problems are overcome, and other advantages are realized, by using the embodiments.

[0007] According to an example of the present disclosure, an image sensor includes: a plurality of photoelectric conversion elements; and a processor configured to process signals read out from the plurality of photoelectric conversion elements, wherein the processor is configured to: read out the signals from the plurality of photoelectric conversion elements multiple times in a single frame after multiple exposures, and obtain a plurality of images at different times in a single frame; detect a moving object in the plurality of images by comparing the plurality of images obtained from the multiple readouts; shift a position of the moving object in each of the plurality of images so that each position of the moving object in the plurality of images is the same position; and combine the plurality of images after each position of the moving object in the plurality of images is the same position.

[0008] According to another example of the present disclosure, an image sensor includes: a plurality of high-sensitivity photoelectric conversion elements; a plurality of low-sensitivity photoelectric conversion elements; and a processor configured to process signals read out from the plurality of low-sensitivity photoelectric conversion elements and the plurality of high-sensitivity photoelectric conversion elements, wherein the processor is configured to: read out the signals from the plurality of low-sensitivity photoelectric conversion elements multiple times in a single frame after multiple exposures, and obtain a plurality of low-sensitivity images at different times in a single frame; detect a moving object in the plurality of low-sensitivity images by comparing the plurality of low-sensitivity images obtained from the multiple readouts; shift a position of the moving object in each of the plurality of low-sensitivity images so that each position of the moving object in the plurality of low-sensitivity images is the same position; and combine the plurality of low-sensitivity images after each position of the moving object in the plurality of low-sensitivity images is the same position, thereby forming a low-sensitivity composite image.

[0009] The present disclosure also relates to an image signal processing method using the above image sensor. BRIEF DESCRIPTION OF DRAWINGS

[0010] Non-limiting and non-exhaustive examples of the present application are described with reference to the following figures.

[0011] Figure 1 A block diagram to show a configuration of an image sensor.

[0012] Figure 2 A configuration of a single pixel is shown.

[0013] Figure 3 A timing chart to show readouts at a high-sensitivity sub-pixel and a low-sensitivity sub-pixel.

[0014] Figure 4 An image in the presence of a moving object is shown.

[0015] Figure 5 An exemplary flowchart to show an operation of an image sensor.

[0016] Corresponding reference numbers indicate corresponding components throughout the several views of the drawings. It will be understood by those within the art that the drawings are not necessarily drawn to scale and that, unless otherwise noted, elements are drawn to approximately the same scale for purposes of simplicity and clarity. For example, the dimensions of some of the elements in the figures can be exaggerated relative to other elements for clarity.

[0017] BRIEF DESCRIPTION OF DRAWINGS

[0018] 10: pixel array;

[0019] 12: row selection circuit;

[0020] 14: column parallel ADC;

[0021] 16: image signal processor;

[0022] 18: moving object processing section;

[0023] 20: MIPI;

[0024] 100: image sensor;

[0025] 101: high-sensitivity sub-pixel;

[0026] 102: low-sensitivity sub-pixel;

[0027] 114-1: high-sensitivity photodiode;

[0028] 114-2: low-sensitivity photodiode;

[0029] 116-1, 116-2: transfer transistor;

[0030] 118-1, 118-2: floating diffusion;

[0031] 120-1, 120-2: reset transistor;

[0032] 124-1, 124-2: source follower transistor;

[0033] 126-1, 126-2: row selection transistor;

[0034] 128-1, 128-2: bit line;

[0035] A, B, C, D, E: exposure;

[0036] RS(0,0), RS(0,1): row selection line;

[0037] RST(0): reset line;

[0038] S11, S12, S13, S14, S15, S16, S17, S18: step;

[0039] TXL(0,0), TXS(0,0): transfer control lines;

[0040] Vsync: vertical synchronization signal. DETAILED DESCRIPTION

[0041] In the following description, numerous specific details are set forth to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details. In other instances, well-known materials or methods have not been described in detail in order to avoid obscuring the present application.

[0042] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearance of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable

[0043] Figure 1 A block diagram showing a configuration of the image sensor 100 of an embodiment. In this embodiment, the image sensor 100 is mounted on a single semiconductor chip, and includes the pixel array 10 and an image signal processor 16 (processor 16) that processes signals read out from the pixel array 10.

[0044] The pixel array 10 includes a plurality of pixels arranged in a matrix. In this embodiment, the pixel array 10 includes a plurality of high-sensitivity photoelectric conversion elements (e.g., high-sensitivity photodiodes) and a plurality of low-sensitivity photoelectric conversion elements (e.g., low-sensitivity photodiodes). The high-sensitivity photoelectric conversion elements can be large photodiodes having large light-receiving areas, and the low-sensitivity photoelectric conversion elements can be small photodiodes having small light-receiving areas.

[0045] It should be appreciated that the expression "high-sensitivity photoelectric conversion element" can also mean a high-sensitivity photodiode. The expressions "high-sensitivity photoelectric conversion element" and "high-sensitivity photodiode" can be used interchangeably. Similarly, the expression "low-sensitivity photoelectric conversion element" can also mean a low-sensitivity photodiode. The expressions "low-sensitivity photoelectric conversion element" and "low-sensitivity photodiode" can be used interchangeably.

[0046] Reference Figure 2, a high-sensitivity photodiode 114-1 can produce a relatively large output for a given amount of light received during an exposure time. A low-sensitivity photodiode 114-2 can produce a relatively small output for the same given amount of light received during the same exposure time. This configuration allows for a single pixel to have a high-sensitivity photodiode and a low-sensitivity photodiode.

[0047] On the other hand, the LOFIC structure can be used to act as the low-sensitivity photodiode 114-2. The LOFIC structure allows for storage of overflow charge and provides a low-sensitivity output. Thus, for the low-sensitivity photodiode 114-2, the same photodiode as the high-sensitivity photodiode 114-1 can be used instead of a photodiode with a small photosensitive area.

[0048] Returning to Figure 1 , a row selection circuit 12 and a column parallel analog-to-digital convertor (ADC) 14 are connected to the pixel array 10. The row selection circuit 12 selects a row of the pixel array 10 and causes the signal from each pixel in the row to be output to a corresponding column bit line, respectively. The column parallel ADC 14 converts the analog signal supplied from the respective column bit line into a digital signal.

[0049] Thus, the image signals (digital signals) from multiple pixels in the selected row are output in parallel from the column parallel ADC 14. In the row selection circuit 12, the rows are sequentially selected, and when all the rows are selected, the image signals of one frame are output from the column parallel ADC 14.

[0050] The image signals from the column parallel ADC 14 are supplied to an image signal processor 16, where the image signals are processed in various ways. In this system, the image signal processor 16 has a moving object processing section 18. The moving object processing section 18 performs positioning correction (e.g., moving position) and other processing on moving objects in the image.

[0051] The image signals processed in the image signal processor 16 are supplied to a mobile industry processor interface (MIPI) 20 and output in a predetermined format. The output digital image data is displayed on a display or transmitted to, for example, the next step in the system.

[0052] Furthermore, in Figure 2In the example shown in the middle, one pixel contains two photodiodes, i.e. a high sensitivity photodiode 114-1 and a low sensitivity photodiode 114-2. The signals from both photodiodes are read out independently. However, it is also possible to provide one photodiode per pixel and to provide only one readout circuit for the photodiodes.

[0053] Figure 2 The configuration of a single pixel is shown. In this example, one pixel has two photodiodes 114-1 and 114-2. For example, photodiode 114-1 is a high sensitivity photodiode and photodiode 114-2 is a low sensitivity photodiode. Figure 2 It is also shown that the pixel contains a high sensitivity sub-pixel 101 comprising the high sensitivity photodiode 114-1 and a low sensitivity sub-pixel 102 comprising the low sensitivity photodiode 114-2.

[0054] The photodiode 114-1 is connected via a transfer transistor 116-1 to a floating diffusion 118-1 having a given capacitance. The photodiode 114-2 is connected via a transfer transistor 116-2 to a floating diffusion 118-2.

[0055] The floating diffusions 118-1 and 118-2 have a predetermined capacitance and accumulate the charges transferred from the photodiodes 114-1 and 114-2, respectively. The gate of a source follower transistor 124-1 is connected to the floating diffusion 118-1. The gate of a source follower transistor 124-2 is connected to the floating diffusion 118-2. The drain of the source follower transistor 124-1 is connected to a power supply and the source is connected to the drain of a row select transistor 126-1. The drain of the source follower transistor 124-2 is connected to a power supply and the source is connected to the drain of a row select transistor 126-2.

[0056] The gate of the row select transistor 126-1 is connected to a row select line RS(0,0) and the gate of the row select transistor 126-2 is connected to a row select line RS(0,1). The source of the row select transistor 126-1 is connected to a bit line 128-1 and the source of the row select transistor 126-2 is connected to a bit line 128-2.

[0057] Thus, an image signal corresponding to the stored charges of the high sensitivity photodiode 114-1 is read out on the bit line 128-1 and an image signal corresponding to the stored charges of the low sensitivity photodiode 114-2 is read out on the bit line 128-2.

[0058] The gates of the reset transistors 120-1 and 120-2 are connected to a reset line RST(0). The drains of the reset transistors 120-1 and 120-2 are connected to a power supply, and the sources of the reset transistors 120-1 and 120-2 are connected to the sources of the transfer transistors 116-1 and 116-2, respectively.

[0059] At the start, RST(0) is set to high (H), turning on the reset transistors 120-1 and 120-2, and then the floating diffusion 118-1 and the floating diffusion 118-2 are reset. Accordingly, the outputs of the source follower transistors 124-1 and 124-2 are also reset. Next, after RST(0) returns to low (L) and after a predetermined exposure time, the transfer control line TXL(0,0) is turned on. This turns on the transfer transistor 116-1, and the accumulated charge of the photodiode 114-1 is read out to the bit line 128-1 through the floating diffusion 118-1.

[0060] Further, after RST(0) returns to L, after a predetermined exposure time, the transfer control line TXS(0,0) is turned on, and the charge of the photodiode 114-2 is read out to the bit line 128-2 through the floating diffusion 118-2. In this way, the readout operation can be performed multiple times in one frame.

[0061] Figure 3 A timing chart showing the readout of the high-sensitivity photodiode 114-1 at the high-sensitivity subpixel 101 and the readout of the low-sensitivity photodiode 114-2 at the low-sensitivity subpixel 102 is shown. Figure 3 The drawing is not to scale. The vertical synchronization signal Vsync is at a low level at the start of each frame. Accordingly, one frame of image is obtained within two low vertical synchronization signals Vsync. For example, a frame time is 16.67 milliseconds.

[0062] At the high-sensitivity subpixel 101, the charge of the photodiode 114-1 is transferred to the floating diffusion 118-1 at a time after a predetermined exposure A, and a voltage corresponding to this transferred charge is read out as an image signal. For example, the exposure time A is 1 millisecond.

[0063] The processor 16 reads out the signal from the high-sensitivity photodiode 114-1 at one time in a single frame after the exposure A, and obtains a high-sensitivity image.

[0064] On the other hand, at the low sensitivity sub-pixel 102, the exposure time of the low sensitivity photodiode 114-2 is relatively long. In this example, the exposure time is B+C+D+E, which can be (but is not limited to) four times the exposure time A of the high sensitivity sub-pixel 101. The longer exposure time causes fast moving objects or images of moving objects to become blurred. In this example, the low sensitivity photodiode 114-2 is read four times, each exposure time being the same as the exposure time A of the high sensitivity photodiode 114-1. Charge transfer and readout are performed after exposure B, exposure C, exposure D, and exposure E, respectively.

[0065] In this example, within a frame, the low sensitivity photodiode 114-2 is read four times, each exposure time being the same as the exposure time of the high sensitivity photodiode 114-1. The processor 16 can read the signal from the low sensitivity photodiode 114-2 multiple times in a single frame after multiple exposures, and obtain multiple low sensitivity images at different times in a single frame.

[0066] Figure 4 An image in the presence of a moving object is shown. In this example, the moving object is indicated by a vertical line moving from the right edge of the screen to the left edge during exposures B through E. For the high sensitivity photodiode 114-1, the object appears at the left edge of the screen because the image is obtained at exposure A. On the other hand, for the low sensitivity photodiode 114-2, the moving object is at the right edge in the image at exposure B, at a more right position in the image at exposure C, at a more left position in the image at exposure D, and at the left edge in the image at exposure E.

[0067] In this example, the image signal processor 16 detects the moving object from each image at the four exposures of the low sensitivity photodiode 114-2, which are the low sensitivity images obtained from the multiple readouts. The moving object is detected by comparing the low sensitivity images obtained from the multiple readouts.

[0068] The position of the detected moving object in the low sensitivity images at exposures B, C, and D moves to the same position as the position of the detected moving object in the image of the last readout (i.e., at exposure E). After each position of the low sensitivity images is the same position, a composite image is formed by combining the four low sensitivity images at the four exposures B through E.

[0069] In this example, the exposure A of the high sensitivity photodiode 114-1 and the exposure E of the low sensitivity photodiode 114-2 are performed simultaneously. A high sensitivity image, e.g., from the high sensitivity photodiode 114-1, and a low sensitivity composite image, e.g., from the low sensitivity photodiode 114-2, are further combined.

[0070] The amount of charge accumulated in the low-sensitivity photodiode 114-2 during one exposure cycle B, C, or D in the B+C+D+E sequence is insufficient. Furthermore, each image from one exposure cycle B, C, or D does not resolve the problem caused by the LED light in the high-sensitivity photodiode 114-1 (i.e., exposure is too short). Therefore, the four images from exposures B to E are combined to form a composite image. The combined four exposures are long enough to resolve the problem caused by the PWM light (such as LED light) in the high-sensitivity photodiode 114-1.

[0071] After the position of the moving object is moved in each of the three low-sensitivity images at exposures B to D, a background image can be added to the blank area. Adding a background image to the blank area after the moving object's position has been moved includes interpolation or other processing.

[0072] In this way, a low-sensitivity composite image and a high-sensitivity image are combined, where the moving object exists in the same position in the low-sensitivity composite image as it does in the high-sensitivity image. Furthermore, only for the low-sensitivity composite image, because the exposure time is long enough, a low-sensitivity composite image can be obtained without any problems caused by PWM light (such as LED light).

[0073] Without correcting the position of moving objects in a low-sensitivity image, a single-frame, single-exposure low-sensitivity image contains a blurred image of the moving object. The method disclosed herein is used to correct the blurred image.

[0074] Figure 5 An exemplary flowchart illustrating the operation of an image sensor is provided. Images are acquired at exposure A of the high-sensitivity photodiode 114-1 and at exposures B, C, D, and E of the low-sensitivity photodiode 114-2 (S11). Moving objects are detected in the acquired images (S12). The detection of moving objects can be performed, for example, by inter-frame comparison of the high-sensitivity images.

[0075] If the determination in S12 is yes, then the coordinate range of the moving object is determined (S13). That is, the coordinate range of the moving object is determined based on the inter-frame comparison of the high-sensitivity image. In one embodiment, steps S12 and S13 can be skipped.

[0076] Next, the B low sensitivity image is compared to the E low sensitivity image to determine if a moving object is detected (S14). In this way, a moving object should be detected within the coordinate range of the moving object in S13. If S12 and S13 have been skipped, then this condition is ignored. If YES in S14, then for each of the B image to the D image, the moving object portion and the stationary object portion are determined (S15).

[0077] If the moving and stationary object portions of each image have been determined, then the position of the moving object portion is aligned with the position of the moving object portion in the E image (S16), and all images are combined to obtain a single composite image (S17). If NO in S12 and S14, then the images from exposure B to exposure E are combined as is (S18).

[0078] If only one photodiode is provided for a pixel, then the one photodiode is considered to be the low sensitivity photodiode 114-2 described above.

[0079] In one embodiment, the pixel array 10 Figure 1 ) includes a plurality of pixels arranged in a matrix. Each pixel includes only one photodiode as a photoelectric conversion element. Figure 2 The low sensitivity photodiode 114-2 is considered to be a photodiode.

[0080] In the above description, it is assumed that the readout from the pixel array 10 is performed row by row. However, it is also possible to read out the image signals from multiple rows of pixels in parallel by setting a parallel readout configuration.

[0081] In each pixel, the signal from the high sensitivity photodiode can be used to capture dark scenes (the high sensitivity photodiode provides a clear image of dark scenes), and the signal from the low sensitivity photodiode can be used to capture bright scenes (the low sensitivity photodiode provides a non-saturated image of bright scenes). The two signals can be combined together.

[0082] While the application has been described herein relative to illustrative embodiments and the best mode for practicing the application, it will be apparent to those skilled in the art that various embodiments, adaptations and variations of the application can be made without departing from the spirit and scope of the application.

[0083] The terms used in the appended claims should not be construed to limit the application to the specific embodiments disclosed in the specification and claims. Rather, the scope of the application should be determined entirely by the appended claims, which should be construed in accordance with the established doctrines of claim interpretation. This detailed description and accompanying drawings are therefore to be taken as illustrative and not restrictive.

Claims

1. An image sensor, comprising: Multiple high-sensitivity photoelectric conversion elements; Multiple low-sensitivity photoelectric conversion elements; as well as A processor for processing signals read from the plurality of low-sensitivity photoelectric conversion elements and the plurality of high-sensitivity photoelectric conversion elements; The processor is configured as follows: After multiple exposures, signals are read out multiple times from the plurality of low-sensitivity photoelectric conversion elements in a single frame, and multiple low-sensitivity images are obtained at different times in the single frame. Moving objects are detected in the plurality of low-sensitivity images by comparing them obtained from multiple readouts. Move the position of the moving object in each of the plurality of low-sensitivity images such that each position of the moving object in the plurality of low-sensitivity images is the same; as well as After each position of the moving object in the plurality of low-sensitivity images is the same, the plurality of low-sensitivity images are combined to form a low-sensitivity composite image. The processor is further configured to read out signals from the plurality of high-sensitivity photoelectric conversion elements in a single frame after exposure, and to obtain a high-sensitivity image in the single frame; as well as The processor is further configured to detect the moving object in the plurality of low-sensitivity images by inter-frame comparison of high-sensitivity images before comparing the plurality of low-sensitivity images obtained from the plurality of readouts.

2. The image sensor of claim 1, wherein the processor is further configured to add a background image in the blank area after the position of the moving object has been moved in each of the plurality of low-sensitivity images.

3. The image sensor of claim 2, wherein adding the background image to the empty area after the position of the moving object has been moved includes interpolation.

4. The image sensor according to claim 1, wherein the one-time readout of the signal from the plurality of high-sensitivity photoelectric conversion elements and the last readout of the plurality of low-sensitivity photoelectric conversion elements occur at the same time.

5. The image sensor of claim 4, wherein the processor is further configured to combine the low-sensitivity composite image and the high-sensitivity image.

6. The image sensor of claim 1, wherein the exposure time of the high-sensitivity image is the same as the exposure time of each of the plurality of low-sensitivity images.

7. The image sensor according to claim 1, wherein: Each of the plurality of highly sensitive photoelectric conversion elements produces a large output for a given amount of received light; and Each of the plurality of low-sensitivity photoelectric conversion elements produces a small output for the given amount of light received.

8. A method for reading out an image sensor, wherein the image sensor comprises a plurality of low-sensitivity photoelectric conversion elements and a plurality of high-sensitivity photoelectric conversion elements, comprising: After multiple exposures, signals are read out multiple times from the plurality of low-sensitivity photoelectric conversion elements in a single frame, and multiple low-sensitivity images are obtained at different times in the single frame. Moving objects are detected in the plurality of low-sensitivity images by comparing them obtained from multiple readouts. Move the position of the moving object in each of the plurality of low-sensitivity images such that each position of the moving object in the plurality of low-sensitivity images is the same; as well as After each position of the moving object in the plurality of low-sensitivity images is the same, the plurality of low-sensitivity images are combined to form a low-sensitivity composite image. Before comparing the plurality of low-sensitivity images obtained from the multiple readouts, the moving object in the plurality of low-sensitivity images is detected by inter-frame comparison of the plurality of high-sensitivity images; as well as After exposure, the signal is read out from the plurality of high-sensitivity photoelectric conversion elements in a single frame, and a high-sensitivity image is obtained in the single frame.

9. The method according to claim 8, further comprising: After the position of the moving object is moved in each of the plurality of low-sensitivity images, a background image is added to the blank area.

10. The method of claim 9, wherein adding the background image to the empty area after the position of the moving object has been moved includes interpolation.

11. The method of claim 8, wherein the one-time readout of the signal from the plurality of high-sensitivity photoelectric conversion elements occurs at the same time as the last readout of the plurality of readout signals from the plurality of low-sensitivity photoelectric conversion elements.

12. The method of claim 11, further comprising: The low-sensitivity composite image and the high-sensitivity image are combined.

13. The method of claim 8, wherein the exposure time of the high-sensitivity image is the same as the exposure time of each of the plurality of low-sensitivity images.

Images