Image sensor and method of reading out signals from an image sensor

Abstract

An image sensor (10) comprises a plurality of pixels (100), each of the pixels (100) being configured to generate an output signal. The image sensor (10) further comprises a controller (112) for controlling a readout circuit (22) configured to readout a signal comprising selectively combining one or a plurality of output signals from adjacent pixels (100) to generate a readout signal representing pixel regions (121, 122, 123) having different binning sizes.

Description

[0001] IMAGE SENSOR AND METHOD OF READING OUT SIGNALS FROM AN IMAGE SENSOR

[0002] Consumer electronic devices such as cameras or smartphones usually employ a CMOS image sensor (CIS) comprising an array of photodiodes and readout circuitry. Incident electromagnetic radiation is converted into charges by the photodiodes. The generated charges are read out and processed by elements comprising an ADC ("analog digital converter") . For example, during readout, a higher power can provide a lower readout noise and a higher dynamic range resulting in an improved image quality. On the other hand, a higher power results in an increased consumption, shorter battery lifetimes and thermal heating.

[0003] According to concepts, pixels may be binned, e.g. the signals read out may be combined, e.g. by summing or averaging their values, e.g. during or after readout. As a result, a signal-to- noise ratio may be improved. On the other hand, a resolution of the output image may be decreased due to binning.

[0004] Accordingly, concepts are sought by which a processing of image signals may be further improved.

[0005] It is an object of the present invention to provide an improved image sensor and an improved method of reading out signals from the image sensor.

[0006] SUMMARY

[0007] According to embodiments, the above object is achieved by the claimed matter according to the independent claims. Further developments are defined in the dependent claims. According to embodiments, an image sensor comprises a plurality of pixels, each of the pixels being configured to generate an output signal, and a controller for controlling a readout circuit configured to readout a signal. Controlling the readout circuit comprises selectively combining one or a plurality of output signals from adjacent pixels to generate a readout signal representing pixel regions having different binning sizes.

[0008] The image sensor may further comprise an upsampling circuit configured to upsample the readout signal to compensate for the different binning sizes. For example, the upsampling circuit may be configured to upsample the readout signal from a pixel region to a higher resolution. Further, due to the upsampling the binning as such may be compensated for. As a result, an image having the resolution of the sensor without binning may be obtained.

[0009] For example, the controller may be configured to determine a binning size of one or more pixel regions corresponding to a number of output signals to be combined. This determination may be accomplished based on an image taken while signals are read out from single pixels.

[0010] According to further embodiments, determining the binning size may be performed at a different stage of an electronic device comprising the image sensor, e.g. in a component processing the generated image data, or in a further component, e.g. of an electronic device comprising the image sensor.

[0011] According to embodiments, each of the pixels comprises sub-pixels configured to detect electromagnetic radiation representing a different color, respectively. For example, output signals from sub-pixels of the same color may be combined. Further embodiments relate to a method of reading out signals from an image sensor comprising a plurality of pixels , each of the pixels being configured to generate an output signal . The method may comprise selectively combining one or a plurality of output signals from adj acent pixels to generate a readout signal representing pixel regions having di f ferent binning si zes , and reading out the combined signals .

[0012] The method may further comprise upsampling the readout signal to compensate for the di f ferent binning si zes . As a result , a high and uni form resolution across the image may be obtained .

[0013] According to embodiments , the method may further comprise taking an image without binning before , for a subsequent image , combining one or a plurality of output signals from adj acent pixels , and determining a binning si ze of regions in dependence from the image taken .

[0014] The method may further comprise determining a brightness of the regions , wherein the binning si ze is determined based on the brightness of the corresponding region .

[0015] The method may further comprise selecting an upsampling method in dependence from whether a high speed or a high image quality is to be obtained .

[0016] According to embodiments , an electronic device comprises the image sensor as described above . For example , the electronic device may be selected from a still camera, a video camera, and a smartphone .

[0017] BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this speci fication . The drawings illustrate the embodiments of the present invention and together with the description serve to explain the principles . Other embodiments of the invention and many of the intended advantages will be readily appreciated, as they become better understood by reference to the following detai led description . The elements of the drawings are not necessarily to scale relative to each other . Like reference numbers designate corresponding similar parts .

[0018] Fig . 1A summari zes a method according to embodiments .

[0019] Fig . IB shows an array of sub-pixels for explaining the method according to embodiments .

[0020] Fig . 2A shows components of an image sensor according to embodiments .

[0021] Fig . 2B shows elements of a pixel that may be a component of the image sensor according to embodiments .

[0022] Fig . 3 shows elements of an image sensor according to embodiments .

[0023] Fig . 4 illustrates an electronic device according to embodiments .

[0024] DETAILED DESCRIPTION

[0025] In the following detailed description reference is made to the accompanying drawings , which form a part hereof and in which are illustrated by way of illustration speci fic embodiments in which the invention may be practiced . In this regard, directional terminology such as "top", "bottom", "front", "back", "over", "on", "above", "leading", "trailing" etc. is used with reference to the orientation of the Figures being described. Since components of embodiments of the invention can be positioned in a number of different orientations, the directional terminology is used for purposes of illustration and is in no way limiting. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope defined by the claims.

[0026] The description of the embodiments is not limiting. In particular, elements of the embodiments described hereinafter may be combined with elements of different embodiments.

[0027] Fig. 1A shows elements of a method of reading out signals from an image sensor comprising a plurality of pixels. Each of the pixels is configured to generate an output signal in response to incident electromagnetic radiation. The method comprises selectively combining (S100) one or a plurality of output signals from adjacent pixels to generate a readout signal. The readout signal may thus represent pixel regions having different binning sizes. The method further comprises reading out (Slid) the combined signals.

[0028] As will be explained in more detail below, according to concepts, depending from characteristics of the image that is taken, a binning size of different pixel regions within the pixel array is chosen. This may be accomplished by analyzing an image and, thereafter, selectively setting different binning sizes for different pixel regions. When subsequently an image or frame is captured, a readout signal that represents pixel regions having different binning sizes is generated. For example, the method may further comprise taking an image (S90) without binning. For example, this may be a live view running which is usually performed during image capturing, e.g. in smartphones. For example, the image taken may be analyzed, e.g. for determining exposure and gain (ISO) of the image taken, e.g. to prevent the image from being overexposed. Moreover, the method may comprise determining (S95) a binning size of regions in dependence from the image taken, for an image that is subsequently to be captured. This process will be explained in more detail with reference to Fig. IB.

[0029] For example, the processes of determining (S95) a binning size of regions in dependence from the image taken, selectively combining (S100) one or a plurality of output signals from adjacent pixels and reading out (Slid) the combined signals may be continuously repeated.

[0030] Further, according to embodiments, an upsampling process (S120) is performed so as to adjust the resolution within the image and to compensate for the different binning sizes. For example, this process may be performed immediately after reading (S110) out the combined signals. For example, in this case, the processes of determining (S95) a binning size of regions in dependence from the image taken, selectively combining (S100) one or a plurality of output signals from adjacent pixels, reading out (Slid) the combined signals and the upsampling process may be continuously repeated. According to further embodiments, the up- sampling process may be performed at a later time, e.g. when a still picture is taken and further processed.

[0031] Fig. IB shows an example of an array of sub-pixels 102 of an image sensor. The term "sub-pixel" may e.g. refer to elements of pixel 100 detecting electromagnetic radiation in a certain wavelength band representing e.g. a specific color. Usually, color filters, in particular a color filter array may be arranged over the pixel array. In more detail, the color filter array may comprise portions which transmit electromagnetic radiation of a certain color so that color information may be obtained. For example, a color filter array may represent a Bayer pattern comprising color filters of R, G, B wherein the ratio of G may be 50 %, the ratio of R may be 25 % and the ratio of B may be 25 %. When performing binning, the image signals of sub-pixels 102 of the same color are combined.

[0032] Fig. IB shows sub-pixels 102 of a first pixel region 121, in which e.g. a high brightness is present. The pixel array may further comprise a second pixel region 122 in which a medium brightness level may be present. The pixel array further comprises a third pixel region 123 in which a very low brightness is present. According to embodiments, a binning size is determined e.g. depending on the brightness. Accordingly, in the first pixel region 121, no binning takes place. Further, in the second pixel region 122 four adjacent sub-pixels 102 are binned. Moreover, in the third pixel region 123, 16 adjacent sub-pixels 102 are binned. Accordingly, for one single image, different binning sizes may be applied.

[0033] For example, the second pixel region 122 and the third pixel region may be binned to increase the signal / noise ratio and to get information of the dark areas. Due to the binning process, the resolution of the second and the third pixel region is reduced .

[0034] In order to achieve the resolution of the non-binned parts of the image, e.g. the readout signals representing the second pixel region 122 and the third pixel region 123 may be upsampled. For example, the upsampling process for the second and third pixel regions may be performed by a method depending on how much time and computing resources are available. For example, algorithmbased approaches may be used for real-time applications. According to further embodiments, e.g. especially, when the upsampling is not performed in real time, further Al ("artificial intelligence") enhancement may be employed.

[0035] A further upsampling method is e.g. described in US publication 2023 / 0027452 Al.

[0036] Generally, the upsampling process may be selected and changed depending on how much effort and resource are to be applied. For example, a video including many frames, e.g. a real-time video may be processed using a faster upsampling method than a still image. On the other hand, a still image may be upsampled using a slow method that achieves a high image quality. According to embodiments, upsampling may be performed in a post-processing step which is performed "offline", e.g. at a later timing. According to further implementations, upsampling may be dispensed with so that the resolution is determined by the binned portions of the image. In this case, the higher resolution parts may be downsampled .

[0037] Due to the method proposed, the resolution and dynamic range of the sensor may be maximized without the need to take several photos and to combine them. Nevertheless, for example, additional photos may be taken resulting in a better resolution. Moreover, the maximum binning size and appropriate upsampling methods may be decided and may be changed according to the application of the image sensor. According to further embodiments, the upsampling methods may be changed to maximize either image quality or processing time.

[0038] Fig. 2A shows a portion of an image sensor including a pixel array according to embodiments. The pixel array comprises a plurality of pixels 100 wherein each pixel 100 may comprise several sub-pixels 102 . For example , as is illustrated, one pixel 100 may comprise one R sub-pixel , two G sub-pixels and one B sub-pixel . In this context , "R" , "G" and "B" refer to color filters that are arranged over the respective sub-pixels 102 . As is clearly to be understood, any other kind of color filters may be employed .

[0039] A column output 15 may be connected to each of the pixel s 100 . The signals of the respective pixels 100 are supplied to a column processing device 118 that may comprise an analog digital converter (ADC ) 115 . The image sensor comprises a plurality of switches 23 . The switches 23 may be configured to selectively combine two column output lines 15 . In more detail , depending on an operating state , the switch 23 may be operable to combine two column output lines or to transmit the signal of one column output line only .

[0040] Accordingly, depending on a state of the associated switch 23 , either single pixels 100 / sub-pixels 102 may be selected or multiple pixels 100 / sub-pixels 102 may be combined . The charges of the selected pixels are combined and can then be read out by the column processing device 118 or the ADC 115 . Using a cascading number of switches 23 , an increasing number of pixels may be combined .

[0041] Accordingly, by selectively activating corresponding ones of the switches 23 , di f ferent binning si zes may be implemented .

[0042] Fig . 2B shows elements of the pixel including the readout circuit in more detail . As is illustrated, the sub-pixel 102 comprises a photodiode 20 for detecting electromagnetic radiation 16 . Charges generated by the photodiode 20 may be trans ferred via the trans fer transistor 21 to the readout circuit 22 . The elements of the readout circuit 22 are well-known and have been described elsewhere . An output of the readout circuit 22 may be transmitted via the column output 15 to the switch 23 .

[0043] Fig . 3 shows a schematic view of an image sensor 10 according to embodiments . The image sensor 10 comprises a plurality of pixels 100 that may be arranged in rows and columns . For example , a column output 15 may be assigned to a column of pixels 100 . The image sensor 10 may further comprise a row selection line driver 114 for selecting corresponding lines of the pixels 100 . The image sensor may further comprise a column processing device 118 for processing signals output by the column output 15 . For example , the column processing device 118 may comprises one or a plurality of ADCs 115 , (not illustrated in Fig . 3 ) . The image sensor 10 may further comprise a controller 112 . The controller 112 may be configured to determine a binning si ze of one or more pixel regions corresponding to a number of output signals to be combined based on an image taken while the signals are read out from the single pixels . For example , the controller 112 may selectively activate the switches 23 ( illustrated in Fig . 2A) .

[0044] According to implementations , the decision which areas to bin to which degree can be made in di f ferent parts of the imaging chain, e . g . in the image sensor 10 itsel f , an image processor that further processes the image data 17 output by the image sensor 10 or the CPU ("central processing unit" ) . The device taking the decision may be an other component of an electronic device comprising the image sensor 10 .

[0045] The image sensor 10 may further comprise a pixel data processing device 116 . For example , the pixel data processing device 116 may comprise an upsampling circuit 110 for upsampling the readout signal to compensate for the di f ferent binning si zes . The pixel data processing device 116 may be configured to process image signals into image data 17 .

[0046] As has been described, the image sensor 10 comprises a plurality of pixels 100 wherein each of the pixels is configured to generate an output signal . The image sensor further comprises a controller 112 for controlling a readout circuit configured to read out a signal . The readout comprises selectively combining one or a plurality of output signals from adj acent pixels . As a result , a readout signal is generated which represents pixel regions 121 , 122 , 123 having di f ferent binning si zes . According to embodiments , the image sensor 10 may further comprise an upsampling circuit 110 that is configured to upsample the readout signal to compensate for the di f ferent binning si zes .

[0047] Fig . 4 shows a schematic view of an electronic device 30 according to embodiments . The electronic device 30 comprises the image sensor 10 that has been described above . For example , the electronic device may be a still camera, a video camera or a smartphone .

[0048] While embodiments of the invention have been described above , it is obvious that further embodiments may be implemented . For example , further embodiments may comprise any subcombination of features recited in the claims or any subcombination of elements described in the examples given above . Accordingly, this spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein . LIST OF REFERENCES

[0049] 10 image sensor

[0050] 15 column output

[0051] 16 electromagnetic radiation

[0052] 17 image data

[0053] 20 photodiode

[0054] 21 trans fer transistor

[0055] 22 readout circuit

[0056] 23 switch

[0057] 30 electronic device

[0058] 100 pixel

[0059] 102 sub-pixel

[0060] 105 readout circuit

[0061] 110 upsampling circuit

[0062] 112 controller

[0063] 114 row selection line driver

[0064] 115 analog digital converter

[0065] 116 pixel data processing device

[0066] 118 column processing device

[0067] 121 first pixel region

[0068] 122 second pixel region

[0069] 123 third pixel region

Claims

CLAIMS1. An image sensor (10) comprising: a plurality of pixels (100) , each of the pixels (100) being configured to generate an output signal; and a controller (112) for controlling a readout circuit (22) configured to readout a signal comprising selectively combining one or a plurality of output signals from adjacent pixels (100) to generate a readout signal representing pixel regions (121, 122, 123) having different binning sizes, further comprising an upsampling circuit (110) configured to upsample the readout signal to compensate for the different binning sizes.

2. The image sensor (10) according to claim 1, wherein the controller (112) is configured to determine a binning size of one or more of the pixel regions (121, 122, 123) corresponding to a number of output signals to be combined based on an image taken while signals are read out from single pixels (100) .

3. The image sensor (10) according to claim 1 or 2, wherein each of the pixels (100) comprises sub-pixels (102) configured to detect electromagnetic radiation (16) representing a different color, respectively, wherein output signals from sub-pixels (102) of the same color are combined.

4. A method of reading out signals from an image sensor (10) comprising a plurality of pixels (100) , each of the pixels (100) being configured to generate an output signal, the method comprising: selectively combining (S100) one or a plurality of output signals from adjacent pixels (100) to generate a readout signal representing pixel regions (121, 122, 123) having different binning sizes; andreading out (S110) the combined signals, further comprising upsampling (S120) the readout signal to compensate for the different binning sizes.

5. The method according to claim 4, further comprising taking (S90) an image without binning before, for a subsequent image, combining (S110) one or a plurality of output signals from adjacent pixels, and determining (S95) a binning size of regions in dependence from the image taken.

6. The method according to claim 5, further comprising determining a brightness of the regions, wherein the binning size is determined based on the brightness of the corresponding region.

7. The method according to claim 5 or 6, further comprising selecting an upsampling method in dependence from whether a high processing speed or a high image quality is to be obtained .

8. An electronic device (30) comprising the image sensor (10) according to any of claims 1 to 3.

9. The electronic device (30) according to claim 8, being selected from a still camera, a video camera, and a smartphone .

Images