Image capture method and apparatus

Abstract

To provide a method of capturing an image having a high dynamic range.SOLUTION: A camera having at least one image sensor is disclosed. The image sensor includes a pixel array comprising narrowband pixels and wideband pixels of at least three colors. The camera is operable in a first mode in which the image sensor has first gain settings and generates an output having a first response. Further, the camera is operable in a second mode in which the image sensor has second gain settings and generates an output having a second response. The image sensor is configured to generate, in the first mode, a digital output in which the wideband pixels have a spectral output response approximately equal to a sum of spectral output responses of the narrowband pixels. The image sensor is further configured to generate, in the second mode, a digital output in which the wideband pixels have a spectral output response greater than in the first mode.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to image capture methods and apparatus. [Background technology]

[0002] High dynamic range (HDR) image capture technology is driving improvements in both digital still imaging and digital video. HDR technology exploits the ability of an image sensor's pixels to operate with different imaging characteristics. The outputs of pixels with different characteristics can be blended using an algorithm that preserves the most appropriately exposed pixel values ​​or image regions.

[0003] Here, pixel "characteristics" can be intrinsic to the pixel (e.g., different color filters), operational characteristics (e.g., different integration times), or signal processing characteristics (e.g., different pixel gains). These characteristics allow different pixels to have different sensitivities. Some HDR techniques require capturing multiple images with different sensing parameters applied to some or all pixels of the image sensor, while other techniques capture a single image in which some pixels within the single image have different characteristics. A combination of both techniques is also possible.

[0004] The systems, devices, methods, and approaches described in this section, as well as components thereof, are known by the inventors, and therefore, unless expressly stated otherwise, it should not be assumed that the systems, devices, methods, approaches, or components thereof are cited as prior art solely by virtue of their description in this section, or that the systems, devices, methods, approaches, and components are generally known to those skilled in the art. Summary of the Invention

[0005] According to a first aspect of the present disclosure, there is provided a camera comprising at least one image sensor, the image sensor having a pixel array including wideband pixels and narrowband pixels of at least three colors.

[0006] The narrowband and wideband pixels are read out with different pixel gains, the wideband pixels being read out with a higher pixel gain than the narrowband pixels. In one embodiment, the narrowband pixels of the camera include red, green and blue pixels, and the wideband pixels are read out with a pixel gain that is H times the pixel gain used to read out the green pixels.

[0007] According to a second aspect of the present disclosure, there is provided a camera comprising at least one image sensor, the image sensor including a pixel array comprising narrowband pixels and wideband pixels of at least three colors, the camera being operable as follows: In the first mode, the image sensor has a first pixel gain setting and produces an output having a first response. In the second mode, the image sensor has a second pixel gain setting and produces an output having a second response. Furthermore, the image sensor is configured as follows: In a first mode, the wideband pixel produces a digital output having a spectral output response that is approximately equal to the sum of the spectral output responses of the narrowband pixels; In the second mode, the wideband pixel produces a digital output having a larger spectral output response than in the first mode.

[0008] In some embodiments of the camera of the second aspect, in the second mode, the spectral output response of the wideband pixels is approximately equal to H times the sum of the spectral output responses of the narrowband pixels.

[0009] According to a third aspect of the present disclosure, there is provided a camera having at least one image sensor. The image sensor includes a pixel array with narrowband pixels and wideband pixels of at least three colors, and the camera is operable in a first mode to read out the image sensor using a first analog gain setting and in a second mode to read out the image sensor using a second analog gain setting. When readout in the second mode, the relative difference between the analog gain applied to the wideband pixels and the analog gain applied to the narrowband pixels is greater than the relative difference between the analog gain applied to the wideband pixels and the analog gain applied to the narrowband pixels during readout in the first mode. In some embodiments of the camera of the third aspect in the second mode, the analog gain applied to the wideband pixels is greater than the analog gain applied to the wideband pixels in the first mode.

[0010] In some embodiments of the camera according to the third aspect, the analog gain applied to the wideband pixels in the second mode is H times the analog gain applied to the wideband pixels in the first mode.

[0011] In some embodiments of the camera according to any one of the first, second or third aspects, H is greater than 2.

[0012] In some embodiments of the camera according to any one of the first, second or third aspects, H is less than 6.

[0013] In some embodiments of the camera according to any one of the first, second or third aspects, H is in the range of 2.5 to 3.5.

[0014] In some embodiments of the camera according to any one of the first, second or third aspects, H is in the range of 2.8 to 3.

[0015] In some embodiments of the camera according to any one of the first, second or third aspects, H is about 2.9.

[0016] According to a fourth aspect of the present disclosure, there is provided a camera having at least one image sensor comprising a pixel array including narrowband pixels and wideband pixels of at least three colors, operable at a first pixel gain setting in a normal mode and at a second pixel gain setting in a high dynamic range mode, in which the wideband pixels are read out with a higher pixel gain than in the normal mode.

[0017] In some embodiments of the camera according to the fourth aspect, in high dynamic range mode the narrowband pixels are read out with a second pixel gain setting that is different from said normal mode.

[0018] In some embodiments of the camera according to the fourth aspect, in high dynamic range mode, the pixel gain setting used for the wideband pixels is increased by a factor in the range of 2 to 6 times compared to the pixel gain setting for at least one color of the narrowband pixels.

[0019] In some embodiments of the camera according to the fourth aspect, in the high dynamic range mode, the pixel gain setting used for the wideband pixels is increased by a factor in the range of 2 to 6 times relative to the pixel gain setting for the narrowband pixels having a green color filter.

[0020] In some embodiments of the camera according to the fourth aspect, the magnification is in the range of 2.5 to 3.5 times.

[0021] In some embodiments of the camera according to the fourth aspect, the magnification is about 2.9x.

[0022] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, the pixel gain is an analog gain, or the pixel gain setting is an analog gain setting.

[0023] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, the pixel gain is a pixel conversion gain, or the pixel gain setting is a pixel conversion gain setting.

[0024] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, the pixel array includes three types of narrowband pixels (T1, T2, T3) and one type of wideband pixel (w).

[0025] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, the pixel array has a unit cell with a 1:1:1 ratio of T1:T2:T3 pixels.

[0026] In some embodiments of a camera according to any one of the first, second, third or fourth aspects, the pixel array has a unit cell with the following pixel arrangement: T1 w T2 w T3 w, w T1 w T2 w T3, T2 w T3 w T1 w, w T2 w T3 w T1, T3 w T1 w T2 w, w T3 w T1 w T2, where "T1" corresponds to a narrowband pixel of type 1, "w" corresponds to a wideband filter, "T2" corresponds to a narrowband pixel of type 2 and "T3" corresponds to a narrowband pixel of type 3.

[0027] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, the camera is further configured to generate one or both of a narrowband luminance value Y based on output from the narrowband pixels; and a wideband luminance value W based on output from the wideband pixels.

[0028] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, the camera is further configured to process the narrowband luminance value and the wideband luminance value to generate a combined luminance value.

[0029] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, the luminance values ​​of the narrow band are equal to or greater than a saturation value Y SAT and the broadband luminance value is saturated at W SAT where W SAT / Y SAT is greater than 3.

[0030] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, W SAT / Y SAT is greater than 6.

[0031] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, W SAT / Y SAT is greater than 8.

[0032] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, W SAT / Y SAT is about 9.

[0033] In some embodiments of the camera according to any one of the first, second, third or fourth aspects, Y=(T1+T2+T3) / 3.

[0034] According to a fifth aspect of the present disclosure, there is provided a method in a camera having at least one image sensor, the image sensor having a pixel array including narrowband pixels and wideband pixels of at least three colors, the method including reading out the narrowband pixels and the wideband pixels with different pixel gains, the wideband pixels being read out with a higher pixel gain than the narrowband pixels.

[0035] In some embodiments of the method of the fifth aspect, the narrowband pixels include red, green and blue pixels, and the method includes reading out the wideband pixels with a pixel gain that is H times the pixel gain used to read out the green pixels.

[0036] According to a sixth aspect of the present disclosure, there is provided a method implemented in a camera having at least one image sensor, the image sensor having a pixel array including narrowband pixels and wideband pixels of at least three colors, the method including: selectively operating the camera in a first mode in which the image sensor has a first pixel gain setting and produces an output having a first response; or selectively operating the camera in a second mode in which the image sensor has a second pixel gain setting and produces an output having a second response, the method further including: in the first mode, the wideband pixels producing a digital output having a spectral output response approximately equal to the sum of the spectral output responses of the narrowband pixels; and in the second mode, the wideband pixels producing a digital output having a spectral output response greater than in the first mode.

[0037] In some embodiments of the method according to the sixth aspect in the second mode, the spectral output response of the wideband pixel is approximately equal to H times the sum of the spectral output responses of the narrowband pixels.

[0038] According to a seventh aspect of the present disclosure, there is provided a method implemented in a camera having at least one image sensor, the image sensor comprising a pixel array including narrowband pixels and wideband pixels of at least three colors, the method including selectively operating the camera in a first mode to read out the image sensor using a first analog gain setting or in a second mode to read out the image sensor using a second analog gain setting, wherein during readout in the second mode, a relative difference between the analog gains applied to the wideband pixels and the narrowband pixels is greater than the relative difference during readout in the first mode.

[0039] In some embodiments of the method according to the seventh aspect in the second mode, the analog gain applied to the wideband pixels is greater than the analog gain applied to the wideband pixels in the first mode.

[0040] In some embodiments of the method according to the seventh aspect, the analog gain applied to the wideband pixel in the second mode is H times the analog gain applied to the wideband pixel in the first mode.

[0041] In some embodiments of the method of the fifth, sixth or seventh aspect, H is greater than 2.

[0042] In some embodiments of the method of the fifth, sixth or seventh aspect, H is less than 6.

[0043] In some embodiments of the method according to the fifth, sixth or seventh aspect, H is between 2.5 and 3.5.

[0044] In some embodiments of the method of the fifth, sixth or seventh aspect, H is between 2.8 and 3.

[0045] In some embodiments of the method according to the fifth, sixth or seventh aspect, H is about 2.9.

[0046] According to an eighth aspect of the present disclosure, there is provided a method performed in a camera having at least one image sensor, the image sensor including a pixel array with narrowband pixels and wideband pixels of at least three colors, the method including selectively operating in either a normal mode or a high dynamic range mode, the image sensor operating in the normal mode at a first pixel gain setting, and the image sensor operating in the high dynamic range mode at a second pixel gain setting in which the wideband pixels are read out with a higher pixel gain than in the normal mode.

[0047] In some embodiments of the method according to the eighth aspect, in the high dynamic range mode the narrowband pixels are read out with a second pixel gain setting that is different from the normal mode.

[0048] In some embodiments of the method according to the eighth aspect, in the high dynamic range mode, the pixel gain setting used for the wideband pixels is increased by a factor in the range of 2 to 6 relative to the pixel gain setting of at least one color of the narrowband pixels.

[0049] In some embodiments of the method of the eighth aspect, in the high dynamic range mode, the pixel gain setting used for the wideband pixels is increased by a factor in the range of 2x to 6x relative to the pixel gain setting of the narrowband pixels having a green color filter.

[0050] In some embodiments of the method according to the eighth aspect, the magnification is between 2.5 and 3.5.

[0051] In some embodiments of the method of the eighth aspect, the magnification is about 2.9.

[0052] In some embodiments of the method according to the fifth, sixth, seventh or eighth aspect, the pixel gain is an analog gain, or the pixel gain setting is an analog gain setting.

[0053] In some embodiments of the method according to the fifth, sixth, seventh or eighth aspect, the pixel gain is a pixel conversion gain, or the pixel gain setting is a pixel conversion gain setting.

[0054] In some embodiments of the method according to the fifth, sixth, seventh or eighth aspect, the method further comprises generating one or both of a narrowband luminance value Y based on output from the narrowband pixels and / or a wideband luminance value W based on output from the wideband pixels.

[0055] In some embodiments of the method according to the fifth, sixth, seventh or eighth aspect, the method further comprises processing the narrowband luminance value and the wideband luminance value to generate a combined luminance value.

[0056] In some embodiments of the method according to the fifth, sixth, seventh or eighth aspect, the narrowband luminance value is equal to or greater than a saturation value Y SAT and the broadband luminance value has a saturation value W SAT W SAT / Y SAT is greater than 3.

[0057] In some embodiments of the method of the fifth, sixth, seventh or eighth aspect, W SAT / Y SAT is greater than 6.

[0058] In some embodiments of the method of the fifth, sixth, seventh or eighth aspect, W SAT / Y SAT is greater than 8.

[0059] In some embodiments of the method of the fifth, sixth, seventh or eighth aspect, W SAT / Y SAT is about 9.

[0060] According to a ninth aspect of the present disclosure, there is provided a method according to any one of the fifth to eighth aspects, wherein the camera is a camera according to any one of the first to fourth aspects.

[0061] While the invention disclosed herein is amenable to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and described in detail. It should be understood, however, that the drawings and detailed description are not intended to limit the invention to the particular forms disclosed. Furthermore, all alternative combinations of two or more of the individual features mentioned or apparent from the text or drawings comprise additional aspects or inventions disclosed, which may be the subject of claims. [Brief explanation of the drawings]

[0062] [Figure 1] FIG. 1 shows the analog responses of R, G, B and broadband pixels in an image sensor with an RGBW (red, green, blue and white (broadband)) color filter array.

[0063] [Figure 2] FIG. 10 illustrates the analog responses of R, G, B and wideband pixels when an image sensor with an RGBW color filter array according to an embodiment of the present disclosure operates in a second mode.

[0064] [Figure 3A] FIG. 1 illustrates the spectral responses of R, G, B and wideband pixels in an image sensor with an RGBW color filter array adapted for use with embodiments of the present disclosure. [Figure 3B] FIG. 1 illustrates the spectral responses of R, G, B and wideband pixels in an image sensor with an RGBW color filter array adapted for use with embodiments of the present disclosure.

[0065] [Figure 4] FIG. 5 illustrates an example spectral response of the sensor of FIG. 4 operating in a second mode according to an embodiment of the present disclosure.

[0066] [Figure 5A] FIG. 2 is a diagram showing details of an RGBW color filter array applied in one embodiment of the present disclosure. [Figure 5B] FIG. 2 is a diagram showing details of an RGBW color filter array applied in one embodiment of the present disclosure. [Figure 5C] FIG. 2 is a diagram showing details of an RGBW color filter array applied in one embodiment of the present disclosure.

[0067] [Figure 6] FIG. 1 is a schematic block diagram of a camera according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0068] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, which should be read in conjunction with the summary set forth above and the accompanying drawings. It will be apparent, however, that the present invention may be practiced without these specific details. Also, in some instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring key details.

[0069] Pixel conversion gain (HDR) systems improve dynamic range by using at least two different analog gains (e.g., pixel conversion gains) in image sensors. Each pixel in an image sensor contains one or more photodiodes that convert light into an electrical charge. When each pixel is readout, the accumulated charge within the pixel is converted into a voltage signal, which is then measured and digitized to generate a pixel value. Pixel conversion gain is a measure of the ratio between the change in accumulated charge caused by a photon incident on the pixel and the change in the readout voltage signal. Pixel circuits can be designed with high pixel conversion gains to improve pixel sensitivity in low-light conditions. However, if lighting conditions are too bright, the photodiodes accumulate excess charge and reach saturation. Pixels with low conversion gains do not saturate easily, making them suitable for high-brightness conditions, but their low sensitivity can prevent them from capturing sufficient image detail in low-light conditions. Devices employing pixel conversion gain (HDR) exploit these different characteristics to expand dynamic range. Such devices can vary and optimize pixel conversion gain depending on the light level, and different pixel conversion gains can be applied to different pixels. As a result, some pixels with low conversion gain are optimized to avoid saturation in bright light conditions, while pixels with high conversion gain optimize performance in low light conditions where saturation is less likely.

[0070] While prior art techniques vary or select these traditional pixel gains to optimize image capture for different pixel types in order to avoid saturation of the image sensor, the present inventors have discovered that, although it may seem counterintuitive, in some circumstances it is useful to use pixel conversion gain HDR techniques without placing emphasis on saturation avoidance.

[0071] Applicant's patent US11,258,992B2 (Buettner), the contents of which are incorporated herein by reference, describes an image sensor having a color filter array containing a pattern of wideband pixels and at least two (usually three) types of narrowband pixels. Using such a filter array, it is possible to generate multiple luminance images adapted to different light levels. A first luminance image Y can be created by combining only narrowband pixel values, and alternatively (or additionally), a second luminance image W can be created using only wideband pixels. Generally, the first luminance image is more suitable for use in bright lighting conditions because the color pixels used to generate it are less likely to saturate, while the second luminance image is more suitable for use in low-light conditions because the wideband pixels used to generate it capture more light in low-light conditions. These two luminance representations can also be blended into a single luminance image according to a combination algorithm.

[0072] Figure 1 shows the analog response of R, G, B, and broadband pixels, illustrating their behavior under natural light and natural light simulated by a white LED (or similar). Each plot shows the analog light response for either a broadband (W), green (G), blue (B), or red (R) pixel. The vertical axis represents pixel output, and the horizontal axis represents light level. The broadband pixel response increases from zero, rising rapidly to W. SAT Each of the other pixel types follows a roughly similar response pattern, but with different saturation levels, i.e., green is saturated at G SAT , blue is B SAT , red is RSAT It can be seen that the wideband pixel is the most sensitive, but saturates earlier than the other pixels.

[0073] As is well known, a typical image sensor with a color filter array (CFA) has pixels arranged in a repeating unit cell of pixels with different color filters. In the following examples, we will assume an image sensor with unit cells having narrowband pixels with a T1:T2:T3 ratio of 1:1:1, where "T1" corresponds to a narrowband pixel of type 1, "T2" corresponds to a narrowband pixel of type 2, and "T3" corresponds to a narrowband pixel of type 3. However, it should be construed that the present disclosure is applicable, mutatis mutandis, to other unit cells.

[0074] An example of such a unit cell contains red, green, and blue pixels and a broadband pixel. In this example, the luminance Y can be calculated as Y=(R+G+B) / 3. Therefore, saturation of the luminance output occurs when Y SAT =(R SAT +B SAT +G SAT ) / 3. Because the broadband pixels have virtually no color selectivity for the spectral range of the R, G, and B pixels, the broadband image can be thought of as representing a second luminance value W. The narrowband luminance saturation point Y SAT The saturation point W of the wideband output SAT Compared to the Y pixel, we can see the increased dynamic range provided by the inclusion of the W pixel in normal operation, which is SAT / W SAT is expressed by

[0075] In one embodiment of the present disclosure, the sensor response is modified to further increase the dynamic range, as shown in FIG. 2. In one embodiment, this is achieved by increasing the pixel gain applied to the wideband pixels relative to the pixel gain applied to the narrowband pixels. It is simplest to fix the pixel gain applied to the narrowband pixels and only vary the pixel gain applied to the wideband pixels. However, there are cases where the pixel gain applied to the narrowband pixels is not fixed. FIG. 2 shows the analog responses of the R, G, B, and wideband pixels in one embodiment of a high dynamic range mode of operation, illustrating their behavior under daylight and simulated daylight illumination by a white LED (or similar). Each plot shows the analog light response of the wideband (W), green (G), blue (B), and red (R) pixels, respectively. As can be seen, the response of the wideband pixels increases from zero, and as the W SAT Each of the other pixel types follows a roughly similar response pattern, but for green, the response SAT , B for blue SAT , R for red SAT In this second high dynamic range operating mode, the wideband pixels have a relatively high pixel gain, allowing them to saturate at W SAT is generated, which is significantly lower than the saturation point of the narrowband pixel. If the gain of the wideband pixel is increased by a factor of "H", the saturation point shifts downward by 1 / H (i.e., occurs at a light level 1 / H that of the normal mode), and WHG = W SAT / H.

[0076] Considering the same example CFA arrangement, this results in the following: Dynamic range (high pixel gain) = Y SAT / (W SAT / H)=H*Y SAT / W SAT

[0077] Figure 3A, a reproduction of a portion of Figure 5 in U.S. Patent No. 11,258,992 B2, shows the spectral response of an image sensor with three narrowband pixel colors (red, green, and blue) and one wideband pixel type (mono). The figure shows the spectral response of each B, G, and R pixel (following the peaks from left to right) as dotted lines, and the spectral response of the wideband pixel (mono) as a solid line. The plot assumes the presence of an infrared cutoff filter. It is understood that the image sensor response shown here is a combination of the sensor pixel response, the filtering effect of the corresponding pixel in the CFA, and the effects of other filters, such as the infrared cutoff filter. As can be seen, the peak response of the pixel corresponding to the wideband filter is centered around the combined spectral response of the pixel with the R, G, and B filters, and the wideband pixel (solid line) has a sensitivity that roughly matches the sum of the spectral responses of the narrowband pixels. For ease of comparison, all values ​​have been normalized to the peak sensitivity of the wideband pixel. These plots represent the pixel's spectral output response in the digital domain, including the effects of analog amplification such as pixel conversion gain applied during pixel readout, and the effect of any subsequent digital gains that may be applied to the R, G, B, and W sensor outputs. In this regard, to achieve a reasonably close match between the Mono (broadband) and R, G, and B composite spectral responses, it is necessary to apply different digital gains to each channel to adjust their spectral response levels.

[0078] In the present invention, in addition to the spectral response of the pixel, Figure 3A also shows the luminance sensitivity Y (dashed line) of a sensor having the same number of R, G, and B pixels as described above. As shown, the narrowband luminance Y is approximately one-third of the broadband value W (corresponding to the broadband luminance value) across the entire spectral range of the sensor. Figure 3B (which reproduces the plot from Figure 5 of U.S. Pat. No. 11,258,992 B2) shows the same data as Figure 3A, but scales the luminance value Y by a factor of three to visualize the appropriate level of match between the combined spectral response of the W pixel and a pixel with R, G, and B filters in one embodiment.

[0079] Figure 4 adds to Figure 3A, showing a plot (dashed-dotted line) of the spectral response output of the wideband pixel in the second high dynamic range mode described in Figure 2. In this mode, the WHG response clearly exceeds the W response output in the first mode across the entire spectral range of the W pixel. Assuming no change in digital gain, WHG is H times higher than W and approximately 3H times higher than Y.

[0080] FIG. 5A is a schematic representation of a two-dimensional array of pixels 50 with color filters and broadband filters, having a unit cell 55 with a 1:1:1 ratio of T1:T2:T3 pixels, which can be used in the systems and methods described herein. This unit cell 55 is a 6x6 pixel pattern that is repeated throughout the filter array. The narrowband filters are labeled "R," "G," and "B," representing filters with passbands centered at red, green, and blue, respectively. The "W" pixel is a broadband pixel with a passband that can generate a spectral output as described in connection with FIG. 3A. FIG. 5B shows a color representation of the sensor array to aid visualization. In this embodiment, there is one "W" pixel for each narrowband pixel.

[0081] A further advantage of the embodiment of FIG. 5A is that it can be operated in a binning mode, which can advantageously improve sensitivity and readout speed. As shown in FIG. 5C, the sensor's pixels can be divided into pairs of wideband filter elements (W) and pairs of identical narrowband filter elements. These pairs are indicated by connected diagonal lines. In binning mode, pixel pairs are read out simultaneously and the pixel values ​​are combined. This binning mode can also be tailored to be usable in both the normal and high dynamic range modes described herein, since pixel pairs share the same gain during readout. Advantageously, half the total number of pixel responses need to be read out from the image sensor, allowing for double the frame rate while maintaining the same field of view compared to a full readout mode in which all pixels are read out individually.

[0082] Figures 7 and 8 of U.S. Patent No. 11,258,992 B2 schematically illustrate an example method for generating a narrowband luminance image from narrowband pixels and an example method for generating a wideband luminance image from wideband pixels of an RGBW sensor element. For example, Figure 8 describes a process that can be used for the pixel arrays of Figures 5A-5C herein. These techniques are broadly applicable to cameras operating in either the high dynamic range mode or the normal dynamic range mode of the present disclosure.

[0083] Embodiments of the present disclosure allow for the generation of two luminance outputs (e.g., two luminance images) with different absolute sensitivities, one generated from narrowband pixels and the other generated from wideband pixels, which can optionally be configured with matching spectral sensitivity profiles.

[0084] In normal operation mode, a relatively high dynamic range can be achieved by mixing these two luminance outputs. This is because the luminance output W from the wideband pixel is prioritized in dark conditions (e.g., lighting conditions or partial scenes), and the luminance output Y from the narrowband pixel is prioritized in bright conditions. In high dynamic range mode, the sensitivity difference between the wideband and narrowband pixels can be increased by applying an increased pixel gain to the output of the wideband pixel relative to the narrowband pixel. An even higher dynamic range can be achieved by mixing the two luminance outputs obtained in this high dynamic range mode. This is because the luminance output WHG from the wideband pixel is prioritized in dark conditions (e.g., lighting conditions or partial scenes), and the luminance output Y from the narrowband pixel is prioritized in bright conditions.

[0085] FIG. 6 is a schematic diagram of a camera 10 capable of implementing the methods described herein. The camera 10 includes an image capture system 12 configured to convert received light into raw image data. In this example, the image capture system 12 includes an image sensor 12A (e.g., a CCD or CMOS image detection chip or similar) and an associated optical filter 12B (e.g., an IR-cut filter or an optical low-pass filter). Additionally, an optical system 14, e.g., a lens, is provided for forming an image on the image sensor. The image sensor includes a CFA (color filter array), which is in the form of a spatial array of filters placed in front of the image sensor chip, allowing it to capture an image consisting of an array of pixels at different locations and generate a color image. The filters of the CFA correspond spatially to each photoelectric conversion element (photosite) of the image sensor, and each photosite has a corresponding spectrally selective filter. In this disclosure, pixels generated by photosites with a particular filter in the CFA are named based on that filter. For example, a pixel generated by a photosite with a "green" color filter is referred to as a green pixel. Examples of filter arrays suitable for use with the camera of FIG. 6 are shown in FIGS. 5A-5C.

[0086] The raw sensor data generated by image capture system 12 is sent to image processing subsystem 18. In an embodiment of the present disclosure, the sensor gain is changed or set in image capture system 12 to allow it to operate in either a first, normal mode, or a second, high dynamic range mode. This can be accomplished by changing the analog gain applied during pixel readout or by modifying the digital gain of the sensor output. In a preferred embodiment, the pixel conversion gain is modified to switch between modes.

[0087] The image processing subsystem 18 may include one or more data processors with ASICs, FPGAs, microprocessors, or associated software and is configured to perform various image processing tasks. These tasks include, but are not limited to, correcting for unwanted optical effects such as pincushion distortion, demosaicing, noise reduction, and correcting pixel-to-pixel variations in the captured video data, for example, by removing dead pixels and correcting for variations in conversion efficiency. A working memory 20 is provided to allow temporary storage of data, software, etc. during image processing and / or image compression and other tasks.

[0088] The image processing subsystem 18 also includes a video encoding system 22. The video encoding system 22 is typically implemented by providing software configured to enable a processor to implement one or more video codecs, which can be used to encode video data in a desired format and, if necessary, compress it.

[0089] The image processing subsystem 18 may include a format conversion system 24, which processes the video output data into a format that can be transmitted via a video transmission system 26. The video transmission system 26 is generally configured to transmit only video data that conforms to one or more video transmission protocols. The format conversion system 24 formats the video data into one of the video transmission formats to enable transmission, which is then passed to the video transmission system 26. This may include transcoding the video data from its original format to one of the appropriate video transmission formats for the video transmission system 26. The video transmission system may transmit (and optionally receive) the video output data via a video interface having at least a video output port. The video interface may be bidirectional and therefore may also include a video input port. By way of example, the video interface may be an SDI interface or a similar interface.

[0090] The camera includes a data storage system in the form of a memory control subsystem 28, which is configured to control persistent storage of video data (and other data) in local non-volatile memory 30. Local memory 30 can be removable memory, such as a memory card or a removable hard drive. However, in the general case, memory control subsystem 28 is configured to send and receive control signals to local memory 30 to control the storage and retrieval of video data on memory 30, as well as to encode or format data for storage. Memory 30 may be a solid-state drive operating according to the Serial ATA protocol, in which case the memory control subsystem operates to control the operation of the SATA drive and manage the reading and writing of data to it.

[0091] Terms expressly defined herein shall govern the interpretation of the terms contained in the appended claims. No limitation, element, property, feature, advantage, or attribute not expressly recited in a claim shall in any way limit the scope of that claim.

[0092] As used herein, the terms "include" and "comprise" (and variations thereof, such as "including," "includes," "comprising," "comprises," and "comprised") are intended to be inclusive and are not intended to exclude additional features, components, technical elements, or steps.

[0093] With respect to aspects of the present disclosure described using flowcharts, a particular flowchart step may be performed by various methods or various devices, systems, or system modules. A particular flowchart step may be divided into multiple steps, and multiple flowchart steps may be combined into a single step, unless expressly stated to the contrary. Additionally, the order of steps may be changed without departing from the scope of the present disclosure, unless expressly stated to the contrary.

Claims

1. 1. A method in a camera having at least one image sensor, the image sensor comprising a pixel array including narrowband pixels and wideband pixels of at least three colors, the method comprising: reading out the narrowband pixels and the wideband pixels with different pixel gains; The method, wherein the wideband pixels are read out with a higher pixel gain than the narrowband pixels.

2. the narrowband pixels include red, green and blue pixels, and the method comprises:

10. The method of claim 1, comprising reading out the wideband pixels with a pixel gain that is H times the pixel gain used to read out the green pixels.

3. 1. A method performed in a camera having at least one image sensor with a pixel array including narrowband pixels and wideband pixels of at least three colors, the method comprising: selectively operating the camera in a first mode in which the image sensor has a first pixel gain setting and produces an output having a first response; or selectively operating the camera in a second mode in which the image sensor has a second pixel gain setting and produces an output having a second response, the method further comprising: in a first mode, the wideband pixel produces a digital output having a spectral output response approximately equal to the sum of the spectral output responses of the narrowband pixels; and in a second mode, the wideband pixel produces a digital output having a greater spectral output response than in said first mode; A method comprising:

4. 4. The method of claim 3, wherein in said second mode, the wideband pixel has a spectral output response approximately equal to H times the sum of the spectral output responses of the narrowband pixels.

5. 1. A method performed in a camera having at least one image sensor with a pixel array including narrowband pixels and wideband pixels of at least three colors, the method comprising: selectively operating the camera in a first mode in which the image sensor is read out using a first analog gain setting; or selectively operating the camera in a second mode in which the image sensor is read out using a second analog gain setting; A method in which the relative difference between the analog gain applied to the wideband pixels and the analog gain applied to the narrowband pixels when readout in the second mode is greater than the relative difference between the analog gain applied to the wideband pixels and the analog gain applied to the narrowband pixels when readout in the first mode.

6. 6. The method of claim 5, wherein an analog gain applied to wideband pixels in the second mode is greater than an analog gain applied to wideband pixels in the first mode.

7. 7. The method of claim 5, wherein the analog gain applied to the wideband pixels in the second mode is H times the analog gain applied to the wideband pixels in the first mode.

8. The H is greater than 2; less than 6; Between 2.5 and 3.5; Between 2.8 and 3; Approximately 2.9; 38. The method of any one of claims 32, 34 or 37, wherein one or more of:

9. 1. A method performed in a camera having at least one image sensor with a pixel array including narrowband pixels and wideband pixels of at least three colors, the method comprising: selectively operating said camera in either a normal mode or a high dynamic range mode; operating the image sensor in a normal mode with a first pixel gain setting; operating the image sensor in a high dynamic range mode with a second pixel gain setting in which wideband pixels are read out with a higher gain than in a normal mode; A method comprising:

10. 10. The method of claim 9, wherein in the high dynamic range mode, narrowband pixels are read out with a second pixel gain setting that is different from the normal mode.

11. In the high dynamic range mode, a pixel gain setting used for the wideband pixels is increased by a factor in the range of 2 to 6 relative to a pixel gain setting used for at least one color of the narrowband pixels; Optionally, the magnification is in the range of 2.5x to 3.5x; More preferably, the magnification is about 2.9 times. The method of claim 10.

12. The method of any one of claims 1 to 11, wherein the pixel gain is an analog gain or the pixel gain setting is an analog gain setting.

13. The method of any one of claims 1 to 12, wherein the pixel gain is a pixel conversion gain or the pixel gain setting is a pixel conversion gain setting.

14. a narrowband luminance value Y based on the output from the narrowband pixel; and a broadband luminance value W based on the output from said broadband pixel; The method of any one of claims 1 to 13, further comprising generating one or both of:

15. The method of any preceding claim, further comprising processing the narrowband luminance value and the wideband luminance value to generate a combined luminance value.

16. The narrowband luminance value is a saturation value Y ＳＡＴ and the broadband luminance value has a saturation value W ＳＡＴ and W ＳＡＴ / Y ＳＡＴ 16. The method of claim 14 or 15, wherein is greater than 3.

17. The W ＳＡＴ / Y ＳＡＴ 17. The method of claim 16, wherein is greater than 6, or greater than 8, or about 9.

18. A camera configured to carry out the method of any one of claims 1 to 17.

19. The camera of any one of claims 1 to 18, wherein the pixel array comprises three types of narrowband pixels (T1, T2, T3) and one type of wideband pixel (w).

20. 20. The camera of claim 19, wherein the pixel array has a unit cell having T1, T2, and T3 pixels in a 1:1:1 ratio.

Images