Image sensor and method for HDR image capture using image sensor system

Abstract

A method and image sensor for high dynamic range (HDR) image capture using an image sensor system is described. For example, an image sensor array is configured according to a red-green-blue-luminance (RGBL) CFA pattern such that at least 50% of the imaging pixels of the array are luminance (L) pixels. In each image capture time window, multiple (e.g., three) luminance enhanced (LE) component images are generated. Each LE component image is generated by exposing the image sensor to incident illumination for a corresponding amount of time, using pixel binning during readout to generate significantly downsampled color and luminance capture frames, generating an upsampled luminance guide frame from the luminance capture frame, and using the upsampled luminance guide frame to guide upsampling of the color capture frame (e.g., and demosaicing). The resulting LE component images can be digitally combined to generate an HDR output image.

Description

[0001] Cross-references

[0002] This application is a non-provisional application of and claims priority to U.S. Provisional Patent Application No. 63 / 226,731, filed on July 28, 2021, entitled “STAGGERED HDR WITH PIXEL BINNING FOLLOWED BY UPSAMPLING WITH AN RGBW CFA,” the entire contents of which are incorporated herein by reference. This application is also related to U.S. Non-Provisional Patent Application No. 17 / 581,964, filed on January 23, 2022, entitled “STAGGERED HIGH-DYNAMIC-RANGE IMAGE CAPTUREWITH LUMINANCE-DRIVEN UPSAMPLING OF PIXEL-BINNED IMAGE SENSOR ARRAY OUTPUT,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments described herein relate to image sensor devices, and more particularly, to interleaved high dynamic range (HDR) image capture with luma-driven upsampling of a binned image sensor array output of an image sensor having a red-green-blue-luminance (RGBL) color filter array (CFA). Background Art

[0004] Modern image sensors with high pixel counts attempt to achieve high frame rates, low read noise, and high dynamic range with minimal power consumption. The basic function of a modern CMOS image sensor (CIS) is to capture photons that are converted into electrons in a photodiode. These captured electrons are read out by a series of ADCs that are part of the sensor. In the readout chain, higher power can provide lower read noise or higher dynamic range, thereby improving image quality. However, higher power consumption also results in shorter battery life and heat generation, which have inherent limitations in consumer products and can lead to poorer sensor performance in a CIS environment.

[0005] In many modern applied CIS, for example, in mobile phones, the dynamic range is inherently limited due to the small pixel size. The small pixel size is dictated by the limited available area and volume of the camera on a small portable electronic device. To achieve a larger dynamic range, multiple frames with different exposure times can be captured, which are digitally combined to create a single image. Using this approach, the dynamic range can be increased by the ratio of the maximum and minimum exposure times. A limitation of this approach is that more time is required to capture multiple frames. Objects that move during the image capture operation will have a larger blur and may also appear at different locations in the frame with different exposure times. This may cause image defects or artifacts in the process of digitally combining multiple frames.

[0006] Various conventional approaches seek to reduce such image defects and artifacts, but these conventional approaches are often limited at least by readout time. For example, in high-resolution image sensors with very high pixel counts, reading out each individual pixel in each image capture frame may consume excessive time and power. For each image capture frame, capturing multiple image frames at different exposure times further increases the already excessively consumed time and power resources. Summary of the invention

[0007] Embodiments provide novel technical methods for efficient interleaved high dynamic range (HDR) output of images captured using high pixel count image sensors, based on pixel merging and subsequent brightness guided upsampling. For example, an image sensor array is configured according to a red-green-blue-brightness (RGBL) CFA pattern so that at least 50% of the imaging pixels of the array are brightness (L) pixels. In each image capture time window, multiple (e.g., three) brightness enhanced (LE) component images are generated. Each LE component image is generated by exposing the image sensor to incident illumination for a corresponding amount of time, using pixel merging during readout to generate significantly downsampled color and brightness capture frames, generating upsampled brightness guided frames from the brightness capture frames, and using the upsampled brightness guided frames to guide the upsampling of the color capture frames (e.g., and de-mosaicing). The resulting LE component images can be digitally combined to generate an HDR output image.

[0008] In one set of embodiments, a method for high dynamic range (HDR) image capture using an image sensor system is provided, the image sensor system having an image sensor array formed of a plurality of imaging pixels, the plurality of imaging pixels being configured according to a red-green-blue brightness (RGBL) color filter array (CFA) pattern and arranged to be addressable in imaging pixel groups. The method comprises: for each of N subframes, each n-th subframe of the N subframes being associated with an n-th exposure window and an n-th readout window, each n-th readout window starting after an n-th exposure time after the start of the n-th exposure window, N being an integer greater than 1: sequentially exposing each imaging pixel group of the image sensor array to incident illumination for the n-th exposure time over the n-th exposure window; and sequentially reading out each imaging pixel group over the n-th readout window using pixel merging, such that each imaging pixel group is read out immediately after the imaging pixel group is exposed to the incident illumination for the n-th exposure time, and each imaging pixel group is read out immediately after the imaging pixel group is exposed to the incident illumination for the n-th exposure time, and each imaging pixel group is read out immediately after the imaging pixel group is exposed to the incident illumination for the n-th exposure time, and each imaging pixel group is read out immediately after the imaging pixel group is exposed to the incident illumination for the n-th exposure time, and each imaging pixel group is read out immediately after the imaging pixel group is exposed to the incident illumination for the n-th exposure time, and each imaging pixel group is read out immediately after the imaging pixel group is read out after the imaging pixel group is read out after the imaging pixel group is read out At the end of the output port, the pixel merging results in a corresponding one of N downsampled HDR subframes having a corresponding RGB capture frame and a corresponding L capture frame; the N brightness enhanced component images are generated by upsampling the corresponding L capture frame for each of the N downsampled HDR subframes to generate a corresponding L guided frame, and applying brightness guided interpolation to the corresponding RGB capture frame based on the corresponding L guided frame to upsample the RGB capture frame to form a corresponding one of the N brightness enhanced component images; and digitally combining the N brightness enhanced component images to generate a single HDR output image.

[0009] In another set of embodiments, an image sensor is provided. The image sensor includes an image sensor array, a binning readout circuit, a sensor controller, an upsampling engine, and an HDR engine. The image sensor array is formed of a plurality of imaging pixels configured according to a red-green-blue brightness (RGBL) color filter array (CFA) pattern and arranged to be addressable in an imaging pixel group. The binning readout circuit is coupled to the image sensor array to read out the imaging pixel group using pixel binning to generate N downsampled HDR subframes, each subframe having a corresponding RGB capture frame and a corresponding L capture frame, N being an integer greater than 1. The sensor controller is configured to generate each of the N downsampled HDR subframes by: over a corresponding exposure window, directing each imaging pixel group of the image sensor array to be sequentially exposed to incident illumination for a corresponding exposure time associated with the downsampled HDR subframe; and over a corresponding readout window, directing each imaging pixel group to be sequentially readout by the merged readout circuitry such that each imaging pixel group is readout immediately after the imaging pixel group is exposed to the incident illumination for the corresponding exposure time, and such that at the end of the corresponding readout window, the pixel merging results in the corresponding RGB capture frame and the corresponding L capture frame having the downsampled HDR subframe. An upsampling engine is coupled to the merged readout circuitry to generate N brightness-enhanced output images by applying brightness-guided interpolation to the corresponding RGB capture frame based on the L capture frame for each downsampled HDR subframe. An HDR engine is configured to generate a single HDR output image by digitally combining the N brightness-enhanced output images.

[0010] Other features and advantages will become apparent to those skilled in the art upon reviewing the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings constitute a part of this disclosure and describe exemplary embodiments of the present invention. Together with the description, they serve to explain the principles of the present invention.

[0012] Figure 1 A block diagram of an imaging sensor environment is shown as context for the various embodiments described herein.

[0013] Figures 2A-2D An example of a non-Bayer red-green-blue-luminance (RGBL) color filter array (CFA) is shown in accordance with various embodiments described herein.

[0014] Figure 3 An example circuit is shown including a set of imaging pixels and illustrative readout circuitry configured to charge combine the outputs of the imaging pixels.

[0015] Figures 4A-4DThe various embodiments described herein are shown. Figures 2A-2D Four examples of pixel binning patterns for a non-Bayer RGBLCFA implementation.

[0016] Figure 5 A representative flow chart of one illustrative stage of generating an LE output image is shown, wherein a downsampled L capture frame is used to generate an upsampled L guide frame.

[0017] Figures 6A-6C A representative flow diagram of illustrative stages of generating an LE output image is shown, wherein an upsampled L-guided frame is used to provide luma-guided upsampling of a downsampled RGB capture frame to generate color planes of an upsampled RGB frame.

[0018] Figure 7 An illustrative timing diagram of a conventional multi-frame high dynamic range (HDR) approach is shown.

[0019] Figure 8 A flow chart showing a conventional interlaced HDR approach that alleviates some of the limitations of multi-frame HDR.

[0020] Fig. 9 A flow chart showing a novel interleaved HDR method for reducing time image sensor readout using the type of pixel binning downsampling described herein.

[0021] Fig.10 A flow chart is shown of an illustrative method of high dynamic range (HDR) image capture using an image sensor system in accordance with various embodiments described herein.

[0022] In the accompanying drawings, similar components and / or features may have the same reference number. In addition, various components of the same type may be distinguished by following the reference number with a second reference number that distinguishes between the similar components. If only the first reference number is used in the specification, the description applies to any one of the similar components having the same first reference number, regardless of the second reference number. DETAILED DESCRIPTION

[0023] In the following description, many specific details are provided to fully understand the present invention. However, it should be understood by those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, features and techniques known in the art will not be described for the sake of brevity.

[0024] Electronic devices that include high-performance image sensors have become ubiquitous, and consumers are demanding ever-increasing resolution from these image sensors. While such image sensor designs continue to evolve toward high pixel counts and image resolutions, the implementation of such features often competes with other desired design features and constraints, such as high frame rates, low read noise, minimal power consumption, etc. For example, in many imaging environments, it is desirable to provide a high dynamic range (HDR) to support high-quality image capture in a wide range of lighting conditions and / or scenes with high brightness variations. However, in many modern applications, such as for small size image sensors integrated into smartphones and / or other portable consumer electronic devices, the dynamic range may be inherently limited due to the drive to continually shrink the image sensor pixel size.

[0025] Among the various types of conventional methods for achieving greater dynamic range, some methods acquire multiple subframes in each image acquisition frame, where each subframe is associated with a different exposure time. For example, a longer exposure time may provide improved performance in low light conditions by allowing a photosensor in an image sensor more time to accumulate photocharge, and a shorter exposure time may provide improved performance in bright conditions by avoiding saturation of the photosensor due to overexposure. Multiple subframes with different exposure times may be captured and digitally combined to create a single output image. Using such conventional methods, the dynamic range may be increased by the ratio of the maximum and minimum exposure times. However, capturing multiple subframes requires more time, and any objects that move during the image capture operation will cause blurring and / or may appear in different locations in the frame at different exposure times. This may manifest as image defects and / or artifacts in the output image generated by digitally combining multiple subframes.

[0026] Various conventional methods attempt to reduce such image defects and artifacts, but these conventional methods are at least limited by the readout time. For example, in high-resolution image sensors with very high pixel counts (e.g., tens of millions of photosensors), reading out each individual pixel in each image capture frame may consume excessive time and power. For each image capture frame, capturing multiple subframes at different exposure times further increases the already excessively consumed time and power resources.

[0027] For context, Figure 7 and Figure 8 An illustration of two different traditional HDR approaches is provided. Figure 7An illustrative timing diagram 700 of a conventional multi-frame HDR method is shown. In diagram 700, the vertical dimension generally represents time and the horizontal dimension generally represents array width. As shown, in the multi-frame HDR method, multiple sub-frames are captured and read out in succession, each with a corresponding exposure time. During the "frame 1 exposure" window, the photosensors of the array are exposed row by row.

[0028] As used herein, a photosensor is considered to be "exposed to" incident illumination when the photosensor is in a state to accumulate photocharge for subsequent readout for image capture purposes. In practice, incident illumination may be permitted to reach the entire image sensor array (e.g., a physical shutter may be fully open) during the entire Frame 1 exposure window, but only a portion of the photosensors in the array may be configured to be "exposed" to that incident illumination at any particular time during the Frame 1 exposure window. For example, circuitry may be used to maintain a photosensor in a state where it does not accumulate photocharge prior to the start of its scheduled exposure, to reset the photosensor so that any accumulated photocharge is cleared prior to the start of its scheduled exposure, or to otherwise selectively configure the photosensor so that, even if illumination is incident on the photosensor, the photosensor is not effectively "exposed to" illumination prior to the start of its scheduled exposure.

[0029] In practice, the readout speed of the readout circuitry defines the minimum duration of the Frame 1 exposure and Frame 1 readout windows. For example, a high pixel count array may have thousands of rows. If all rows of the array were to begin exposure at the same time (i.e., if the height of the Frame 1 exposure time corresponds to a single row time), then the last row of the array would have to remain exposed for much longer than Exposure Time 1 while waiting for the readout circuitry to read out the thousands of previous rows.

[0030] In a multi-frame HDR approach, a second readout window ("frame 2 readout") cannot begin until the frame 1 readout window ends, and a third readout window ("frame 3 readout") cannot begin until the frame 2 readout window ends. For example, the readout circuit reads all rows of the array in each readout window, and then restarts with the first row of the array in the next readout window. As shown, each subsequent exposure window may begin before the end of the previous readout window. For example, the frame 2 exposure window may begin before the end of the frame 1 readout window (i.e., the first row of the array may be reset and may begin to be re-exposed while the last row of the array is still at its first exposure time). However, the start of an exposure window must precede the start of its corresponding readout window by a corresponding exposure time, not before. For example, the frame 2 exposure window precedes the start of the frame 2 readout window by exactly exposure time 2. Therefore, the earliest start time of the frame 2 exposure window is exposure time 2 before the end of the frame 1 readout window (or slightly later to account for setup time and / or other practical considerations).

[0031] The readout time is typically dependent on the readout circuitry, the number of rows of the array, and the width of each row; not the exposure time. Thus, the readout window has the same duration for all subframes, and the total image capture time is at least limited by the product of the number of subframes and the full array readout time. For example, in the case shown, for three subframes, the total image capture time is at least three times the full array readout time. As an example, assume that the readout circuitry has a subframe readout time of 30 milliseconds (ms). To implement multi-frame HDR with three subframes having exposures of 10 ms, 1 ms, and 100 microseconds, respectively, the total image capture time for all three frames will take at least 90 ms.

[0032] For any particular row of the array in this example, the times are as follows: 10ms exposure (exposure time 1 during the frame 1 exposure window), 20ms wait time (for frame 1 readout), 1ms exposure (exposure time 2 during the frame 2 exposure window), 29ms wait time (for frame 2 readout), and 100us (exposure time 3 during the frame 3 exposure window). For any particular row, more than 60ms elapse between the start of exposure in the first subframe and the start of readout in the third subframe. During such long image capture times, any motion of scene objects will produce various image defects and artifacts.

[0033] Figure 8 A flow chart 800 is shown of a conventional interlaced HDR method that alleviates some of the limitations of multi-frame HDR. Figure 7 As shown, in diagram 800, the vertical dimension generally represents time and the horizontal dimension generally represents array width. Figure 7 By comparison, flowchart 800 represents a similar situation where HDR is performed with three subframes, each with a corresponding exposure time. Figure 7 The readout windows appear consecutively without any overlap. Figure 8 The interleaved HDR method shown in has overlapping readout windows.

[0034] For example, during the "frame 1 exposure" window, the photosensors of the array are exposed row by row. After a period of "exposure time 1", the row-by-row readout of the array begins. The readout of this first subframe takes some time, defining the height of the "frame 1 readout" window and the height of the frame 1 exposure window, as shown in reference Figure 7 The readout circuitry can be configured such that readout of each subsequent subframe (e.g., a "frame 2 readout" window) begins while readout of one or more previous subframes (e.g., a frame 1 readout window) is still occurring, such that the readout is "interleaved." This interleaved readout can be implemented in a variety of ways, for example, using parallelization in the readout circuitry.

[0035] Some conventional interleaved HDR approaches temporally interleave the frames on a row-by-row basis, or over a small group of rows. In this way, less time may elapse between the start of exposure in the first subframe of any particular row and the start of readout in the last subframe of that particular row, which may allow any motion artifacts on any given row to see the benefits of interleaved HDR operation. However, typically, such interleaving results in the readout time for any particular row being increased by a multiple corresponding to the number of subframes. For example, regardless of the amount of overlap between the readout frames, in the case shown, the same row is read out three times by the same readout circuitry, which effectively increases the width of the array by a factor of three. Compare Figure 7 and Figure 8 , it can be seen that Figure 8 The readout window ratio in Figure 7 The readout window in is about three times higher.

[0036] One consequence of the increased sub-frame readout window duration is that the total image capture time may not be noticeably improved. Although any particular readout window may begin sooner in an interleaved HDR environment than in a multi-frame HDR approach, the duration of each readout window is also considerably longer. For example, again comparing Figure 7 and Figure 8 , it can be seen that the total time (i.e., the total length of the arrow labeled "Time") is the same. Another consequence of the increase in the subframe readout time is that, while motion artifacts in a single row can be minimized, it takes longer to read out each subframe. For example, each subframe readout time may now correspond to a rolling shutter aperture of the imaging system, which increases the readout time from top to bottom and may produce (or exacerbate) other motion artifacts and defects. In this way, conventional interlaced HDR can provide better dynamic range with fewer motion artifacts than simple multi-frame HDR methods; but still tends to produce more motion artifacts than standard single-frame (i.e., non-HDR) image capture methods.

[0037] As described above, a limitation of conventional HDR is that the readout of multiple subframes per image capture window increases the total readout time of the image capture window, which can manifest as motion-related image artifacts and defects. In practice, various conventional techniques can be used to reduce the subframe readout time, for example, by reducing the resolution of the image sensor. For example, some high-performance image sensors use conventional downsampling techniques to reduce the amount of data read out from the sensor array in each readout window. While this approach can produce a more efficient readout (e.g., faster, lower power, etc.), this approach also reduces the resolution of the output image. In theory, advanced image processing algorithms can be used to upscale the resolution of the image with reduced resolution. Conventional techniques tend to rely heavily on computationally intensive neural networks or similar types of algorithmic structures. At least because these techniques tend to take more time to perform resolution upscaling than the image sensor generates the image, these techniques are typically performed "offline" (i.e., separate from the image sensor path and after the image sensor captures and outputs the image). For example, if a mobile phone can use these types of techniques to generate a magnified image of a single image frame over a period of several seconds or even longer, such capability may be useful for high-quality photography applications or for applying digital zoom or image resolution enhancement to an image previously captured and output from an image sensor (i.e., as a post-editing feature, etc.). However, these types of techniques are often impractical or infeasible for real-time applications, such as real-time magnification of an image or video (when the image or video is captured and generated), real-time magnification of an image preview pattern, etc. Even if a computational engine can be implemented using conventional techniques to support such real-time magnification, such a device may consume more power and / or other resources than simply generating a higher resolution image locally in the image sensor itself. Thus, the use of conventional techniques may reduce the HDR subframe readout time, but often also results in reduced image quality of each HDR subframe (and thus reduced image quality of the final output image) and / or other performance degradation. Because the purpose of providing HDR features in an imaging system is generally to support high-quality, high-performance image capture (e.g., under a wide range of lighting conditions), the application of such conventional techniques will tend to be incompatible with the provision of HDR features.

[0038] Embodiments described herein provide a novel type of interlaced high dynamic range (HDR) image capture in which downsampled readout followed by brightness-guided upsampling significantly reduces readout time and associated artifacts and defects while retaining high-resolution image data on the color plane. For example, an image sensor is configured according to a red-green-blue-luminance (RGBL) color filter array (CFA) pattern. In each image capture time window, multiple (e.g., three) brightness-enhanced (LE) component images are generated. Each LE component image is generated by exposing the image sensor to incident illumination for a corresponding amount of time, using pixel merging during readout to generate significantly downsampled color and brightness capture frames, generating upsampled brightness-guided frames from the brightness capture frames, and using the upsampled brightness-guided frames to guide upsampling of the color capture frames (e.g., and de-mosaicing). The resulting LE component images can be digitally combined to generate an HDR output image.

[0039] Steering Figure 1 , a block diagram of an imaging sensor environment 100 is shown as a background for various embodiments described herein. The imaging sensor environment 100 is shown as including a complementary metal oxide semiconductor (CMOS) imaging sensor (CIS) system 110 in communication with a processor-controlled system 160. The imaging sensor environment 100 can be used to implement a digital imaging system in any suitable application environment. For example, the CIS system 110 and the processor-controlled system 160 can be implemented in a smartphone, a digital camera, a wearable device, an implantable device, a laptop, a tablet computer, an e-reader, an Internet of Things (IoT) device, or any other suitable environment. In addition to the features of the CIS system 110, the processor-controlled system 160 is generally intended to represent any suitable one or more systems that provide any suitable features of the imaging sensor environment 100. For example, in a smart phone, the processor-controlled system 160 may include subsystems for providing phone and communication features, display features, user interaction features, application processing features, etc.

[0040] As shown, the CIS system 110 includes a sensor array 120, a merged readout circuit 130, and an upsampling engine 140. The time, signal, and / or other features of the components of the CIS system 110 are controlled by a sensor controller 150. The sensor controller 150 may include any suitable processor and / or circuit for directing the operation of the CIS system 110 components and engaging with other systems (e.g., a processor-controlled system 160). Some embodiments of the sensor control circuit 150 are implemented as or include a matching integrated circuit (IC) with an integrated interface component, a storage component, and a processing component. For example, the processing component of the sensor control circuit 150 may include one or more central processing units (CPUs), application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), graphics processing units (GPUs), physical processing units (PPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, microcontroller units, reduced instruction set computers (RISC) processors, complex instruction set computers (CISC) processors, microprocessors, etc., or any combination thereof.

[0041] The basic function of the sensor array 120 is to capture photons using photosensors and convert them into electrons. Therefore, the sensor array 120 is implemented as an array of any suitable photosensors. For example, each photosensor can be implemented as a single photodiode or a group of photodiodes (e.g., multiple photodiodes sharing a common floating diffusion node). In some cases, the sensor array 120 is a high pixel count array, for example, a 48 million pixel array. Each photosensor in the sensor array 120 generates a sensor output level in response to exposure to incident illumination. For example, when the sensor array 120 is exposed during an integration time window, the photosensor accumulates an amount of photocharge corresponding to the illumination intensity it receives. At the end of the integration time window, the amount of photocharge accumulated by the photosensor is displayed as the sensor output level of the photosensor.

[0042] Each photosensor in the sensor array 120 is optically integrated with an optical device, including a corresponding color filter that allows light energy in one or more frequency bands. For example, a photosensor integrated with a "red" color filter will receive light in the red frequency band allowed by the color filter, so that the sensor output level generated by the photosensor will indicate the received intensity of red light at the corresponding position in the sensor array 120. The color filter array optically integrated with the photosensor array is arranged according to a pattern called a color filter array (CFA) pattern or simply CFA. Terms such as "imaging pixel" or "image sensor pixel" are used herein to refer to a specific imaging position in the sensor array 120, which corresponds to the light received by the corresponding photosensor via the corresponding color filter.

[0043] Many conventional CFAs implement red (R), green (G), and blue (B) color filter patterns, where each filter color can be viewed as a color plane of reduced density. For example, a typical Bayer RGB CFA may be arranged with 50% G imaging pixels, 25% B imaging pixels, and 25% R imaging pixels. Some conventional CFAs have sought to adapt the CFA to increased brightness response. For example, it has been found that yellow (Y) imaging pixels or white (W) imaging pixels tend to provide increased brightness information. Thus, some conventional designs attempt to modify the Bayer pattern by replacing G imaging pixels with higher brightness Y or W imaging pixels (e.g., forming a RYYBCFA or RWWBCFA). However, such designs are often unsuccessful, at least because for any particular light condition, large brightness differences between adjacent imaging pixels tend to produce poor response characteristics (e.g., signal-to-noise ratio, pixel conversion gain, etc.) on the array.

[0044] The embodiments described herein use a non-Bayer color pattern that includes various luminance (L) imaging pixel densities in addition to RGB imaging pixels. Such a CFA is referred to herein as an RGBL CFA. In some embodiments, the L imaging pixels (of any of the different RGBL CFAs described herein) are implemented as W imaging pixels so that the CFA follows an RGBW pattern. Luminance pixels can be used primarily to capture broad-spectrum luminance (e.g., light intensity) information, rather than specific color information.

[0045] Figures 2A-2D An example of such a non-Bayer RGBL CFA 200 is shown in accordance with various embodiments described herein. Figure 2A A first RGBL CFA implemented as a so-called "six-ten RGBL" CFA is shown. This CFA includes 8×8 subarrays of repeated imaging pixels, each subarray having equally distributed L imaging pixels and RGB imaging pixels. In particular, each subarray (of 64 imaging pixels) includes 32 RGB imaging pixels (16 G imaging pixels, 8 R imaging pixels, and 8 B imaging pixels), with the 32 L imaging pixels evenly dispersed in a checkerboard pattern. Figure 2B A second RGBL CFA is shown implemented as a so-called "quad RGBL" CFA. This CFA includes 4×4 subarrays of repeated imaging pixels, each subarray also having equally distributed L imaging pixels and RGB imaging pixels. In particular, each subarray (of 16 imaging pixels) includes 8 RGB imaging pixels (4 G imaging pixels, 2 R imaging pixels, and 2 B imaging pixels), and the 8 L imaging pixels are evenly distributed in a checkerboard pattern. It can be seen that in Figure 2A The six-ten RGBL CFA and Figure 2BIn a four-RGBL CFA of , half of all pixels in each subarray are L imaging pixels. This distribution is referred to herein as having a 50% brightness density (or L density).

[0046] Figure 2C and 2D An RGBL CFA with 75% luminance density is shown. Figure 2C A third RGBL CFA is shown implemented as a so-called "sparse six-ten RGBL" CFA. This CFA is similar to Figure 2A The six-by-ten RGBL CFA has only L imaging pixels in every other row. Thus, each subarray (i.e., each 8×8 subarray of 64 imaging pixels) includes 48 L imaging pixels interspersed with 16 RGB imaging pixels (8 G imaging pixels, 4 R imaging pixels, and 4 B imaging pixels). Figure 2D A fourth RGBL CFA is shown implemented as a so-called "sparse quad-RGBL" CFA. This CFA is similar to Figure 2B The four RGBL CFAs are shown in FIG. 1 , except that every other row has only L imaging pixels. Thus, each subarray (i.e., each 4×4 subarray of 16 imaging pixels) includes 12 L imaging pixels interspersed with 4 RGB imaging pixels (2 G imaging pixels, 1 R imaging pixel, and 1 B imaging pixel). Figure 2C The sparse six-ten RGBL CFA and Figure 2D The sparse four-RGBL CFAs are referred to herein as 75% brightness density CFAs because three-quarters of the imaging pixels in each of their corresponding subarrays are L imaging pixels.

[0047] Although embodiments are described herein with reference to specific CFA patterns, the new techniques described herein can be applied to any suitable type of CFA pattern that interleaves a sufficiently high density of L imaging pixels with color imaging pixels. For example, specific references to red, green, and blue planes herein are intended only to illustrate one set of possible implementations; and some or all of these color planes can be replaced with other color planes to interleave with L imaging pixels. As an example, embodiments described herein can be applied to a CFA pattern that interleaves cyan, magenta, and / or yellow planes with a high density of L imaging pixels.

[0048] Back to Figure 1, the CIS system 110 can be used to generate images (e.g., a single image, one of a series of consecutive images, a video frame, etc.) in imaging time windows. For example, each imaging time window can include at least an integration time window, a readout time window, and an upsampling time window. During the integration time window, the sensor array 120 is exposed to incident illumination (e.g., during which the shutter, focusing optics, and / or other components are controlled to facilitate such exposure). In response to such exposure, each imaging pixel accumulates an amount of photocharge related to a corresponding illumination intensity received by a corresponding photosensor of the imaging pixel via a corresponding color filter of the imaging pixel. The corresponding amount of accumulated photocharge of each imaging pixel is manifested as a corresponding sensor output level.

[0049] As described above, reading out each individual sensor output level at full resolution, particularly for a high pixel count sensor array 120, can consume a significant amount of power, time, and / or other resources. Thus, it may be desirable to downsample the sensor array 120 output during readout. An embodiment of the sensor controller 150 directs the binning readout circuit 130 to read out the respective sensor output levels via pixel binning. Pixel binning may be performed in any suitable manner that: (a) combines respective sensor output levels from adjacent imaging pixels of the same color to produce a reduced number of binned sensor output levels; and (b) produces separate downsampled RGB capture frames 132 and downsampled L capture frames 134 for each sensor array 120 readout window. In some embodiments, groups of imaging pixels are coupled to the same floating diffusion node to be read out via the same source follower path. Such an embodiment may be referred to as "charge binning" and may result in an additive combination of adjacent same-color sensor output levels. In other embodiments, each imaging pixel is coupled to a respective floating diffusion node and a source follower path, and the source follower path drives a common readout line to produce an average combination of adjacent same-color sensor output levels.

[0050] For example, Figure 3 An example circuit 300 is shown that includes a set of imaging pixels and an illustrative readout circuit configured to charge combine the outputs of the imaging pixels. The illustrated readout circuit may be Figure 1An embodiment of a merged readout circuit 130 is shown. As shown, the readout circuit may include at least one reset (RST) device, a source follower (SF) device, and a select (SEL) device. The concept of pixel charge merging is to provide the electron charge from all photodiodes at once to a shared floating diffusion node, labeled "FD". In this case, the sum of all shared imaging pixels can be read out by a single analog-to-digital (ADC) operation. This can reduce the total energy for reading out the entire array, and can reduce the circuit noise applied to the shared pixel cluster at the expense of reduced resolution. In the circuit shown, four imaging pixels are shown as "Pixel 1" to "Pixel 4". Each imaging pixel is shown as having a corresponding photodiode (PD) that is coupled to a shared FD node via a corresponding transmission gate (TX).

[0051] Before the integration time window, the RST device can be turned on to effectively reset the FD node (e.g., to ground). The photodiode can also be reset by turning on the transmission gate at the same time as the RST device is turned on. During the integration time window, the transmission gate is closed and the photodiode accumulates photocharge. After the integration time window is completed, the accumulated photocharge can be read out. For example, during the readout time window, turning on any transmission gate will cause the accumulated photocharge of the corresponding photodiode to be transferred to the FD node. Similarly, turning on all transmission gates at the same time can cause the accumulated photocharge of all photodiodes to be transferred to the FD node in an aggregated (e.g., added) manner.

[0052] The SF device is disposed between the source follower voltage (VDD_SF) and the SEL device, whose gate is coupled to the FD node. Thus, the SF device allows the flow of an amount of current corresponding to the amount of accumulated photocharge that has been transferred from the imaging pixel to the shared FD node. The SEL device has a gate coupled to a row select line and is disposed between the SF device and a voltage readout line, which is also coupled to a current source (IBIAS). Thus, when the row is selected (i.e., the SEL device is turned on), a combined sensor output level (i.e., an analog output voltage signal V OUT ), corresponding to the amount of accumulated photocharge transferred to the FD node. The simulated V OUT The signal may be converted by an analog-to-digital converter ADC into a digital binned pixel value for readout via a readout line (eg a column readout line).The illustrated apparatus may be implemented in any suitable manner, for example using NMOS transistors and / or PMOS transistors.

[0053] As shown, some embodiments include data storage, such as registers or other storage devices for storing readout digital binned pixel values. For example, the storage device can be used to collect various binned pixel values ​​from the sensor array 120 to generate the RGB capture frame 132 and the L capture frame 134. A controller is also shown for context. The controller can be composed of Figure 1 The sensor controller 150 is implemented to provide any suitable clock signals, control signals, etc. to achieve the desired pixel binning mode. For example, the controller outputs control signals to open or close the transmission gate, RST device, SEL device and / or other components to achieve the characteristics of pixel binning readout.

[0054] Although Figure 3 The embodiment shows four imaging pixels performing charge merging, but similar techniques can be used for various numbers and / or combinations of pixels. For example, substantially the same circuit 300 can be used with two imaging pixels, eight imaging pixels, or any other suitable number of imaging pixels to the extent supported by other device components in circuit 300. Similarly, the same circuit 300 can be used to charge merge a selected subset of imaging pixels coupled to a shared FD node. For example, in a rolling shutter type of readout, a first readout cycle can transfer charge from pixel 1 and pixel 2 simultaneously for readout (i.e., by turning on TX1 and TX2), and the next readout cycle can transfer charge from pixel 3 and pixel 4 simultaneously for readout (i.e., by turning on TX3 and TX4).

[0055] Figures 4A-4D The various embodiments described herein are shown. Figures 2A-2D Four examples of pixel binning patterns 400 for non-Bayer RGBLCFA 200 implementations. First, go to Figure 4A , Figure 2A The six-by-ten RGBLCFA 200a is shown as being simultaneously pixel-merged into a 2×2 RGB capture frame 132a and a 2×2 L capture frame 134a. This is a 4×4 pixel merging mode that causes the resolution to be downsampled by a factor of 4 in both the horizontal and vertical dimensions. For example, the imaging pixels in the upper left quadrant of the six-by-ten RGBLCFA 200a (i.e., 16 of a total of 64 imaging pixels) are indicated by the dashed rectangle 410a. The eight G imaging pixels in quadrant 410a are pixel-merged (e.g., by charge merging or other suitable techniques) to produce merged pixels 415a in the corresponding quadrant of the RGB capture frame 132a, and the eight L (e.g., W) imaging pixels in quadrant 410a are pixel-merged (e.g., by the same technique) to produce merged pixels 425a in the corresponding quadrant of the L capture frame 134a.

[0056] exist Figure 4B middle, Figure 2B The four-RGBLCFA 200b is shown as being simultaneously pixel-merged into a 4×4 RGB capture frame 132b and a 4×4 L capture frame 134b. This is a 2×2 pixel merging mode that causes the resolution to be downsampled by a factor of 2 in both the horizontal and vertical dimensions. For example, the upper left quadrant of the imaging pixels (i.e., 4 of a total of 64 imaging pixels) of the four-RGBLCFA 200b is indicated by the dashed rectangle 410b. The two G imaging pixels in quadrant 410b are pixel-merged to produce a merged pixel 415b in a corresponding position of the RGB capture frame 132b, and the two L imaging pixels in quadrant 410b are pixel-merged to produce a merged pixel 425b in a corresponding position of the L capture frame 134b.

[0057] exist Figure 4C middle, Figure 2C The sparse six-ten RGBLCFA 200c is shown as being simultaneously pixel-binned into a 2×2 RGB capture frame 132c and a 2×2 L capture frame 134c. This is another type of 4×4 pixel binning mode, similar to Figure 4A Here, however, the upper left quadrant 410c includes only four G imaging pixels, but includes 12 L imaging pixels. The four G imaging pixels are pixel-binned to produce a binned pixel 415c in the corresponding quadrant of the RGB capture frame 132c, and the 12 L imaging pixels in the quadrant 410c are pixel-binned to produce a binned pixel 425c in the corresponding quadrant of the L capture frame 134c.

[0058] exist Figure 4D middle, Figure 2D The sparse quad RGBLCFA 200d is shown as being simultaneously pixel-merged into a 4×4 RGB capture frame 132d and a 4×4 L capture frame 134d. This is another type of 2×2 pixel-merging mode, similar to Figure 4B Here, however, the upper left quadrant of the upper left quadrant (the four pixels indicated by the dashed rectangle 410d) includes only a single G imaging pixel and three L imaging pixels. In this case, the RGB capture frame 132d is generated without pixel binning, and the three L imaging pixels in the quadrant 410d are pixel binned to produce a binned pixel 425d in the corresponding position of the L capture frame 134d. Figures 4A-4D Only some possible pixel binning methods that can be used for RGBL CFA are described. Some other examples of pixel binning methods that can be used for RGBL CFA are described in U.S. Patent Application No. 17 / 382,354, filed on July 22, 2021, entitled “PIXEL BINNING FOR HEXA-DECA RGBW COLOR FILTER ARRAYS,” the entire contents of which are incorporated herein by reference.

[0059] Return again Figure 1 , the binning readout circuit 130 performs a binning readout of the sensor output levels from the sensor array 120 during the readout time window of each imaging time window to generate two downsampled capture frames to represent each of the plurality of downsampled HDR subframes 135. The binning readout may be performed according to a global shutter readout, a rolling shutter readout, or in any other suitable manner to support HDR features until all sensor output levels are read out from the sensor array 120 for the plurality of downsampled HDR subframes 135 (i.e., as part of a binning set of sensor output levels). In some embodiments, interleaved HDR is supported by implementing a binning readout with row interleaving on the exposure subframes, as described above. In some embodiments, the binning readout circuit further includes a parallel readout structure to allow different rows of different exposure subframes to be read simultaneously.

[0060] As shown, after completing the readout time window, the merged readout circuit 130 has generated a full (downsampled) RGB capture frame 132 and a full (downsampled) L capture frame 134 for each downsampled HDR subframe 135. In some embodiments, each downsampled HDR subframe 135 is represented by a separate RGB capture frame 132 and L capture frame 134. In other embodiments, multiple subframes can be viewed as a single image capture frame spanning an array of N times the width, where N is the number of subframes. In such embodiments, a single wide RGB capture frame 132 and wide L capture frame 134 can represent the data of all N downsampled HDR subframes 135. As described above with reference to Figure 3 As described above, the binned pixel values ​​making up the captured frame may be temporarily stored in any suitable data storage, such as registers or other storage devices integrated with the CIS system 110 (eg, on the same chip as the binned readout circuit 130).

[0061] Still in the same imaging window (i.e., in the same image sensor output path), an embodiment of the sensor controller 150 directs the upsampling engine 140 to generate a brightness enhanced (LE) component image 145 based on the RGB capture frame 132 and the L capture frame 134. As described herein, each generated LE component image 145 corresponds to one of the downsampled HDR sub-frames 135. In this way, each LE component image 145 is actually one of a plurality of sub-frame images associated with one of a plurality of corresponding exposure times to support subsequent digital combination into an HDR output image 175. The sensor array 120 has a certain maximum "native" resolution defined at least by the number of imaging pixels. For example, a sensor array 120 having 48 million imaging pixels can be said to have a maximum native resolution of 48 million pixels. As described above, the downsampled RGB capture frame 132 and the L capture frame 134 each have a resolution significantly lower than the maximum resolution of the sensor array 120. For example, if a 2×2 pixel binning mode is applied to a sensor array 120 having a maximum native resolution R, each of the RGB capture frame 132 and the L capture frame 134 can have a resolution of approximately R / 4 (e.g., if some imaging pixels at the edge of the sensor array 120 cannot be binned in the same manner as those pixels away from the edge, the result may not be exactly R / 4). Similarly, if a 4×4 pixel binning mode is applied to a sensor array 120 having a maximum native resolution R, each of the RGB capture frame 132 and the L capture frame 134 can have a resolution of approximately R / 16. Embodiments of the upsampling engine 140 can generate the LE component image 145 to have any suitable resolution higher than the resolution of the downsampled capture frame (i.e., each of the RGB capture frame 132 and the L capture frame 134). In some embodiments, the upsampling engine 140 generates the LE component image 145 to have a resolution equal to the maximum native resolution of the sensor array 120. In other embodiments, the upsampling engine 140 generates the LE component image 145 to have a resolution between the resolution of the downsampled capture frame and the maximum native resolution of the sensor array 120. In other embodiments, the upsampling engine 140 generates the LE component image 145 to have a resolution higher than the maximum native resolution of the sensor array 120.

[0062] Traditional upsampling methods are often applied to RGBCFA by interpolating between pixels in the same color plane. However, in some CFAs, these pixels are relatively far apart, and the interpolation accuracy decreases as the spacing between the measured pixel values ​​increases. For example, in a traditional Bayer RGB array, each 2×2 imaging pixel block can follow the following pattern: {R, G; G, B}. In this pattern, each R pixel can be separated from any adjacent R pixel (e.g., the R pixel in the right adjacent block) by a single G pixel. Scaling this pattern by a factor of two can result in a 4x4 array, such as: {R, ? , G, ? ; ? , ? , ? , ? ; G, ? , B, ? ; ? , ? , ? , ?} (where '?' represents an undefined value for the array). Now, each R pixel is separated from any adjacent R pixel by three pixel positions. Other CFA patterns and / or greater degrees of magnification produce even larger spacing between pixels sharing a color plane. Some other conventional upsampling methods applied to conventional CFA attempt to extrapolate between pixels on different color planes. However, such extrapolation between different color planes often cannot distinguish between color changes and intensity changes.

[0063] Embodiments described herein use a non-Bayer RGBL CFA (e.g., non-Bayer RGBL CFA 200) to provide an additional luminance plane that spatially corresponds to, but is color-separated from, all of the RGB color planes. In effect, the introduction of the L plane in the RGBL CFA reduces the overall resolution of the RGB color planes compared to a sensor array with only RGB pixels of the same native resolution. However, the luminance information provided by the dense population of L pixels in the sensor array 120 is suitable for upsampling, at least because of the large amount of light collected by the L imaging pixels relative to the amount collected by the RGB imaging pixels, the ability to distinguish between the overall luminance information of the L imaging pixels and the color-specific intensity information of the RGB imaging pixels, and the spatial and temporal correspondence between the data captured in the L plane and the data captured in the RGB plane (i.e., because the L imaging pixels and the RGB imaging pixels are in the same sensor array 120 and are exposed to the same incident light in the same imaging time window).

[0064] Figure 5-6C A representative flow chart of an illustrative manner in which upsampling engine 140 generates LE component image 145 by upsampling and demosaicing RGB capture frame 132 using L capture frame 134 is shown. In general, upsampling engine 140 can generate upsampled L pilot frame 520 from L capture frame 134. Based on L pilot frame 520, a demosaicing operation is performed on each color plane of RGB capture frame 132. Figure 5-6C The representation assumption Figure 2BThe pixel binning mode 200b is used to generate the RGB capture frame 132 and the L capture frame 134, but similar techniques can be applied to any other suitable pixel binning mode.

[0065] First go to Figure 5 , shows a first representative flow chart 500, starting with L capture frame 134 (e.g., Figure 2B L capture frame 134b). L capture frame 134 is upscaled to produce upscaled L frame 510. Upscaled L frame 510 has a size configured to support a target resolution of LE component image 145 (e.g., which may also be a target resolution of a final HDR output image after digitally combining LE component image 145), which is greater than the resolution of L capture frame 134 (e.g., an integer multiple thereof). In this way, upscaled L frame 510 includes more array positions than L capture frame 134. As described above, L capture frame 134 is an array of merged L pixel values ​​425, each pixel value being generated by combining information from multiple photosensor outputs. In this way, each merged L pixel value 425 approximates the brightness information of the corresponding position in the captured scene (i.e., the incident illumination to which sensor array 120 is exposed).

[0066] When zoomed in, each merged L pixel value 425 may be considered to indicate brightness information at a point in space that may not correspond to any particular array position (i.e., an enlarged L pixel 515) of the enlarged L frame 510. For example, the enlarged L frame 510 shown is zoomed in by a factor of two in each of the horizontal and vertical dimensions, so that each single merged L pixel value 425 may be considered to be zoomed in to a corresponding 2×2 block of four enlarged L pixels 515. As shown, each merged L pixel value 425 may be considered to be the value at the center of its corresponding 2×2 block of four enlarged L pixels 515. In some embodiments, each merged L pixel value 425 is directly mapped to a corresponding one of the enlarged L pixels 515 in a consistent manner. In the illustrated embodiment, each merged L pixel value 425 is directly mapped to the upper left enlarged L pixel 515 of its corresponding 2×2 block. For example, merged L pixel value 425-1 is mapped to enlarged L pixel 515a, and merged L pixel value 425-2 is mapped to enlarged L pixel 515b. After enlargement, enlarged L frame 510 may include substantially the same resolution as L capture frame 134, but with a reduced pixel density. For example, L capture frame 134 includes 16 "known" brightness values ​​in 16 possible array positions to produce 100% pixel density; enlarged L frame 510 includes the same 16 "known" brightness values, but in 64 possible array positions, producing 25% pixel density. The remaining 48 array positions of enlarged L frame 510 are "unknown", so that their corresponding data are undefined, empty, etc.

[0067] Each unknown array position of the upscaled L frame 510 may then be filled in by upsampling to produce an upsampled L pilot frame 520. Embodiments may use any suitable interpolation technique to generate values ​​for unknown positions of the upscaled L frame 510. In some implementations, for each unassigned position, the values ​​of some or all of the known nearest neighbor array positions (e.g., positions that directly correspond to merged L imaging pixel data in the L capture frame 134) are used to estimate the pixel value of the unassigned position. Other embodiments calculate the estimated L pixel 525 values ​​using a bilinear technique. Other embodiments calculate the estimated L pixel 525 values ​​using a bicubic technique.

[0068] In the upsampled L-guided frame 520, the cells with known luminance values ​​in the enlarged L-frame 510 are indicated in bold as L, and the cells with upsampled luminance values ​​calculated from the known luminance values ​​are indicated in italics as L'. For ease of reference, certain cells are also numbered. As an example of some of the interpolation techniques mentioned above, interpolation can be performed in an iterative manner. In one iteration, for example, by averaging four known values ​​to obtain a single unknown value, the value of cell L'1 can be interpolated from the values ​​of its nearest cell squares L1, L2, L4, and L5. Similarly, the unknown value of cell L'3 can be interpolated from known value cells L2, L3, L5, and L6; the unknown value of cell L'6 can be interpolated from known value cells L4, L5, L7, and L8; and the unknown value of cell L'8 can be interpolated from known value cells L5, L6, L8, and L9. In the next iteration, the interpolated values ​​from the previous iteration can now be used to interpolate additional unknown values. For example, the nearest neighbors of L'2 are a diamond consisting of the originally known cells L2 and L5 and the previously interpolated cells L'1 and L'3. The values ​​of cells L'4, L'5, and L'7 can be interpolated using a similar method. Using such a technique, the entire L-guide frame 520 can be upsampled with a combination of enlarged L pixels 515 and estimated L pixels 525 to achieve 100% L pixel density in the L-guide frame 520 with the target resolution of the LE component image 145.

[0069] Go to Figures 6A-6C , Figures 6A-6C 600 of a partial demosaicing operation for a corresponding color plane of an RGB capture frame 132 is shown. Figures 6A-6C An upsampling sequence of green then red then blue is shown, but the RGB color planes can be upsampled in other orders. Figures 6A-6CThe result of demosaicing is an LE component image 145 of a specific resolution and following a standard Bayer RGB pattern, but demosaicing can be adapted to produce LE component images 145 of other resolutions and / or other RGB patterns. Embodiments of the upsampling engine 140 can be configured to produce LE component images 145 according to a defined output image mode (or a set of predefined output image modes), such as a specific resolution, CFA pattern, etc. For example, the LE component images 145 are configured to be digitally combined into a single HDR output image 175, and the output image mode defines the target resolution, CFA pattern, etc. of the HDR output image 175. The HDR output image 175 can be image data output by the CIS system 110 to a downstream component (e.g., the processor control system 160). In some embodiments, the processor controlled system 160 is designed to receive output image frames formatted according to a specific output image mode from the CIS system 110. For example, processor-controlled system 160 may be designed with standard physical and logical interfaces configured to receive image frames having a defined resolution and a conventional (e.g., standard Bayer RGB) CFA pattern from CIS system 110. In such an embodiment, upsampling engine 140 is configured to generate LE component images 145 to support such image frame parameters.

[0070] exist Fig. 6A , a representative flowchart 600a for upsampling the green plane is shown. Flowchart 600a begins with an RGB capture frame 132 (e.g., Figure 2B RGB capture frame 132b). RGB capture frame 132 is enlarged by assigning known merged G imaging pixels 415g of RGB capture frame 132 to corresponding positions of enlarged G frame 610g to generate enlarged G frame 610g. Enlarged G frame 610g is configured to have the same size as enlarged L frame 510 to support the target resolution of LE component image 145. Similar to the above reference Figure 5, each merged G imaging pixel 415g approximates the green information of the corresponding position in the captured scene, which can generally be considered to correspond to a point in space relative to the enlarged G frame 610g (but not necessarily to any particular array position in the enlarged G frame 610g). In this way, an embodiment may choose to map each merged G imaging pixel 415g to a particular enlarged G pixel 615g of the 2×2 block of corresponding enlarged G pixels 615g in the enlarged G frame 610g. In order to avoid introducing certain artifacts, it may be desirable to map values ​​in the L and RGB planes in a consistent manner. For example, if each merged L pixel value 425 is mapped to the upper left enlarged L pixel 515 in its corresponding 2×2 block of the enlarged L frame 510, then an embodiment may similarly map each merged G imaging pixel 415g to the upper left enlarged G pixel 615g in its corresponding 2×2 block of the enlarged G frame 610g. After enlargement, the enlarged G frame 610g can have substantially the same resolution as the G color plane of the RGB capture frame 132, but with a reduced pixel density. For example, the RGB capture frame 132 includes 8 "known" G values ​​in 16 possible array positions to produce a pixel density of 50%; the enlarged G frame 610g includes the same 8 "known" G values, but in 64 possible array positions, producing a pixel density of 12.5%. The remaining 56 array positions of the enlarged G frame 610g are "unknown", such that their corresponding data are undefined, empty, etc.

[0071] The unknown G values ​​in the enlarged G frame 610g can be filled by upsampling to produce an upsampled G frame 620g. In some embodiments, the upsampled G frame 620g is generated and stored as a separate array of G values. In other embodiments, the upsampled G frame 620g represents the G color plane data of the upsampled RGB frame 620, which includes the upsampled G frame 620g and the upsampled R and B frames, as described below. The upsampling of the enlarged G frame 610g may include at least two features. The first feature is that the upsampling interpolates the G color values ​​for a specific portion of the unit of the upsampled G frame 620g according to a predefined target CFA pattern. The second feature is that the upsampling is a brightness-guided interpolation based on the upsampled L-guided frame 520.

[0072] Regarding the first feature, as described above, the output image mode may define a target resolution and a target CFA pattern for the LE component image 145 (and / or the resulting HDR output image 175). These mode definitions may effectively define which cells of the upsampled G frame 620g (or upsampled RGB frame 620) are allocated to the G color plane. For example, Figures 6A-6CThe output image pattern shown employs a standard Bayer RGB pattern that includes 2×2 blocks of repeating R, G, G, B pixels. According to such a pattern, every other cell in the frame is assigned to the G color plane in a checkerboard pattern. In this way, upsampling of the enlarged G frame 610g can be performed in a manner that interpolates G color values ​​only for those cells assigned to the G color plane according to the output image pattern. As shown, the upsampled G frame 620g resulting from the upsampling operation has a G pixel density of 50%, corresponding to the target color plane resolution and CFA pattern (Bayer RGB pattern) of the illustrative LE component image 145.

[0073] With respect to the second feature, the upsampling is referred to herein as "luminance-guided" interpolation because the interpolation is guided by the L-guide frame 520. As described above, the L-guide frame 520 is upsampled to have a luminance information density of 100% so that each cell of the upsampled G frame 620g (e.g., more generally, the upsampled RGB frame 620) can have a one-to-one spatial correspondence with a corresponding one of the upsampled L-guide frame 520. Thus, the unknown G values ​​in the upsampled G frame 620g are interpolated based on neighboring known (or previously interpolated) G values ​​and local luminance information from the L-guide frame 520. Embodiments may apply luminance-guided techniques to any suitable interpolation method, such as nearest neighbor, bicubic, or other methods. Luminance-guided interpolation may use the upsampled L-guide frame 520 information to weight and / or normalize the G color information in any suitable manner to improve the interpolation in the G color plane. Some embodiments implement brightness-guided interpolation for a particular cell by using neighboring G color plane information to obtain an initial G color value for the cell, and then weighting the initial G color value based on one or more local L values ​​in a corresponding position of the L guide frame 520 to produce an estimated G pixel 625g for the cell. Other embodiments implement brightness-guided interpolation for a particular cell by weighting the neighboring G color plane information based on local L values ​​in a corresponding position of the L guide frame 520 to obtain an initial G color value for the cell, and then re-weighting the initial G color value based on the local L values ​​in a position of the L guide frame 520 corresponding to the cell to produce an estimated G pixel 625g for the cell.

[0074] As an example of brightness guided interpolation, interpolation can be performed in an iterative manner. In one iteration, the estimated G pixel 625g value of unit G'4 can be interpolated from the values ​​of its nearest unit diamonds G2, G3, G4 and G5. In one embodiment, the four known values ​​are averaged to obtain a single initial value for G'4. The initial value can now be adjusted by the difference in brightness information of the corresponding units of the L guided frame 520. For example, the units G2, G3, G4, G5 and G'4 of the upsampled G frame 620g correspond spatially to the units L2, L4, L6, L8 and L5 of the L guided frame 520, respectively. Some embodiments use a color ratio method to multiply the average value of the values ​​of the units G2, G3, G4 and G5 by the ratio of the value of the unit L5 to the average value of the values ​​of the units L2, L4, L6 and L8. Other embodiments use a color difference method to find the difference between the value of cell L5 and the average of the values ​​of cells L2, L4, L6, and L8, and add that difference to the average of the values ​​of cells G2, G3, G4, and G5. The color difference method may be preferred in some embodiments because it can be implemented without a hardware division operation. In the color ratio method and the color difference method, the spatially corresponding L values ​​are used to adjust the interpolation in the color plane to account for local variations in brightness that would otherwise not be captured by the specific color plane information. Continuing with the example of brightness-guided interpolation, the same method can be used to generate an estimated G pixel 625g value for cell G'5 based on cells G3, G5, G6, and G7 and the spatially corresponding cells from the L-guided frame 520. In subsequent iterations, the previously interpolated value can be used in the same manner for further interpolation. For example, the same method may be used to generate an estimated G pixel 625g value for cell G'14 based on cells G3, G'4, G'5 and G6 (their nearest neighbors) and spatially corresponding cells from L lead frame 520 (L4, L5, L7, L8 and L'6).

[0075] Steering Figure 6B , a representative flowchart 600b for upsampling the red plane is shown. Flowchart 600b begins with an RGB capture frame 132 (e.g., Figure 2BRGB capture frame 132b). The RGB capture frame 132 is enlarged by assigning known merged R imaging pixels 415r of the RGB capture frame 132 to corresponding positions of the enlarged R frame 610r to produce the enlarged R frame 610r. The enlarged R frame 610r is configured to have the same size as the enlarged G frame 610g. The RGB capture frame 132 can be enlarged to produce the enlarged R frame 610r in the same manner as described for producing the enlarged G frame 610g. For example, each merged R imaging pixel 415r can be mapped to the top left enlarged R pixel 615r in the 2×2 block of the corresponding enlarged R frame 610r. In the illustrated case, the RGB capture frame 132 includes 4 "known" R values ​​in 16 possible array positions to produce a 25% pixel density for the R color plane; the enlarged R frame 610r includes the same 4 "known" R values, but in 64 possible array positions, producing a 6.25% pixel density. The remaining 60 array positions of the enlarged R frame 610r are "unknown", such that their corresponding data are undefined, empty, etc.

[0076] The unknown R values ​​in the upscaled R frame 610r may be filled by upsampling to produce an upsampled R frame 620r. In some embodiments, the upsampled R frame 620r is generated and stored as a separate array of R values. In other embodiments, the upsampled R frame 620r represents the R color plane data of the upsampled RGB frame 620, which includes the upsampled R frame 620r and the upsampled G and B frames, as described below. For context, the G color plane units are indicated without shading in the upsampled R frame 620r (e.g., as previously described in Fig. 6A ). Fig. 6A As described by the G color plane in FIG. 14 , the upsampled upscaled R frame 610 r may include at least two features: the upsampling interpolates R color values ​​for a specific portion of the cells of the upsampled R frame 620 r according to a predefined target CFA pattern of the LE component image 145, and the upsampling is based on the brightness-guided interpolation of the upsampled L-guided frame 520. Regarding the first feature, Figure 6B The upsampling shown in is shown to produce R color values ​​only for those cells assigned to the R color plane according to the output image mode. The illustrative result is an upsampled R frame 620r with a 25% density of R pixels, corresponding to the target color plane resolution and CFA pattern (Bayer RGB pattern) of the illustrative LE component image 145.

[0077] Regarding the second feature, it can be seen from the reference Fig. 6A The brightness-guided interpolation is performed in a similar manner to any of the methods described. However, Figure 6B The specific case shown in Fig. 6A and Figure 6B There are some differences between the interpolation in . First, in Figure 6B In (with Figure 6B 610r), none of the known value locations of the upsampled R frame 610r directly coincide with R color plane cells in the upsampled R frame 620r. For reference, the locations of the upsampled R pixels 615r are shown relative to the upsampled R frame 620r, and it can be seen that the corresponding known values ​​do not map directly to those locations (in fact, those locations just correspond to cells in the G color plane). Thus, all of the R values ​​in the upsampled R frame 620r are estimated R pixel 625r values ​​resulting from luma-guided interpolation.

[0078] Second, in Figure 6B (different from Figure 6B ), the interpolated pixels are offset relative to their neighboring pixels in the upsampled R frame 620r. For example, the estimated R pixel 625r, represented as R'1, is closer to its nearest known values ​​615r1 and 615r3 than its nearest known values ​​615r2 and 615r4. Embodiments may account for the offset in various ways, for example, by using a weighted average based on linear distance. For example, the initial value of cell R'1 may be calculated as (3*(R1+R3)+(R2+R4)) / 8, so that closer nearest neighbors have an increased influence on the average (e.g., an increase of 3 times). The initial value may then be adjusted using a color ratio or color difference method based on the L-guided frame 520 to calculate the estimated R pixel 625r value for cell R'1. For example, cells 615r1, 615r2, 615r3, 615r4, and R'1 in the upsampled R frame 620r spatially correspond to cells L1, L3, L7, L9, and L'4 of the L-guide frame 520, respectively. Some embodiments use a color ratio method to multiply the initial value (i.e., the weighted average of the values ​​of cells R1, R2, R3, and R4) by the ratio of the value of cell L'4 to the average of the values ​​of cells L1, L3, L7, and L9. Other embodiments use a color difference method to find the difference between the value of cell L'4 and the average of the values ​​of cells L1, L3, L7, and L9, and add the difference to the initial value (i.e., the weighted average of the values ​​of cells R1, R2, R3, and R4). A similar method can be used to generate estimated R pixel 625r values ​​for all 16 R color plane cells of the upsampled R frame 620r.

[0079] Steering Figure 6C , a representative flowchart 600c for upsampling the blue plane is shown. Flowchart 600c begins with an RGB capture frame 132 (e.g., Figure 2B610b). The RGB capture frame 132 is enlarged by assigning the known merged B imaging pixels 415b of the RGB capture frame 132 to the corresponding positions of the enlarged B frame 610b to produce the enlarged B frame 610b. The enlarged B frame 610b is configured to have the same size as the enlarged G frame 610g and the enlarged R frame 610r. Enlarging the RGB capture frame 132 to produce the enlarged B frame 610b can be performed in the same manner as described for generating the enlarged G frame 610 and the enlarged R frame 610r. For example, each merged B imaging pixel 415r can be mapped to the upper left enlarged B pixel 615b in the 2×2 block of the corresponding enlarged B frame 610b. In the illustrated case, RGB capture frame 132 includes 4 "known" B values ​​in 16 possible array positions to produce a 25% pixel density for the B color plane; the enlarged B frame 610b includes the same 4 "known" B values, but in 64 possible array positions, producing a 6.25% pixel density. The remaining 60 array positions of the enlarged B frame 610b are "unknown", so that their corresponding data are undefined, empty, etc.

[0080] The unknown B values ​​in the upscaled B frame 610b may be filled by upsampling to produce an upsampled B frame 620b. In some embodiments, the upsampled B frame 620b is generated and stored as a separate array of B values. In other embodiments, the upsampled B frame 620b represents the B color plane data of the upsampled RGB frame 620, which includes the upsampled B frame 620b and the upsampled G and R frames, as described below. For context, the G and R color plane units are indicated without shading in the upsampled B frame 620b (e.g., as previously described in Fig. 6A and 6B ). Fig. 6A and Figure 6B As described by the G and R color planes in the upsampled B frame 610b, the upsampling of the upsampled B frame 610b may include at least two features: the upsampling interpolates B color values ​​for specific portions of the cells of the upsampled B frame 620b according to a predefined target CFA pattern of the LE component image 145, and the upsampling is based on the luma-guided interpolation of the upsampled L-guide frame 520.

[0081] Regarding the first feature, Figure 6C The upsampling shown in FIG. 6 is shown to produce B color values ​​only for those cells assigned to the B color plane according to the output image mode, resulting in an upsampled B frame 620 b having a 25% density of B pixels positioned according to the target color plane resolution and the CFA pattern (Bayer RGB pattern) for the illustrative LE component image 145. With respect to the second feature, the B color value may be obtained in a manner similar to that of reference 620 b. Fig. 6A and 6BThe luma-guided interpolation is performed in a manner similar to any of the methods described above. Similar to the R color plane, it can be seen that none of the known value locations of the upscaled B frame 610b directly coincide with the B color plane cells in the upsampled B frame 620b (the known value locations in the B color plane just correspond to cells in the G color plane). Thus, all B values ​​in the upsampled B frame 620b are estimated B pixel 625b values ​​generated by luma-guided interpolation.

[0082] In addition, similar to the R color plane, the interpolated B pixels are offset relative to the neighboring pixels in the upsampled B frame 620b. In this way, a weighted averaging technique can be applied. Some embodiments may use a weighted bicubic method, similar to the reference Figure 6B The method described herein adjusts the weights based on the relative linear distance between each target cell and its nearest neighbor. Where appropriate, other embodiments may use a weighted bilinear method to interpolate based only on the two nearest neighbors. For example, pixel B'1 is adjacent to known value position 615b4 and only one pixel is removed from known value position 615b2, but is significantly away from known value position 615b1 or 615b3. In this way, a more reliable interpolation of cell B'1 can be achieved using a weighted bicubic method based on known values ​​B2 and B4 and a brightness-guided adjustment based on cells L5, L11, and L'10. A similar method may be used to generate estimated B pixel 625b values ​​for all 16 B color plane cells of the upsampled B frame 620b.

[0083] exist Figures 6A-6C After all the upscaling and upsampling operations, the result is a set of upsampled color frame data (i.e., upsampled G frame 620g, upsampled R frame 620r, and upsampled B frame 620b), which are interleaved to form an upsampled RGB frame 620. As described above, the upsampled RGB frame 620 has a resolution and CFA pattern corresponding to the output image mode defined for the LE component image 145 (and / or HDR output image 175). In some cases, strict application of the above-mentioned upsampling techniques can lead to edge artifacts. Embodiments include edge detection and / or edge sensing algorithms to adapt the upsampling to edge areas. For example, these algorithms can detect gradients in different directions indicating edge behavior, and these algorithms can adjust the interpolated pixel weights accordingly. In fact, before the brightness-guided adjustment of the initial value, the algorithm often affects the weight and / or calculation used to obtain the initial value of a particular unit. Therefore, embodiments can use the various brightness-guided adjustment methods described above (e.g., based on color ratio, color difference, etc.) to achieve brightness-guided interpolation near edges.

[0084] Return again Figure 1, the output of the upsampling engine 140 is a set of LE component images 145, each component image representing the same image of the same scene at the same time, but with different corresponding exposure times. As shown, an embodiment of the CMOS image sensor system 110 may include an HDR engine 170. An embodiment of the sensor controller 150 may direct the HDR engine 170 to digitally combine the LE component images 145 to generate an HDR output image 175. As described above, the pixel-binning downsampling implemented by the binning readout circuit 130 facilitates brightness-guided upsampling by the upsampling engine 140, which can significantly reduce the readout time of the sensor array 120 without significantly reducing the image quality (e.g., retaining high-resolution color and brightness information on the sensor array). In this way, these techniques can mitigate motion-related artifacts and defects within and / or between subframes while producing high-quality LE component images 145 to support the generation of high-quality HDR output images 175. Embodiments can generate HDR output images 175 to comply with any output image mode, storage considerations, communication protocols, and / or other characteristics or constraints to support downstream use of the HDR output images 175. For example, the HDR output image 175 is transmitted to any suitable processor-controlled system 160 for downstream processing, further communication, etc.

[0085] Fig. 9 Flowchart 900 shows a novel interleaved HDR method that uses the type of pixel binning downsampling described herein to reduce time for image sensor readout. Figure 7 and 8 As mentioned above, the vertical dimension usually represents time and the horizontal dimension usually represents array width. During the "frame 1 exposure" window, the photosensors of the array are exposed row by row. After a period of "exposure time 1", the array begins to read out row by row. The readout of this first subframe takes some time, which defines the height of the "frame 1 readout" window. As mentioned above with reference to Figure 7 and 8 As described above, the duration of the exposure window is also determined by the sub-frame readout time to ensure that all rows of the sensor array (from the first row to the last row) are exposed for the same amount of time before being readout. Pixel binning during each row readout significantly reduces the number of values ​​read out in each row (i.e., the number of ADC read operations), thereby significantly reducing the sub-frame readout time, regardless of which interleaved readout method is used (e.g., row interleaving, etc.). For example, downsampling by a factor of two (e.g., generating each RGB capture frame 132 and each L capture frame 134 to have one-fourth the native resolution of the sensor array) results in half the photosensor output level being read out per row.

[0086] The more the row-by-row readout time is reduced, the less waiting time is required between each subframe exposure. For example, in the frame 1 exposure window, the first row of imaging pixels begins exposure at time T0, and the readout of the first row of imaging pixels is read out at time T0 + exposure time 1 (frame 1 readout window starts). After the readout of the first row of the first subframe is completed, the imaging pixels in the row can be reset and re-exposure corresponding to exposure time 2 of the second subframe can begin. As shown in the figure, the frame 2 exposure window can start shortly after the start of the frame 1 readout window, and the frame 3 exposure window can start shortly after the start of the frame 2 readout window. Therefore, reducing the row-by-row readout time can both shorten the total duration of each subframe readout window (for example, the frame 1 readout window is shortened) and shorten the waiting time between each subframe exposure of the same row of pixels. For comparison, Fig. 9 The total image capture exposure and readout duration is represented by the vertical arrow labeled “Time”, while Figure 7 and Figure 8 The corresponding total image capture exposure and readout duration for the conventional case of is represented by the vertical dashed arrow adjacent to the arrow labeled "Time".

[0087] Although exposure, readout, etc. are described herein as being performed row by row, the novel techniques described herein may be similarly applicable to other pixel architectures. As one example, image sensor circuitry may be designed to address, expose, and readout multiple rows of imaging pixels at a time, rather than addressing, exposing, and reading out a row of imaging pixels at a time. As another example, some image sensor systems may be configured to address, expose, readout, etc. in a row-by-row manner, a column-by-column manner, a segment-by-segment manner (e.g., row by row or column by column by quadrant), etc.

[0088] Fig.10A flow chart of an illustrative method 1000 for high dynamic range (HDR) image capture using an image sensor system according to various embodiments described herein is shown. As described herein, an image sensor system has an image sensor array formed of a plurality of imaging pixels configured according to a red-green-blue-luminance (RGBL) color filter array (CFA) pattern. The RGBL CFA pattern may include at least 50% luminance pixels (e.g., 50%, 75%, etc.). Each imaging pixel may include at least one photosensor optically coupled to a color filter. The image sensor array may be configured such that the imaging pixels are arranged to be addressable in imaging pixel groups to facilitate pixel binning. For example, a sensor controller may be used to simultaneously set a group of adjacent imaging pixels to an exposure mode, wherein the imaging pixel group simultaneously accumulates photocharge in response to exposure to incident illumination; and the sensor controller may be used to simultaneously set a group of adjacent imaging pixels to a readout mode, wherein the photocharge accumulated by the imaging pixel group is simultaneously read out as one or more pixel binning output values. Thus, a particular group of imaging pixels may be a group of pixel binning groups on one or more rows and one or more columns of the image sensor array. For example, a group of imaging pixels may correspond to a particular "row" of an image sensor array, which may include multiple adjacent rows of imaging pixels to facilitate pixel merging across multiple rows (e.g., pixel merging of diagonally adjacent imaging pixels of the same color).

[0089] An embodiment of method 1000 may begin by sequentially repeating stages 1004 and 1008 for all imaging pixel groups and each of N subframes, where N is an integer greater than 1. Every nth subframe of the N subframes (i.e., n=1 to N) is associated with an nth exposure window and an nth readout window, and every nth readout window begins after an nth exposure time after the start of the nth exposure window. For example, Fig. 9 A frame 1 exposure window is shown that is earlier than the frame 1 readout window by exposure time 1, a frame 2 exposure window is earlier than the frame 2 readout window by exposure time 2, and a frame 3 exposure window is earlier than the frame 3 readout window by exposure time 3. Each nth exposure time is a different amount of time. For example, N can represent three subframes associated with three different exposure times, and all imaging pixels of the image sensor array are sequentially exposed in each nth exposure window and read out in each nth readout window in each subframe iteration.

[0090] In each iteration of stage 1004, the next group of one or more groups of pixels is sequentially exposed to incident illumination over the nth exposure window for an nth exposure time. For example, the imaging pixels can be signaled to cause a rolling exposure of one or more groups of imaging pixels at a time. In each iteration of stage 1008, the same group of imaging pixel groups are sequentially read out using pixel merging over the nth readout window, so that each imaging pixel group is read out immediately after the imaging pixel group is exposed to incident illumination for the nth exposure time. As shown, at the end of each nth readout window, pixel merging produces a corresponding one of N downsampled HDR subframes. Each downsampled HDR subframe has a corresponding RGB capture frame and a corresponding L capture frame. For example, when color imaging pixel groups are read out sequentially using pixel merging in an iteration of stage 1008, the pixel-merged color values ​​are stored in a data frame representing the corresponding RGB capture frame; and when brightness imaging pixel groups are read out sequentially using pixel merging in an iteration of stage 1008, the pixel-merged brightness values ​​are stored in a data frame representing the corresponding L capture frame; so that after all imaging pixel groups are read out using pixel merging, the result is a complete RGB capture frame and a complete L capture frame.

[0091] Although stages 1004 and 1008 are shown as each iterating over all pixel groups in a serial manner, stages 1004 and 1008 actually occur in an overlapping manner to ensure that each imaging pixel group is exposed for the same amount of time before being read out in each subframe. For example, the first imaging pixel group begins exposure at an exposure start time (Tes1) and begins readout at a readout start time (Trs1), which is the nth exposure time immediately after Tes1 (i.e., Trs1=Tes1+nth exposure time). The second imaging pixel group begins exposure at Tes2 and begins readout at Trs2, where Trs2=Tes2+nth exposure time. Assume that there are G imaging pixel groups in the image sensor array. Because the image sensor system takes a fixed amount of readout time (Tro) to read out each imaging pixel group, each Trsg (representing the gth readout start time of the gth imaging pixel group, where g=1 to G) occurs one Tro after each Trs(g-1). To ensure that each imaging pixel group is exposed for the same amount of time before being read out in each subframe, each Tesg also has a Tro after each Tes(g-1). Assuming that the nth exposure time is less than G*Tro (i.e., the total time to read out all imaging pixel groups of the entire image sensor array), when stage 1008 begins to iterate (read out in sequence) the first group of pixels, stage 1004 will still iterate (expose in sequence) the remaining imaging pixel groups, and when stage 1004 has completed iterating all imaging pixel groups, stage 1008 will still iterate the remaining imaging pixel groups.

[0092] At stage 1012, an embodiment may generate N brightness-enhanced component images from the N downsampled HDR subframes. For example, for each of the N downsampled HDR subframes, an embodiment may upsample the corresponding L capture frames to generate corresponding L guide frames. An embodiment may then apply brightness-guided interpolation to the corresponding RGB capture frame based on the corresponding L guide frame to upsample the corresponding RGB capture frame to form a corresponding one of the N brightness-enhanced component images. In some embodiments, upsampling the corresponding L capture frame to generate the corresponding L guide frame includes: upsampling the corresponding L capture frame to generate the corresponding upscaled L frame having a target frame size defined by an output image mode of an HDR output image (to be generated in stage 1016 below), filling a portion of the corresponding upscaled L frame based on known brightness values ​​from the corresponding L capture frame; and upsampling the corresponding upscaled L frame to generate the corresponding L guide frame based on interpolation of the known brightness values, so that the corresponding L guide frame has a brightness information density of 100%. In some embodiments, applying luma-guided interpolation to the corresponding RGB capture frame includes, for each color plane of the RGB capture frame: upscaling the color plane to generate a corresponding upscaled color frame so that the corresponding upscaled color frame has a target frame size defined by an output image mode of the HDR output image and filling a portion of the corresponding upscaled color frame based on corresponding known color values ​​from the color plane; and upsampling the corresponding upscaled color frame by applying luma-guided interpolation to the corresponding known color values ​​based on the corresponding L guide frame to generate a corresponding upsampled color frame. In some embodiments, for each of the N downsampled HDR subframes: the corresponding L guide frames include an array of L value cells, each value cell having a corresponding luma value; and applying luma-guided interpolation to the corresponding RGB capture frame includes generating the corresponding upsampled RGB frame to have an array of color value cells by: generating an initial value for the color value cell based on interpolating an adjacent set of known color values ​​from the corresponding RGB capture frame; and calculating an estimated color pixel value for the color value cell by adjusting the initial value based on the luma values ​​of those L value cells that spatially correspond to the positions of the color value cell and the adjacent set of known color values. In some embodiments, the adjustment includes: calculating the color ratio or color difference between the L value unit that spatially corresponds to the color value unit and those L value units that spatially correspond to positions of an adjacent set of known color values; and adjusting the initial value of the color value unit based on the color ratio or color difference.

[0093] At stage 1016, an embodiment may digitally combine the N brightness-enhanced component images to generate a single HDR output image. In some embodiments, the image sensor array has a native resolution; for each of the N downsampled HDR subframes, a corresponding RGB capture frame may be generated to have a downsampled resolution less than or equal to one-quarter of the native resolution; and each of the N brightness-enhanced component images is generated to have an output resolution at least three times the downsampled resolution. The output resolution of each brightness-enhanced component image may be the same as the resolution of the generated HDR output image. For example, generating the HDR output image may include digitally combining full-resolution data from multiple brightness-enhanced component images. In some embodiments, the resolution of the generated HDR output image may be the same as the native resolution of the image sensor array. At stage 1020, some embodiments may output the HDR output image to a processor-controlled system separate from the image sensor system.

[0094] It should be understood that when an element or component is referred to as "connected to" or "coupled to" another element or component in this article, the element or component can be connected or coupled to another element or component, or there may be an intermediate element or component. On the contrary, when an element or component is referred to as "directly connected to" or "directly coupled to" another element or component, there is no intermediate element or component between them. It should be understood that although the terms "first", "second", "third", etc. can be used to describe various elements and components in this article, these elements, components, and regions should not be limited by these terms. These terms are only used to distinguish an element or component from another element or component. Therefore, without departing from the teaching of the present invention, the first element and component discussed below can be referred to as the second element and component. The terms "logical low", "low state", "low level", "logical low level", "low" or "0" used herein are interchangeable. The terms "logical high", "high state", "high level", "logical high level", "high" or "1" are interchangeable.

[0095] The terms "a", "an", and "the" as used herein may include singular and plural references. It should also be understood that when used in this specification, the terms "include", "comprises", "has", and variations thereof specify the presence of the features, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or groups thereof. In contrast, when used in this specification, the term "consisting of" specifies the stated features, steps, operations, elements, and / or parts, and excludes additional features, steps, operations, elements, and / or parts. In addition, the term "and / or" as used herein may refer to and include any possible combination of one or more of the associated listed items.

[0096] Although the present invention is described herein with reference to illustrative embodiments, the description is not intended to be interpreted in a limiting sense. On the contrary, the purpose of the illustrative embodiments is to enable those skilled in the art to better understand the spirit of the present invention. In order not to obscure the scope of the present invention, many details of well-known processes and manufacturing techniques are omitted. With reference to the specification, various modifications of the exemplary embodiments and other embodiments will be apparent to those skilled in the art. Therefore, the appended claims are intended to include any such modifications.

[0097] In addition, some features of the preferred embodiments of the present invention may be used without the corresponding use of other features. Thus, the foregoing description should be considered as merely an illustration of the principles of the present invention, rather than a limitation of the present invention. Those skilled in the art will appreciate that variations of the above-described embodiments fall within the scope of the present invention. As a result, the present invention is not limited to the specific embodiments and illustrations discussed above, but is defined by the following claims and their equivalents.

Claims

1. A method of high dynamic range (HDR) image capture using an image sensor system having an image sensor array formed of a plurality of imaging pixels configured according to a red-green-blue-luminance (RGBL) color filter array (CFA) pattern and arranged to be addressable in groups of imaging pixels, the method comprising: For each of the N subframes, every nth subframe of the N subframes is associated with an nth exposure window and an nth readout window, and every nth readout window starts after an nth exposure time after the start of the nth exposure window, where N is an integer greater than 1, sequentially exposing each imaging pixel group of the image sensor array to incident illumination on the nth exposure window for the nth exposure time; as well as sequentially read out each imaging pixel group over the nth readout window using pixel binning, such that each imaging pixel group is read out immediately after the imaging pixel group is exposed to the incident illumination for the nth exposure time, and at the end of the nth readout window, the pixel binning produces a corresponding one of N downsampled HDR subframes having a corresponding RGB capture frame and a corresponding L capture frame; generating N brightness-enhanced component images by: for each of the N down-sampled HDR sub-frames, up-sampling the corresponding L capture frame to generate a corresponding L-guided frame, and applying brightness-guided interpolation to the corresponding RGB capture frame based on the corresponding L-guided frame to up-sample the RGB capture frame to form a corresponding one of the N brightness-enhanced component images; as well as The N brightness enhanced component images are digitally combined to generate a single HDR output image.

2. The method according to claim 1, wherein: Every (n+1)th readout window starts before every nth readout window ends.

3. The method according to claim 1, wherein: For each imaging pixel group, using pixel merging to read out the group of imaging pixels consumes a group readout time R, so that the duration of each readout window is R times the total number of imaging pixel groups of the image sensor array; Every nth exposure window starts at a corresponding exposure start time Tes(n); and Each Tes(n+1) is greater than the sum of each Tes(n), the nth exposure time and R, and is less than the sum of each Tes(n), the nth exposure time and 2R.

4. The method according to claim 1, further comprising: The HDR output image is output to a processor controlled system separate from the image sensor system.

5. The method according to claim 1, wherein: The up-sampling the corresponding L capture frame to generate the corresponding L guide frame comprises: upscaling the corresponding L capture frame to generate a corresponding upscaled L frame, the corresponding upscaled L frame having a target frame size defined by an output image mode of the HDR output image, a portion of the corresponding upscaled L frame being populated based on known luminance values ​​from the corresponding L capture frame; and The corresponding enlarged L frame is up-sampled based on interpolating the known luminance values ​​to generate the corresponding L guide frame, so that the corresponding L guide frame has luminance information of 100% density.

6. The method according to claim 1, wherein: The applying brightness-guided interpolation to the corresponding RGB capture frame comprises: for each color plane of the RGB capture frame, upscaling the color plane to generate a corresponding upscaling color frame, such that the corresponding upscaling color frame has a target frame size defined by an output image mode of the HDR output image, and a portion of the corresponding upscaling color frame is filled based on a corresponding known color value from the color plane; and The corresponding upscaled color frame is upsampled by applying the luma-guided interpolation to the corresponding known color values ​​based on the corresponding L-guided frame to generate a corresponding upsampled color frame.

7. The method according to claim 1, wherein: For each of the N downsampled HDR subframes, The corresponding L-guide frame includes an array of L-value units, each L-value unit having a corresponding luminance value; and The applying the brightness-guided interpolation to the corresponding RGB captured frame comprises generating a corresponding upsampled RGB frame having an array of color value cells by: for each color value cell of a portion of the color value cells, generating an initial value of the color value unit based on interpolating a set of adjacent known color values ​​according to the corresponding RGB capture frame; as well as An estimated color pixel value for the color value unit is calculated by adjusting the initial value based on the luminance values ​​of those of the L value units that spatially correspond to the positions of the color value unit and the adjacent set of known color values.

8. The method according to claim 7, wherein: The adjustments include: Calculating color ratios or color differences between L value units spatially corresponding to the color value unit and those L value units spatially corresponding to positions of the adjacent set of known color values; and The initial value of the color value unit is adjusted based on the color ratio or the color difference.

9. The method according to claim 1, wherein: The image sensor array has a native resolution; For each of the N downsampled HDR subframes, the corresponding RGB capture frame has a downsampled resolution less than or equal to one quarter of the original resolution; and Each of the N brightness-enhanced component images is generated to have an output resolution that is at least three times the downsampled resolution.

10. The method according to claim 9, wherein: The output resolution is equal to the original resolution.

11. The method according to claim 1, wherein: The RGBL CFA pattern of the image sensor array has a 50% density of brightness imaging pixels and a 50% density of color imaging pixels.

12. The method according to claim 1, wherein: The RGBL CFA pattern of the image sensor array has a 75% density of brightness imaging pixels and a 25% density of color imaging pixels.

13. The method according to claim 1, wherein: N is at least 3.

14. An image sensor, comprising: an image sensor array formed of a plurality of imaging pixels configured according to a red-green-blue-luminance (RGBL) color filter array (CFA) pattern and arranged to be addressable in groups of imaging pixels; a binning readout circuit coupled to the image sensor array to read out the imaging pixel groups using pixel binning to generate N downsampled high dynamic range (HDR) subframes, each HDR subframe having a corresponding RGB capture frame and a corresponding L capture frame, N being an integer greater than 1; A sensor controller configured to generate each of the N down-sampled HDR subframes by: directing each imaging pixel group of the image sensor array to be sequentially exposed to incident illumination over a corresponding exposure window for a corresponding exposure time associated with the downsampled HDR subframe; as well as directing readout of each imaging pixel group by the binning readout circuitry in sequence over a corresponding readout window such that each imaging pixel group is read out immediately after the imaging pixel group is exposed to the incident illumination for the corresponding exposure time, and such that at the end of the corresponding readout window, the pixel binning produces the corresponding RGB capture frame and the corresponding L capture frame for the downsampled HDR subframe; an upsampling engine coupled to the binning readout circuitry to generate N brightness-enhanced output images by, for each downsampled HDR subframe, applying brightness-guided interpolation to the corresponding RGB capture frame based on the L capture frame; as well as An HDR engine is provided for generating a single HDR output image by digitally combining the N brightness-enhanced output images.

15. The image sensor according to claim 14, wherein: The respective readout windows of the N downsampled HDR sub-frames overlap in time.

16. The image sensor according to claim 14, wherein: For each imaging pixel group, reading out the imaging pixel group using pixel merging consumes a group readout time R, so that the duration of each readout window is R times the total number of imaging pixel groups of the image sensor array; A respective nth exposure window of each of the N downsampled HDR subframes starts at a respective exposure start time Tes(n); and Each Tes(n+1) is greater than the sum of each Tes(n), the nth exposure time and R, and is less than the sum of each Tes(n), the nth exposure time and 2R.

17. The image sensor according to claim 14, wherein: The sensor controller is also coupled to the upsampling engine to guide the generation of the N brightness-enhanced component images by: for each of the N downsampled HDR subframes, upsampling the corresponding L capture frame to generate a corresponding L guided frame, and based on the corresponding L guided frame, applying brightness-guided interpolation to the corresponding RGB capture frame to upsample the RGB capture frame to form a corresponding one of the N brightness-enhanced component images.

18. The image sensor according to claim 17, wherein: The upsampling engine upsamples the L capture frame to generate the L guide frame by: upscaling the L capture frame to generate an upscaled L frame having a target frame size defined by an output image mode of the HDR output image, a portion of the upscaled L frame being filled based on known luminance values ​​from the L capture frame; as well as The up-scaled L frame is up-sampled based on interpolating the known luminance values ​​to generate the L pilot frame, so that the L pilot frame has luminance information of 100% density.

19. The image sensor according to claim 17, wherein: For each of the N downsampled HDR sub-frames, the upsampling engine applies the luma-guided interpolation to the corresponding RGB capture frame by: for each color plane of the corresponding RGB capture frame, upscaling the color plane to generate a corresponding upscaling color frame, such that the corresponding upscaling color frame has a target frame size defined by an output image mode of the HDR output image and a portion of the corresponding upscaling color frame is filled based on known color values ​​from the color plane; as well as The corresponding upscaled color frame is upsampled by applying the luma-guided interpolation to the known color values ​​based on the L-guided frame to generate a corresponding upsampled color frame.

20. The image sensor according to claim 14, wherein: The image sensor array has a native resolution; For each of the N downsampled HDR subframes, the corresponding RGB capture frame has a downsampled resolution less than or equal to one quarter of the original resolution; and The output resolution of each of the N brightness-enhanced component images is at least three times the downsampled resolution.

Images