Brightness adaptive processing method for hexadecimal RGBW color filter array

Abstract

This document describes techniques for brightness adaptive processing of a hexadecimal red-green-blue-white (RGBW) color filter array (CFA) in a digital imaging system. Raw image data is acquired by a sensor array configured according to a hexadecimal RGBW CFA pattern, which also acquires relevant ambient brightness information. The ambient brightness information is used to detect one of a plurality of predetermined brightness conditions. Based on the detected brightness condition, embodiments may determine whether and to what extent the raw image data should be downsampled as part of a readout from the sensor array (such as using a binning technique), and whether and to what extent the downsampled data should be re-stitched and / or upsampled to generate an RGB output array for communication with other processing components of the imaging system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a U.S. non-provisional application with application number 63 / 225,524 filed on July 25, 2021, and claims priority, the entire contents of which are incorporated herein by reference.

[0003] This application also claims priority to U.S. application No. 17 / 388,016, filed on July 29, 2021, entitled “LUMINANCE-ADAPTIVE PROCESSING OF HEXA-DECA RGBW COLOR FILTER ARRAYS IN CMOSIMAGE SENSORS,” the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates to digital imaging systems, and more particularly to brightness adaptive processing of hexadecimal RGBW color filter arrays in CMOS image sensors, such as those used in smartphone cameras and / or other digital cameras. Background Art

[0005] Many electronic devices include digital imaging systems. For example, most modern smartphones include one or more digital cameras. Modern image sensors with high pixel counts (e.g., 48-megapixel sensors, etc.) attempt to achieve high frame rates, low read noise, high dynamic range, and / or other features with minimal power consumption (e.g., to extend battery life, reduce heat generation, etc.). The basic function of a modern CMOS image sensor (CIS) is to capture photons that are converted into electrons in a photodetector (e.g., a photodiode). These captured electrons are read out by a series of analog-to-digital converters (ADCs) that are part of the sensor.

[0006] CMOS image sensors often struggle to operate well in low-light conditions for several reasons. One reason is that CIS detection of multiple colors often requires covering the photodetector with a color filter array (CFA), an array of color filters arranged in a specific pattern. While the CFA allows the CIS to distinguish between different colors, the nature of the color filters reduces the number of photons collected by the photodetector. Another reason is that CMOS image sensors often struggle to operate well in low-light conditions, driven by the pursuit of smaller sensor sizes and increased pixel counts in many modern applications. Consequently, pixel size continues to decrease. However, with smaller pixels, fewer photons per pixel reach the active photodiode area to generate electron-hole pairs. In low-light conditions, only a relatively small number of photons may need to be collected. Consequently, the number of collected photons is further reduced due to CFA filtering, reduced pixel size, and / or other factors, resulting in very low signal levels. In some cases, the signal level is too low to support reliable reconstruction of imaging information, such as when the signal level is too low to be reliably distinguished from noise.

[0007] There are several conventional approaches to improving low-light performance. One example of a conventional approach is to increase the power of the readout chain, which can provide lower readout noise and / or higher dynamic range, thereby improving image quality. However, higher power consumption also shortens battery life, increases heat, and can have other adverse consequences on the performance of the sensor and / or more generally on consumer product implementations. Another example of a conventional approach is to use pixel merging to combine the outputs of multiple photodetectors in each color plane to effectively increase the signal level of each color plane. While this pixel merging can increase readout speed and reduce noise without increasing power consumption, there is a significant trade-off in terms of reduced sensor resolution. Consequently, conventional CIS designs still strive to achieve high performance in low-light conditions while maintaining other features such as high resolution, fast readout, low noise, and low power consumption. Summary of the Invention

[0008] Embodiments provide systems and methods for brightness adaptive processing of a hexadecimal red-green-blue-white (RGBW) color filter array (CFA) in a digital imaging system. Raw image data is acquired by a sensor array configured according to a hexadecimal RGBW CFA pattern, which also acquires relevant ambient brightness information. The ambient brightness information is used to detect one of a plurality of predetermined brightness conditions. Based on the detected brightness condition, embodiments may determine whether and to what extent the raw image data should be downsampled as part of a readout from the sensor array (such as using a binning technique), and whether and to what extent the downsampled data should be re-stitched and / or upsampled to generate an RGB output array for communication with other processing components of the imaging system.

[0009] According to one set of embodiments, a method for brightness adaptive processing for a hexadecimal RGBW CFA CIS system is provided. The method includes acquiring raw image data via a photodetector array configured according to a hexadecimal RGBW CFA having a raw array resolution; detecting a brightness condition, the brightness condition being one of a set of predetermined brightness conditions including at least a high brightness condition and a low brightness condition, for acquiring the raw image data via the photodetector array; and generating a sensor output signal representing a Bayer-RGB output array based on the raw image data and the brightness condition by: in response to detecting the brightness condition as the high brightness condition, re-stitching the raw image data to convert the hexadecimal RGBW CFA to the Bayer-RGB output array at an output array resolution; and in response to detecting the brightness condition as the low brightness condition, downsampling the raw image data into a downsampled Bayer array and a downsampled brightness array, and upsampling the downsampled Bayer array based on the downsampled brightness array to generate the Bayer-RGB output array at the output array resolution.

[0010] According to another set of embodiments, an image sensor system is provided. The system includes: one or more processors coupled to a photodetector array, the photodetector array configured to acquire raw image data at a raw array resolution based on a hexadecimal red, green, blue, and white (RGBW) color filter array (CFA); and a non-volatile memory having instructions stored thereon that, when executed, cause the one or more processors to perform the following steps, including: detecting a luminance condition associated with the acquisition of the raw image data by the photodetector, the luminance condition being detected as one of a set of predetermined luminance conditions, including at least a high luminance condition and a low luminance condition; and generating a sensor output signal representing a Bayer-RGB output array based on the raw image data and the luminance condition by: in response to detecting the luminance condition as the high luminance condition, re-stitching the raw image data to convert the hexadecimal RGBW image data into a Bayer-RGB output array. The invention also provides a method for converting the raw image data into the Bayer-RGB output array at an output array resolution by CFA; and in response to detecting the luminance condition as the low luminance condition, guiding downsampling the raw image data into a downsampled Bayer array and a downsampled luminance array, and upsampling the downsampled Bayer array based on the downsampled luminance array to generate the Bayer-RGB output array at the output array resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present invention.

[0012] Figure 1 A block diagram of an image sensor environment is shown in the context of various embodiments described herein.

[0013] Figure 2 A simplified schematic diagram of a portion of an example image sensor including pixels and readout circuitry supporting one or more binning schemes for use with embodiments described herein is shown.

[0014] Figure 3 An example of a hexadecimal red, green, blue, and white (RGBW) color filter array (CFA) under a specific binning scheme is shown according to various embodiments described herein.

[0015] Figure 4 An illustrative system for brightness adaptive processing of a hexadecimal RGBW CFA is shown, in accordance with various embodiments.

[0016] Figure 5 According to various embodiments, the use of Figure 4 Data flow chart of the system performing brightness adaptive processing on hexadecimal RGBW CFA.

[0017] Figure 6 A flow chart of an illustrative method for brightness adaptive processing of a hexadecimal RGBW CFA in a CIS system is shown, according to various embodiments.

[0018] In the accompanying drawings, similar components and / or features may have the same reference numerals. In addition, components of the same type may be distinguished by following the reference numeral with a second numeral that distinguishes between similar components. If only the first reference numeral is used in the specification, the description applies to any similar component having the same first reference numeral, without regard to the second reference numeral. DETAILED DESCRIPTION

[0019] The following description provides many specific details for a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, for the purpose of brevity, features and techniques known in the art are not described.

[0020] See also Figure 1 , a block diagram of an image sensor environment 100 in the context of various embodiments described herein is shown. Image sensor environment 100 is shown as including a processor 110 in communication with a processor control system 120 and a complementary metal-oxide semiconductor (CMOS) image sensor (CIS) system 130. Image sensor environment 100 can be used to implement a digital imaging system in any suitable application environment. For example, processor 110, processor control system 120, and CIS system 130 can all be implemented in a smartphone, a digital camera, a wearable device, an implantable device, a laptop, a tablet, an e-reader, an Internet of Things (IoT) device, or any other suitable device.

[0021] Processor control system 120 is generally intended to represent any suitable system or systems that provide any suitable features of image sensor environment 100, rather than features of CIS system 130. For example, in a smartphone, processor control system 120 may include subsystems for providing call and communication features, display features, user interaction features, application processing features, etc. Embodiments of image sensor environment 100 may include one or more processors 110. In some embodiments, one or more processors 110 are shared between processor control system 120 and CIS system 130. In other embodiments, processor control system 120 utilizes one or more processors 110, while CIS system 130 has one or more dedicated processors 110.

[0022] An embodiment of the CIS system 130 includes a sensor array 140 and a sensor control circuit 150. As described below, the sensor array 140 and the sensor control circuit 150 can communicate via an interface channel 145. The sensor array 140 can be implemented as an array of photodetector elements 142, which can be implemented by any suitable photosensitive component or group of components. In some cases, the sensor array 140 is a high-pixel-count array, such as a 48-megapixel array. In some implementations, each photodetector element 142 can include a photodiode and a filter for detecting light energy in one or more frequency bands. The photodetector elements 142 can implement a color filter array (CFA), allowing the sensor array 140 to detect light energy within at least the visible spectrum and output corresponding electrical signals. The specific CFA arrangement is assumed herein to be a so-called hexadecimal red, green, blue, and white (RGBW) CFA. As described below, such a CFA includes red, green, and blue (RGB) pixels, with white (W) pixels (also known as "luminance pixels") uniformly distributed at a density of approximately 50 percent.

[0023] Although not explicitly shown, the sensor array 140 also includes a readout circuit. As described below, the readout circuit typically may include a readout line to selectively feed the analog output signal from the photodetector element 142 to an analog-to-digital converter (ADC), which may convert the analog output signal into a digital output signal for output to the sensor control circuit 150 via the interface channel 145. For illustration, Figure 2 A simplified schematic diagram 200 of a portion of an exemplary image sensor including pixels and readout circuitry that supports one or more binning schemes for use with embodiments described herein is shown. The portion of the image sensor may be Figure 11. The partial implementation of sensor array 140 in FIG. The illustrated partial array is shown as having four pixels, but the features illustrated by schematic 200 can be extended to any suitable number of pixels.

[0024] Each pixel is shown to include a photodiode (PD) or a photosensitive element, and a transfer transistor (TX) coupled to the PD. The TXs of multiple pixels are coupled to a floating diffusion node (FD) of the readout circuit. The readout circuit includes a reset transistor (reset signal, RST) having a drain node coupled to a reset voltage reference (VDD_RST), a source node coupled to the FD (i.e., to the TX), and a gate node controlled by the reset signal (RST). When RST is turned on, RST is used to charge the FD to VDD_RST, thereby resetting the FD. Each PD can be reset together with the FD by turning on the TX corresponding to each PD (e.g., by asserting or deasserting the TXn control signal). The readout circuit also includes a source follower transistor (SF) having a drain node coupled to a source follower voltage reference (VDD_SF), a gate node coupled to FD (i.e., to TX), and a row select transistor (SEL) having a source node coupled to the drain node. SEL has a gate node coupled to a row select line and a source node coupled to a voltage readout line that provides an analog output pixel signal (Vout) to the ADC for data conversion. The source node of SEL is also coupled to a current source (IBIAS). In the illustrated implementation, TX, RST, SF, and SEL are NMOS transistors. PMOS transistors and / or other suitable component designs may also be used for implementation.

[0025] As shown, embodiments may include a controller for generating clock and control signals. The controller may be implemented using any suitable hardware, firmware, etc. In some implementations, the controller is integrated with the sensor array as a sensor component (e.g., as part of the sensor array 140). In other implementations, the controller is implemented as a sensor external component (e.g., by the sensor control circuit 150) by a separate controller or processor. In other implementations, the features of the controller are distributed between one or more sensor components and one or more sensor external components. For example, the sensor control circuit 150 (external to the sensor) may generate instructions, guide timing and / or generate specific control signals by the on-sensor controller. In some embodiments, the controller may include processing circuits, logic state machines, phase-locked loops, etc. to provide clock and control signals to the image sensor.

[0026] In addition, embodiments may include a data memory for storing digital data representing pixel signals after pixel conversion by the ADC. In some implementations, the data memory includes buffers and / or registers for temporarily storing readout data before transmitting the data to other processing components (e.g., before transmitting to the sensor control circuit 150 via the interface channel 145). For example, the readout data is buffered in the data memory for fast, serial transmission to the sensor control circuit 150.

[0027] The embodiments described herein dynamically use one or more merging schemes in response to detected ambient brightness conditions. For example, merging may not be used under a first ambient brightness condition (e.g., a high brightness condition), while a merging scheme may be used under a second ambient brightness condition (e.g., a low brightness condition). Alternatively, a first merging scheme may be used for a first ambient brightness condition (e.g., a high brightness condition), and a second merging scheme may be used for a second ambient brightness condition (e.g., a low brightness condition). As used herein, "merging" refers to so-called charge merging, in which the charges of multiple pixels (e.g., corresponding to the amount of light detected by the pixels, the number of photons, etc.) are added, averaged, or otherwise combined onto one or more levels of readout lines.

[0028] As shown in the figure, each pixel (e.g., a pixel on each row or column) has its own TX. The TXn control signal can be generated by a controller with specific timing to support different merging schemes. For illustration, schematic 200 can be used to transfer charge from a single PD (e.g., PD1) to the FD by using a single TX (e.g., TX1) to enable readout without merging (e.g., turning on RST before turning on TX to reset the FD to VDD_RST). The charge on the FD is then transferred to the readout line and ADC via SF and SEL so that the data is converted into digital data, which can then be stored in a data memory. Similarly, schematic 200 can be used to transfer charge from multiple PDs to the FD together by simultaneously controlling multiple TXs (e.g., TX1-TX4) to enable their respective PDs (e.g., PD1-PD4), enabling readout with merging (e.g., turning on RST before turning on multiple TXs to reset the FD to VDD_RST). The combined charge on the FD is now then transferred to the readout line and ADC via SF and SEL to convert the data into digital data, which can then be stored in a data memory. Different merging schemes can be supported based on the type of pixels sharing a common FD, the number of pixels sharing a common FD, the capabilities of the controller, etc. In some embodiments, one or more merging schemes may involve multi-stage merging. For example, a first merging scheme charges 32 RGB pixels together to produce 16 merged RGB pixel outputs, and a second merging scheme may re-merge the 16 merged RGB pixel outputs to produce 4 re-merged RGB pixel outputs. Although the above refers to charge merging, the embodiments may be implemented with other suitable types of merging, such as digital merging, merging in a later post-processing stage, etc.

[0029] Back to Figure 1The sensor control circuit 150 may include any suitable processor and / or circuitry for directing the operation of the sensor array 140, processing signals received from the sensor array 140, and interfacing with other systems (e.g., the processor 110). Some implementations of the sensor control circuit 150 are implemented as or include an integrated circuit (IC) having integrated interface components, storage components, and processing components. For example, the processing components of the sensor control circuit 150 may include one or more central processing units (CPUs), application-specific integrated circuits (ASICs), application-specific instruction-set processors (ASIPs), graphics processing units (GPUs), physics processing units (PPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, microcontroller units, reduced instruction set computers (RISC) processors, complex instruction set computers (CISC) processors, microprocessors, etc., or any combination thereof.

[0030] As described herein, the CIS system 130 is configured to provide novel hexadecimal RGBWCFA brightness adaptation processing within the image sensor environment 100. For example, a sensor array acquires raw image data from an image capture of a scene. The CIS system 130 (e.g., the sensor array 140, a separate brightness sensor 155, or any other suitable component) also captures ambient brightness data that influences the raw image data acquired by the sensor array. Brightness conditions are determined, such as detecting high brightness conditions or low brightness conditions. In high brightness conditions, embodiments generally assume that a sufficient number of photons reach the active photodiode areas of each pixel in the sensor array 140 to support relatively straightforward generation (e.g., with little or no downsampling and upsampling) of a high brightness image at the full sensor array resolution. In low brightness conditions, embodiments generally assume that a sufficient number of photons do not reach the active photodiode areas of each pixel in the sensor array 140, and one or more binning schemes may be used to improve the response at lower brightness levels. The binning schemes may be used in conjunction with upsampling techniques to utilize the brightness data of the W pixel color plane to efficiently generate low brightness images at the same full sensor array resolution. Some embodiments may support a greater number of brightness conditions, such as medium brightness conditions.

[0031] As described above, embodiments are described with reference to a hexadecimal RGBW CFA. Many conventional CFAs are designed based on a Bayer pattern. Over the years, image sensor array designers have explored different variations of Bayer and non-Bayer CFA patterns, including using different levels of brightness, each design having its own characteristics and limitations. For example, some early Bayer patterns used RGGB CFAs, describing the green (G) pixels as providing additional brightness (relative to the red (R) and blue (B) pixels). Other prior and current CFA designs use yellow (Y) pixels, such as the RYYB pattern, to provide additional brightness relative to the G pixels. However, although some existing designs have attempted to further increase brightness by adding white (W) (for example, in the RWWB color pattern, etc.), these attempts have often been unsuccessful. For example, due to the large difference in the amount of light received by the white and color pixels under any particular lighting conditions, such existing designs generally cannot produce the desired response characteristics (for example, signal-to-noise ratio, pixel conversion gain, etc.) across the entire array.

[0032] The Hexadecimal RGBW CFA is a recently developed non-Bayer pattern in which W pixels are evenly interspersed among the RGB pixel blocks, supporting various specialized techniques to produce the desired response characteristics. Figure 3An example of a hexadecimal RGBW CFA 310 under a particular binning scheme according to various embodiments described herein is shown. The illustrated hexadecimal RGBW CFA 310 includes an 8 by 8 array (e.g., which can be implemented as a quadrant of a larger 16 by 16 array, etc.) of photodiodes, including approximately 50% W pixels and approximately 50% RGB pixels. As can be seen, each position in the array alternates between an RGB pixel and a W pixel, such that the W pixels are evenly interspersed with the RGB pixels. The illustrated hexadecimal RGBW CFA 310 can be considered to be a block of a larger array, such as Figure 1 For example, the entire CIS may include millions of total pixels, implemented as thousands of instances of the hexadecimal RGBW CFA 310.

[0033] As mentioned above, pixel binning generally involves arranging the readout circuitry to effectively group pixels on the same color plane so that the responses of those pixels (i.e., corresponding to the amount of light detected) can be summed, averaged, or otherwise combined. Different types of CFAs can perform binning in different ways, and new binning techniques have been invented for new types of CFAs. For example, in a conventional Bayer pattern CFA, pixels of the same color can be binned before ADC readout; in a conventional quad-Bayer pattern, 2×2 pixel clusters can be binned before ADC readout, and so on.

[0034] This conventional Bayer-style binning method cannot be directly applied to a hexadecimal RGBW CFA because the W pixels are dispersed in a specific pattern among the RGB pixels throughout the array. Various methods for binning pixels for hexadecimal RGBW CFAs 310 are described and illustrated in U.S. Patent Application No. 17 / 382,354, entitled "Pixel Binning for Hexadecimal RGBW Color Filter Arrays," which is incorporated herein by reference in its entirety. Generally speaking, such binning schemes use the concept of diagonal pixel binning to combine localized R, G, B, and W pixels while seeking to optimize circuit noise performance.

[0035] exist Figure 3, the 8×8 hexadecimal RGBW CFA blocks 310 are merged to generate a 4×8 downsampled block 320. Each labeled box in the hexadecimal RGBW CFA block 310 represents an unmerged pixel 315. Each labeled box in the downsampled block 320 represents a merged pixel 325. Each label represents the color as "R," "G," "B," or "W," representing red, green, blue, or white, respectively. Each label also indicates a merge index. For example, the pixel at the bottom right position in the illustrated array is represented as "R0," indicating that the pixel is red and belongs to the 0th merge group; the pixel at the top left position in the illustrated array is illustrated as "B24582," indicating that the pixel is blue and is part of the 24582nd merge group. Figure 3 The particular merge index number shown in may represent a situation where the illustrated 8 by 8 hexadecimal RGBW CFA block 310 is the lower rightmost block of a larger array that includes thousands of blocks 310 .

[0036] As shown, each unbined pixel 315 in each row of the hexadecimal RGBW CFA block 310 has a different binning index than every other unbined pixel 315 in that row. For example, the bottom row of the hexadecimal RGBW CFA block 310 includes eight unbined pixels 315 with indices "0" through "7." However, each unbined pixel 315 in every second row of the hexadecimal RGBW CFA block 310 shares a binning index with a diagonally adjacent unbined pixel 315 in the row immediately below it. For example, the unbined pixel 315a in the top row (i.e., the eighth row) of the hexadecimal RGBW CFA block 310 is a white pixel in the 24583rd binning group, and the diagonally adjacent unbined pixel 315b in the immediately adjacent row of the hexadecimal RGBW CFA block 310 (i.e., the seventh row) is also a white pixel in the 24583rd binning group. It is intended to be understood that binning pixels in the hexadecimal RGBW CFA block 310 involves combining unbinned pixels 315 having the same index (e.g., by turning on these pixels simultaneously to pass the combined charge to the readout circuitry, as described with respect to FIG. Figure 2 For example, unbinned pixel 315a is binned with diagonally adjacent unbinned pixel 315b to be read out as a single binned pixel 325a in downsampling block 320.

[0037] Using the method shown in the figure, it can be seen that the result of the merging is essentially the same number of columns, but half the number of rows. Therefore, merging allows the entire array to be read out with half the number of read operations. This merging method can be called "1Hx2V" merging, which means that the horizontal dimension is divided by "1" (i.e., the array is not downsampled in the horizontal dimension), and the vertical dimension is divided by "2" (i.e., the array is downsampled by a factor of 2 in the vertical dimension). The same merging result can also be viewed as two 4×4 arrays: a downsampled (4×4) RGB output array consisting of alternating columns of merged RGB pixel data, and a downsampled (4×4) luminance output array consisting of alternating columns of merged white pixel data. Using the method shown in the figure, the downsampled RGB output array is a Bayer-type array, specifically a quad-Bayer array.

[0038] Figure 4 An illustrative system 400 for brightness adaptive processing of a hexadecimal RGBW CFA is shown in accordance with various embodiments. An embodiment of the system 400 includes a sensor array 140 coupled to a CIS system 130. The sensor array 140 may include an array of photodetectors 142 implemented as a hexadecimal RGBW CFA for acquiring raw image data at a raw array resolution. The raw array resolution corresponds to the number of pixels in the array (e.g., a 48-megapixel array may be said to have a raw array resolution of 48 megapixels). Notably, this may be different from the output resolution of the image sensor. For example, after processing the image to combine color planes, etc., a 48-megapixel hexadecimal RGBW CFA may only support a final imaging resolution of 12 megapixels, or some other resolution that is less than the raw array resolution.

[0039] Embodiments may include one or more processors. In some embodiments, the one or more processors implement an on-sensor controller 410, which may be integrated with the sensor array 140 (e.g., including its circuitry). The one or more processors may also implement an off-sensor controller 420, which may be part of the CIS system 130. The one or more processors may implement various engines of the CIS system 130, such as one or more re-stitching engines 440, one or more amplification engines 450, and one or more guided filter engines 460.

[0040] Embodiments of the CIS system 130 may also include one or more interfaces, such as a sensor input interface 430 and an RGB output interface 470. The sensor input interface 430 may include any suitable components for receiving imaging data from the sensor array 140 and transmitting commands and / or other information to the sensor array 140, such as via the interface channel 145. The imaging data received by the sensor array 140 may be the sensor array output data 405. The RGB output interface 470 may include a processor for transmitting the imaging data to other processing components (e.g., Figure 1 The imaging data output by the CIS system 130 may be RGB output data 480.

[0041] Although not explicitly shown, the various controllers and engines implemented by one or more processors operate based on instructions stored in one or more processor-readable non-volatile memories. For example, the on-sensor controller 410 and the off-sensor controller 420 may each have and / or be coupled to one or more memories having firmware, software, and / or other instructions stored thereon. Execution of the stored instructions may configure the processors and / or other circuits to perform brightness adaptive processing on the raw image data from the array of photodetectors 142 to generate RGB output data 480.

[0042] Embodiments of the off-sensor controller 420 include a brightness detector 425 to detect ambient brightness conditions associated with the acquisition of raw image data by the sensor array 140. The ambient brightness condition is detected (e.g., or identified) as one of a set of predetermined brightness conditions, which may include at least a high brightness condition and a low brightness condition. In some embodiments, the predetermined set of brightness conditions also includes a medium brightness condition, and / or one or more other brightness conditions. In some embodiments, the brightness detector 425 is coupled to the brightness sensor 155, which may be integrated with the sensor array 140, integrated with the CIS system 130, implemented separately from the sensor array 140 and the CIS system 130, or implemented in any other suitable manner. The brightness sensor 155 may include any component suitable for obtaining one or more ambient brightness measurements affecting some or all areas of the array of photodetectors 142, such as one or more light level meters, photodiodes, etc. In other embodiments, the brightness detector 425 derives brightness information from the sensor array output data 405, such as from white pixel data.

[0043] In some embodiments, brightness detector 425 outputs a single determination of the brightness condition for the entire photodetector 142 array associated with at least the current image capture frame. In other embodiments, brightness detector 425 outputs a separate brightness condition determination for each of multiple regions of the photodetector 142 array, such as by performing block-level determinations by dividing the photodetector 142 array into quadrants or regions, into 16×16 pixel hexadecimal RGBW CFA blocks (e.g., or any other appropriately sized blocks), etc. In some embodiments, each brightness condition determination made by brightness detector 425 is valid only for the current image capture frame. In other embodiments, some or all brightness condition determinations made by brightness detector 425 are associated with a validity window, such that the same determination can be used across multiple image capture frames. The validity window can be based on a certain amount of time, a certain number of image capture frames, a threshold change in the overall lighting condition detected (e.g., by brightness detector 425, by brightness sensor 155, etc.), etc. In some embodiments, brightness detector 425 determines the brightness condition based on brightness information acquired concurrently with the current image capture frame. For example, current ambient brightness information is obtained from brightness sensor 155 while sensor array output data 405 is obtained from sensor array 140. In other embodiments, brightness detector 425 determines brightness conditions based on brightness information obtained before the current image acquisition frame. For example, in one or more previous image acquisition frames, brightness detector 425 uses pixel data obtained from white (brightness) pixels of a hexadecimal RGBW CFA to derive brightness data and determine brightness conditions for one or more subsequent image acquisition frames.

[0044] The off-sensor controller 420 uses the brightness conditions determined by the brightness detector 425 to generate various control signals to direct brightness adaptive processing of the raw image data from the array of photodetectors 142. Embodiments of the off-sensor controller 420 implement such brightness adaptive processing by adaptively indicating whether and / or the degree of downsampling to be applied to the raw image data by the on-sensor controller 410, such as by using one or more binning schemes; by adaptively indicating whether and / or the degree of upsampling to be applied to the downsampled image data by the upscaling engine 450; and by adaptively indicating whether and / or the degree of re-stitching to be performed on the raw and / or downsampled image data by the re-stitching engine 440.

[0045] To explain more clearly, Figure 5 According to various embodiments, the use of Figure 4 A data flow diagram 500 of the system 400 performing brightness adaptive processing on a hexadecimal RGBW CFA is shown. Figure 5The illustrative embodiment assumes three detectable ambient light conditions and associated processing paths: high light conditions processed according to a high light path 510, medium light conditions processed according to a medium light path 520, and low light conditions processed according to a low light path 530. The input to each processing path is the same raw image data 512 at the raw array resolution acquired by the sensor array 140, and the output of each processing path is RGB output data 480.

[0046] In some cases, the brightness detector 425 detects the brightness condition as a high brightness condition. In response to this detection, an embodiment of the off-sensor controller 420 can direct the sensor array 140 to output the sensor array output data 405 at the original array resolution: as raw image data 512 (e.g., unbinned). The unbinned raw image data 512 can be received by the CIS system 130 via the sensor input interface 430 and passed to the first of the re-stitching engines 440a, where the pixel data arranged in the hexadecimal RGBW CFA can be re-stitched into a standard Bayer-RGB output array format for communication with other processing components (as RGB output data 480). As used herein, "Bayer-RGB" generally refers to the standard RGGB Bayer CFA pattern expected by the standard output interface, while "quad-Bayer" specifically refers to a Bayer CFA pattern having 2×2 blocks of the same color pixels arranged in a 4×4 RGGB pattern.

[0047] As used herein, "re-stitching" generally refers to converting an input array in an input CFA format to an output array in an output array format by estimating corresponding values ​​in the output array using pixel values ​​of the input array. Assume that the first and second adjacent pixels of the input array are the "R" and "W" pixels of a hexadecimal RGBW CFA pattern; the corresponding two adjacent pixels of the output array are the 'R' and 'G' pixels of a Bayer-RGB CFA pattern. The re-stitching in this case may directly use the "R" at the first position of the input array as the "R" at the first position of the output array. However, the "G" pixel at the second position of the output array is not directly obtained from the corresponding "W" pixel of the input array; instead, the re-stitching may estimate the "G" pixel from surrounding information, such as from nearby "G" pixels, other color pixels, brightness information, and / or other information (e.g., using interpolation, filtering, and / or other techniques). Any suitable re-stitching method may be used.

[0048] As described above, the high light path 510 can efficiently process pixel information under the assumption that the pixel receives sufficient light to achieve a good signal-to-noise ratio. For low light conditions, the corresponding processing path assumes different conditions and uses a different merging scheme accordingly. In response to detecting any lower light conditions, an embodiment of the off-sensor controller 420 can direct the sensor array 140 to downsample the raw image data 512 into a downsampled Bayer array 542 and a downsampled luminance array 544, and upsample the downsampled Bayer array 542 based on the downsampled luminance array 544 to generate RGB output data 480 at the output array resolution (e.g., the raw array resolution).

[0049] In some cases, the brightness detector 425 detects the lower brightness condition as a medium brightness condition. In response to this detection, an embodiment of the off-sensor controller 420 can direct the sensor array 140 (e.g., the on-sensor controller 410) to read out the sensor array output data 405 according to a first binning scheme. In practice, the on-sensor controller 410 can reconfigure the readout circuitry into a first binning engine 525a (labeled as a hexa-deca binning engine (HDBE)). In the illustrated embodiment, the first binning engine 525a is Figure 3 The 1Hx2V binning scheme shown configures the readout circuitry to produce a first downsampled Bayer array 542a (generated by binning the binned RGB pixels to produce a quad-Bayer array) and a first downsampled luma array 544a (generated by binning the W pixels). Each of the first downsampled Bayer array 542a and the first downsampled luma array 544a is downsampled by a factor of 2 by diagonally binning pairs of pixels in the same color plane.

[0050] The off-sensor controller 420 can direct the second respinning engine 440b (labeled as the Quad-Bayer respinning engine (QB RE)) to convert the first downsampled Bayer array 542a from the Quad-Bayer CFA pattern to the standard Bayer-RGB pattern array 522. The off-sensor controller 420 can then direct the first upscaling engine 450a (labeled as the Bayer x2 upscaling engine, or Bx2 upscaling engine (SUE)) to upscale the respinned Bayer-RGB pattern array 522 by a factor of two. This upscaling effectively generates a template array for subsequent upsampling. In one implementation, this upscaling generates an array with twice the horizontal dimension and twice the vertical dimension by inserting a placeholder value between each pixel value in the respinned Bayer-RGB pattern array 522. For example, if a row of the respinned Bayer-RGB pattern array 522 is RRGG, the top row of the corresponding upscaled array can be RPGPRPGP, where 'P' represents a placeholder value. In another implementation, the upscaling generates an array with twice the horizontal dimension and twice the vertical dimension by inserting a cell between each cell in the re-stitched Bayer-RGB pattern array 522 and copying each pixel value in the re-stitched Bayer-RGB pattern array 522 to its adjacent inserted cell in the scaled array. For example, if a row of the re-stitched Bayer-RGB pattern array 522 is RRGG, the corresponding top row of the upscaled array can be RRGGRRGG. Any other suitable technique can be used to upscale the array. The off-sensor controller 420 can direct the second upscaling engine 450b (labeled as the Luma x2 Upscaling Engine, or Lx2 SUE) to upscale the first downsampled luma array 544a by a factor of 2, generating an upscaled luma array 546. The upscaling of the first downsampled luma array 544a can be performed using the same or different techniques as those used to upscale the first downsampled Bayer array 542a. Importantly, the upscaled Bayer array and the upscaled luma array 546 have the same dimensions.

[0051] An embodiment of the off-sensor controller 420 can direct the guided filter engine 460 to effectively upsample the upscaled Bayer array based on the upscaled luma array 546 to RGB output data 480. In some embodiments, the guided filter engine 460 can perform this upsampling based on the same or similar techniques as those used by the re-stitching engine 440. An implementation of the guided filter engine 460 can generate pixel values ​​for some or all of the pixel values ​​in the RGB output data 480 by, for each target pixel, identifying the values ​​of the same-color and luma pixels local to the target pixel location (based on their respective upscaled arrays), using the values ​​of these identified pixels and their distances from the target pixel to generate weighting factors, and applying one or more filtering algorithms to interpolate the target pixel value. For example, the luma values ​​of W pixels of the upscaled luma array 546 and their distances from a particular target pixel can be used to calculate weights, and these weights, along with the values ​​of the neighboring same-color pixels, can be input to a joint bilateral filter to calculate the target (upsampled) pixel value for the RGB output data 480.

[0052] In some cases, the brightness detector 425 detects the lower brightness condition as a low brightness condition (i.e., a brightness lower than the medium brightness condition). In response to this detection, an embodiment of the off-sensor controller 420 can direct the sensor array 140 (e.g., the on-sensor controller 410) to read out the sensor array output data 405 according to a second binning scheme. In practice, the on-sensor controller 410 can reconfigure the readout circuitry into a series of first and second binning engines 525 (including the first binning engine 525a and the second binning engine 525b described above, labeled as 2x2 binning engines, 2x2 BE). In the illustrated embodiment, the first binning engine 525a, as described above, is Figure 3 The readout circuitry is configured for the 1Hx2V binning scheme shown in FIG, generating a first downsampled Bayer array 542a and a first downsampled luma array 544a. A second binning engine 525b configures the readout circuitry for a 2Hx2V binning scheme, generating a second downsampled Bayer array 542b and a second downsampled luma array 544b. Each of the first downsampled Bayer array 542a and the first downsampled luma array 544a is downsampled by a factor of 2 relative to the first downsampled Bayer array 542a and the first downsampled luma array 544a. For example, each 2×2 color block of the four Bayer CFA pattern of the first downsampled Bayer array 542a can be binned into a single pixel of the second downsampled Bayer array 542b. The second downsampled Bayer array 542b and the second downsampled luma array 544b can also be considered downsampled by a factor of 4 relative to the original image data 512.

[0053] The off-sensor controller 420 can then instruct the third upscaling engine 450c (labeled as the Bayer x4 Upscaling Engine, or Bx4 SUE) to upscale the second downsampled Bayer array 542b by a factor of four, and the fourth upscaling engine 450d (labeled as the Luma x4 Scale-Up Engine, or Lx4 SUE) to similarly upscale the second downsampled luma array 544b by a factor of four. As described above for the midlight path 520, upscaling in the lowlight path 530 results in a scaled-up Bayer array and a scaled-up luma array 546, both of the same size (also the same size as the RGB output data 480). Embodiments of the lowlight path 530 can proceed in much the same manner as the midlight path 520; embodiments of the off-sensor controller 420 can direct the filtering engine 460 to upsample the scaled Bayer array to the RGB output data 480 based on the scaled-up luma array 546, as described above.

[0054] Figure 6 A flow chart is shown of an illustrative method 600 for brightness adaptation processing of a hexadecimal RGBW CFA in a CIS system, according to various embodiments. Embodiments of method 600 may begin at stage 604 by acquiring raw image data from a photodetector array configured with a hexadecimal RGBW CFA having a raw array resolution. At stage 608, embodiments may detect a brightness condition used to acquire the raw image data via the photodetector array. As described herein, the brightness condition may be detected as one of a set of predetermined brightness conditions, such as at least a high brightness condition and a low brightness condition. In some embodiments, the raw image data is acquired by the photodetector array in stage 604 during an image acquisition frame, and detecting the brightness condition in stage 608 is based on brightness data acquired during a previous image acquisition frame. For example, detecting the brightness condition in stage 608 includes extracting brightness pixel data from previous image data acquired by the photodetector array during a previous image acquisition frame, and detecting the brightness condition based on the brightness pixel data. In some embodiments, the photodetector array includes a plurality of array regions (e.g., quadrants, etc.), and detecting the brightness condition in stage 608 includes detecting a respective region-level brightness condition for each of at least a portion of the plurality of array regions. In some such embodiments, each array region is one of a plurality of hexadecimal RGBWCFA blocks.

[0055] The detection at stage 608 can be considered a decision block, whereby method 600 can proceed along different paths depending on the detection result. In particular, method 600 can continue to generate sensor output signals representing a Bayer-RGB output array based on the raw image data and the luminance condition, according to stage 612 or 616. Method 600 can proceed to stage 612 in response to detecting that the luminance condition at stage 608 is a high luminance condition. In response, embodiments can re-stitch the raw image data to convert the hexadecimal RGBW CFA to a Bayer-RGB output array at the output array resolution. In response to detecting that the luminance condition at stage 608 is a low luminance condition, method 600 can proceed to stage 616. In response, embodiments can downsample the raw image data into a downsampled Bayer array and a downsampled luminance array, and can upsample the downsampled Bayer array based on the downsampled luminance array to generate a Bayer-RGB output array at the output array resolution.

[0056] In some embodiments, the output array resolution is equal to the original array resolution. In other embodiments, the output array resolution is the same for all detected luminance conditions, but is different from the original array resolution. In some embodiments, the low luminance condition detected in stage 608 is one of a plurality of low luminance conditions, such as a first low luminance condition (e.g., a medium luminance condition) and a second low luminance condition (e.g., a minimum luminance condition). In response to detecting the luminance condition as the first low luminance condition, downsampling is performed according to a first downsampling factor such that each of the downsampled Bayer array and the downsampled luminance array has a first downsampled resolution that is less than the original array resolution by the first downsampled factor. In response to detecting the luminance condition as the second low luminance condition, downsampling is performed according to a second downsampling factor such that each of the downsampled Bayer array and the downsampled luminance array has a second downsampled resolution that is less than the original array resolution by the second downsampled factor, the second downsampled factor being greater than the first downsampled factor.

[0057] In some embodiments, in response to detecting the luminance condition as a first low luminance condition, downsampling includes diagonally binning the original image data to generate a quad Bayer array at a first downsampled resolution, generating a downsampled luminance array at the first downsampled resolution, and re-stitching the quad Bayer array into the downsampled Bayer array. In such embodiments, upsampling may include upscaling the downsampled Bayer array based on a first downsampling factor to generate an upsampled Bayer-RGB array, upsampling the downsampled luminance array based on the first downsampling factor to generate an upsampled luminance array, and upsampling the upscaled Bayer-RGB array based on the upscaled luminance array to generate a Bayer-RGB output array at the output array resolution.

[0058] In some embodiments, in response to detecting the luminance condition as a second low luminance condition, downsampling includes diagonally binning the original image data to generate a quad-Bayer array at the first downsampled resolution and a pre-downsampled luminance array at the first downsampled resolution, and rebinning the quad-Bayer array and the pre-downsampled luminance array to generate a downsampled Bayer array and a downsampled luminance array at the second downsampled resolution. The diagonal binning and rebinning downsample the original image data by the second downsampling factor. In such embodiments, upsampling may include upscaling the downsampled Bayer array based on the second downsampling factor to generate an upscaled Bayer-RGB array, upscaling the downsampled luminance array based on the second downsampling factor to generate an upscaled luminance array, and upsampling the upscaled Bayer-RGB array based on the upscaled luminance array to generate a Bayer-RGB output array at the output array resolution.

[0059] In some embodiments, in response to detecting the luminance condition as a second low luminance condition, downsampling includes diagonally binning the original image data to generate a downsampled Bayer array at the second downsampled resolution and generating a downsampled luminance array at the second downsampled resolution. In such embodiments, upsampling may include upscaling the downsampled Bayer array based on the second downsampling factor to generate an upscaled Bayer-RGB array, upsampling the downsampled luminance array based on the second downsampling factor to generate an upscaled luminance array, and upsampling the upscaled Bayer-RGB array based on the upscaled luminance array to generate a Bayer-RGB output array at the output array resolution.

[0060] Although this patent document contains many details, none of these details should be construed as limitations on any invention or the scope of what is claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable partial combination. Furthermore, although the features described above work in certain combinations and originally claimed combinations, one or more features from a claimed combination may, in some cases, be separated from the combination, and a claimed combination may be directed to a partial combination or a variation of a partial combination.

[0061] Similarly, while operations may be described in a particular order in the drawings, this should not be understood as requiring that these operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed, in order to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring that they be separated in all embodiments.

[0062] This patent document only describes several implementations and examples, and other implementations, improvements, and variations may be made based on the descriptions and illustrations in this patent document.

[0063] Unless specifically indicated to the contrary, references to "a," "an," or "the" are intended to mean "one or more." Ranges may be expressed herein as from "about" one specified value and / or to "about" another specified value. The term "about," as used herein, means approximately, within the range of, roughly, or around. When the term "about" is used in conjunction with a numerical range, it modifies the range by extending the boundaries above and below the values ​​listed above. Generally, the term "about" is used herein to modify a numerical value above and below the stated value by 10% variance. When such a range is expressed, another embodiment includes from one specific value and / or to another specific value. Similarly, when a value is expressed as an approximation by using the antecedent "about," it is understood that the specified value forms another embodiment. It is also understood that the endpoints of each range are included in the range.

[0064] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. No document is admitted to be prior art.

Claims

1. A method for performing brightness adaptive processing on a hexadecimal red, green, blue, and white (RGBW) color filter array (CFA) in a complementary metal oxide semiconductor (CMOS) image sensor (CIS) system, the method comprising: Acquire raw image data using a photodetector array configured according to a hexadecimal RGBW CFA with native array resolution; detecting a brightness condition, the brightness condition being used to acquire the raw image data via the photodetector array, the brightness condition being detected as one of a set of predetermined brightness conditions, including at least a high brightness condition and a low brightness condition; as well as Generate a sensor output signal to represent a Bayer-RGB output array based on the raw image data and the brightness condition by the following steps: In response to detecting that the brightness condition is the high brightness condition, re-stitching the original image data to convert the hexadecimal RGBW CFA to the Bayer-RGB output array at an output array resolution; as well as In response to detecting that the brightness condition is the low brightness condition, downsampling the original image data into a downsampled Bayer array and a downsampled luminance array, and upsampling the downsampled Bayer array based on the downsampled luminance array to generate the Bayer-RGB output array at the output array resolution; In response to detecting that the brightness condition is the low brightness condition, the downsampling includes: diagonally binning the original image data to generate a quad-Bayer array at a first downsampled resolution, and generating the downsampled luminance array at the first downsampled resolution; and re-stitching the quad Bayer array into the downsampled Bayer array; and In response to detecting that the brightness condition is the low brightness condition, the upsampling includes: upscaling the downsampled Bayer array based on a first downsampling factor to generate an upscaled Bayer-RGB array, the first downsampled resolution being smaller than the original array resolution by the first downsampling factor; upscaling the downsampled luminance array based on the first downsampling factor to generate an upscaled luminance array; and The enlarged Bayer-RGB array is upsampled based on the enlarged luminance array to generate the Bayer-RGB output array at the output array resolution.

2. The method according to claim 1, wherein The output array resolution is equal to the original array resolution.

3. The method according to claim 1, wherein The low brightness condition is a first low brightness condition; The predetermined group of brightness conditions further includes a second low brightness condition; In response to detecting that the luminance condition is the second low luminance condition, the downsampling is performed according to a second downsampling factor such that each of the downsampled Bayer array and the downsampled luminance array has a second downsampled resolution, the second downsampled resolution being smaller than the original array resolution by the second downsampled factor, and the second downsampled factor being higher than the first downsampled factor.

4. The method according to claim 3, wherein: In response to detecting that the brightness condition is the second low brightness condition, the downsampling includes: diagonally binning the original image data to generate a quad-Bayer array at the first downsampled resolution and a pre-downsampled luminance array at the first downsampled resolution; and Recombining the quad Bayer array and the pre-downsampled luminance array to generate the downsampled Bayer array and the downsampled luminance array at the second downsampled resolution, wherein the diagonal merging and the re-merging downsample the original image data by the second downsampling factor; and In response to detecting that the brightness condition is the second low brightness condition, the upsampling includes: upscaling the downsampled Bayer array based on the second downsampling factor to generate an upscaled Bayer-RGB array; upscaling the downsampled luminance array based on the second downsampling factor to generate an upscaled luminance array; and The enlarged Bayer-RGB array is upsampled based on the enlarged luminance array to generate the Bayer-RGB output array at the output array resolution.

5. The method according to claim 3, wherein In response to detecting that the brightness condition is the second low brightness condition, the downsampling includes: diagonally binning the original image data to generate the downsampled Bayer array at the second downsampled resolution, and generating the downsampled brightness array at the second downsampled resolution; and In response to detecting that the brightness condition is the second low brightness condition, the upsampling includes: upscaling the downsampled Bayer array based on the second downsampling factor to generate an upscaled Bayer-RGB array; upscaling the downsampled luminance array based on the second downsampling factor to generate an upscaled luminance array; and The enlarged Bayer-RGB array is upsampled based on the enlarged luminance array to generate the Bayer-RGB output array at the output array resolution.

6. The method according to claim 3, wherein: The first downsampling factor reduces the horizontal resolution of the original image data by a factor of two and reduces the vertical resolution of the original image data by a factor of two; and The second downsampling factor reduces the horizontal resolution of the original image data by a factor of four and reduces the vertical resolution of the original image data by a factor of four.

7. The method according to claim 1, wherein The photodetector array collects the raw image data in an image collection frame; and The detecting the brightness condition is based on brightness data acquired before the image acquisition frame.

8. The method according to claim 7, wherein: The detecting the brightness condition includes: extracting, by the photodetector array, brightness pixel data from previous image data acquired in a previous image acquisition frame, and detecting the brightness condition based on the brightness pixel data.

9. The method according to claim 1, wherein The photodetector array includes a plurality of array regions; as well as The detecting the brightness condition includes detecting a respective region-level brightness condition for each of at least a portion of the plurality of array regions.

10. The method according to claim 9, wherein: Each of the array regions is one of a plurality of hexadecimal RGBW CFA blocks.

11. An image sensor system comprising: one or more processors coupled to a photodetector array configured to acquire raw image data at a native array resolution based on a hexadecimal red, green, blue, and white (RGBW) color filter array (CFA); as well as a non-volatile memory having instructions stored thereon, which, when executed, cause the one or more processors to perform the following steps, including: detecting a brightness condition associated with the photodetector acquiring the raw image data, the brightness condition being detected as one of a group of predetermined brightness conditions, including at least a high brightness condition and a low brightness condition; and Generate a sensor output signal to represent a Bayer-RGB output array based on the raw image data and the brightness condition by the following steps: In response to detecting that the brightness condition is the high brightness condition, re-stitching the raw image data to convert the hexadecimal RGBW CFA to the Bayer-RGB output array at an output array resolution; and In response to detecting that the brightness condition is the low brightness condition, guiding downsampling of the original image data into a downsampled Bayer array and a downsampled luminance array, and upsampling the downsampled Bayer array based on the downsampled luminance array to generate the Bayer-RGB output array at the output array resolution; In response to detecting that the brightness condition is the low brightness condition, the guiding downsampling includes: Directing the original image data to be diagonally binned to generate a quad-Bayer array at a first downsampled resolution, and generating the downsampled luminance array at the first downsampled resolution; and re-stitching the quad Bayer array into the downsampled Bayer array; and In response to detecting that the brightness condition is the low brightness condition, the upsampling includes: upscaling the downsampled Bayer array based on a first downsampling factor to generate an upscaled Bayer-RGB array, the first downsampled resolution being smaller than the original array resolution by the first downsampling factor; upscaling the downsampled luminance array based on the first downsampling factor to generate an upscaled luminance array; and The enlarged Bayer-RGB array is upsampled based on the enlarged luminance array to generate the Bayer-RGB output array at the output array resolution.

12. The image sensor system of claim 11 , further comprising: The photodetector array includes a plurality of photodetectors, each of the color filters being disposed over an associated one of the plurality of photodetectors to form the hexadecimal RGBW CFA; as well as a readout circuit coupled between the photodetector array and at least one of the one or more processors and configured to: in response to detecting that the brightness condition is the high brightness condition, reading out the raw image data from the photodetector array at the raw array resolution; as well as In response to detecting that the brightness condition is the low brightness condition, merged image data is read out from the photodetector array at a downsampled resolution.

13. The image sensor system of claim 11 , further comprising: a brightness sensor coupled to the one or more processors to sense ambient brightness of the photodetector array, Wherein, the detecting of the brightness condition is based on the ambient brightness.

14. The image sensor system of claim 11 , further comprising: The low brightness condition is a first low brightness condition; The predetermined group of brightness conditions further includes a second low brightness condition; In response to detecting that the luminance condition is the second low luminance condition, downsampling is directed by a second downsampling factor such that each of the downsampled Bayer array and the downsampled luminance array has a second downsampled resolution that is smaller than the original array resolution by the second downsampling factor, the second downsampling factor being higher than the first downsampling factor.

15. The image sensor system of claim 14, wherein: In response to detecting that the brightness condition is the second low brightness condition, the directing downsampling includes: performing guided diagonal binning on the original image data to generate a quad-Bayer array at the first downsampled resolution and a pre-downsampled luminance array at the first downsampled resolution; and Recombining the quad Bayer array and the pre-downsampled luminance array to generate the downsampled Bayer array and the downsampled luminance array at the second downsampled resolution, wherein the diagonal merging and the re-merging downsample the original image data by the second downsampling factor; and In response to detecting that the brightness condition is the second low brightness condition, the upsampling includes: upscaling the downsampled Bayer array based on the second downsampling factor to generate an upscaled Bayer-RGB array; upscaling the downsampled luminance array based on the second downsampling factor to generate an upscaled luminance array; and The enlarged Bayer-RGB array is upsampled based on the enlarged luminance array to generate the Bayer-RGB output array at the output array resolution.

16. The image sensor system of claim 14, wherein: In response to detecting that the brightness condition is the second low brightness condition, the directing downsampling includes: directing the original image data to be diagonally combined, generating the downsampled Bayer array at the second downsampled resolution, and generating the downsampled brightness array at the second downsampled resolution; and In response to detecting that the brightness condition is the second low brightness condition, the upsampling includes: upscaling the downsampled Bayer array based on the second downsampling factor to generate an upscaled Bayer-RGB array; upscaling the downsampled luminance array based on the second downsampling factor to generate an upscaled luminance array; and The enlarged Bayer-RGB array is upsampled based on the enlarged luminance array to generate the Bayer-RGB output array at the output array resolution.

17. The image sensor system of claim 11, wherein: The raw image data is collected by the photodetector array in an image collection frame; and The detecting the brightness condition is based on brightness data acquired before the image acquisition frame.

18. The image sensor system of claim 11, wherein: The photodetector array includes a plurality of array regions; as well as The detecting the brightness condition includes detecting a respective region-level brightness condition for each of at least a portion of the plurality of array regions.

Images