Multi-density pixel array for high resolution sensor

Abstract

A system comprises: an image sensor including a plurality of color filters arranged in a color filter array, the plurality of color filters being associated with a plurality of pixels formed by photoelectric conversion elements; and an image processing pipeline, including at least one processing unit, wherein the at least one processing unit is configured to generate two types of images based on electrical signals outputted by the plurality of pixels; wherein the color filter array includes a square array pattern including a first plurality of filters corresponding to a first color for capturing luminance signals and a second plurality of filters corresponding to two or more second colors different from the first color for capturing chrominance signals, the square array pattern being repeatedly arranged in horizontal and vertical directions of the color filter array, and wherein, in the square array pattern.

Description

MULTI-DENSITY PIXEL ARRAY FOR HIGH RESOLUTION SENSORCross References to Related Applications

[0001] This application claims the benefit of priority of United States Provisional Application No. 63 / 557,825, filed on February 26, 2024. The foregoing application is incorporated herein by reference in its entirety.BACKGROUND

[0002] As technology continues to advance, the goal of a fully autonomous vehicle that is capable of navigating on roadways is on the horizon. Advanced driver assistance (ADAS) and autonomous vehicles (AV) may use sensors, such as one or more cameras, to detect the road surface. Cameras may include complementary Metal-Oxide-Semiconductor (CMOS) image sensors (CIS), which may include pixels, analog and digital circuits, and a transmitter. These devices are used in a wide range of applications including automotive applications.

[0003] An optimized system for an ADAS or AV system may first detect objects or road surfaces without the benefit of color information (e.g., in low light or dark conditions). Consequently, the initial detection of objects in dark scenes may rely on the ability of an imaging system to produce a low-noise signal within such dark scenes. After the detection of objects within the images, the system may further analyze the images to identify color information. Because the colors of objects in the environment (e.g., objects on a road) typically require a broad color separation to avoid misclassification by drivers, the requirements of color accuracy for imaging systems are lower than that of human viewing.

[0004] In addition to their use in object detection, the images captured by ADAS and AV systems may also serve secondary purposes, such as video recording for insurance documentation or entertainment. To support these applications, the output images from the sensors may be usable by a classical image signal processor (ISP) pipeline that can output an RGB888 or YUV422 image that can be provided to a video compression device to store compressed videos or be output to a video display.

[0005] Traditional imaging systems, however, may face limitations in resolution and sensitivity under low-light conditions, which can hinder the performance of computer vision techniques used for object detection and classification. Addressing these challenges may involve modifying the structure of the pixel array in image sensors to enhance low-light performance or achieve higher resolution. While such modifications can improve detection accuracy, they may also create incompatibilities with standard image formats commonly used for video recording, compression, and display.

[0006] In this context, there is a growing demand for image sensors with increased sensitivity and smaller pixel sizes to increase resolution capabilities and to improve the accuracy of systems that analyze images to detect objects, while mitigating one or more of the shortcomings inherent in extant solutions. The present disclosure describes solutions to alleviate or overcome one or more of the above-stated problems, among others, with conventional systems.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

[0008] FIG. 1A is a diagrammatic representation of an exemplary system consistent with the disclosed embodiments.

[0009] FIG. IB is a flowchart showing an exemplary process for generating two types of images based on electrical signals outputted by the plurality of pixels consistent with the disclosed embodiments.

[0010] FIG. 2 is an illustration of an exemplary structure for a color fdter array consistent with the disclosed embodiments.

[0011] FIGS. 3A-3E are illustrations of exemplary square array patterns including color fdters consistent with the disclosed embodiments.

[0012] FIG. 4A is a flowchart showing exemplary processes 400 for generating a first image type, consistent with the disclosed embodiments.

[0013] FIG. 4B is a flowchart showing exemplary processes 450 for generating a second image type, consistent with the disclosed embodiments.

[0014] FIG. 5A is a diagrammatic representation of an image processing pipeline configured to generate a first image type consistent with the disclosed embodiments.

[0015] FIG. 5 A is a diagrammatic representation of an exemplary image processing pipeline configured to generate a second image type consistent with the disclosed embodiments.

[0016] FIG. 6 is a diagrammatic representation of an exemplary image processing pipeline applying High Dynamic Range (HDR) techniques and Piecewise Linear (PWL) companding techniques consistent with the disclosed embodiments.

[0017] FIGS. 7A and 7B are diagrammatic representations of image processing sequences for an image signal processor consistent with the disclosed embodiments.SUMMARY

[0018] Embodiments consistent with the present disclosure provide systems and methods for acquiring an image.

[0019] In an embodiment, a system for acquiring an image is disclosed. The system may comprise an image sensor including a plurality of color fdters arranged in a color filter array, the plurality of color filters being associated with a plurality of pixels formed by photoelectric conversion elements; and an image processing pipeline, including at least one processing unit, wherein the at least one processing unit is configured to generate two types of images based on electrical signals outputted by the plurality of pixels; wherein the color filter array includes a square array pattern including a first plurality of filters corresponding to a first color for capturing luminance signals and a second plurality of filters corresponding to two or more second colors different from the first color for capturing chrominance signals, the square array pattern being repeatedly arranged in horizontal and vertical directions of the color filter array, and wherein, in the square array pattern, a number of the first plurality of filters is greater than 1.25 times a number of the second plurality of filters; and wherein the two types of images include a first image type and a second image type, the first image type having a first resolution and being generated based on a first color schema pattern, and the second image type having a second resolution and being generated based on a second color schema pattern.

[0020] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.DETAILED DESCRIPTION

[0021] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components illustrated in the drawings, and the illustrative methods described herein may be modified by substituting, reordering, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the proper scope is defined by the appended claims.

[0022] The disclosed embodiments include a system for acquiring an image. Such a system may be designed to enhance computer vision performance. For example, the system may be designed to capture images at an increased resolution in order to provide more accurate computer vision results. For example, the disclosed embodiments may include an image sensor with smaller pixels. The disclosed image sensor may be, for example, an 18-MegaPixel (MP) image sensor. The choice of sensor resolution may involve balancing factors such as detection range, image processing capabilities, and system cost. Higher-resolution sensors provide more detailed imagery, beneficial for complex object recognition tasks, but they also demand greaterprocessing power and data storage. Conversely, lower-resolution sensors may suffice for specific applications, offering advantages in terms of reduced computational load and cost. In the context of ADAS and AV systems, traditional image sensors have a low (1.2 MP) to high resolution (8 MP - 4K).

[0023] Additionally, computer vision performance may be further enhanced by designing the system with more numerous higher-sensitivity pixels, enabling superior detection capabilities in low-light environments. For example, in some embodiments, the color pattern employed within an image sensor may be modified such that more light is provided to the higher-sensitivity pixels to improve low-light detection. However, altering the color pattern from traditional configurations may complicate subsequent applications such as image processing, storage, display, or human viewing. To address this challenge, the disclosed imaging system may include an image processing pipeline capable of converting the modified color pattern to a conventional one, such as those used for human vision or standard image signal processors (ISPs). This ensures compatibility with broader applications, including image storage and display, while maintaining enhanced performance for computer vision tasks.

[0024] FIG. 1A is a diagrammatic representation of an exemplary system 100 for acquiring an image, consistent with the disclosed embodiments. System 100 may include various components depending on the requirements of a particular implementation. In some embodiments, system 100 may include an image sensor 110. An image sensor refers to an electronic device that converts an optical image (i.e., an optical signal) into an electronic signal. An image sensor may capture light and convert it into electrical signals, which may then be further processed to form a digital image. The image sensor 110 may include a plurality of color filters arranged in a color filter array 116. The plurality of color filters may be associated with a plurality of pixels 118 formed by photoelectric conversion elements. A photoelectric conversion element refers to a component that converts light into electrical signals. Examples of photoelectric conversion elements used for imaging applications include but are not limited to photodiodes, Charge-Coupled Devices (CCD), or Complementary Metal-Oxide-Semiconductor (CMOS) Sensors. While these elements are effective at detecting light intensity, they may lack the ability to inherently distinguish between different wavelengths, making them incapable of independently capturing color information. To overcome this limitation, image sensor 110 may include a plurality of color filters arranged in a Color Filter Array (CFA) 116. A CFA is a mosaic of color filters organized in a specific pattern and placed over the pixel sensors of the image sensor. Color filters are transparent materials that selectively absorb certain wavelengths of light while allowing others to pass through. By filtering the light before it reaches the photoelectric conversion elements, color filters enable the capture of color information. The CFA allows thefiltered light to be processed into color data, which can subsequently be reconstructed into a fullcolor image. In some embodiments, the dimensions of the color filters arranged in the CFA 116 may be substantially the same as the one of the photoelectric conversion elements 118. This arrangement, combining photoelectric conversion elements with a CFA, may enable the image sensor to capture both light intensity and color information, ensuring accurate and detailed digital imaging.

[0025] In some embodiments, image sensor 110 may include one or more lenses 112. These lenses 112 may facilitate the collection, focusing, and directing of incoming light onto the photoelectric conversion elements 118 of the sensor. By precisely shaping and guiding the light, the lenses 112 may enable the formation of a well-defined image onto the surface of the photoelectric conversion elements 118. The photoelectric conversion elements 118, which are responsible for transforming light into electrical signals, may thereby receive a sharp and accurate representation of the scene being captured. For example, as shown in FIG. 1A, an image the scene 105 being captured may be formed at the surface of the photoelectric conversion elements 118. Additionally, the inclusion of multiple lenses may allow for advanced optical configurations, such as reducing aberrations, enhancing image resolution, or enabling functionalities like zoom or depth sensing. These features may collectively contribute to improved image quality and enhanced performance of the image sensor in various applications.

[0026] In some embodiments, image sensor 110 may include one or more filters 114 placed over the CFA 116. These “macroscopic” filters may be designed to modulate the overall spectrum of light reaching the CFA and, subsequently, the photoelectric conversion elements beneath it. By selectively filtering the light across the entire surface of the CFA, these filters may enable precise control over the wavelengths of light that are allowed to pass through. For example, in some embodiments, the image sensor 110 may include an Infrared (IF) filter, and an Ultra-Violet (UV) filter placed over the color filter array 116. By selectively filtering the light across the entire surface of the CFA, these filters may enable precise control over the wavelengths of light that are allowed to pass through. This filtering capability may serve several purposes. For instance, it may enhance the sensor’s ability to capture images in specific lighting conditions or environments by blocking unwanted wavelengths, such as ultraviolet (UV) or infrared (IR) light, that could otherwise distort color accuracy or reduce image quality. Additionally, these filters may be used to tailor the sensor’s spectral sensitivity for specialized applications. By ensuring that only the desired wavelengths reach the CFA 116 and the photoelectric conversion elements 118, the macroscopic filters 114 may contribute to improved image fidelity, better contrast, and more accurate color reproduction.

[0027] In some embodiments, the color filter array 116 may include a square array pattern. The square array pattern may be repeatedly arranged in horizontal and vertical directions of the color filter array 116. The square array pattern may serve as a fundamental unit, repeated periodically across the entire CFA 116. By being systematically arranged in both horizontal and vertical directions, a consistent and uniform structure may be created for the CFA 116. The square array pattern in the CFA 116 may cover a specific number of photoelectric conversion elements, effectively corresponding to a defined number of pixels arranged in a square matrix. This pattern may serve as a modular unit within the CFA, with each square grouping designed to manage and filter light for the underlying photoelectric conversion elements. By defining the spectral characteristics of light reaching each pixel, the square matrix may ensure consistent color representation and alignment across the image sensor. The periodic repetition of this square array pattern across the CFA may enable a scalable and systematic structure, supporting uniform light filtering and color differentiation for high-quality image capture.

[0028] FIG. 2 illustrates an exemplary structure for a CFA 200. CFA 200 includes a square array pattern (e.g., 210, 220, or 230), repeatedly arranged in both horizontal and vertical directions of CFA 200. As illustrated, the square array pattern may correspond to a different number of pixels. For example, in some embodiments, the square array pattern may correspond to 2 x 2 pixels (square array pattern 210), 3 x 3 pixels (square array pattern 220), or 6 x 6 pixels (square array pattern 230). It is to be appreciated that in FIG. 2, the square array patterns — 210, 220, and 230 — are depicted with the same overall size, indicating that as the number of pixels within the square array pattern increases, the size of each individual pixel or photoelectric conversion element 118 decreases. This results in an increase in resolution. For example, the resolution achieved by the 6 x 6 square array pattern 230 is three times greater than that of the 2 x 2 square array pattern 210, demonstrating how smaller pixel sizes can enable finer image detail. Various other pixel configurations are also conceivable for the square array pattern, such as a 4 x 4 pixel arrangement. This flexibility allows the square array pattern to be adapted to different application requirements, balancing factors like resolution, pixel size, and sensor performance. It is also to be appreciated that while CFA 200 is illustrated in FIG. 2 with sixteen square array patterns, the actual number of square array patterns in a CFA can vary and may be significantly higher than shown — for example, extending to several million patterns in practical implementations.

[0029] In some embodiments, the square array pattern may include a first plurality of filters corresponding to a first color for capturing luminance signals and a second plurality of filters corresponding to two or more second colors different from the first color for capturing chrominance signals. Luminance signals represent the intensity or brightness of light in animage. These signals may be used for detailing the structure, contrast, and fine details of a scene, as the human visual system is particularly sensitive to variations in brightness. In some embodiments, the first color may include one of Green (G) or Yellow (Y). These color filters may be used to capture luminance information due to the high sensitivity of the human eye to green and yellow wavelengths (via M and L cones), which dominate the perception of brightness. Chrominance signals on the other hand may convey the color information of an image. These signals may be derived from the differences in light intensity among various colors. In some embodiments, the two or more second colors may include two or more of Red (R), Magenta (M), Blue (B), Cyan (C), or Green (G). By combining the chrominance signals with the luminance signals, the full color spectrum of the image may be reconstructed.

[0030] A color filter corresponding to the first color or one of the two or more second colors may selectively absorb specific wavelengths of light while allowing wavelengths corresponding to its designated color to pass through. For instance, a color filter corresponding to yellow (Y) may absorb optical signals with wavelengths below 490 nm and allow those above 490 nm to pass through. Similarly, a color filter corresponding to red (R) may absorb optical signals with wavelengths below 570 nm and either allow all wavelengths above 570 nm or restrict the passband to a range between 570 nm and 670 nm. Likewise, a color filter corresponding to blue (B) would absorb optical signals with wavelengths above 500 nm and restrict the passband to a range between 400 nm and 500 nm. This selective filtering may ensure that only the desired spectral components reach the photoelectric conversion elements, enabling accurate color differentiation and faithful image reproduction. In some embodiments, the passband of one or more filters placed over the color filter array (CFA) may be selected in relation to the passband of the color filters within the CFA, or conversely, the passband of the color filters within the CFA may be tuned with respect to the passband of the one or more filters placed over it. For example, in some embodiments, an Infra-Red (IF) filter may have a wider passband (e.g., up to 700 nm) than that of a second filter of the color filter array corresponding to the color Red (R) (e.g., between 570 and 650 nm). This broader IF filter passband may enable pixels associated with other colors, such as yellow (Y) pixels, to capture a wider spectrum of light, including wavelengths up to 700 nm. By allowing more optical signals to reach these pixels, the image sensor may maintain enhanced sensitivity in low-light conditions, thereby improving the detection of objects and overall image quality in dimly lit environments. This may be particularly advantageous in reddish light conditions, such as those involving halogen light sources. On the other hand, the red color filter's passband may be aligned with the color sensitivity of the human eye, which does not detect red light beyond 650 nm. This alignment may enhance color detection accuracy relative to human perception.

[0031] FIG. 3A illustrates an exemplary square array pattern 300a corresponding to 2 x 2 pixels. As illustrated, in square array pattern 300a, two filters corresponding to the color Green (G) are positioned along the diagonal of the square matrix, while two filters corresponding to the colors Red (R) and Blue (B) are arranged along the anti -diagonal. This configuration, known as a Bayer filter, is a commonly used square array pattern in traditional image sensors. The Bayer filter is composed of 50% green filters, 25% red filters, and 25% blue filters, a configuration often referred to as RGGB pattern. The use of twice as many green filters as red or blue is designed to emulate the physiology of the human eye, which is more sensitive to green wavelengths and brightness, enhancing the sensor's ability to capture luminance details.

[0032] While Bayer filters are widely used and effective across a variety of applications, they can be modified and optimized to improve performance in specific computer vision tasks. Such modifications may include adjusting the distribution of colors, reconfiguring the filter patterns, or incorporating additional filter types to better align with the requirements of tasks like object detection. In certain embodiments, the square array pattern used in the color filter array (CFA) 116 of the image sensor 110 may deviate from the traditional Bayer RGGB pattern to better suit computer vision applications. For instance, in some embodiments, the square array pattern in CFA 116 may employ color filters with broader passbands, allowing a greater quantity of light to reach the photoelectric elements, thereby enhancing signal capture and improving overall sensor performance for specific tasks.

[0033] FIG. 3B illustrates another exemplary square array pattern 300b corresponding to 2 x 2 pixels. In this configuration, two colors from the pattern shown in FIG. 3A are changed. Specifically, the green filters are replaced with yellow filters (i.e., green + red), and the blue filters are replaced with cyan filters (i.e., green + blue). This arrangement is composed of 50% yellow filters, 25% red filters, and 25% cyan filters may be referred to as RYYCy pattern. The yellow and cyan filters may provide a broader passband, allowing the sensor to capture more light. This can be attributed to the spectral composition of most artificial light sources, which are typically most intense in the 500-600 nm wavelength range. As a result, the blue pixel may become cyan (green + blue), and the green pixel may become yellow (green + red), thus exposing both pixels to a wider range of light. In this setup, the colors of objects captured by the image sensor may be identified using three color components — red, yellow, and cyan. These can then be mapped to the standard red / green / blue (RGB) color representation used in traditional displays, with the following high-level correspondences: “green = yellow - red,” “blue = cyan - (yellow - red),” and “red = red.” However, while this RYYCy pattern may provide a broader spectrum of light capture, the traditional RGGB pattern may make it easier to distinguishdifferent colors of objects, as the traditional RGGB arrangement has a more defined separation of color components.

[0034] As previously mentioned, computer vision tasks can benefit from higher- resolution images, particularly for detecting smaller objects within a scene. Accordingly, in some embodiments, the square array pattern used in the CFA 116 may be designed to include a greater number of pixels within the same area as a traditional 2 x 2 pixel Bayer filter. For instance, as illustrated in FIG. 2, a square array pattern can achieve a higher pixel density within the same overall size by reducing the height and width of individual pixels. This enables the resolution of a 2 x 2 filter array to be increased by transforming the square array pattern into configurations such as 3 x 3 pixels, 4 x 4 pixels, or even 6 x 6 pixels.

[0035] FIG. 3C illustrates an exemplary square array pattern 300c corresponding to 6 x 6 pixels. In this arrangement, the size of the pixels shown in FIGS. 3A and 3B is reduced. Specifically, the pixel width and height may be scaled down by a factor of three. For example, if the original pixel measured 4.2 x 4.2 pm, the new pixel size may be reduced to 1.4 x 1.4 pm. As a result, nine 1.4 pm pixels can fit into the space previously occupied by a single 4.2 pm pixel, effectively tripling the resolution. The size of the pixels may correspond to the size of the color filters used in the CFA. Accordingly, in some embodiments, a size of each of the first plurality of filters and each of the second plurality of filters may be equal to 1.4 x 1 .4 pm.

[0036] While square array pattern 300c is shown using the RYYCy pattern detailed in FIG. 3B, it should be noted that other configurations, such as the RGGB Bayer pattern or alternative designs, may also be utilized within the square matrix. Specifically, when an image sensor with smaller pixels is employed, the color distribution in a square array pattern may be adjusted from the traditional 2 x 2 pattern to a larger pattern (e.g., a 6 x 6 pattern). In such configurations, more light may be allocated to the higher-sensitivity pixels — those dedicated to capturing the luminance signal — compared to the pixels used for capturing the chrominance signal, thereby enhancing low-light performance. Consequently, in some embodiments, in the square array pattern the number of filters corresponding to the first color, which is responsible for capturing the luminance signal, may exceed the number of filters corresponding to the two or more second colors used for capturing the chrominance signal. For example, some embodiments may involve in the square array pattern a number of the first plurality of filters being greater than 1.25 times a number of the second plurality of filters. In certain configurations, the imbalance between the filters for capturing luminance and chrominance signals may exceed 25%, allowing for enhanced capture of the luminance signal and enhanced sensitivity in low-light conditions. For example, in some embodiments, in the square array pattern, the number of the first plurality of filters may be equal to eight times the number of the second plurality of filters. This approachcontrasts with patterns like the Bayer RGGB or RYY Cy patterns, where the number of filters assigned to the first color (i.e., first plurality of filters) for luminance capture (e.g., two filters green (G) or Yellow (Y)) is equal to the number of filters corresponding to the two or more second colors (i.e., second plurality of filters) for chrominance capture (e.g., two filters, one for Red (R) and one for Cyan (Cy)).

[0037] In some embodiments, the square array pattern corresponds to 6 x 6 pixels, and the square array pattern may include two first arrays and two second arrays corresponding to 3 x 3 pixels. Each first array may include one interior filter corresponding to a first one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter. Each second array may include one interior filter corresponding to a second one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter. The two first arrays and the two second arrays may be alternately arranged in horizontal and vertical directions of the square array pattern. Additionally, in some embodiments, in the square array pattern, the number of the first plurality of filters may be equal to sixteen times a number of filters corresponding to the first or the second one of the two or more second colors. In other words, in this configuration, the percentage of luminance signal collected may represent about 89% of the total optical signal collected.

[0038] FIG. 3D illustrates another exemplary square array pattern 300d corresponding to 6 x 6 pixels, including the two first arrays and two second arrays above described. With respect to the square array patterns shown in FIGS. 3A-3C, the color distribution has been changed from a classical 2 x 2 pattern to a 6 x 6 pattern. In each of the two first or second arrays corresponding to 3 x 3 pixels, the interior filter may correspond to Red (R) or Cyan (Cy), while the other outer filter may correspond to Yellow (Y). Accordingly, in this configuration 300d, the number of filters corresponding to the color yellow (i.e., thirty-two) is equal to sixteen times the number of filters corresponding to either the color red or cyan (i.e., two of each). In other words, square array pattern 300d consists of approximately 89% yellow filters, 6% red filters, and 6% cyan filters. The significant increase in the proportion of yellow filters may enhance the image sensor's sensitivity to yellow light, which may be particularly beneficial for computer vision applications, particularly in the context of ADASs or AVs. Since the yellow spectrum encompasses much of the artificial lighting encountered by vehicles during driving (e.g., streetlights, headlights), this boost in sensitivity may improve the imager's performance in low- light or dark conditions, enhancing object detection in such environments. Additionally, the reduction in pixel size by one-third may increase the detection range, enabling the identification of objects at distances up to three times farther.

[0039] Alternatively, in some other embodiments, wherein the square array pattern corresponds to 6 x 6 pixels, the square array pattern may include a first array, a second array, and two third arrays corresponding to 3 x 3 pixels. The first array may include one interior filter corresponding to a first one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter. The second array may include one interior filter corresponding to a second one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter. Each third array may include one interior filter corresponding to a third one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter. The first array and the second array may be arranged along a diagonal of the square array pattern and the two third arrays may be arranged along an anti-diagonal of the square array pattern. Additionally, in some embodiments, the number of the first plurality of filters may be equal to sixteen times a number of filters corresponding to the third one of the two or more second colors and may be equal to thirty-two times a number of filters corresponding to the first or the second one of the two or more second colors. In this configuration, the percentage of luminance signal collected may also represent about 89% of the total optical signal collected.

[0040] FIG. 3E illustrates another exemplary square array pattern 300e corresponding to 6 x 6 pixels, including the first, second, and two third arrays above described. Compared to the square array pattern 300d shown in FIG. 3D, the color distribution in this pattern has been modified to incorporate a fourth color — green (G) — along with the previously used colors (Y ellow (Y), Red (R), and Cyan (Cy)). In configuration 300e, the number of yellow filters (i.e., thirty-two) is sixteen times the number of green filters (i.e., two) and thirty-two times the number of either red or cyan filters (i.e., one of each). In other words, square array pattern 300e consists of approximately 89% yellow filters, 6% green filters, 3% red filters, and 3% cyan filters. The addition of green pixels may be beneficial for computer vision applications in the context of ADAS or AVs, as it may enhance the detection of colors in the range between green and red. This may be particularly useful for detecting orange or amber traffic lights. By incorporating green, the system may gain two degrees of freedom for distinguishing colors in this range (e.g., red vs. green, red vs. cyan), whereas a red-yellow-cyan pattern would only provide one degree of freedom (e.g., cyan vs. red).

[0041] In some embodiments, system 100 may include an image processing pipeline 120 including at least one processing unit 125. The at least one processing unit 125 may be configured to generate two types of images based on electrical signals outputted by the plurality of pixels, as illustrated in process 150 shown in FIG. IB. An image processing pipeline refers toa collection of hardware components configured to execute a series of computational processes and algorithms that transform raw image data, such as electrical signals generated by an image sensor's pixels, into refined, viewable images or datasets tailored for specific applications. Pipeline 120 may be designed to perform tasks such as signal processing, noise reduction, color correction, and image enhancement, ensuring the captured images achieve the desired quality and accuracy. It is to be appreciated that the processing unit 125 within the image processing pipeline 120 may encompass a wide range of hardware components suited for imaging tasks, ranging from general-purpose processors to highly specialized hardware. For instance, the processing unit 125 may include one or more of the following: CPUs (Central Processing Units), GPUs (Graphics Processing Units), DSPs (Digital Signal Processors), ISPs (Image Signal Processors), ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or CMOS microcontrollers. Additionally, the processing unit 125 may integrate high-speed memory and storage to support efficient image processing workflows.

[0042] Consistent with the disclosed embodiments, processing unit 125 within the image processing pipeline 120 may be configured to generate two distinct types of images based on the electrical signals output by the plurality of pixels 118. For example, as shown in FIG. 1A, two different images 130 of captured scene 105 are generated by image processing pipeline 120. The two types of images may have different properties and may be tailored for different purposes. For example, one image type may be a full-color representation optimized for human viewing, generated by applying operations such as demosaicking, white balance adjustment, gamma correction, and tone mapping. The other type of image may be a raw or minimally processed dataset intended for machine learning models, computer vision algorithms, or other analytical purposes.

[0043] In some embodiments, the two types of images may include a first image type and a second image type. The first image type may have a first resolution and be generated based on a first color schema pattern. The second image type may have a second resolution and be generated based on a second color schema pattern. The first and second resolutions may be either identical or different. In some embodiments, the second resolution may be lower than the first resolution. For example, in some embodiments, the second resolution may be equal to one-third of the first resolution.

[0044] A color schema pattern refers to a mathematical framework or representation used to describe colors as a combination of signals or channels. These channels may encode the color information of an image in terms of specific components, such as Red, Green, Blue, Cyan, Magenta, and / or Yellow (CMY). A schema pattern may also encompass chroma components (e.g., U and V or Cb and Cr) to encode color differences (or color variations) from a lumacomponent (e.g. Y). In some embodiments, the first color schema pattern may include the first color and two or more second colors different from the first color. For example, the first color schema pattern may correspond to Yellow (Y) - Red (R) - Green (G) - Cyan (Cy) or to Yellow (Y) - Red (R) - Cyan (Cy). In some embodiments, the second color schema pattern may include the first color and two second colors different from the first color. For example, the second color schema pattern may correspond to Red (R) - Green (G) - Blue (B) or to Yellow (Y) - Magenta (M) - Cyan (Cy). In some embodiments, the first and second color schema patterns may be identical or different. For example, the first color schema patterns may include four colors while the second color schema may only include three.

[0045] FIG. 4A is a flowchart showing exemplary processes 400 for generating a first image type, consistent with the disclosed embodiments. In accordance with the disclosed embodiments, such a process may be executed by processing pipeline 120 via the implementation of processing unit 125. Further, process 400 is not necessarily limited to the steps shown in FIG. 4A, at least some of the steps shown in FIG. 4A may be optional, and any steps or processes of the various embodiments described throughout the present disclosure may also be included in process 400.

[0046] At step 402, image processing pipeline 120 may be configured to generate the first image type by converting, via an analog-to-digital converter, the electrical signals outputted by the plurality of pixels into a set of digital pixel signals having the first resolution and based on the first color schema pattern. In other words, the pixel array (i.e., an association of the CFA 116 and the photoelectric conversion element 118) may capture light and convert it into electrical signals, which may then be directly processed (e.g., conversion performed using an analog-to- digital converter (ADC)) and outputted as an image. Each pixel in the array may correspond to a specific point in the image, and its signal value may reflect the light intensity and color information captured at that point. By generating the image directly from the pixel array in this manner, the system maintains the original resolution and color information as captured by the sensor. Step 402 is schematically illustrated in FIG. 5 A, where a pixel array 500 is directly converted into an image 510 (set of digital pixel signals). In this example, the pixel array 500 consists of four square array patterns outlined by bold lines, similar to the square array pattern 300e depicted in FIG. 3E. Consequently, the first color schema pattern utilized in this example includes Yellow (Y), Red (R), Green (G), and Cyan (Cy). Although only four square array patterns are shown within the pixel array 500 for simplicity, it should be understood that the actual number of square array patterns in a practical implementation may vary and is typically much larger — potentially consisting of several million patterns in high-resolution applications.

[0047] At step 404, image processing pipeline 120 may be configured to crop the first image type. Cropping, in this context, refers to the process of selecting and extracting a specific portion of the first image type, effectively removing the unwanted regions. This operation may be performed within the image processing pipeline to focus on a particular area of interest in the image while discarding the rest. In practical terms, cropping might involve specifying a rectangular area or set of coordinates within the original image. The cropped image may retain the selected section, while the rest of the data is discarded. For example, if the first image type contains a full scene captured by the pixel array, cropping could focus on a region where an object of interest (e.g., a traffic light or pedestrian) may be located, ensuring that subsequent processing steps deal only with the relevant portion of the image.

[0048] FIG. 4B is a flowchart showing exemplary processes 450 for generating a second image type, consistent with the disclosed embodiments. In accordance with the disclosed embodiments, such a process may be executed by processing pipeline 120 via the implementation of processing unit 125. Further, process 450 is not necessarily limited to the steps shown in FIG. 4B, at least some of the steps shown in FIG. 4B may be optional, and any steps or processes of the various embodiments described throughout the present disclosure may also be included in process 450.

[0049] At step 452, image processing pipeline 120 may be configured to convert, via an analog-to-digital converter, the electrical signals outputted by the plurality of pixels into a first set of digital pixel signals and a second set of digital pixel signals. The first and second sets of digital pixel signals may be identical, with the first resolution, and based on the first color schema pattern. This process may correspond to the one described above concerning step 402 shown in FIG. 4A, where a pixel is directly converted into a set of digital pixel signals. However, in this situation, it results in two sets of digital pixel signals that could be identical.

[0050] At step 454, image processing pipeline 120 may be configured to replace in the first set of digital pixel signals, via interpolation, pixel signals corresponding to the two or more second colors with pixel signals corresponding to the first color, thereby generating an interpolated first set of digital pixel signals. Using the data from the first color, image processing pipeline 120 may calculate (interpolate) new values to replace the pixel signals for the two or more second colors. Interpolation refers to a mathematical technique where missing or replaced data is estimated using surrounding or related data points. The result thereby obtained is a new or "interpolated" set of digital pixel signals where the data for the two or more second colors is replaced by first color information. Step 454 is schematically illustrated in FIG. 5B, where pixel signals corresponding to the two or more second colors in a first set of digital pixel signals 510 are replaced by pixel signals corresponding to the first color leading to interpolated first set ofdigital pixel signals 502. In this example, the pixel array translated into the first set of digital pixel signals 510, consists of four square array patterns outlined by bold lines, similar to the square array pattern 300e depicted in FIG. 3E. Consequently, the first color schema pattern utilized in this example includes Yellow (Y), Red (R), Green (G), and Cyan (Cy). In this configuration, pixel signals corresponding to the color red, green, or cyan are replaced with signals corresponding to the color yellow, leading to an interpolated set of digital pixel signal 510 including only pixel signals corresponding to the color yellow. Although only four square array patterns are shown, it should be understood that the actual number of square array patterns in a practical implementation may vary and is typically much larger — potentially consisting of several million patterns in high-resolution applications.

[0051] At step 456, image processing pipeline 120 may be configured to downscale the interpolated first set of digital pixel signals from the first resolution to the second resolution, thereby generating a downscaled first set of digital pixel signals. Downscaling refers to the process of reducing the resolution of a set of digital signals (such as an image or video) by decreasing the number of pixels used to represent it. This may involve resampling the data to convert it from a higher resolution (e.g., the first resolution) to a lower resolution (e.g., the second resolution) while preserving as much of the original visual information as possible. Step 456 is schematically illustrated in FIG. 5B, where the interpolated first set of digital pixel signals 502 is downscaled from the first resolution to the second resolution, thereby generating a downscaled first set of digital pixel signals 504. Specifically, in this example, the second resolution is one-third of the first resolution, effectively simulating an increase in pixel size. For instance, this is equivalent to tripling the dimensions of the pixels, such as increasing from 1.4 pm pixels to 4.2 pm pixels.

[0052] At step 458, image processing pipeline 120 may be configured to demosaick the second set of digital pixel signals to estimate for each digital pixel signal color levels for all color components using a third color schema pattern, thereby generating a demosaicked second set of digital pixel signals. Demosaicking refers to a process used in digital image processing to reconstruct a full-color image from the incomplete color data captured by an image sensor using a CFA. Sensors typically capture only one color per pixel (red, green, or blue), and demosaicking may estimate the missing color values for each pixel by analyzing neighboring pixels and applying interpolation techniques. The second set of digital pixel signals, which is based on a color filter pattern (e.g., square array pattern 300e), only contains partial color information for each pixel. The demosaicking process estimates the missing color levels for each pixel by using a third color schema pattern and applying algorithms to interpolate the missing data. The result is a demosaicked second set of digital pixel signals, where every pixel has complete colorinformation (values for all color components of the third color schema pattern). In some embodiments, the third color schema pattern may be either identical or different from the second color schema pattern. For example, the second color schema pattern may correspond to Red (R) - Green (G) - Blue (B).

[0053] At step 460, image processing pipeline 120 may be configured to downscale the demosaicked second set of digital pixel signals from the first resolution to the second resolution, thereby generating a downscaled second set of digital pixel signals. Steps 458 and 460 are illustrated schematically in FIG. 5B. In this process, the second set of digital pixel signals 510 is first demosaicked using a third color schema pattern (in this case, Red (R), Green (G), and Blue (B)). FIG. 5C illustrates three exemplary demosaicking patterns 520-1, 520-2, and 520-3, that may be employed in the scenario illustrated in FIG. 5B to estimate the missing color values for each pixel by analyzing neighboring pixels. The demosaicked second set of digital pixel signals is then downscaled from the first resolution to the second resolution, resulting in a downscaled second set of digital pixel signals 506. Similar to step 454 described earlier, the demosaicking process in step 458 may be effectively equivalent to tripling the pixel dimensions, such as increasing the pixel size from 1.4 pm to 4.2 pm.

[0054] In some embodiments, image processing pipeline 120 may be further configured to convert the third color schema pattern used by the demosaicked second set of digital pixel signals to a fourth color schema pattern. For example, in the scenario illustrated in FIG. 5B the third color schema pattern (i.e., RBG) used by the demosaicked second set of digital pixel signals 506 is converted into a fourth color schema pattern (in this example YUV), thereby obtaining a new demosaicked second set of digital pixel signal 508 based on based on a new format (e.g., YUV444 format).

[0055] At step 462, image processing pipeline 120 may be configured to remove information on the first color from the downscaled second set of digital pixel signals. In this context, removing information on the first color from the downscaled second set of digital pixel signals may involve eliminating or discarding the data associated with the first color component from the signal, while retaining the remaining color information. Step 462 is schematically illustrated in FIG. 5B, where information on the first color (Y) from the downscaled second set of digital pixel signals 508, which uses the fourth color schema pattern (YUV), is discarded, while the remaining color information (UV) is retained. For example, the luminance (Y) information is removed from the image, and the chrominance information (UV) is normalized to the luminance (Y) pixel values, such that UVnorm = UV / Y. By this process, a second set of digital pixel signals without first color / luminance information 512 is generated. In some embodiments, an optional chromatic denoise may be applied to the second set of digital pixelsignals without first color / luminance information 512 (UV information). Chromatic denoise refers to a process used to reduce or eliminate noise specifically in the chroma (color) components of an image, without affecting the luminance (brightness) information. Noise refers to random variations in pixel values that can distort the visual quality of an image. Alternatively, in some other embodiments, a denoise filter may be applied to the downscaled second set of digital pixel signals 506 prior to the color schema pattern conversion and removal of the first color / luminance information. A denoise filter refers to a processing technique used to reduce or eliminate unwanted noise from an image or signal.

[0056] At step 464, image processing pipeline may be configured to combine the downscaled first set of pixel digital pixel signals and the downscaled second set of digital pixel signals into a single set of pixel digital signals. In other words, the downscaled first set of digital pixel signals, which contains only the first color / luminance (Y) information, may be merged with the downscaled second set of digital pixel signals, which contains only the second color / chrominance (UV) information. This combination may result in a complete set of pixel data with full color information, enabling the creation of a fully colored image. Step 464 is schematically illustrated in FIG. 5B, where the downscaled first set of digital pixel signals 504 (containing Y information) is combined with the downscaled second set of digital pixel signals 512 (containing UV information) leading to a single set of digital pixel 516. For example, the UVnorm (normalized chrominance data) may be rescaled to match the luminance values, such that the chrominance components (UV) are adjusted to reflect the appropriate color information (UV = Y*UVnorm).

[0057] At step 466, image processing pipeline 120 may be configured to mosaick the single set of digital pixel signals using the second color schema pattern. Additionally, in some embodiments, image processing pipeline 120 may be further configured to convert the fourth color schema pattern used in the single set of digital pixel signals back to the third color schema pattern before mosaicking. For example, as illustrated in FIG. 5B the fourth color schema pattern (YUV) used in the single set of digital pixel signals 514 is converted back to the third color schema pattern (RGB) leading to single set of digital pixel signal 516. Mosaicking refers to the opposite process of demosaicking and may involve separating or partitioning a full-color image into its individual color components, using a CFA pattern (such as a Bayer pattern). Each pixel from the full-color image would be split into different channels corresponding to specific colors and only a specific channel will be retained for each pixel. Step 466 is schematically illustrated in FIG. 5B, where the single set of digital pixel signals 516 is mosaicked (separated) using a Bayer pattern. As a result, the second image type, corresponding to the mosaicked single set of digital pixel signals 518, is generated. It is to be appreciated that in some alternative scenarios,the CFA patern used in the mosaicking process may be different from a Bayen patern. For example, a modified Bayer patern wherein cyan is used instead of blue as one of the second colors for the chrominance signal may be used.

[0058] The second image type, which represents a full-color image, may be beter optimized for human viewing compared to the first image type. In the first image type, the pixels corresponding to the two or more second colors (e.g., red, blue, and green or red, cyan, and green) may have lower sensitivity (e.g., I / 9th sensitivity) and can be noisier when viewed on their own. However, in the second image type, the combination of this data with the much more sensitive first color pixels (e.g., yellow pixels), along with the optional color denoise filter, can improve the signal-to-noise ratio, resulting in a clearer and higher-quality output image (e.g., red-green-blue or red-green-cyan image).

[0059] In some embodiments, image processing pipeline 120 may be further configured to apply High Dynamic Range (HDR) techniques and Piecewise Linear (PWL) companding techniques on the first and second image types. High Dynamic Range (HDR) refers to a technique used in imaging to enhance the contrast between the lightest and darkest areas of an image. HDR enables the capture and display of a wider range of brightness levels than traditional imaging techniques, making the image appear more natural and visually rich. Standard imaging may work with a limited dynamic range, which may cause details to be lost in extremely bright or dark areas. HDR solves this issue by combining multiple exposures of the same scene (each with different exposure levels) to capture both highlights and shadows in detail. The goal of HDR is thereby to create images that more closely resemble what the human eye perceives, which can handle a broader range of brightness. Piecewise Linear (PWL) companding is a method of compression and expansion used to modify the signal range. Companding stands for compressing and expanding the dynamic range of a signal to optimize how it is represented or transmited. In this method, the signal may be divided into segments, and each segment is compressed or expanded independently using linear functions. The overall result is a nonlinear transformation that maintains smooth transitions between different levels of the signal while focusing on compressing high dynamic range data or expanding it into a more manageable range. In imaging, PWL can be used to compress high dynamic range data so that it can be efficiently stored or transmited, while still retaining important detail. Then, when the data is displayed, the PWL expansion may help to restore the original contrast and dynamic range. For example, HDR images may use PWL techniques to compress their range for efficient storage or transmission and then expand them during display to optimize the image for viewing.

[0060] FIG. 6 illustrates an exemplary image processing pipeline that employs HDR and PWL companding techniques. This pipeline may be positioned after the previously describedpipelines, which are configured to output the first and second image types. As shown, the input to this pipeline can include 2x, 3x, or 4x individual images (either the first or second image type) from any of the pixel arrays described earlier (e.g., a pixel array with a CFA, such as the square array pattern 300e), using a 10-bit or 12-bit analog -to-digital converter (ADC). The outputs of the pipeline may be fed into one or two high dynamic range (HDR) combination circuits (step 610), which can combine the 2x-4x images into a single 24-bit HDR image. After the HDR combination, the image may undergo compression, reducing the pixel size to less than 12 bits per pixel using PWL companding techniques (step 620).

[0061] In some embodiments, image processing pipeline 120 may be further configured to transmit the first and second image types. For example, image processing pipeline 120 may include a transmitter such as a MIPI transmitter, i.e., a device or component that transmits data using the MIPI (Mobile Industry Processor Interface) standard. Additionally, in some embodiments system 100 may further comprise a receiver configured to receive at least one of the first and second image types and an image signal processor. An Image Signal Processor (ISP) refers to a specialized processor designed to handle and process digital image data from various sources, such as image sensors. An ISP may be responsible for converting raw image data into a final processed image that is suitable for display or further analysis. For example, as shown in FIG. 1A, system 100 includes a receiver 140 configured to receive at least one of the first and second image types 130, with an image signal processor 145 integrated within receiver 140 in this scenario.

[0062] In some embodiments, image signal processor 145 may be configured to when receiving the first image type, linearize the first image type, and interpolate the first image type to generate a monochrome image only including information for the first color and a color image only including information for the two or more second colors. Linearization refers to the process of converting the non-linear image data into a linear format, where the pixel values are directly proportional to the light intensity captured by the sensor. In many imaging systems, sensors produce non-linear responses due to factors like exposure, gamma correction, or other preprocessing applied by the camera. Linearizing the image may ensure that the data accurately reflects the original light levels. Additionally, most image processing algorithms, such as color correction, contrast adjustment, and other enhancements, work more effectively when the image data is linear. Linearization could be performed because PWL companding techniques were applied earlier in the image processing pipeline.

[0063] As described earlier, interpolation in imaging refers to the process of estimating missing or unknown pixel values based on known neighboring values. In this context, interpolation may be used to create two distinct images: a monochrome image containing onlythe information for the first color / luminance signal, and a color image that includes information for the two or more second colors / chrominance signals. This image processing sequence for image signal processor 145 is shown in FIG. 7A, where the received first image type is first linearized (step 710) and then interpolated (step 720) to generate a monochrome image containing only the yellow color information (luminance) and a color image containing the red, green, and cyan color information (chrominance).

[0064] In some embodiments, image signal processor 145 may be configured to when receiving the second image type, linearize the second image type. This image processing sequence for image signal processor 145 is shown in FIG. 7B, where the received second image type is linearized (step 710).

[0065] In some embodiments, image signal processor 145 may be further configured to detect objects in the monochrome image and detect colors via the color image or the second image type. As mentioned earlier, a monochrome image, which includes information about the luminance (brightness) of the scene, may be more suitable for detecting objects due to its higher sensitivity to intensity variations. Luminance information may provide a clearer representation of the shapes and edges of objects, making it easier for algorithms to identify and segment different objects in the image. Once the objects are detected based on the luminance data, image signal processor 145 may then refer to the color image (or second image type) to associate specific colors with these objects. By combining the object contours or regions from the monochrome image with the color information from the color image, the system may accurately determine the object’s color and enhance the overall analysis. For example, the monochrome image may be used to isolate and identify distinct regions corresponding to different objects, while the color image or second image type may then be used to provide precise color attributes for each detected object, facilitating more detailed object recognition and analysis. This dual approach leverages the strengths of both luminance and chrominance data for more efficient and accurate object detection.

[0066] Image processing pipeline 120 used in conjunction with image sensor 110 may allow a custom pixel array (e.g., a pixel array including a CFA comprising a 6 x 6 pixels square array pattern such as 300d or 300e) to output image sensor information in a classical 2x2 format with red, green, and blue / cyan colors. Image signal processor 145 can then receive this information and produce a final output image suitable for human viewing, using formats like YUV422 or RGB888. This configuration allows the processor to benefit from increased sensitivity while maintaining compatibility with ISPs commonly used for human-viewable images. Altering the arrangement of color-sensitive pixels, including the addition of yellow pixels, as described herein, can improve contrast (rather than color), thereby enhancing detectionrange. Furthermore, the disclosed embodiments allow for the generation of both human-viewable color images and machine-vision-specific images from the output of a single sensor. In some embodiments, narrowing the color filter range may result in images that are particularly well- suited for detecting color properties.

[0067] The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Additionally, although aspects of the disclosed embodiments are described as being stored in memory, one skilled in the art will appreciate that these aspects can also be stored on other types of computer-readable media, such as secondary storage devices, for example, hard disks or CD ROM, or other forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray, or other optical drive media.

[0068] Computer programs based on the written description and disclosed methods are within the skill of an experienced developer. The various programs or program modules can be created using any of the techniques known to one skilled in the art or can be designed in connection with existing software. For example, program sections or program modules can be designed in or by means of .Net Framework, .Net Compact Framework (and related languages, such as Visual Basic, C, etc.), Java, C++, Objective-C, HTMU, HTMU / AJAX combinations, XMU, or HTMU with included Java applets.

[0069] Moreover, while illustrative embodiments have been described herein, the scope of any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations, and / or alterations as would be appreciated by those skilled in the art based on the present disclosure. The limitations in the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application. The examples are to be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including by reordering steps and / or inserting or deleting steps. It is intended, therefore, that the specification and examples be considered as illustrative only, with a true scope and spirit being indicated by the following claims and their full scope of equivalents.

Claims

WHAT IS CLAIMED IS:

1. A system for acquiring an image, the system comprising: an image sensor including a plurality of color filters arranged in a color filter array, the plurality of color filters being associated with a plurality of pixels formed by photoelectric conversion elements; and an image processing pipeline, including at least one processing unit, wherein the at least one processing unit is configured to generate two types of images based on electrical signals outputted by the plurality of pixels; wherein the color filter array includes a square array pattern including a first plurality of filters corresponding to a first color for capturing luminance signals and a second plurality of filters corresponding to two or more second colors different from the first color for capturing chrominance signals, the square array pattern being repeatedly arranged in horizontal and vertical directions of the color filter array, and wherein, in the square array pattern, a number of the first plurality of filters is greater than 1.25 times a number of the second plurality of filters; and wherein the two types of images include a first image type and a second image type, the first image type having a first resolution and being generated based on a first color schema pattern, and the second image type having a second resolution and being generated based on a second color schema pattern.

2. The system of claim 1, wherein the second resolution is lower than the first resolution.

3. The system of claim 2, wherein the second resolution is equal to one-third of the first resolution.

4. The system of claim 1, wherein the first color schema pattern includes the first color and two or more second colors different from the first color.

5. The system of claim 1, wherein the second color schema pattern includes the first color and two second colors different from the first color.

6. The system of claim 1, wherein the first color includes one of Green (G) or Yellow (Y).

7. The system of claim 1, wherein the two or more second colors include two or more of Red (R), Magenta (M), Blue (B), Cyan (C), or Green (G).

8. The system of claim 1, wherein, in the square array pattern, the number of the first plurality of filters is equal to eight times the number of the second plurality of filters.

9. The system of claim 1, wherein the square array pattern corresponds to 6 x 6 pixels.

10. The system of claim 9, wherein the square array pattern includes: two first arrays corresponding to 3 x 3 pixels, wherein each first array includes one interior filter corresponding to a first one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter; two second arrays corresponding to 3 x 3 pixels, wherein each second array includes one interior filter corresponding to a second one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter; and wherein the two first arrays and the two second arrays are alternately arranged in horizontal and vertical directions of the square array pattern.

11. The system of claim 10, wherein, in the square array pattern, the number of the first plurality of filters is equal to sixteen times a number of filters corresponding to the first one or the second one of the two or more second colors.

12. The system of claim 9, wherein the square array pattern includes: a first array corresponding to 3 x 3 pixels, wherein the first array includes one interior filter corresponding to a first one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter; a second array corresponding to 3 x 3 pixels, wherein the second array includes one interior filter corresponding to a second one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter;two third arrays corresponding to 3 x 3 pixels, wherein each third array includes one interior filter corresponding to a third one of the two or more second colors at a center, and eight outer filters corresponding to the first color, arranged to surround the one interior filter; and wherein the first array and the second array are arranged along a diagonal of the square array pattern and the two third arrays are arranged along an anti-diagonal of the square array pattern.

13. The system of claim 12, wherein the number of the first plurality of filters is equal to sixteen times a number of filters corresponding to the third one of the two or more second colors, and is equal to thirty-two times a number of filters corresponding to the first one or the second one of the two or more second colors.

14. The system of claim 1, wherein a size of each of the first plurality of filters and each of the second plurality of filters is equal to 1.4 x 1.4 pm.

15. The system of claim 1, wherein the image processing pipeline is configured to generate the first image type by converting, via an analog-to digital converter, the electrical signals outputted by the plurality of pixels into a set of digital pixel signals having the first resolution and based on the first color schema pattern.

16. The system of claim 1, wherein the image processing pipeline is further configured to crop the first image type.

17. The system of claim 1, wherein the image processing pipeline is configured to generate the second first image type by: converting, via an analog-to digital converter, the electrical signals outputted by the plurality of pixels into a first set of digital pixel signals and a second set of digital pixel signals, wherein the first and second sets of digital pixels signals are identical, with the first resolution, and based on the first color schema pattern; replacing in the first set of digital pixel signals, via interpolation, pixel signals corresponding to the two or more second colors by pixel signals corresponding to the first color, thereby generating an interpolated first set of digital pixel signals;downscaling the interpolated first set of digital pixel signals from the first resolution to the second resolution, thereby generating a downscaled first set of digital pixel signals; demosaicking the second set of digital pixel signals to estimate for each digital pixel signal color levels for all color components using a third color schema pattern, thereby generating a demosaicked second set of digital pixel signals; downscaling the demosaicked second set of digital pixel signals from the first resolution to the second resolution, thereby generating a downscaled second set of digital pixel signals; removing information on the first color from the downscaled second set of digital pixel signals; combining the downscaled first set of pixel digital pixel signals and the downscaled second set of digital pixel signals into a single set of pixel digital signals; and mosaicking the single set of digital pixel signals using the second color schema pattern.

18. The system of claim 17, wherein the image processing pipeline is further configured to convert the third color schema pattern used by the demosaicked second set of digital pixel signals to a fourth color schema pattern and convert the fourth color schema pattern used in the single set of digital pixel signals back to the third color schema pattern before mosaicking.

19. The system of claim 1, wherein the image processing pipeline is further configured to apply High Dynamic Range (HDR) techniques and Piecewise Linear (PWL) companding techniques on the first and second image types.

20. The system of claim 1, wherein the image processing pipeline is further configured to transmit the first and second image types, and the system further comprises: a receiver configured to receive at least one of the first and second image type and an image signal processor.

21. The system of claim 20, wherein the image signal processor is configured to:when receiving the first image type, linearize the first image type, and interpolate the first image type to generate a monochrome image only including information for the first color and a color image only including information for the two or more second colors; and when receiving the second image type, linearize the second image type.

22. The system of claim 21, wherein the image signal processor is further configured to detect objects in the monochrome image, and detect colors via the color image or the second image type.

23. The system of claim 1, wherein the image sensor further includes an Infrared (IF) filter, and an Ultra-Violet (UV) filter placed over the color filter array.

24. The system of claim 23, wherein the IF filter has a wider passband than that of a second filter of the color filter array corresponding to the color Red (R).

Images