Color image sensor, method, and system

Abstract

The noise performance of the image sensor is improved by using inter-pixel comparisons, which can number in the hundreds and all contribute to information about a given pixel. This enables accurate determination of image chromaticity even at extremely low signal-to-noise ratios. In other embodiments, filters with unconventional spectral transmission functions are employed to reduce metamerism, enabling the discrimination of scene information invisible to the human eye. Yet another embodiment involves fabricating a sparse array of transparent pedestals on the image photosensor array. When a color resist is subsequently applied, these pedestals cause variations in the thickness of the resulting resist layer, leading to pixels with different spectral responses despite using the same color resist.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) In the United States, this application is a continuation application of International Application PCT / US2023 / 073352, filed on September 1, 2023, which claims priority to the following U.S. provisional applications, namely, "Color Filter Arrays and Methods", filed on January 5, 2023, Serial No. 63 / 478,527; "Color Filter Arrays Employing Filters with Non - Normative Transmission Functions", filed on January 6, 2023, Serial No. 63 / 478,728; "Hue Recovery from Noisy Color Photosensor Data", filed on January 12, 2023, Serial No. 63 / 479,572; "Color Filter Arrays Having Improved Modulation Transfer Functions", filed on January 25, 2023, Serial No. 63 / 481,390; "Color Image Sensors With N - Byte Pixel Signature Mechanism to Increase Manufacturing Yields", filed on March 2, 2023, Serial No. 63 / 487,941; "Color Image Sensors Employing Neighborhood - Variant Color Correction Kernels", filed on May 4, 2023, Serial No. 63 / 500,089; and "Color Into Infrared Image Sensing", filed on July 25, 2023, Serial No. 63 / 515,577. These applications are hereby incorporated by reference into this specification.

[0002] The subject matter of this application also relates to U.S. application 18 / 056,704, filed on 17 November 2022, and its priority applications, namely, application 63 / 280,898, filed on 18 November 2021; application 63 / 267,892, filed on 11 February 2022; application 63 / 269,194, filed on 11 March 2022; application 63 / 362,508, filed on 5 April 2022; application 63 / 365,482, filed on 27 May 2022; application 63 / 367,999, filed on 8 July 2022; and application 63 / 381,639, filed on 31 October 2022. These applications are also incorporated herein by reference. [Background technology]

[0003] Digital imaging systems have become widespread in recent years, including in consumer, medical, agricultural, automotive, and other machine vision applications. Many systems rely on Bayer pattern color filter arrays, which were first developed in the 1970s.

[0004] The emergence and widespread adoption of neural networks have expanded the capabilities of digital imaging systems and created new demands for this technology.

[0005] Despite the advancements made in digital imaging systems since the 1970s, many shortcomings persist. This technology plays a role in addressing various of these shortcomings.

[0006] One of the drawbacks concerns the noise of digital imaging systems in low-light environments. When the signal-to-noise ratio (SNR) drops to 10:1, 3:1, and even lower, current imaging systems cannot adequately cope, particularly when discerning scene chromaticity. This impairs the human enjoyment of the images and hinders the machine vision use of the data. [Overview of the project] [Means for solving the problem]

[0007] According to one aspect of this technology, a digital image sensor is provided that offers superior low-light performance.

[0008] One particular embodiment is an image sensor comprising a semiconductor substrate processed to define a plurality of pixels, the cell containing N pixels. The cell includes, in a spatial group, a first pixel at a first location, a second pixel at a second location, a third pixel at a third location, and so on, up to the Nth pixel at the Nth location. Each pixel has a distinct spectral response, and at least two pixels in the cell have different spectral responses.

[0009] The semiconductor substrate is further processed to define a hardware network configured to (a) compare scene values ​​associated with a first pair of pixels in a cell to obtain first pixel pair data, (b) compare scene values ​​associated with a second pair of pixels in the cell that are different from the first pair of pixels to obtain second pixel pair data, and (c) form query data based on the first and second pixel pair data. In some embodiments, such comparisons are performed between all pairings of pixels in a cell. In other embodiments, such comparisons are further extended to pairings between the target cell and surrounding cells. The resulting query data (which in some embodiments may be based on tens or hundreds of pixel pair values) is provided as input data to a color reconstruction module that determines color information about the pixels in the cell based on the query data.

[0010] The aforementioned and many other features of this technology will become more readily apparent from the following detailed description, which will proceed with reference to the accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1] Figures 1 and 1A illustrate two embodiments. [Figure 1A]Figures 1 and 1A illustrate two embodiments.

[0012] [Figure 2] Figures 2 and 3 detail prior art spectral transmission curves. [Figure 3] Figures 2 and 3 detail prior art spectral transmission curves.

[0013] [Figure 4] Figure 4 illustrates another embodiment.

[0014] [Figure 5] Figure 5 shows a pixel identification array.

[0015] [Figure 6A] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6B] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6C] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6D] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6E] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6F] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6G] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6H] Figures 6A - 6I detail spectral transmission curves for an exemplary filter. [Figure 6I] Figures 6A - 6I detail spectral transmission curves for an exemplary filter.

[0016] [Figure 7] Figure 7 details prior art spectral transmission curves.

[0017] [Figure 8] Figure 8 compares the curves in Figures 6A and 6B.

[0018] [Figure 9] Figure 9 identifies the characteristics of the filter spectral transmission curve.

[0019] [Figure 10A] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10B] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10C] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10D] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10E] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10F] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10G] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10H] Figure 10-10I details the spectral transmission curve for a more illustrative filter. [Figure 10I] Figure 10-10I details the spectral transmission curve for a more illustrative filter.

[0020] [Figure 11] Figures 11, 11A, 11B, and 11C illustrate pixel fields and their use in one embodiment. [Figure 11A] Figures 11, 11A, 11B, and 11C illustrate pixel fields and their use in one embodiment. [Figure 11B] Figures 11, 11A, 11B, and 11C illustrate pixel fields and their use in one embodiment. [Figure 11C] Figures 11, 11A, 11B, and 11C illustrate pixel fields and their use in one embodiment.

[0021] [Figure 12] Figure 12 illustrates the variation in spectral transmission function due to filter thickness.

[0022] [Figure 13] Figure 13 illustrates the spectral transmission functions for illustrative red, green, blue, cyan, magenta, and yellow filters with a thickness of 1 micron.

[0023] [Figure 14] Figure 14 illustrates how filters of different thicknesses can contribute to the diversity of spectral transmission functions, even when using the same color resist.

[0024] [Figure 15] Figure 15 shows the first derivative of the function in Figure 14.

[0025] [Figure 16] Figure 16 shows a sparse array of transparent pedestals fabricated (for example, from clear photoresist) on a photosensor array.

[0026] [Figure 17] Figure 17 shows the arrangement of Figure 16 after the application of the resist layer, resulting in resist layers of different thicknesses.

[0027] [Figure 18] Figure 18 depicts a green filter on a transparent pedestal.

[0028] [Figure 19A]Figures 19A-19E illustrate different arrays of sparse pedestals on a photosensor array. [Figure 19B] Figures 19A-19E illustrate different arrays of sparse pedestals on a photosensor array. [Figure 19C] Figures 19A-19E illustrate different arrays of sparse pedestals on a photosensor array. [Figure 19D] Figures 19A-19E illustrate different arrays of sparse pedestals on a photosensor array. [Figure 19E] Figures 19A-19E illustrate different arrays of sparse pedestals on a photosensor array.

[0029] [Figure 20] Figures 20 and 21 illustrate filter cells employing both relatively thick and relatively thin filters. [Figure 21] Figures 20 and 21 illustrate filter cells employing both relatively thick and relatively thin filters.

[0030] [Figure 22] Figure 22 shows the spectral transmission curve for the filter in Figure 21.

[0031] [Figure 23] Figure 23 illustrates an additional filter cell that employs both relatively thick and relatively thin filters.

[0032] [Figure 24] Figure 24 shows the spectral transmission curve for the filter in Figure 23.

[0033] [Figure 25] Figures 25-28 illustrate other filter cells that employ both relatively thick and relatively thin filters. [Figure 26] Figures 25-28 illustrate other filter cells that employ both relatively thick and relatively thin filters. [Figure 27] Figures 25-28 illustrate other filter cells that employ both relatively thick and relatively thin filters. [Figure 28] Figures 25-28 illustrate other filter cells that employ both relatively thick and relatively thin filters.

[0034] [Figure 29] Figure 29 shows spectral transmission curves for six filter cells employing both relatively thick and relatively thin filters.

[0035] [Figure 30] Figure 30 shows the spectral transmission curve for a prior art color image sensor and the spectral transmission curve for a monochrome version of the same sensor.

[0036] [Figure 31] Figures 31, 32, 32A, 33, and 34 detail exemplary arrangements that can be made to have a spatial relationship that varies across the image sensor, including the photosensor and the color filter. [Figure 32] Figures 31, 32, 32A, 33, and 34 detail exemplary arrangements that can be made to have a spatial relationship that varies across the image sensor, including the photosensor and the color filter. [Figure 32A] Figures 31, 32, 32A, 33, and 34 detail exemplary arrangements that can be made to have a spatial relationship that varies across the image sensor, including the photosensor and the color filter. [Figure 33] Figures 31, 32, 32A, 33, and 34 detail exemplary arrangements that can be made to have a spatial relationship that varies across the image sensor, including the photosensor and the color filter. [Figure 34] Figures 31, 32, 32A, 33, and 34 detail exemplary arrangements that can be made to have a spatial relationship that varies across the image sensor, including the photosensor and the color filter.

[0037] [Figure 35] Figure 35 shows a color filter array employing two different 2x3 filter cells, one consisting of relatively thick layers and the other of relatively thin layers.

[0038] [Figure 36] Figure 36 illustrates how the spectral transmission function of a red filter can deviate from nominal values ​​such as the average spectral transmission function of all red filters on the image sensor.

[0039] [Figure 37] Figure 37 details how the deviation in the nominal spectral transmission function for the red filter can vary between pixels of the image sensor.

[0040] [Figure 38] Figure 38 illustrates fundamental functions that can parameterize the spectral transmission function of a filter.

[0041] [Figure 39] Figure 39 illustrates how the color correction matrix can vary depending on the local position of the filter cell (or filter).

[0042] [Figure 40] Figure 40 shows how a reference pixel (A) is compared to two ornate pixels (B and C), resulting in two pixel-pair data.

[0043] [Figure 41] Figure 41 is a block diagram of an image sensor embodiment.

[0044] [Figure 42] Figure 42 shows how a reference pixel can be compared to any other pixel in the cell.

[0045] [Figure 43]Figure 43 illustrates that each pixel within a cell can serve as a reference pixel for comparison with one or more other pixels within the cell.

[0046] [Figure 44] Figures 44-46 illustrate that the comparison with the reference pixel can extend beyond the filter cell of the reference pixel. [Figure 45] Figures 44-46 illustrate that the comparison with the reference pixel can extend beyond the filter cell of the reference pixel. [Figure 46] Figures 44-46 illustrate that the comparison with the reference pixel can extend beyond the filter cell of the reference pixel.

[0047] [Figure 47] Figure 47 shows nine reframes of a 3x3 pixel cell, each with a different pixel at its center.

[0048] [Figure 48] Figure 48 illustrates a side view of one embodiment in which a 2x2 Bayer filter cell is reframed as a 3x3 cell.

[0049] [Figure 49A] Figures 49A and 49B compare the performance of one embodiment with that of the prior art. [Figure 49B] Figures 49A and 49B compare the performance of one embodiment with that of the prior art.

[0050] [Figure 50] Figure 50 identifies the filter location within a 2x2 cell. [Modes for carrying out the invention]

[0051] Detailed explanation This specification begins by detailing a first embodiment that introduces certain aspects of the technology, which will then be further discussed in later embodiments. Such aspects include, for example, improvements in manufacturing, calibration, and image processing procedures to reduce manufacturing tolerance constraints on image sensor color filter arrays while maintaining or improving colorimetric, contrast, and / or machine vision performance.

[0052] This first embodiment relates to a sensor including a color filter array (CFA) organized as 3x3 pixel cells (tiles), but other embodiments may, of course, employ CFAs of other configurations. Pigment-based CFAs are used in this embodiment, but other filtering materials (e.g., dyes, quantum dots, etc.) can also be used. Fujifilm and Toppan are well-known suppliers of suitable pigments. All such materials, regardless of their technical type, are referred to as “pigments,” “inks,” “resists,” or “color resists.”

[0053] Referring to Figure 1, the 3x3 CFA of the first embodiment consists of four different commercially available color resist products laid out in the depicted pattern (e.g., Fujifilm Color Mosaics). (登録商標) A trademarked product is employed. Six separate photolithography masks are used, each requiring a sequence of process steps (e.g., cleaning, resist coating / firing, exposure in a lithography tool, development, hard firing, with associated measurement and inspection operations, as is well known to those skilled in the art). For the purposes of the specific embodiment, such a color filter array can be placed on a 1,200 × 1,600 or 3,000 × 4,000 pixel image sensor, each pixel outputting a 16-bit brightness value. The pixels may have sides of less than 10 microns, less than 2 microns, or less than 1 micron.

[0054] Figure 1A illustrates an alternative embodiment.

[0055] The first layer to be processed is the diagonal of the three magenta pixels, S7, S5, and S3, in Figure 1. The specification for this first layer is to aim for a magenta (M) thickness of 0.5 microns. After development and hard firing, the measurement and inspection specifications will allow for relaxed (larger) tolerances compared to modern commercial-grade standards for CFA-based CMOS image sensors. Such relaxed tolerances include, for example, higher cross-mixing rates of physical materials beyond the nominally specified color resist material for any given pixel. That is, for example, in a pixel that would otherwise be "red," instead of allowing only 2 percent residual "magenta" resist, the manufacturer may instead allow up to 10 percent. These figures are used here only to illustrate aspects of what is intended by the relaxed tolerances. Such relaxed tolerances allow for lower-cost and more environmentally friendly chemical choices than those made possible within existing tolerance standards.

[0056] The second layer defines the photolithography mask, which corresponds to the two other corners of the 3x3 pixel cells, namely S1 and S9. These two pixel cells will use the same CFA magenta pigment M from, for example, the same vial and chemical delivery system, but this second layer will be defined to be 1 micron thick, in contrast to the 0.5 microns of layer 1. The photolithography mask of the second layer will be manufactured such that very slight physical contact will be allowed, for example, at the corners of the two cells of the second layer when they contact the three cells of the first layer. For example, if the pixels are 2 microns square, there may be only 100 nm of "overlap" on the sensor, with the layer 2 pixel material covering the layer 1 material. After the second layer is completed, during the measurement and inspection steps, these contacts between the layer 1 cells and the layer 2 cells can be quantified, for example, as effective nanometer overlap. Similar to layer 1, the tolerances of layer 2 will also be relaxed compared to modern standards. This relaxation is for the same reasons stated for Layer 1. For example, current standards may assume tolerances for a 15% standard deviation in color resist thickness and only a 3% cross-material residue rate. Embodiments of this technology increase one or both of these figures to 50%, 100%, 200%, or more.

[0057] The third layer will define a general green (G) as used in Bayer filter CFA sensors and will target only a single cell in the 3x3 cell array, i.e., S2. There are two slight differences from current practices for laying the green: A) the defined thickness of this G layer will be approximately 10-40 percent thinner than that typical in Bayer RGB sensor green layers, and B) measurement and inspection tolerances will be more relaxed than in modern practices, as with layers 1 and 2. For this specific design, 0.7 microns will be used as the defined thickness for this third layer. The main point is to select a thickness for the green that is thinner than those selected in modern Bayer CFA designs.

[0058] The fourth layer employs commercially available cyan (C) and fills the left-center divot, i.e., pixel location S4, of the 3x3 cell structure. The thickness specification for this fourth layer of cyan is 1 micron. Relaxed tolerances are employed, as with the previous three layers. The fifth layer uses the same cyan and this time fills the cell divot, i.e., S6, of the right-center pixel in the 3x3 cell pattern. This will be adjacent to the cyan pixel deposited in the fourth layer of the adjacent 3x3 cell to the right. The thickness of this fifth layer will be 0.5 microns, rather than 1 micron for the fourth layer. In some embodiments, the physical mask used for layer 4 is rotated and used as the mask for layer 5.

[0059] Finally, the sixth layer is the yellow (Y) mask and color resist layer at pixel location S8. The thickness specification for this sixth layer is 1 micron, with relaxed tolerances as described above.

[0060] Figures 2 and 3 show the spectral curves associated with these color resists. These curves are based on information published by Fujifilm that describes the properties of their Color Mosaic pigments. Figure 2 shows the transmittance of cyan, magenta, and yellow pigments at different layer thicknesses, with the solid line representing 0.9 microns (nominal), the dotted line representing 0.7 microns, and the dashed line representing 1.1 microns. Note that, in relation to different thicknesses, the curves do not simply shift up or down with the same profile. Rather, their shapes change. For example, the slope of the curves and the shape of their stop bands change, as does the width of the notches. Figure 3 shows the transmittance of red, green, and blue pigment filters at nominal layer thicknesses.

[0061] The final result after a sensor (or a large wafer containing many sensors) has undergone these process steps can be described as a very slightly wrinkled mosaic-like assembly of color resist cells, where each cell is very slightly different from all of its neighbors, and indeed different from all 3x3 cells in the entire sensor. All cells share a stably and roughly defined pattern at the 3x3 pattern level, but all differ by measurable quantities in surface measurements at the nanometer level. Furthermore, statistically non-negligible amounts (trace amounts) of material from each layer will be present in isolated pixels of assigned cells in one or more other layers (and in some cases, all assigned cells in the other layers). Considering the three diagonal cells of layer 1 as one example, electron microscopy can reveal that if the atomic volume of the intended material of layer 1 inside its intended cell is normalized to a value of 1.0, then the volumes of the materials of layers 2, 3, 4, 5, and 6 will also occupy that cell, possibly in the range of 0.01 to 0.05, and sometimes in the range of 0.005 to 0.1 or greater. In plain terms, the material of all layers will be found within every pixel, at some measurable level. Typical “layered distributions” of these trace amounts can also be expected, and such layered distributions can be expected to mimic the layering order.

[0062] The curves in Figures 2 and 3 represent the visible spectral axis range of 400–700 nanometers. Extensions to near-infrared (NIR) and near-ultraviolet (NUV) are required in all designs and applications where more than just "good color images for human viewing" is desired. As taught in previous disclosures, a balance is needed to optimize color image quality (or vice versa) while maintaining the specified quality of multi-channel information useful for machine vision applications.

[0063] In the embodiments detailed above, the six layers of filtering can have at least some transmittance in the NUV and NIR bands. This enables estimation of the NUV channel optical signal and the NIR channel optical signal. This differs from, for example, enabling estimation of two separate NIR optical signals, such as 700nm to 750nm and 750nm, along with 750nm to 800nm, which can be done in other embodiments. That is, here, the far-red to NIR band of approximately 690nm to 780nm is treated as one single channel, and the dark blue to NUV band of approximately 360nm to 410nm is treated as another single channel. The six-layer filtering described above enables diverse filtering behavior with respect to these two new bands, which are conveniently referred to as NIR and NUV.

[0064] The fundamental quantum efficiency (QE) of a silicon detector attenuates to lower levels as blue light transitions to NUV and as far-red light transitions to NIR. Therefore, in both cases, the fundamental behavior of the sensor causes the photoelectron signal level to shift downward. On the NUV side of the spectrum, there is additional attenuation introduced by both the typical lens, which is often used, and by any glass that is a cover for the sensor itself. Therefore, with respect to NUV, this normally occurring attenuation can be used to provide a full-pixel NUV cutoff frequency in the wavelength range of approximately 350 nm to 360 nm. Similarly, with respect to NIR, the quantum efficiency of silicon decreases with increasing wavelength, and a full-pixel NIR cutoff can be produced through either pigment interpolation or an explicit glass surface or other average values. The first embodiment employs a full-pixel NIR cutoff of any of approximately 750 nm to approximately 800 nm.

[0065] Once the full pixel cutoffs for both NIR and NUV are established as described above, it is generally desirable that the spectral transmission properties of the six layers, with respect to the NUV and NIR bands, differ slightly among the six layers. Commercially available color resists of the four M, C, Y, and R colors already achieve this in essence. Only if a practitioner cannot find a commercially available color resist that implements this diversity of transmittance in the NIR and NUV bands remains the option of adding a small amount of pigment to an existing color resist that will be transparent primarily across the approximately 400 nm to approximately 700 nm visible range, but will add some opacity in one or both of the NUV and / or NIR bands.

[0066] Another embodiment is shown in Figure 4. This 3x3 color filter array includes three conventional red, green, and blue filters, plus two each of cyan, yellow, and magenta. Each of these latter filters is fabricated in two different thicknesses, namely thin and thick layers (indicated as "N" for thin layers and "T" for thick layers in the figure), resulting in two different spectral transmission filter curves for each of these three colors. Thin layers can be less than 1.0 micron, such as 0.9, 0.8, or 0.7 microns, while thick layers can be greater than 0.8 microns, such as 0.9, 1, 1.2, or 1.5 microns (appropriately paired with the thin layer filter of that color to maintain the thin / thick layer relationship).

[0067] The embodiments detailed above are, of course, merely illustrative. One fundamental concept is that a color filter array may include elements formed from the same color resist but with different thicknesses to achieve diversity in filtering action. In one described embodiment, one magenta filter layer is 0.5 microns thick, while another magenta filter layer is 1 micron thick. In one embodiment, there are three different magenta filter layers with thicknesses of 0.4, 0.6, and 0.8 microns. Such thickness ratios are merely illustrative. In a different embodiment, one layer may be only 10%, 20%, or 50% thicker than another layer of the same color. Or one layer may be 100%, 200%, or more than that thicker than another layer of the same color. One embodiment employs different thicknesses for only one color, while other embodiments employ different thicknesses for multiple colors. As shown, some embodiments deposit a single photoresist with two or more different thicknesses.

[0068] Later sections detail methods in which pedestals may be used to vary the filter thickness. Embodiments detailed in that discussion can also be implemented without pedestals by directly fabricating filter layers of different thicknesses.

[0069] (Tolerance compensation, calibration, and algorithm implementation: pixel-by-pixel correction) Using the first embodiment described above, in which four distinct color resists (C, M, Y, G) are defined, each photosite will contain a certain finite measurable amount of each of the four pigments, i.e., its assigned color and trace amounts of the other three. Six different nominal surface thicknesses are defined for these four color resists. Each photosite has a nominal surface thickness value ranging from a few tenths of a micron to more than one micron, which is called the nominal thickness of the color resist layer.

[0070] Due to the relaxation of conventional manufacturing tolerances, "mixing" of the other three color resist pigments is expected in all of the photosites of a particular sensor. Such relaxation accelerates sensor processing and reduces costs.

[0071] The normalized pigment concentration level of the pigment assigned to the photosite is defined as the global average value of the pigment in the sensor after the sensor has been manufactured and packaged. This global sensor average value is normalized or set to 1.0 (i.e., not in microns).

[0072] For each photosite, the detailed measurements will exhibit a variation of 1.0 from the global mean of this sensor, and this variation will be partly a function of the degree of tolerance relaxation in the manufacturing of the sensor. This variation can range from a very low standard deviation of 0.03 or less, centered around a mean of 1.0, in situations where higher (stricter) tolerances are practiced in manufacturing, to a significantly higher standard deviation of 0.1 or greater, in situations where the experiment "pushes to the limits" of what is acceptable, but can still be designed to function properly. We will refer to this standard deviation of the assigned pigment as the "normalized slop".

[0073] In the first embodiment, for each photosite, there are three (contamination) pigments different from the pigment assigned to the photosite. Each of these three pigments will have a global average value for a certain sensor, measured across the entire manufactured and packaged sensor, even within the cell to which the pigment is not assigned. This global average value for each of the three different pigments may be called a “mixed average value.” All three pigments have a unique mixed average value, ranging from values ​​in a normalized unit of a few hundredths (e.g., 0.015, 0.03, or 0.05) for higher tolerance manufacturing practices to even higher values ​​such as 0.06, 0.1, 0.15, or higher normalized units for experiments attempted up to a limit.

[0074] Similarly, with respect to the standard deviation of nominally assigned pigments, the mixed mean values ​​of these unassigned pigments will have their own standard deviations, which are referred to as “mixture slops” (experimental practice is expected to show that for many sensor designs, the mixed mean values ​​and mixture slop values ​​will correlate via a simple square root relationship, but this disclosure keeps these figures independent).

[0075] Across the 3x3 CFA, we recognize that there are four pigments and six defined thicknesses for those pigments (two thicknesses for two of the pigments, and one thickness for every other two of the pigments, giving a total of six types). Then, for the six photosite types, there are six normalized 1.0 levels, each with its own normalized slop, and thus three more mixed mean values ​​for each of the six photosite types, each of which has an associated mixed slop number. Thus, with respect to the baseline design, we have the following:

[0076] Six normalized slop values, 18 normalized mixed mean values, and 18 normalized mixed slop values. This results in a total of 42 calibration parameters applicable to any given manufactured sensor. This set of values ​​can be measured and then stored in memory on the sensor chip at the pixel-level or cell-level. One illustrative approach is to measure global mean and values, fit the histogram to the individual pixel behavior, and then digitize the histogram values ​​so that the individual behavior of a given pixel is divided into one of these coded histogram bins.

[0077] During the normal operation of the sensor, which generates "digital numbers" to represent the detected photoelectron counts during a given exposure for all photosites, the task is to properly interpret these digital numbers in the context of their calibration numbers. This context includes many elements, including: 1) A unique characteristic of the global average value of a photosite. 2) Related to this, estimation of the mixing ratio of the four pigments for each photosite. 3) The absolute radiance level as a function of the light spectrum, which gives a given global average value and therefore relates the normalized 1.0 value to the sensor irradiance level. 4) The relationship between the characteristics of a photosite and its immediate neighborhood, consisting of eight surrounding photosites, i.e., essentially the 3x3 CFA cells surrounding the photosite itself. [An alternative definition of the CFA pattern also exists, in which each photosite can claim ownership of a "central pixel," and as a result, the surrounding neighborhoods determined for each photosite settle in their corresponding locations; note that this is a useful algorithmic definition and differs from the definition used in the introductory CFA of baseline design, and may be called a photosite-centered CFA.] 5) Detailed relationship between a central CFA of a photosite and eight 3x3 photosite central CFAs surrounding that central CFA.

[0078] This list of five, while not comprehensive, is sufficient for further explanation of pixel-per-watt correction (PWC).

[0079] One embodiment of the present technology is a color imaging sensor having a plurality of pixels, each pixel being associated with a plurality of byte memory, the memory storing data relating one or more parameters of a pixel to similar parameters across the sensor.

[0080] Each photosite has its own unique signature for these 42 calibration parameters, which presents challenges in measuring and using these parameters.

[0081] The first problem is addressed by what is called Chromabath and FuGaI illumination. The second problem is addressed by per-pixel correction.

[0082] (Chromabath and FuGaI) This disclosure is built upon the Chromabath technology already taught in applications 63 / 267,892 and 18 / 056,704, and replaces a multi-LED (e.g., 10 or 12 LED) lighting system, referred to as a monochromator and a Full Gamut Illuminator (FuGaI), used therein. However, the monochromator array retains its role in being used to train and calibrate the use of FuGaI in scaled production lines.

[0083] Further parameters, such as the dark median, can be added to the parameters above. Thus, each photosite can be characterized by parameters including 1) its dark median in digital number, 2) its nominally uniform white light gain in digital number (which is therefore related to irradiance (the level of light hitting the photosite)), and 3)-5) the CYMG mixing ratio (the sum of the ratios is 1.0, and only three parameters are required to fully define their ratios) (some of which depend on the type of photosite layer).

[0084] We begin with the spectral curves of the C, Y, M, and G photosites. These curves interact with the quantum efficiency curves of the fundamental silicon sensor photosites. These spectral curves also reflect the Beers-Lambert behavior of the photosites, so that knowledge of thickness and mixing ratio refines the spectral estimations. Next, we characterize the five described spectroscopic measurement parameters for each photosite.

[0085] The measurement of the dark median originates from the "dark frame" characterization in astronomy. Light is not allowed to strike the sensor, and many frames of data are captured. The long-term global average of these frames is stored and sometimes associated with metadata indicating the sensor's temperature. Such data is then used to correct subsequent measurements, for example, by subtracting the dark frame data on a pixel-by-pixel basis. Many existing CMOS image sensors have some form of dark level adjustment and / or correction.

[0086] In some embodiments, the applicant uses the median of a series of dark frames for correction, rather than the average value. This is thought to aid in certain calculations that employ neighbor photosite comparison.

[0087] The isowhite light gain values ​​for the sensor's photosite are typically measured after applying a correction for each dark median of the photosite. A "flat-field" imaging procedure can be used to measure these gain values.

[0088] Measuring the CMYG mixing ratio is more complex. Various methods can be employed. The illustrative method detailed below is suitable for low-cost mass production and is designed for application at the annual unit volume level of millions of sensors.

[0089] Four "color calibration sensors" are fabricated to employ only one of the C, M, Y, and G color resists. These sensors undergo all the steps required to fabricate the final CFA-based CMOS imaging sensor, except that only one color resist coating step is applied during the CFA color resist process stage of manufacturing. However, the thickness of the color resist is anticipated and varied in different areas of the sensor, from a thin thickness of about a few tens of microns (nominal) to sometimes a thickness of 1 or 2 microns. Spatial patterns such as sinusoids and squares, or photosite level masks, can also be applied. The resulting color calibration sensors will be used to characterize the global spectroscopic properties of the sensor, showing how all specific choices of C, M, Y, and G interact with the silicon-driven quantum efficiency (QE) sensitivity spectral profile with respect to the photosite size (pitch) of this particular class.

[0090] Why are there four calibration sensors, one for each of the four C, M, Y, and G pigments? This is because we do not want any cross-mixing of any pigments. A complete Beers-Lambert model of a single isolated color resist can be measured in situ within the same photosite structure of the final design / construction. If cost constraints are present, the color resist can be modeled according to the supplier's spectral curve data, which may yield data with an accuracy of around 90%, but it is preferable to apply the color resist to the exact photosite structure in which it is used, thin those resist layers across a wide range of thickness values, and then measure the output of the digital numbers.

[0091] Next, each of these color calibration sensors is passed through (preferably) a monochromator-based Chromabath. As noted, “Chromabath” is a neologism from a prior provisional application. Chromabath is a procedure in which, once the designer has selected the full effective spectral range for the image sensor array, such as 350 nm to 800 nm, monochromatic light is moved from one end of the spectral range to the other, illuminating all sensors. In the case of a monochromator-based Chromabath, four color calibration sensors can be used, with a lambda step size of 1 nanometer, assuming 451 wavelength steps / acquired image. In good practice and consistently, it is assumed that the light field of illumination is uniform (e.g., within the low single-digit percentage range, optimally below 1%). The same applies to absolute irradiance values ​​at all wavelengths from 350 to 800. Multiple measurement sessions can be conducted over several hours to collect data.

[0092] The results of such data collection would be a Beer-Lambert law-like spectral curve covering a range of selected wavelengths. Figure 2 illustrates this, revealing the variation in spectral response associated with different photosites of different thicknesses on the color calibration sensor. These curves are, of course, isolated for C, Y, M, and G respectively, and will typically reveal the properties of the photosites, including photosite variation silicon response quantum efficiency effects. Again, a lower-cost approach would use modeling instead of measurement, but since measurement is a one-time pre-production laboratory step, the effort is well amortized, even for low-volume production.

[0093] With these pseudo-Beer-Lambert spectral curves for the four color calibration sensors obtained, FuGaI training is then performed using N different production samples of the CFA CMOS image sensor.

[0094] These N sample production sensors (where N may be, for example, 5) will serve as proxies for the sensor production process and will function as what is called a Pigment Mixing Truth Calibration Sensor (distinguished from a single resist color calibration sensor). Similar to color calibration sensors, we will begin by measuring these sensors and determining the central dark and central flat white gain values ​​for each photosite on each sensor.

[0095] Furthermore, the molar or thickness-reduced concentration values ​​of C, Y, M, and G are determined for each photosite. This is done by performing Chromabath measurements on each sensor, similar to the color calibration sensors. The aim is to determine the "thickness-reduced" concentration levels (c, m, y, and g) of all four resist pigments C, Y, M, and G for each photosite. These are sometimes referred to as these four true value vectors. The nonlinear least-squares fitting for each photosite can be performed using the following equation as the basic equation. [ka]

[0096] Here, the lowercase 'bl' indicates that these are either models for individual pigments (lower-cost scenarios) or experimentally measured pseudo-Beer-Lambert curves. "Pseudo" simply acknowledges that empirical measurements take precedence over theory.

[0097] For any given photosite within a baseline 3x3 CFA, there exists a nominal thickness of the assigned color resist. This nominal value should be approximately the average of calibration measurements across all sensors for that photosite type. Therefore, the other three values ​​will typically be less than 10% of this nominal value.

[0098] Based on monochromator measurements and nonlinear least-squares fitting in the Chromabath process, thickness-converted pigment values ​​are obtained for each photosite among our N pigment-mixed true-value calibrated sensors. In our production process, we want equivalent data for each sensor, but without such a time-consuming process. This is where FuGal comes in.

[0099] FuGaI, again, is an acronym for Full Gamut Illuminator. The illustrative illuminator features a narrowband emitter of 10 or 12 LEDs, each with a bandwidth typically within a 20–50 nanometer full width at half maximum range. The center wavelengths are selected such that all but two are spaced across the visible spectrum of light, with the remaining two placed within the NIR spectrum. These LEDs are preferably tested to assume that they are temporally stable at their center wavelengths and their brightness. Wavelength stability within a single order of magnitude of nanometers is desired. Brightness variation in the low single order of magnitude or even less than 1% is desired.

[0100] For example, with an illumination distance of 40 cm and a frosted white cover placed on top of the LED illuminator, the individual LEDs of the FuGaI system are turned on sequentially, illuminating N pigment-mixed true-value calibration sensors one at a time, or all at once. Many images per LED state are captured using the pigment-mixed true-value calibration sensors, which are numbered, for example, hundreds or thousands. The median value of each "pixel" is recorded across these 100 to 1,000 images per LED state.

[0101] The terms "pixel" and "photosite" are used somewhat synonymously. "Pixel" is sometimes preferred when referring to a digital number associated with a photosite, while "photosite" is sometimes preferred when discussing the physical properties of a photosensor site. Note that a photosite produces a raw digital number (DN), and each photosite can be associated with calibration information that can be used to adjust the raw photosite DN. The resulting adjusted number, i.e., digital number + calibration data, can be called DN + Cal. Thus, each pixel will produce a DN value within the Cal context. The FuGaI Chromabath procedure plays a role in producing this Cal information for each pixel, which is trained by mixed true value data associated with each of the N pigment mixed true value sensors.

[0102] Each pigment-mixed true-value sensor produced 12 images of the central DN value during the FuGaI Chromabath process. This yielded a 12-dimensional vector for each pixel in each of the N sample sensors, which is referred to as "R12" for real-12D (the light field non-uniformity of the FuGaI units themselves would affect the flat white gain value measurement, but their non-uniformity would have little effect on the mixing coefficients (C, Y, M, and G) which would be measured by the set of 12 images).

[0103] Each pixel in each pigment-mixed true-value sensor also has an associated 4-dimensional vector, which is generated by the least-squares curve fitting process detailed above, based on 451 monochromator Chromabath measurements (with values ​​c_truth, m_truth, y_truth, and g_truth, “R4”).

[0104] The task then involves establishing a mapping that converts values ​​in the 12D vector (R12) space to values ​​in the 4D vector (R4) space. A key consideration in such a mapping problem is the so-called one-to-one mapping problem, specifically applied to the inverse mapping of a single R4 point into the R12 space: Does both of two distinct points in the R4 space map to a single point in the R12 space? Related to this is the issue of common noise: if the R12 measurement points contain excessive noise, does this result in unacceptable fluctuations in the R4 solution values? For example, does this mean that within the assigned green pixels, there are excessive errors in the measurement of trace amounts of cyan, magenta, and yellow?

[0105] Experimental evidence suggests that these are merely theoretical concerns, and in practice, trace amounts of the three unassigned pigments can be measured as a ratio to the assigned pigments. As an example from a numerical perspective, consider a pixel assigned cyan with a nominal color resist thickness of 0.5 microns. Can the FuGaI Chromabath process actually measure thickness equivalents in the 10-100 nanometer range for magenta, yellow, and green? Tests have shown that this is possible.

[0106] The thickness conversion index for the three unassigned pigments is, intuitively, a good option for describing the pigment mixture. While it does not technically describe the nanoscale physical reality of the photosite, it serves our purpose. In exemplary embodiments, the thickness conversion calibration approach is employed to obtain a thickness conversion index measured in nanometer units and associated with the nominal thickness of the assigned color resist measured in micron units.

[0107] Turning to the final step of the FuGaI Chromabath process applied to the pigment-mixed true value sensor, it should be noted that the first embodiment has six pixel types within a 3x3 CFA. These are referred to as follows: Layer 1-M0.5 (Magenta color, 0.5 micron thickness) Layer 2-M1 (Magenta, 1 micron thick) Layer 3-G0.7 (green, 0.7 micron thick) Layer 4-C1 (cyan, 1 micron thick) Layer 5-C0.5 (Cyan, 0.5 micron thick) Layer 6-Y1 (yellow, 1 micron thick)

[0108] Across all pixels of all N pigment-mixed true-value sensors, the mapping function between the R12FuGaI vector and the R4 true-value vector is experimentally solved to determine the trace mixing ratio. Each of the six pixel types described above requires its own mapping function. In the Chromabath process, the flat white gain value and dark median value may also be measured. These are measured during the production line use of the FuGaI Chromabath procedure, and therefore they can also be measured here.

[0109] One objective of the FuGaI Chromabath process on mixed true-value sensors is to calibrate the mixing ratio measurement capability of the FuGaI Chromabath process itself for use in mass-production scale production volumes. The applicant prefers FuGaI-based Chromabath testing of production sensors over monochromator-based Chromabath testing for reasons including cost, simplification, scaled manufacturing, and integration considerations into existing sensor testing procedures.

[0110] Solving equation (1) across all pixels and all N sensors yields six 4x12 arrays of mapping function coefficients. These matrices can then be used to estimate the per-pixel trace pigment mixing ratio in each mass-produced sensor.

[0111] In particular, testing of each mass-produced sensor involves performing the FuGaI Chromabath process and obtaining an R12 vector for each pixel. For each pixel of type X (layers 1-6), the R12 vector measured during post-production testing is multiplied by the xth 4x12 matrix, thereby calculating the trace pigment mixing ratio of that pixel. Just as each individual pixel in modern imaging sensors has associated dark offset and gain parameters, each pixel may have its own associated pigment mixing ratio. This data is written to non-volatile memory on the sensor chip. Although sometimes considered a defect in the manufacturing process, this "slop" in the manufacturing process for each pixel is used to compensate for various downstream processing steps using demosaicing, one exemplary downstream step.

[0112] (Pixel-by-pixel correction) This disclosure then turns to the use of pigment mixing ratio data, i.e., per-pixel correction.

[0113] Such corrections employ stored calibration data related to the sensor chip. The detailed array achieves efficient use of memory storage while providing improved images (e.g., contrast, color accuracy, color gamut, richness of machine learning channels, etc.).

[0114] A 3-byte / pixel calibration storage scheme is used in one embodiment. Four bits of one byte are reserved for the pixel's dark median, and four bits of the same byte are dedicated to the white gain correction value. These two stored values ​​can represent the difference value from the global mean for each of the six pixel types (the layers described above). These values ​​correspond to the bins of a histogram, representing deviations in positive and negative ranges around the global pixel type mean (the data for each of the six mean values ​​and the bins in the six histograms are stored separately on the sensor chip). The 16 values ​​are usually sufficient to cover histogram values ​​that are limited to a relatively narrow range.

[0115] The remaining two bytes represent the pigment mixture value for a specific pixel of one of the six pixel types. Here again, a 4-bit histogram centered on a global mean algorithm may be used. Each trace ratio has a global mean, and the histogram is used to describe how the population of individual pixels of that pixel type fluctuates around that mean. A particular 4-bit value represents one of the histogram bins, which is accessed from memory and represents the corresponding calibration value that is applied to adjust the DN value. There are four such 4-bit values ​​associated with a given pixel: one for the magenta trace ratio, one for cyan, one for green, and one for yellow (referring here and everywhere to the four “trace” ratios; however, one of these colors is the assigned photosite color and therefore above “trace,” but all are treated similarly as far as the histogram representation is concerned). Different histograms are associated with different ones of the six layer types.

[0116] Naturally, the arrays detailed above are merely illustrative. Different applications have different considerations, including cost, physical footprint, complexity, power consumption, etc. 3 bytes / pixel is illustrative, as is the use of a histogram showing the difference from the corresponding mean. It is proposed to use one of the 3 bytes for immediate offset / gain correction. The other 2 bytes + offset / gain correction DN can be used in the demosaicing stage, where demosaiced color data is derived pixel by pixel using either an algebraic algorithm or an AI / ML / CNN algorithm. Static 2-byte trace ratio values ​​would simply be static metadata addition to the otherwise normal algorithmic calculations of demosaicing.

[0117] (First exemplary filter set) According to another aspect of this technology, a novel set of different filters is selected for a color filter array (CFA) that forms part of a photosensor array and filters the light incident on its pixels. As shown, a color filter array conventionally comprises square cells of multiple filters, which are repeatedly tiled together with other such cells across the photosensor. In other arrays, cells of two or more different filter patterns may be tiled in a mosaic manner. In some arrays, each filter within a cell may have a different spectral transmission function T (sometimes referred to as spectral profile or transmission function) from the other filters within the cell. In other arrays, a certain filter type, such as two greens in a 2x2 Bayer cell, may be repeated within the cell. Non-square cells are sometimes employed, including rectangular, triangular, and hexagonal cells.

[0118] Figure 5 shows a color filter cell embodiment with nine distinct filters, identified as filters A, B, C, D, E, F, G, H, and I. These may be selected by a process as described in application No. 18 / 056,704, filed November 17, 2022, but other selection processes may also be used. The transmission function for filter AI is shown in Figure 6A-I. Table data detailing the transmission functions of the filters at wavelengths spaced 10 nm apart is provided in Table I below. [Table 1-1] [Table 1-2]

[0119] The transmission function data was measured without near-infrared or near-ultraviolet filtering, as found in some embodiments.

[0120] Note that the maximum transmittance in Table I is 0.9643, i.e., at 690 nm for filter C. Sometimes, it is useful to normalize filter transmittance values ​​relative to the maximum transmittance within a set of filters in order to provide an index that can be compared across datasets. In this case, each value in Table I is divided by 0.9643 to obtain the data in Table II. Such data is referred to as "group normalization." The transmittance for filter C at 690 nm is here 1.0, and all other data are proportionally larger. This group normalization is well known to practitioners, and it has been taught that in the actual operation of sensors and, for example, the use of these curves in color correction matrices, these normalizations revert to their denormalized forms. Since much of the following discussion concerns wavelengths between 400 and 700 nm, the tables are also limited to this data (extensions for wavelengths below 400 or above 700 nm are omitted to simplify this section of the disclosure). [Table 2-1] [Table 2-2]

[0121] The filters detailed above differ in their transmission functions from filters commonly encountered in the prior art, namely red, green, blue, cyan, magenta, and yellow filters. Fujifilm is one of the suppliers of such prior art filters. The transmission functions for their 5000 series "Color Mosaic" red, green, and blue filters and their 4000 series "Color Mosaic" cyan, magenta, and yellow filters are considered standard. The published curves have been digitized, and the resulting transmission function data for these six filters are detailed in Table III below (as with the previous table, this data does not include infrared or ultraviolet filtering). [Table 3]

[0122] Any filter having a transmission function value comparable to those given in the "R" column of Table III is considered a conventional or standard red filter. "Comparable" here means that two arrays of transmission values ​​between 400 and 700 nm have a mean squared error between them of less than 0.05 when each array is normalized to have a peak value of 1.0. Similarly, filters whose transmission function values ​​are comparable to those given in the G, B, C, M, and Y columns of Table III are defined as standard (conventional) green, blue, cyan, magenta, and yellow filters.

[0123] Again, the transmission functions being compared are pure filter transmission values, without near-infrared (NIR) or near-ultraviolet (NUV) filtering or silicon quantum efficiency shaping. Often, the curves are published to depict the camera's filter response. Such curves are, of course, shaped by the spectral response of the camera sensor (usually silicon) and can also be shaped by NIR or NUV filtering. Figure 7 (based on published data from Canon for its 35MMFHDXS family of camera sensors) illustrates this effect. The red, green, and blue (R, G, B) filter curves are influenced by the panchromatic (P) camera response curve, i.e., silicon efficiency. Often, the panchromatic curve is omitted in published data. A clue that the color filter curves are influenced by the panchromatic sensor response is that the green peak is higher than the red and / or blue peaks, due to the decrease in silicon sensitivity at the red and blue ends of the visible spectrum (the same note applies to the published cyan, magenta, and yellow curves).

[0124] Essentially, a filter that is flat across the 400–700 nm spectrum, that is, one whose average transmission values ​​vary by less than + / - 3% across this spectrum, is considered a conventional (panchromatic) filter in this specification.

[0125] The color filter cells and arrays that embody aspects of this technology may typically include red, green, blue, cyan, magenta, yellow, and / or panchromatic filters. Any filter that is not typically red, green, blue, cyan, magenta, yellow, or panchromatic is referred to as a “non-standard” filter. Each of the filters described in Table I is a non-standard filter.

[0126] In one embodiment, some or all of the filters are selected to be diverse. Diversity can manifest in many forms and can be characterized using many factors. Specific forms of diversity preferred by the applicant are detailed below.

[0127] (Dot product) One factor that can characterize the diversity of spectral response curves is the dot product index, also known as the inner product. The dot product index is calculated by multiplying the corresponding pairs of transmission function values ​​obtained from two filters at each of several wavelengths (e.g., at 10 nm intervals) and summing them up. The applicant prefers to calculate the dot product index at 10 nm intervals over the range of 400–700 nm, but other intervals and other ranges can also be used. Group-normalized data is used as shown in Table II. In this embodiment, the dot product index between filters A and B is calculated by summing the product of their individual transmission functions at 400 nm and the products of their individual transmission functions at 410 nm...700 nm. That is, it is as follows: The dot product index (A,B) = (0.6521)(0.5620) + (0.6445)(0.4670) + ... + (0.0490)(0.5556) = 11.2017

[0128] The dot product often takes the form of both a denormalized dot product and a normalized dot product. This disclosure discusses both, but for the discussion immediately following, we will use the denormalized dot product.

[0129] Within the nine filters, there are 36 different filter pairs, from which 36 dot product indices can be calculated. For the nine filters detailed in Table II, the 36 dot products are as shown in Table IV below. [Table 4-1] [Table 4-2]

[0130] For any given set of filters, there will be a pair of transparency functions that are more or less similar than other pairs of transparency functions. This is evident from the variation in the dot products in Table IV. For example, these dot products range from a minimum of 3.4899 to a maximum of 15.1875. The maximum is 4.35 times the minimum. The mean of all 36 dot products is 8.24, and their standard deviation is 2.99.

[0131] Some embodiments include a color filter cell characterized in that the dot product calculated between the group-normalized transmission functions of any possible pairings of different filters within the cell at 10 nm intervals from 400 to 700 nm ranges from a minimum value to a maximum value that is less than 5 times the minimum value or less than 4.5 times the minimum value.

[0132] Some embodiments include a color filter cell characterized in that the dot product calculated between the group-normalized transmission functions of any possible pairings of different filters within the cell at 10 nm intervals from 400 to 700 nm ranges from a minimum value to a maximum value that is at least 3 times, at least 4 times, or at least 4.3 times the minimum value.

[0133] Some embodiments include a color filter cell characterized in that the dot product calculated between the group-normalized transmission functions of two different filters within the cell at 10 nm intervals from 400 to 700 nm is less than 5, less than 4, or less than 3.5.

[0134] Some embodiments include a color filter cell characterized in that the dot product calculated between the group-normalized transmission functions of two different filters in the cell at 10 nm intervals from 400 to 700 nm is at least 10, at least 12, or at least 15.

[0135] Some embodiments include a color filter cell characterized in that the maximum dot product calculated between the group-normalized transmission functions of all different filter pairings within the cell at 10 nm intervals from 400 to 700 nm is less than 17, less than 16, or less than 15.5.

[0136] Some embodiments include a color filter cell in which the dot product calculated between group-normalized transmission functions of pairings that can be considered as any possibility of different filters in the cell at 10 nm intervals from 400 to 700 nm yields a set of values, wherein at least 10% of these values ​​are less than 5.

[0137] Some embodiments include a color filter cell characterized in that the dot product calculated between group-normalized transmission functions of pairings, which can be considered as any possible pairings of different filters within the cell at 10 nm intervals from 400 to 700 nm, yields a set of values, where less than 6 of these values ​​are greater than 20% or 25%.

[0138] Some embodiments include a color filter cell characterized in that the dot product calculated between group-normalized transmission functions of pairings, which can be considered as any possible pairings of different filters within the cell at 10 nm intervals from 400 to 700 nm, yields a set of values, where 40% or more of these values ​​are less than 7.

[0139] Some embodiments include a color filter cell characterized in that the dot product calculated between group-normalized transmission functions of pairings that can be considered as any possibility of different filters in the cell at 10 nm intervals from 400 to 700 nm yields a set of values, where at least 10 of these values ​​are greater than 20% or 25% or 25%.

[0140] Some embodiments include a color filter cell characterized in that the set of dot products between the group-normalized transmission functions of all different filters in the cell at 10 nm intervals from 400 to 700 nm has an average value of at least 6, at least 7, or at least 8.

[0141] Some embodiments include a color filter cell characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has an average value of less than 10, less than 9, or less than 8.5.

[0142] Some embodiments include a color filter cell characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has a standard deviation of at least 2.6 or at least 2.9.

[0143] Some embodiments include a color filter cell characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has a standard deviation of less than 3.5 or less than 3.

[0144] Each of the embodiments detailed above may consist, in part or in whole, of a non-standard filter.

[0145] (Top Code) Another useful metric for characterizing filter diversity is the so-called top code. The top code is an array of numbers indicating which of two filters has a greater transmission value at each wavelength within a range of uniformly increasing wavelengths. An exemplary top code is a binary sequence where "1" indicates that the first of the two filters has a greater transmission value at a particular wavelength, and "0" indicates that the second of the two filters has a greater transmission value at that wavelength. Again, we consider the transmission function, sampled in 10 nm increments, particularly from 400 to 700 nm.

[0146] Regarding the words "code" and its variations such as "codebin," this word forms an explicit connection between signal measurements represented by pixel data and classical "coding theory." Coding theory provides a useful and powerful tool when dealing with low-light and high-noise measurement systems, such as typical visible light cameras, which are employed in very dark and dim light environments, including cases where the signal-to-noise ratio approaches 1:1 and even lower.

[0147] Referring to Table I, we can see that filter A has a transmission value of 0.5500 at 380 nm, and filter B has a transmission value of 0.7069. Therefore, the first bit of the top code (AB), starting from 380 nm, is "0". Filter A has a transmission value of 0.5886 at 390 nm, and filter B has a transmission value of 0.6174, and therefore the second bit of the top code (AB) is also "0". At 400 nm, filter A switches to having a higher transmission function than filter B (i.e., 0.6288 vs. 0.5420), and therefore the third bit of the top code (AB) is "1". Continuing this method through all 41 wavelength samples from 380 nm to 780 nm yields a complete top code (A,B) for this range. [ka]

[0148] Table V shows the top code values ​​for all 36 pairings of the nine filters in Table I, spanning the wavelength range of 380 to 780 nm. [Table 5-1] [Table 5-2]

[0149] The 380–780 nm spectral range of top codes in Table V can be narrowed by deleting bits at the beginning or end. For example, to produce top codes for the spectral range of 400–700 nm, simply delete the first two bits and the last eight bits of each top code, changing the code length from 41 bits to 31 bits. The top codes for the filter set in Table I for 400–700 nm are shown in Table VI. [Table 6-1] [Table 6-2]

[0150] (Cross code) Within any topcode string (vector), a transition between "1" and "0" indicates that one of the two transmission function curves crosses the other. For example, in the topcode (A,B) from Table VI, there is a transition from "1" to "0" at the 10th bit position, which corresponds to 490 nm. This indicates that the transmission function value of filter A is lower than that of filter B at some point between 480 and 490 nm. This can be seen in Figure 8, which presents the transmission functions of filters A and B (shown individually in Figures 6A and 6B) over the 400–700 nm range in a superposition scheme. A transition from "0" to "1" signals that the first curve has risen above the second curve.

[0151] By sequentially examining the 31 bits of the top code string to search for transitions, a 30-bit string representing a curve intersection, which can be called the intersection code, can be derived. For each consecutive pair of bits in the top code string that have the same value ("1" or "0"), the intersection code has a value of "0". When a transition occurs within the top code string, the intersection code has a value of "1".

[0152] Taking the 31-element top-code string (A,B) as an example, there is a single-bit transition from "1" to "0" at the 10th position. The corresponding 30-element intersection code is therefore all "0" except for the "1" at the 9th position, i.e., as follows: [ka]

[0153] The cross codes corresponding to the top codes in Table VI are presented in Table VII. [Table 7-1] [Table 7-2]

[0154] The number of "1"s in each string indicates the number of intersections between the two curves. For example, the intersection code (A,G) contains four "1"s, indicating that these curves intersect each other four times. The same applies to curves H and I.

[0155] Some embodiments include a color filter cell characterized in that multiple pairs of filter spectral transmission curves, defined by samples at 10 nm intervals from 400 to 700 nm, intersect each other at least four times.

[0156] Some embodiments include a color filter cell characterized in that multiple pairs of filter transmission curves, defined by samples at 10 nm intervals from 400 to 700 nm, intersect each other exactly once or exactly zero times.

[0157] As described above, each of the embodiments detailed above may consist of a non-standard filter, either partially or entirely.

[0158] (Cross-code histogram) The number of curve intersections within each wavelength band can be determined by summing the corresponding cross-code bit position values ​​for all 36 cross-codes. For example, at the first bit position, all values ​​for all 36 cross-codes are "0". Therefore, none of the filter transmission function curves intersect with any other in the 400-410 nm band. The 36 cross-codes have a total of three "1"s at the second bit position (410-420 nm), indicating intersections between three pairs of curves (i.e., (B,G), (C,F), and (D,E) curve pairs). The complete set of such data is shown in Table VIII. [Table 8-1] [Table 8-2]

[0159] This vector, or set of data elements, serves as a histogram of curve crossovers across 30 wavelength bands. This may be referred to as a crossover histogram. Within this crossover histogram, the mean value of the set of data elements is 2.17, and the standard deviation is 2.05. The crossover histogram does not have any adjacent 10 nm wavelength bands where both curve crossover counts are zero. That is, within each 20 nm range identified in Table VIII, the transmission function curves for at least one pair of filters intersect each other.

[0160] Some embodiments include a color filter cell comprising three or more different filters, each with an associated transmission curve, wherein the count of the intersections between all pairs of the curves in each of 30 10nm bands from 400 to 700nm results in a vector of 30 count values, the average of which is at least 2.

[0161] Several embodiments include a color filter cell comprising three or more different filters, each with associated transmission curves, wherein the count of intersections between all pairs of the curves in each of 30 10 nm bands from 400 to 700 nm yields a vector of 30 count values, the standard deviation of which is at least 2.

[0162] Several embodiments include a color filter cell comprising three or more different filters, each with associated transmission curves, wherein the count of the intersections between all pairs of the curves in each of 30 10 nm bands from 400 to 700 nm yields a vector of 30 count values, one or more of which have a value of at least 6, or at least 8, or at least 9.

[0163] Several embodiments include a color filter cell comprising three or more different filters, each with associated transmission curves, wherein the count of the intersections between all pairs of the curves in each of 30 10nm bands from 400 to 700nm results in a vector of 30 count values, where neither of the two consecutive count values ​​in the vector is equal to zero.

[0164] As described above, each embodiment detailed in the present invention may consist of a non-standard filter, either in part or in whole.

[0165] (Hamming distance) The difference between two cross codes can be represented by the so-called Hamming distance between their bit strings. The Hamming distance between two strings of equal length is the number of positions where the corresponding bits are different. While the Hamming distance is used in various fields, the applicant is not aware of its use in relation to the component filters of a color filter array in a color image sensor.

[0166] The Hamming distance between cross codes (A,B) and (A,C) is determined by comparing their strings and counting the number of bit positions where they differ. From Table VII, this is obtained as follows: [ka]

[0167] You will notice that these two strings differ in three bit positions. Therefore, the Hamming distance between cross code (A,B) and cross code (A,C) is 3.

[0168] Referring back to Table VII, the nine different filters can be paired in 36 different ways, resulting in a set of 36 cross codes. That is, filter A can be compared with the other eight (BI), filter B can be compared with the other seven (CI), filter C can be compared with the other six (DI), and so on, until filter H can be compared with only one other (I). The number of such pairing combinations among the nine items is sometimes referred to as the 9-summatorial (i.e., 8+7+6...+1=36).

[0169] When calculating the Hamming distance, the 36 cross codes in Table VII can be paired using the 36-summatorial method. That is, there are 630 Hamming distances between the 36 cross codes in Table VII. 630 values ​​is too many to list here. We will limit the values ​​to the range of 0 to 8, with a mean of 3.29 and a standard deviation of 1.23.

[0170] In fact, within the set of 630 values, there are three Hamming distances of 0: between cross codes [(A,D) and (E,G)], between cross codes [(A,H) and (G,H)], and between cross codes [(B,H) and (D,F)]. Examination of Table VII shows that each of these pairs of 30-bit cross code strings is the same. The last of these pairings can be called trivial, as both cross codes are all zero. However, the other two pairs are non-trivial. In particular, the first pairing contains three "1"s in each cross code, and the second pairing contains one "1" in each cross code.

[0171] The Hamming distances of 8 are cross codes (A,G) and (H,I). Among the 630 values, there are two Hamming distances of 7.

[0172] Some embodiments include a color filter cell characterized in that multiple Hamming distances between all possible cross codes defined between different filters within the cell have a value of zero. One or more of these zero Hamming distances may be accompanied by cross codes that are not all zero. At least one of these zero Hamming distances may be accompanied by cross codes that contain at least three "1"s.

[0173] Some embodiments include a color filter cell characterized in that multiple Hamming distances between all possible cross codes defined between different filters within the cell have values ​​greater than 5 or greater than 7.

[0174] Some embodiments include a color filter cell characterized in that the average Hamming distance between all possible cross codes defined between different filters within the cell is at least 3.

[0175] Some embodiments include a color filter cell characterized in that the standard deviation of all Hamming distances between all possible cross codes defined between different filters within the cell is at least 1.2.

[0176] Some embodiments include a color filter cell characterized in that the standard deviation of all Hamming distances between all possible cross codes defined between different filters within the cell is less than 1.25.

[0177] As described above, each of the embodiments detailed above may consist of a non-standard filter, either partially or entirely.

[0178] (efficiency) Another metric related to filter diversity is efficiency. The efficiency of a filter across a given spectrum can be approximated as the average of the transmission function values ​​at uniform intervals across the spectrum. As an example, for filter "A" in Table I, the sum of 31 transmission functions in the range of 400-700 nm (i.e., 0.6288 + 0.6214 + ... + 0.473), when divided by 31, shows an efficiency of 0.43 or 43%. The efficiencies of the nine filters "A"-"I" detailed in Table I are given in Table IX. [Table 9]

[0179] Some embodiments include a color filter cell characterized in that the average efficiency across all non-standard filters within the cell is at least 40%. In some such embodiments, the average efficiency of all non-standard filters is at least 50%, or at least 60%, or at least 70%.

[0180] The efficiency of individual filters within a cell can vary considerably. In Table IX, the efficiency ranges from less than 25% to more than 65%. That is, one filter may have an efficiency more than 2.65 times that of the second filter in the same cell.

[0181] Some embodiments include a color filter cell characterized by including a first non-standard filter having an efficiency at least 2.0 times or at least 2.5 times that of a second non-standard filter within the cell.

[0182] Some embodiments include a color filter cell characterized in that at least one-third of the multiple different non-standard filters within the cell have an efficiency of at least 50%.

[0183] Some balancing act exists that affects efficiency and diversity. High efficiency can be achieved by having a transmission function in which each filter stays near 100%. This will result in near 100% efficiency but will offer relatively little diversity. On the other hand, high diversity can be supported by a filter curve that fluctuates across the entire range of possible values, sometimes above 0.9 and sometimes below 0.1. However, these latter filters tend to have relatively low efficiency and reduce overall imager sensitivity.

[0184] Some embodiments are characterized by comprising a color filter cell that includes at least one non-standard filter having a group-normalized transmission function that remains above 0.4 in the wavelength range of 400 to 700 nm.

[0185] Some embodiments are characterized by comprising a color filter cell that includes one or more non-standard filters having a group-normalized transmission function that remains above 0.2 in the wavelength range of 400 to 700 nm.

[0186] Some embodiments are characterized by comprising a color filter cell that includes at least one filter having a group-normalized transmission function that remains below 0.7 in the 400–700 nm range.

[0187] Some embodiments are characterized by comprising a color filter cell that includes multiple filters having a group-normalized transmission function that remains below 0.75 in the 400-700 nm range.

[0188] Some embodiments are characterized by a color filter cell comprising three filters having a group-normalized transmission function that remains below 0.8 in the 400–700 nm range.

[0189] As described above, each embodiment detailed in the present invention may consist of a non-standard filter, either in part or in whole.

[0190] (correlation) Another useful metric for characterizing filter diversity is the sample correlation coefficient. Given two arrays of n filter transmission function sample values, namely x and y (for example, 31 values ​​for filters A and B detailed in Table I), the sample correlation coefficient r (hereafter simply referred to as "correlation") is calculated as follows: [ka]

[0191] The correlations between different pairs of filter AIs in the 400–700 nm wavelength range are detailed in Table X. [Table 10-1] [Table 10-2]

[0192] Some embodiments include a color filter cell characterized in that the correlation calculated between the transmission functions of two different filters within the cell, at least one of which is non-standard, in a 10 nm interval between 400 and 700 nm, is negative.

[0193] Some embodiments include a color filter cell characterized in that the correlation calculated between the transmission functions of two different filters within the cell at 10 nm intervals from 400 to 700 nm is at least 0.8, at least 0.9, or at least 0.95.

[0194] Some embodiments include a color filter cell characterized in that the correlation calculated between the transmission functions of possible pairings of different filters within the cell at 10 nm intervals from 400 to 700 nm yields a set of values, where at least one-quarter of these values ​​is at least 0.75. In another embodiment, such a condition applies to possible pairings of different non-standard filters within the cell.

[0195] The mean of the correlation values ​​in Table X is 0.5596. The standard deviation is 0.2885. Of the 36 items in the table, 11 items have values ​​below the mean, within one standard deviation (i.e., between 0.2712 and 0.5596). 14 items have values ​​above the mean, within one standard deviation (i.e., between 0.5596 and 0.8308).

[0196] Some embodiments include a color filter cell characterized in that the correlation calculated between group-normalized transmission functions of pairings, which can be considered as any possible pairing of different filters in the cell at 10 nm intervals from 400 to 700 nm, results in a set of values, where a first count of correlation values ​​is within one standard deviation, above the mean of all values ​​in the set, and is different from a second count of correlation values, which is within one standard deviation, below the mean, and is above the smaller of the first and second counts (25%).

[0197] (Transparency curve slope, extreme values) Other metrics useful in characterizing filter diversity relate to the shape as indicated by local maxima and minima (extrema) in the transmission function, i.e., the slope of the transmission curve going towards and away from such extrema. As a general principle, the slope in an individual spectral response function gives rise to discrimination of color / spectrum within a higher-dimensional spectral data space such as a 9-dimensional data space in a 3×3 nine-channel sensor. Each value derived by the sensor defines one dimension within that 9-dimensional space. Thus, when one of the pixel data values changes rapidly as the spectral component of what it is sensing changes, this gives rise to a slope in the same way within this 9-dimensional space. When one pixel is moving in one direction within this 9-dimensional space and another pixel is moving in the opposite direction assuming the same movement in the sensed spectrum, the result is the spectral discrimination caused by this pair of pixels. Thus, the "opposite slope" for each pair of pixels also becomes a useful diversification metric.

[0198] The modifier "local" indicates the spectral transmission function extrema within the vicinity of a threshold-sized wavelength. An exemplary vicinity extends from 60 nm, i.e., + and - 30 nm from the central wavelength. To make this clear, sometimes a local maximum or local minimum is referred to as, for example, a local maximum or local minimum within a 60 nm range.

[0199] To avoid misidentifying small noise artifacts in the measured transmission function as local maxima, in the following discussion, a feature within the transmission function curve of the filter is considered a local maximum only if the value of its group normalization is 0.05 higher than another transmission function value within 60 nm vicinity centered on the feature. Similarly, for a minimum value, it must have a value 0.05 lower than another transmission function value within 60 nm vicinity. If the transmission function is high or low at both ends of the curve (e.g., as in the case of the left edge in Fig. 6A), since the subsequent value is unknown, thus, for the purposes of this discussion, it is not referred to as a local maximum or local minimum.

[0200] The local maximum is considered "broad" if its transmittance function drops less than 25% from its maximum value within a 40 nm spectrum (sampled at 10 nm intervals) centered on the maximum wavelength. That is, the maximum value shall have a broad peak. Relatedly, for a notch, the notch is considered broad if the value of its transmittance function at the bottom of the notch drops less than 25% from the maximum transmittance function value within a 40 nm spectrum centered on the notch wavelength.

[0201] The opposite of a broad extremum is a narrow extremum, which again applies to both local maxima and local minima. That is, a local maximum is considered "narrow" if its transmittance function drops more than 25% above the maximum value within a 40 nm spectrum (10 nm intervals) centered on the wavelength of the maximum value. That is, the maximum value shall have a narrow peak. Relatedly, for a local minimum, the minimum value is considered narrow if the value of its transmittance function at the bottom drops more than 25% from the maximum value within a 40 nm spectrum centered on the notch wavelength.

[0202] The curves in Table I and FIGS. 6A - 6I give examples. A broad local maximum is found at 490 nm in Filter A. Within + / - 20 nm of the wavelength of the maximum value, its maximum drop from the maximum value is only 5.5% (i.e., from 0.767 at the peak to 0.725 at 470 nm). Out of the nine filters A - I, a total of seven broad local maxima exist. No narrow local maxima exist among these filters.

[0203] A broad local minimum is found at 590 nm in filter D (Figure 6D). This notch is only 19% lower than the maximum value found within 20 nm (i.e., the transmission function at 590 nm is 0.400, while the maximum value within the 40 nm window is 0.493 at 610 nm). This is the only broad local minimum in the detailed set of nine filters. Other transmission functions that appear broad at first glance do not technically meet the given definition because their steep tails are within the 40 nm center window, raising the transmission function value high enough that the minimum is more than 25% lower. See, for example, the minimum at 640 nm in Figure 6E. This is not a broad notch under the given definition because its value (0.07060) is 25% lower than the value of 0.1827 found at 660 nm.

[0204] The preceding statement emphasizes the fact that the defined distinction between narrow and wide extrema largely depends on the slope of the filter curve in the vicinity of the extremum. That is, extrema in the 60nm range adjacent to a steep slope are considered narrow extrema, while extrema in the 60nm range adjacent only to a gradual slope are considered wide extrema.

[0205] A narrow local minimum is found at 450 nm in filter B. Its notch is 61.2% lower (i.e., 0.203 vs. 0.524) than another transmission function value within the central 40 nm window. Seven narrow local minimums exist within the detailed set of filters (including the minimum discussed above in Figure 6E).

[0206] Some embodiments include a color filter cell characterized in that the local minimum count in a narrow 60 nm range exceeds the local maximum count in a narrow 60 nm range. Some such embodiments are characterized in that the local minimum count in a narrow 60 nm range is at least 150%, at least 200%, at least 300%, or at least 400% of the narrow local maximum count in the 60 nm range.

[0207] Some embodiments include a color filter cell characterized in that the local minimum count in a narrow 60 nm range exceeds the local minimum count in a broad 60 nm range. Some such embodiments are characterized in that the local minimum count in the narrow 60 nm range is at least 150%, at least 200%, at least 300%, or at least 400% of the local minimum count in the broad 60 nm range. Some such embodiments are characterized in that the local minimum count in the narrow 60 nm range is at least 7 times the local minimum count in the broad 60 nm range.

[0208] Some embodiments include a color filter cell characterized in that the count of local maximums in a broad 60 nm range exceeds the count of local maximums in a narrow 60 nm range. Some such embodiments are characterized in that the count of local maximums in a broad 60 nm range is at least 150%, at least 200%, at least 300%, or at least 400% of the count of local maximums in a narrow 60 nm range.

[0209] Some embodiments include a color filter cell characterized in that the count of local maximums over a broad 60 nm range exceeds the count of local minimums over a broad 60 nm range. Some such embodiments are characterized in that the count of local maximums over a broad 60 nm range is at least 150%, at least 200%, at least 300%, or at least 400% of the count of local minimums over a broad 60 nm range. Some such embodiments are characterized in that the count of local maximums over a broad 60 nm range is at least 7 times the count of local minimums over a broad 60 nm range.

[0210] In one other embodiment, none of the criteria in the preceding four paragraphs are met.

[0211] Some embodiments include a color filter cell characterized in that one or more non-standard filters within the cell have a local maximum in the 60 nm range between 430 and 670 nm and have a broad peak (i.e., with a decrease in transmission function of 25% or less than 25% from the local maximum in the 60 nm range over + / - 20 nm from the local maximum in the 60 nm range).

[0212] Some embodiments include a color filter cell characterized in that one or more non-standard filters within the cell include a local maximum value in the 60 nm range between 430 and 670 nm, and most of the N filters have broad peaks.

[0213] Some embodiments include a color filter cell characterized in that multiple non-standard filters within the cell include a local maximum value in the 60 nm range between 430 and 670 nm, and all of these filters have a transmission function reduction of less than 50% relative to the transmission value at the local maximum value over a range of + / - 20 nm from the maximum value.

[0214] Some embodiments include a color filter cell characterized in that one or more non-standard filters within the cell have a transmission function that includes exactly one local maximum value in the 60 nm range.

[0215] Some embodiments include a color filter cell characterized in that one or more non-standard filters within the cell have a transmission function that does not include a local maximum value in the 60 nm range.

[0216] Some embodiments include a color filter cell characterized in that one or more non-standard filters within the cell have a transmission function that includes exactly one local minimum in the 60 nm range.

[0217] Some embodiments include a color filter cell characterized in that one or more non-standard filters within the cell have a transmission function that does not include a local minimum in the 60 nm range.

[0218] Some embodiments feature a color filter cell in which one or more non-standard filters within the cell have a transmission function that includes exactly one local minimum in the 60 nm range and no local maximum in the 60 nm range.

[0219] Some embodiments feature a color filter cell in which one or more non-standard filters within the cell have a transmission function that includes exactly one local maximum in the 60 nm range and no local minimum in the 60 nm range.

[0220] Some embodiments feature a color filter cell in which one or more non-standard filters within the cell have a transmission function that includes exactly one local maximum in the 60 nm range and one local minimum in the 60 nm range.

[0221] Some embodiments feature a color filter cell with a plurality of different non-standard filters, wherein the count of wide maxima in the non-standard filters exceeds the count of wide minima in the non-standard filters.

[0222] Some embodiments feature a color filter cell with a plurality of different non-standard filters, wherein the count of narrow minima in the non-standard filters exceeds the count of narrow maxima in the non-standard filters.

[0223] As described, the slope of the filter curve connecting to the extreme values can vary. The diversity can be assisted by the diversity in the slope of the transmission curve.

[0224] The slope of the curve is defined as the change in group-normalized transmission over a 10 nm range (i.e., 400–410 nm, 410–420 nm, etc.). Although determined over 10 nm intervals, the slope is expressed in nanometer units. For example, at 690–700 nm, the group-normalized transmission value of filter A changes from 0.0403 to 0.0490, which is a difference of 0.0087 over a 10 nm range. Therefore, it has a slope of 0.00087 / nm. The slope can be positive or negative, depending on whether the curve rises or falls as the wavelength increases.

[0225] Table XI gives the slopes of filters "A" - "I" as described in Table II. [Table 11-1] [Table 11-2]

[0226] Some embodiments include a color filter cell comprising at least one non-standard filter, characterized in that when the slopes of all group-normalized filter transmission functions are calculated over 10 nm intervals from 400 to 700 nm, they yield a set of values, and at least 60% of the values ​​are negative.

[0227] Filter curves can also be characterized in part by the absolute value of their slope.

[0228] Some embodiments include a color filter cell comprising at least one non-standard filter, characterized in that when the absolute slopes of all group-normalized filter transmission functions of the non-standard filter are calculated over 10 nm intervals from 400 to 700 nm, they yield a set of values, and at least 50% of the values ​​are less than 0.01 / nm or less than 0.005 / nm.

[0229] Some embodiments include a color filter cell comprising at least one non-standard filter, wherein when the absolute slopes of all group-normalized filter transmission functions of the non-standard filter are calculated over 10 nm intervals from 400 to 700 nm, a set of values ​​is obtained, characterized in that at least 20% of the values ​​are less than 0.001 / nm.

[0230] When characterizing the diversity of filter sets, it is sometimes useful to categorize the state of the transmission curve within a particular region into one of three classifications: (1) the vicinity of the peak, (2) the vicinity of the trough, or (3) an intermediate vicinity between the two.

[0231] Peaks and troughs can be local maximum and minimum values ​​in the 60 nm range as previously defined. The vicinity of a peak may consist of its points (sampled at 10 nm intervals), whose transmission values ​​are within 0.15 of the local maximum. The vicinity of a trough may consist of its points (sampled at 10 nm intervals), whose transmission values ​​are within 0.15 of the local minimum.

[0232] Figure 9 shows the transmission function curve for filter A after group normalization (comparison with Figure 6A shows the same curve shape, but the values ​​in Figure 9 are scaled so that one filter in the set, in this case filter C, reaches a peak value of 1.0). A trough exists at 450 nm, with a value of 0.5497 according to Table II. The associated trough vicinity includes the transmission function values ​​of filter A at 400, 410, 420, 430, 440, 450, and 460 nm, because these values ​​are each within 0.15 of 0.5497.

[0233] Similarly, a peak exists at 490 nm with a value of 0.7952. The associated peak vicinity includes the filter A transmission function values ​​at 470, 480, 490, 500, 510, 520, and 530 nm, because these values ​​are within 0.15 of 0.7952, respectively. As can be understood, the peak and trough vicinity may overlap in some cases.

[0234] It should be noted that local extrema in the 60nm range are defined by referencing a 60nm wide neighborhood, i.e., + / -30nm from the center wavelength. Since transmission function data beyond the 400-700nm range is sometimes unavailable, local extrema are typically defined to start at 430nm and end at 670nm.

[0235] On the right side of Figure 9, there is a second trough and a vicinity of the second trough. The filter curve is shown to have a local minimum of 0.0403 for the transmission function at 690 nm. It is unclear whether this value meets the definition of a local minimum (i.e., a trough) in the 60 nm range, as it is unknown whether the curve drops further at, for example, 710 nm or 720 nm. Nevertheless, 620, 630, 640, 650, 660, 670, 680, 690, and 700 nm can all be identified as falling within the vicinity of the trough, as they all have group-normalized values ​​below 0.15. That is, regardless of the transmission function value at 710 nm or 720 nm, it is known that such a value is at least 0, and therefore the wavelengths listed above with values ​​of 0.15 or less will necessarily be within 0.15 of the minimum transmission value. Therefore, valleys can sometimes be identified even if it is not possible to identify a specific valley (i.e., the minimum value in the 60 nm range). The same applies to peaks; that is, any transmission function sample with a group-normalized value of 0.85 or greater than 0.85 will necessarily be within the peak vicinity.

[0236] Applying these definitions of peak and trough vicinity to the group-tuned filtered transmission data in Table II allows us to identify which filter function corresponds to which type of vicinity at which wavelength. This information is represented in Table XII by "MAX" and "MIN" notation. [Table 12-1] [Table 12-2]

[0237] The blank spaces in the table above correspond to regions of the filter curve that are neither near the peak nor the trough (wherein "neighborhood" as used herein is an average value within 0.15 based on group-normalized filter transmission values). As described, these regions between the peak and trough neighborhoods constitute a third class of the transmission function region. In these intermediate regions, the 10 nm wavelength range of each "third class" is characterized by a slope value, which can be positive or negative, as detailed in Table XI.

[0238] In one nine-filter embodiment, which includes more than three different filters, in each of at least 16 of 24 10nm wavelength bands between 430 and 670 nm, the first group of filters 1-5 are all at or near local extrema, the second group of filters 1-5 all have a positive slope, and the third group of filters 1-5 all have a negative slope.

[0239] In the second and third groups, the magnitude of the slope preferably includes a range of values, generally within the range of 0.001 / nm to 0.1 / nm. For example, in the 470-480nm band, the filters in the second group have slopes of 0.019 / nm and 0.033 / nm (i.e., filters B and I), and the filters in the third group have slopes of -0.0022 / nm, -0.0064, and -0.0089 / nm (i.e., filters E, F, and C). As will be understood, the different ones in the nine filters correspond to the different ones in the groups within different wavelength bands.

[0240] Examination of Tables XI and XII reveals that among the nine filters and 25 wavelengths sampled at 10 nm intervals from 430 to 670 nm (i.e., 9 × 25, i.e., 225 filter wavelengths), 88 filter wavelengths lie near the trough and 64 filter wavelengths lie near the peak. The remaining 73 filter wavelengths (i.e., the blanks in Table XII) define the endpoints of a third class of 10 nm ranges with positive or negative slopes (some of these ranges begin with a filter wavelength that is the last element near the peak or trough). In particular, 28 of these third class ranges have a positive slope (0.0005 / nm to 0.0837 / nm range), with more than one-third of this number having values ​​below 0.01 and more than one-third having values ​​above 0.01. 71 of these third class ranges have a negative slope (ranging from -0.00009 to -0.0134), more than one-third of these numbers have values ​​above -0.0005, and more than one-third of these numbers have values ​​below -0.0005.

[0241] The study further shows that for every 25 sampled filter values ​​between 430 and 670 nm, there is at least one filter whose transmission value lies near the peak. Similarly, for every 25 sampled filter values, there is at least one filter whose transmission value lies near the trough (in fact, for every 25 sampled filter values, there are at least four filter functions lies near the extremum, and one range, namely 660 nm, has eight filter functions lies near the extremum). For every 25 sampled filter values, there is also at least one filter that is neither near the peak nor the trough, but rather in a "third class" (for one wavelength, namely 550 nm, there are five filter functions lies in this third class).

[0242] Some embodiments of the present technology include a color filter cell comprising N different filters, where N is 3 or more than 3, 4 or more than 4, 9 or more than 9, or 16 or more than 16 filters, and may include one or more non-standard filters. Each filter is characterized by a group-normalized transmission function having values ​​sampled at 10 nm wavelength intervals from 400 to 700 nm, where some sampled values ​​are within 0.15 of the local maximum value in the 60 nm range for the individual filter and are referred to as peak-near elements, and others sampled values ​​are within 0.15 of the local minimum value in the 60 nm range for the individual filter and are referred to as trough-near elements. Some filter functions include 10 nm ranges that are not overall in either the peak or trough-near elements with respect to 24 different 10 nm wavelength ranges extending from 430 to 670 nm. Some of these 10 nm ranges have a positive slope value with increasing wavelength, and some of these 10 nm ranges have a negative slope value with increasing wavelength. These embodiments are characterized by the following: For every 25 wavelengths sampled at 10nm intervals from 430 to 670nm, there is at least one filter whose transmission function is near the peak, and / or For every 25 wavelengths sampled at 10nm intervals from 430 to 670nm, there is at least one filter whose transmission function lies near the valley, and / or For each of the 24 different 10nm ranges from 430 to 670nm, there exists at least one filter whose transmission function is not in the vicinity of either a peak or a trough overall, but instead has a negative slope value, and / or For every 24 different 10nm ranges between 430 and 670nm, there exists at least one filter whose transmission function is not generally within either a peak or trough, but instead has a negative slope value, and / or Across N different filters, each has a transmission function value sampled at 10 nm intervals from 430 to 670 nm, thereby defining 25N filter wavelengths, where at least 25% of the 25N filter wavelengths are not within the vicinity of either a peak or a trough, and / or At least 65% of the 25N filter wavelengths are near the peak or trough, and / or One or more of these positive slopes have a value less than 0.001 / nm, and / or One or more of these positive slopes have a value of at least 0.05 / nm, and / or Those with a positive slope of one-third or more than one-third have a value of less than 0.01 / nm, and / or Those with a positive slope of one-third or more than one-third have a value of at least 0.01 / nm, and / or One or more of these negative slopes have a value of -0.0001 / nm to zero, and / or One or more of these negative slopes have a value of -0.01 / nm to -0.1 / nm, and / or Those exceeding one-third or one-third of the negative slope have a value of -0.005 / nm to zero, and / or Those with a negative slope of one-third or more than one-third have a value of -0.005 / nm to -0.1 / nm.

[0243] The filters detailed in Table I are readily available and were used for proof-of-concept testing. In practical application, filters with more complex transmission curves may be used. An "M" shaped curve, i.e., a camel shape with two humps, is one example of a more complex filter transmission curve. Such a curve rises from a low value in the blue or ultraviolet spectrum to a first peak at a first wavelength, then descends to a trough, then rises to a second peak at a second wavelength, and finally descends again into the infrared or red spectrum. Another exemplary curve is "W" shaped, starting at a single value in the blue or ultraviolet spectrum, then descending to a first trough, then rising to a peak, then descending to a second trough, and then rising again towards infrared or red wavelengths.

[0244] In one embodiment employing a non-standard filter with an M-shaped transmission function, the two local peaks have separate transmission values ​​that are within 0.25 of each other.

[0245] In other embodiments, one or more filter curves are further more complex and include three local maximum or minimum values ​​in the 60 nm range (e.g., a camel with three humps).

[0246] While complex transmission functions are beneficial, the applicant has found that overly complex transmission functions can be detrimental. For example, metamerism effects become more common when one or more filters in a color filter cell of N different filters have a relatively large number of local maximums and minimums (as defined above). Such metamerism effects can interfere with accurate color measurements. The applicant has found that such a problem arises when the combined count of local minimums and maximums in the transmission function of a filter exceeds (N-2) or (N-2), which is the “relatively large” referred to above. Thus, in a color filter cell of nine different filters, this is a concern when one or more filters have seven or more local extremes. In a color filter cell of six different filters, this is a concern when one or more of the filters have four or more local extremes, and so on. Therefore, although not essential, within the color filter cells of N different filters, the applicant generally employs filters, each having N-3 or fewer local extrema.

[0247] The curves shown in Figure 6A-6I and detailed in Table I relate to polyester filters containing pigments commercially available from Rosco Laboratories. In particular, the filters are as follows: 4390 CalColor 90 cyan 3304 Tough Plusgreen 343 Neon Pink 55 Lilac 65 Daylight blue 365 Tharon Delft blue 370 Italian blue 386 Leaf green 389 Chroma green

[0248] Implementation on photosensors can employ pigment-containing polyester filters, although the pixels will generally be relatively large. Photosensor arrays more commonly use pigment filters. Interference filters, dichroic, and quantum dot (nanoparticle) based filters can also be used. Some embodiments are implemented by mixing pigment powder / paste or nanoparticles in a (negative) photoresist support. Different pigments and nanoparticles absorb different wavelengths, causing notches in the resulting transmission spectrum. The higher the concentration, the deeper the notch.

[0249] (Second exemplary filter set) One of the applicant's aims in selecting filters was to obtain a set of filters whose diversity would result in an optimized modulation transfer function (MTF) compared to alternatives. This led to the development of the first set of filters discussed above, and the second set of nine filters detailed below. Again, these nine filters were selected from the following Rosco catalog. AA: Deep Straw (#15) BB: True Pink (#337) CC: Cal30 green (#4430) DD: Turquoise (#92) EE: GalloGold (#316) FF: Cal60 Pink (#4860) GG: GaslightGrn (#388) HH: Cal60 cyan (#4360) II: Azure blue (#72)

[0250] The transparency data for this second filter set is charted in Figure 10A-10I and detailed in Table XIII. [Table 13-1] [Table 13-2]

[0251] As explained earlier, when group normalized, this second filter set has a transmission function in the wavelength range of 400–700 nm, as detailed in Table XIV. [Table 14-1] [Table 14-2]

[0252] Many of the features characterizing this second filter set are similar to or identical to those of the first filter set. Some of the features characterizing this second filter set are discussed below. Other features can be easily determined from the data provided in the previously taught format.

[0253] The dot products of the second set of nine different filters range from a minimum of 6.33 to a maximum of 17.14. The maximum value is 2.7 times the minimum value. The mean of the 36 dot product values ​​is 11.11, and the standard deviation is 2.72. The complete set of dot products is shown in Table XV. [Table 15-1] [Table 15-2]

[0254] Some embodiments include a color filter cell comprising, in whole or in part, non-standard filters, characterized in that the dot product calculated between the group-normalized transmission functions of any possible pairings of different filters within the cell at 10 nm intervals from 400 to 700 nm ranges from a minimum to a maximum that is less than 3 times the minimum or less than 2.75 times the maximum.

[0255] Some embodiments include a color filter cell comprising, in whole or in part, non-standard filters, characterized in that the dot product calculated between the group-normalized transmission functions of any possible pairings of different filters within the cell at 10 nm intervals from 400 to 700 nm ranges from a minimum to a maximum that is at least 2 or at least 2.5 times the minimum.

[0256] Some embodiments include a color filter cell comprising a partially or entirely non-standard filter, characterized in that the dot product calculated between the group-normalized transmission functions of two different filters within the cell at 10 nm intervals from 400 to 700 nm is less than 7 or less than 6.5.

[0257] Some embodiments include a color filter cell comprising a partially or entirely non-standard filter, characterized in that the dot product calculated between the group-normalized transmission functions of two different filters within the cell at 10 nm intervals from 400 to 700 nm is at least 14, at least 16, or at least 17.

[0258] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings that can be considered as any possibility of different filters in the cell at 10 nm intervals from 400 to 700 nm yields a set of values, the mean of which is at least 9, at least 10, or at least 11.

[0259] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings that can be considered as any possibility of different filters in the cell at 10 nm intervals from 400 to 700 nm yields a set of values, the mean of which is less than 13, less than 12, or less than 11.5.

[0260] Some embodiments include a color filter cell comprising, partially or entirely, non-standard filters, characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has a standard deviation of at least 2 or at least 2.5.

[0261] Some embodiments include a color filter cell comprising, partially or entirely, non-standard filters, characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has a standard deviation of less than 2.75 or less than 3.

[0262] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings that can be considered as any possibility of different filters in the cell at 10 nm intervals from 400 to 700 nm yields a set of values, wherein at least 20% of these values ​​are less than 9.

[0263] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings that can be considered as any possibility of different filters in the cell at 10 nm intervals from 400 to 700 nm yields a set of values, where at least 10% of these values ​​are at least 15.

[0264] The top code, cross code, and cross histogram for the second filter set can be determined in the manner described in detail earlier. The cross histogram for the second filter set is shown in Table XVI. [Table 16-1] [Table 16-2]

[0265] The mean value in this histogram is 1.97, and the standard deviation is 1.60.

[0266] Several embodiments comprise a color filter cell, each containing three or more different filters, each associated with a transmission curve, wherein the filter consists of non-standard filters characterized in that, whole or in part, the count of the intersections between all pairs of the curves in each of 30 10nm bands from 400 to 700nm results in a vector of 30 count values, the average value of which is less than 2.

[0267] Several embodiments comprise a color filter cell, each containing three or more different filters with associated transmission curves, wherein the filters consist of non-standard filters characterized in that, whole or in part, the count of intersections between all pairs of the curves in each of 30 10nm bands from 400 to 700nm results in a vector of 30 count values ​​with a standard deviation of less than 1.7.

[0268] As discussed earlier, the difference between two cross codes can be expressed by the Hamming distance. The 36 cross codes associated with the second filter set, similar to the first filter set (Table I), can be paired in 630 ways. For the second filter set, the Hamming distances range from 0 to 7, with a mean of 3.065 and a standard deviation of 1.24. Eleven Hamming distances of 0 exist within the set of 630 values. Four of these are associated with a Hamming distance of 7.

[0269] Some embodiments include a color filter cell characterized in that the average Hamming distance between all possible cross codes defined between different filters within the cell is less than 3.1.

[0270] The efficiency of the second set of filters can be calculated as detailed above. The results are shown in Table XVII. [Table 17-1] [Table 17-2]

[0271] As can be seen from the filter efficiency in this second set, it varies from less than 40% to almost 70% over the 400-700nm range. The average is 48.5%.

[0272] Some embodiments include a color filter cell characterized in that at least 85% of the multiple different non-standard filters within the cell have an efficiency of at least 40%.

[0273] Some embodiments include a color filter cell characterized in that at least one non-standard filter within the cell has an efficiency greater than 66%.

[0274] Some embodiments include a color filter cell comprising non-standard filters, characterized in that the average efficiency calculated across all different filters within the cell, either partially or entirely, exceeds 48%.

[0275] The diversity of the second filter set can also be characterized in part by its extreme values ​​in the narrow and wide 60nm range. There are nine 60nm range minimums, six of which are wide and three are narrow (the latter being at 450nm in filter CC, 450nm in filter EE, and 640nm in filter II). There are six 60nm range maximums, four of which are wide and two are narrow (the latter being at 490nm in filter FF and 500nm in filter EE).

[0276] Some embodiments include a color filter cell with one or more non-standard filters, characterized in that the local minimum count in a broad 60 nm range exceeds the local minimum count in a narrow 60 nm range. Some such embodiments are characterized in that the local minimum count in the broad 60 nm range is at least 150% or at least 200% of the local minimum count in the narrow 60 nm range.

[0277] Some embodiments include a color filter cell with one or more non-standard filters, characterized in that the local minimum count over a broad 60 nm range exceeds the local maximum count over a broad 60 nm range. Some such embodiments are characterized in that the local minimum count over a broad 60 nm range is at least 150% of the local maximum count over a broad 60 nm range.

[0278] Some embodiments include a color filter cell with one or more non-standard filters, characterized in that the filter count having an extreme value in a broad 60 nm range exceeds the filter count having an extreme value in a narrow 60 nm range. Some such embodiments are characterized in that the local extreme value count in the broad 60 nm range is at least 150% or at least 200% of the local extreme value count in the narrow 60 nm range.

[0279] (Third exemplary filter set) Transmission data for the third filter set in the wavelength range of 400–700 nm are detailed in Table XVIII. [Table 18]

[0280] As explained earlier, when group normalized, this third filter set has a transparency function as detailed in Table XIX. [Table 19]

[0281] The filters in this third set are again primarily stained filters made by Rosco. The exception is the last filter, Filter III, which is made by Lee Filters USA. AAA: Neon Pink (#343) BBB: Cal90 yellow (#4590) CCC: Azure blue (#72) DDD: True Pink (#337) EEE: Deep Straw (#15) FFF: Cal30 green (#4430) GGG:Cal30 cyan (#4330) HHH: Cal60 cyan (#4360) III: Medium Amber (Lee #20)

[0282] Five of these filters are common to the second set discussed above. The tabular data for the second set (Table XIII) was scanned from printed datasheets, while the tabular data for the third set (Table XVIII) was measured using a spectrometer. Some variations will be noted.

[0283] Many of the features characterizing this third filter set are similar to or identical to those of the first and / or second filter sets. Some of the features characterizing this third filter set are discussed below. Other features can be easily determined from the data provided in the manner previously taught.

[0284] The dot products of the third set of nine different filters range from a minimum of 7.79 to a maximum of 21.26. The maximum is again 2.7 times the minimum. The mean of the 36 dot product values ​​is 13.8, and the standard deviation is 3.61. The complete set of dot products is shown in Table XX. [Table 20-1] [Table 20-2]

[0285] Some embodiments include a color filter cell comprising a partially or entirely non-standard filter, characterized in that the dot product calculated between the group-normalized transmission functions of two different filters within the cell at 10 nm intervals from 400 to 700 nm is at least 18, at least 20, or at least 21.

[0286] Some embodiments include a color filter cell comprising a partially or entirely non-standard filter, characterized in that the dot product calculated between the group-normalized transmission functions of two different filters within the cell at 10 nm intervals from 400 to 700 nm is less than 10, less than 9, or less than 8.

[0287] Some embodiments include a color filter cell comprising, partially or entirely, non-standard filters, characterized in that the minimum dot product calculated between the group-normalized transmission functions of all different filter pairings within the cell at 10 nm intervals from 400 to 700 nm is at least 6, at least 7, or at least 7.5.

[0288] Some embodiments include a color filter cell comprising, partially or entirely, non-standard filters, characterized in that a set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has an average value of at least 10, at least 12, or at least 13.

[0289] Some embodiments include a color filter cell comprising, partially or entirely, non-standard filters, characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has an average value of less than 17, less than 15, or less than 14.

[0290] Some embodiments include a color filter cell comprising, partially or entirely, non-standard filters, characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has a standard deviation of at least 3 or at least 3.5.

[0291] Some embodiments include a color filter cell comprising, partially or entirely, non-standard filters, characterized in that the set of dot products between the group-normalized transmission functions of all different filters within the cell at 10 nm intervals from 400 to 700 nm has a standard deviation of less than 4 or less than 3.7.

[0292] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings, which can be considered as any possible different filters at 10 nm intervals from 400 to 700 nm, yields a set of values, where at least 10% of these values ​​are less than 8.5.

[0293] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings, which can be considered as any possibility of different filters at 10 nm intervals from 400 to 700 nm, results in a set of values, where at least 25% of these values ​​are less than 11.

[0294] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings, which can be considered as any possible different filters at 10 nm intervals from 400 to 700 nm, results in a set of values, where at least 20% of these values ​​are greater than 16.

[0295] Some embodiments include a color filter cell comprising a color filter cell that is partially or entirely composed of non-standard filters, characterized in that the dot product calculated between group-normalized transmission functions of pairings, which can be considered as any possible pairing of different filters at 10 nm intervals from 400 to 700 nm, results in a set of values, where at least 10% of these values ​​are greater than 19.

[0296] (Formation of good images from problematic data) Noise can impair the usefulness of color images. A well-known example is underexposed images (captured, for example, in low light or using short exposure intervals) which exhibit low color saturation and contain pixels of the wrong color (readout noise is generally a problem in low-light conditions; in other situations, shot noise can be prominent if there is a shortage of photons).

[0297] One aspect of this technology is that it provides a highly reliable technique for restoring accurate and low-noise color information to otherwise high-noise images. The color direction of "hue" represents one aspect of color measurement. In extremely dark scenes, the signal-to-noise ratio of the pixel data itself approaches 1:1, and classical processing of RGB data has challenges even in measuring the major color directions on the red-green and yellow-blue axes of hue. The approach described below works well even as light levels decrease, and still determines the hue angle and major direction.

[0298] First, we consider an image free of noise. We then further examine a region within the imaged scene that has a spectral hue at 400 nm. Each pixel imaging this region will produce an output signal that depends on the transmission function of its filter at 400 nm. Referring to Table I, and in particular to pixels overlaid with filters A and B (illustrated in Figures 6A and 6B), we can see that pixels filtered with filter A will produce relatively high values, while pixels filtered with filter B will produce relatively low values. This is because the former filter function allows more energy to pass through at 400 nm than the latter, i.e., transmission values ​​of 0.6288 versus 0.5420. However, pixels overlaid with filter A will produce relatively lower values ​​than pixels overlaid with filter C, due to transmission values ​​of 0.6288 versus 0.7063.

[0299] In a similar scheme, filter A will allow more or less light to pass through each of the other six filters DI to image the 400nm scene region, depending on whether the Table I transmission value for filter A is higher or lower at 400nm than the transmission value for each of the other filters.

[0300] Similarly, filter B will allow more or less light to pass through than each of the other seven filters CI to image the 400nm scene region, depending on their individual transmission values ​​at that wavelength. And filter C will allow more or less light to pass through than each of the other six filters DI to image the 400nm scene region, depending on their transmission values ​​at that wavelength, and so on.

[0301] Therefore, in a properly exposed scene, different filtered pixels will produce different output signals from the 400 nm scene region according to their individual transmission values ​​at that wavelength. The output signals from different filtered pixels can be compared based on their transmission values ​​in Table I and ranked in order of magnitude, and at 400 nm, they will be found to be ordered EDCFABGHI. There are 9 different orders to which the filtered pixels can be ranked, i.e., 36 different orders.

[0302] At different wavelengths traversed through the spectrum, the ranked sequential sequence of the filter will differ. At 410 nm, it is identical to 400 nm, i.e., EDCFABGHI. However, at 420 nm, it switches to DEFCAGBHI. At 430 nm, it remains identical, but again, at 440 nm, it switches to DEFACGBHI. Each segment of the spectrum is associated with a unique sequential sequence of the filter's output signal. Among the 31 sampled wavelengths in the range of 400–700 nm, there are 26 unique sequential sequences of the filter. Overlaps are expected to occur adjacent to each other. The complete set of sequential sequences is given in Table XXI. These 9-letter sequential sequences can be referred to as spectral reference strings. [Table 21]

[0303] If the scene illumination is dim, or the camera exposure is very short, the output signal from the pixels will be noisier. Some will be higher than they should be, and others lower. However, some of the remaining filtered outputs in the sequence of nine different wavelength-based filters will remain unchanged (consider the 400nm sequence again. At both ends of the ordered sequence of filters, filter E has a transmission value of 0.8075 at 400nm, while filter I has a transmission value of 0.0426. That is, a pixel filtered by filter E will typically produce an output signal that is almost 20 times stronger than a pixel filtered by filter I in a 400nm scene. Even at extreme noise levels, it is unlikely that the latter pixel would produce a higher output signal than the former).

[0304] In noise recovery mode, the output signals from pixels within a 3x3 pixel area of ​​the photosensor are ranked in order of magnitude, and can represent a corresponding ordered sequence of the individual transmission values ​​of those filters at an unknown wavelength of incident light. This ordered sequence of filters will be somewhat distorted by the effects of noise, but the sequence will correspond more closely to some of the spectral reference strings in Table XXI than to others. The closest match in Table XXI can be used to represent the spectral hue of the incident light.

[0305] Various known string matching algorithms can be used. One is based on the Hamming distance. First, an ordered sequence of outputs from nine different filtered pixels is determined in a low-light scene. This nine-element sequence is called the query string. Next, this query string is compared with each of the 36 spectral reference strings in Table XXI. The number of positions in each spectral reference string where the query vector has a different character at a given string position is counted. The smaller this count, the better the query string matches the spectral reference string. The spectral reference string that best matches the query string (i.e., the string with the smallest character difference count from the query string) represents the hue in that region within the photosensor.

[0306] If a query string best matches two spectral reference strings, the query string can be considered to match the wavelength between the two wavelengths indicated by the two spectral reference strings. For example, if a query string best matches the spectral reference string EDCFABGHI, and this reference string is found in Table XXI for both 400 nm and 410 nm, then the query string can be associated with the hue of 405 nm.

[0307] The approach mentioned above does not use the 31 nine-character strings as in Table XXI. Rather, it uses the 36 three-symbol top-code sequences from Table VI.

[0308] Recall that each top code in Table VI represents a pair of filters that have higher transmission values ​​at 31 wavelengths between 400 and 700 nm. The first item in Table VI compares filters A and B. The second item in Table VI compares filters A and C, and so on. Therefore, there are 9 total (36) possible combinations.

[0309] Within each top code, the first number indicates a paired filter with a higher transmittance at 400 nm. The second number indicates a paired filter with a higher transmittance at 410 nm, and so on. Therefore, there are a total of 31 sampled wavelengths.

[0310] When the top code data in Table VI is read in vertical columns instead of horizontal rows, a 36-digit sequence is obtained for each wavelength. Each of the 36 digits represents the filter of a separate pair of filters that have the maximum output at the target wavelength. Each such 36-digit sequence is referred to as the reference hue code. Upon completion of this method, the reference hue codes in Table XXII are obtained. [Table 22]

[0311] Referring to the reference hue code for 400nm, the first symbol "1" indicates that filter A has a higher transmittance value than filter B. The second symbol "0" indicates that filter A has a transmittance value less than filter C. The next three "0"s in the reference hue code indicate that filter A has a transmittance value less than filters D, E, and F, respectively. The following symbol "1" indicates that filter A has a higher transmittance value than filter G. The next two symbols are both "0", indicating that filter A has a higher transmittance value than filters H and I. Subsequent symbols in the 400nm reference hue code continue in this manner, comparing filter B with all the others, then filter C, and so on.

[0312] The binary reference hue codes described above in Table XXII are the counterparts to the spectral reference strings in Table XXI. Each hue code corresponds to a specific spectral wavelength, and the code derived from the noise image data serves as a reference, which can be compared to assign a spectral hue to each pixel in the noise image data. Again, the Hamming distance can be used to compare the reference hue code against a query code derived from the noise image data for a particular pixel to determine the best match (i.e., the minimum Hamming distance). The best-match reference hue code indicates the most likely hue for the query pixel.

[0313] Consider a 3x3 cell of pixels, using the filters detailed in Table I, to image a uniformly colored scene region under low light. The pixels will produce different output signals (data) due to both their different filters and underexposure-related noise. Nine output data are considered in 36 pairs, and for each pairing, identify the filter that yields the larger output signal. If the filtered pixel A produces an output signal that is larger than that of the filtered pixels B, C, and D, but smaller than that of the filtered pixels E, F, G, H, and I, the query code begins with the digit 11100000... Further bits of the query code are generated by comparing the output signals from the filtered pixel B, then the filtered pixel C, and so on with others. A 36-bit query code is thus produced. By comparing this query code with the reference hue codes in Table XXII, it can be found that the query code is closest to the reference hue code for 430nm, namely 110000110000011000011111111111111111. Therefore, the pixel values ​​for this region in the noisy image frame are replaced with RGB (or CMY) pixel values ​​corresponding to the 430nm hue.

[0314] In one embodiment, a lookup table in memory stores corresponding red, green, and blue (RGB) values ​​that define the color of a pixel for each hue code. This mapping of hue to RGB values ​​can first be performed by identifying the CIE XYZ chromaticity coordinates for each hue of interest. These XYZ coordinates are then re-transformed into the desired RGB space by a 3x3 transformation matrix. One preferred matrix corresponding to the sRGB standard with D65 illumination is as follows: [ka] This matrix and other suitable transformation matrices can be found on the website brucelindbloom. <dot>It is available at com / index.html?Eqn_RGB_XYZ_Matrix.html.

[0315] Accordingly, one embodiment of the present technology involves the step of receiving values ​​corresponding to output signals from several photosensors in a local neighborhood, the photosensors being overlaid by filters having different spectral transmission functions. The method then includes the step of providing, for example, a set of multiple color values ​​(e.g., R / G / B or XYZ) for the target photosensors in the neighborhood, from memory, based on the received values. Such a method may also include the step of determining, for each of several photosensors, whether the output signal corresponding to one photosensor is greater than or less than the output signal corresponding to another photosensor. Often this involves the step of determining, for each pair of two photosensors in a neighborhood, which of the two has a larger received value corresponding to it. In many embodiments, each of the multiple color values ​​provided corresponds to a specific hue, and such color values ​​are available only with respect to discrete sampling of the hue and lack other hues between the discrete samplings of the hue.

[0316] In the aforementioned array, which uses both binary hue codes and the previously referenced spectral reference strings, it was assumed that the underexposed scene contained a region of a particular color such that nine different filtered pixels, each using the transmission function of its individual filter, sampled the same color with noise. In most images, this is not a far-fetched approximation, as the closest neighboring pixels will be similar in chromaticity. Thus, in some embodiments, each pixel is considered to be at the center of a cell, and its hue is determined based on a comparison of its value with the values ​​of other pixels in that cell (if the cell does not have a center pixel, neighboring pixels can also be used).

[0317] Other arrays can also be used as alternatives. Consider a tiled photosensor with an array of 3x3 color filter cells, each having the array shown in Figure 5. The pixel of interest (i.e., the target pixel) is filtered by one of the nine filters, for example, filter "A" shown in bold in Figure 11. The other pixels surrounding the target pixel are filtered by different filters according to the cell and tiling pattern.

[0318] Without assuming a constant or nearly constant color region, to perform one of the methods described above (based on binary hue codes or spectral reference strings), the values ​​of the filtered pixels B, C, D, E, F, G, H, and I are interpolated onto the location of the filtered pixel of target "A". Then, employing the previously described method, eight such interpolated filtered pixel values ​​are used at the target pixel, and the ninth is the actual filtered pixel value used at that target pixel. Various interpolation techniques can also be used.

[0319] The simple technique is linear interpolation. To predict the value filtered by B at the location of the target pixel, we look for the values ​​of the two nearest B filtered pixels (i.e., within the same row). These two pixels, identified as B1 and B2, are shown in Figure 11A. Their values ​​are weighted according to the reciprocal of their distance from the target pixel, and the weighted sum is 1. In this example, the interpolated "B" value at the location of the target pixel is as follows: B1 / 3 + 2B2 / 3

[0320] Linearly interpolated values ​​for the photosensor, with other filters (i.e., filters C, D, and G) found in the same row and column as the target pixel, are similarly calculated. The same applies to filtered pixels that are on the same diagonal as the target pixel, such as pixels E1 and E2 in Figure 11B.

[0321] For filters that are not in the same row or column as the target pixel, different interpolation methods, such as bilinear or bicubic interpolation, can be used. The embodiment shown in Figure 11C is for predicting the filtered pixel value of F at the target location. Bilinear interpolation is illustrated and involves performing three linear interpolations. First, the values ​​of the two upper "F" pixels F1 and F2 are combined with a weight of 2 / 3, 1 / 3 to obtain the linearly interpolated filtered pixel value of F at the location indicated by the opposing arrows of the upper pair. The values ​​of the two lower "F" pixels F3 and F4 are combined in the same manner to obtain the linearly interpolated filtered pixel value of F at the location indicated by the opposing arrows of the lower pair. These upper and lower interpolated values ​​themselves are then combined in a ratio of 1 / 3, 2 / 3 to obtain the interpolated value for the "F" pixel at the target location.

[0322] The target location therefore contains the filtered pixel value of "A" measured by the photosensor, and eight interpolated pixel values ​​for the other eight filters. These nine values ​​are then compared with each other to produce a nine-symbol ordered query string or a 36-bit binary query code. Each of these strings / codes is compared against a reference spectral string or hue code to find the best match, thereby determining the hue to be assigned to the target pixel. This operation is performed for all pixels (or only underexposed pixels) in the photosensor array.

[0323] It will be recognized that the process described above for assigning color to a pixel / photosensor stems from limited discrete sampling of hues. For example, in one embodiment detailed earlier, there were 26 unique reference sequence sequences (i.e., reference spectral strings) of the filter, which correspondingly represent 26 hues. In contrast, the photosensor, providing 8 bits each of red, green, and blue output data (i.e., a 24-bit output color space), is 2 24 It has a hue that can be considered as a possibility. Therefore, the available sampling is very sparse, and 2 of the available colors 19 This is equivalent to less than 1 / 1. Naturally, the sparseness of different embodiments may vary depending on the implementation. However, many embodiments will still not be able to produce anything less than 1%, or less than 0.001%, or less than 0.000001% of the hue that can be represented in the output color space.

[0324] As a consequence of the above, a many-to-one mapping exists between the photosensor value and the indicated hue value (and corresponding RGB output value) in the neighborhood surrounding the target photosensor (pixel). The ranking of photosensor output values ​​and the results of comparisons between pairs of pixels are not affected by some variation in the photosensor value. For example, a photosensor overlaid with filter A will have a greater output signal than a photosensor overlaid with filter B, regardless of whether the former photosensor would have an output value of 20 or 200, or whether the latter photosensor would have an output value of 10.

[0325] In addition to the noise reduction provided by this array, many embodiments can be implemented without complex arithmetic operations. For example, the step of determining whether the output value for one photosensor is greater than the output value for another photosensor is a simple operation that can be performed by an analog comparator (if performed before the accumulated photoelectron charge of the photosensors is converted to a digital form) or a digital comparator (if performed afterward). Such operations can be implemented using hardware networks that are simpler than arithmetic operations (which may include multiplication or division) commonly used in image denoising processes.

[0326] The aforementioned array provides hue output data but does not mention luminance. Luminance can be estimated based on the magnitude of the photosensor signal at the target pixel. Alternatively, a weighted average of neighboring photosensor signals can be employed, with the target pixel typically being given more weight than other pixels. Nonlinear weighting can also be employed to reduce the influence of outlier signal values. If the average transmittance values ​​of various filters differ, the photosensor signal can be normalized as appropriate, for example, so that a filter that passes a small fraction of panchromatic light is counted more than a filter that passes a relatively larger fraction of panchromatic light when estimating local image luminance. In yet another embodiment, local luminance within an area of ​​the image is estimated based on a statistical distribution of (normalized) values, as low-light images exhibit larger deviations. Different RGB values ​​can be stored in a lookup table memory for different combinations of hue and luminance. Alternatively, a single set of RGB values ​​can be stored for each hue, and the values ​​can then be scaled up or down based on the estimated luminance.

[0327] Accordingly, according to one embodiment, a value associated with a first pixel having a first spectral response function is compared with values ​​associated with several other pixels in the neighborhood having spectral response functions different from the first spectral response function, and a color or hue is assigned to the first pixel based on the result of the comparison. In some embodiments, the comparison is performed without any multiplication or division operations. In some embodiments, the comparison is used to determine the ordered sequence of the pixels. Some embodiments include the step of assigning a color or hue based on the Hamming distance or based on the result of a string matching operation.

[0328] Naturally, the embodiments taught above can also be practiced using filters and filter cells having the attributes detailed above.

[0329] (CMYRGB filter array) Assume that a color filter array designer has only six available color resist options: red (R), green (G), blue (B), cyan (C), magenta (M), and yellow (Y). These colors are generally available from commercial suppliers. They are therefore economical, and the manufacturing processes using such resists are well understood. Table III gives an exemplary set of transmission functions for such filters.

[0330] Using such resists, the transmission function of the filter can be varied by varying their thickness according to the Beer-Lambert law. Such variations may include changing the range of the maximum and minimum values, and changing the filter slope. Figure 12 shows an example of the transmission function of a magenta resist at various wavelengths, 450 nm, 850 nm, and a thickness of 1.0 micron. Different filter functions can be achieved by fabricating multiple filters within a filter cell using identical but differently thick color resists, thereby expanding the filter palette from, for example, an initial set of six resists.

[0331] Figure 13 shows exemplary spectral response functions for six CMYRGB colors, each applied with a thickness of 1 micron (units are arbitrary, where 1,000 indicates complete transparency and 0 indicates complete opacity). Note that the yellow slope in the 450-500 nm range is approximately the same as the green slope within the same wavelength range. Informationally, this is non-ideal. Similar redundancy occurs with other pairings of the filter.

[0332] Figure 14 illustrates how varying thicknesses can produce significantly different linearly independent spectral filter functions. The two blue, two cyan, and two red curves are all duplicated and accompanied by curves that depict the transmission function for resists with thicknesses from 800 nm to 350 nm, respectively. This set of nine curves is produced using only six color resists. The arrays described anywhere in this disclosure that employ nine diverse filter functions can therefore be realized using only six resists.

[0333] Figure 15, though slightly more abstract, simply depicts the first derivative (i.e., slope) of the curve in Figure 14. This is the interaction of slopes, by which spectral information is derived, and here we can see that there is a desirable diversification of the slope that results in thick / thin layer bifurcation. For example, the thick-layer red "slope peak" may be observed to be shifted by approximately 10 nm from the thin-layer red peak, and this diversification is the primary factor behind color accuracy. Various physical and manufacturing approaches exist that can be used to produce such thick / thin layer bifurcation (or even trifurcation) of a single-color resist.

[0334] One approach is to form a layer of clear, stabilized (cured), positive or negative photoresist that does not contain pigment at each of the filter locations within the cell. This can create a raised clear pedestal (platform) on which subsequent layers of resist can be applied, and can serve to thin the resist layer applied to other locations that lack a clear resist.

[0335] Figure 16 illustrates the concept. In this embodiment, a transparent resist is applied to the photosensor substrate 171, exposed through a mask, developed, and washed (sometimes collectively referred to as "masking") to form a transparent pedestal 172 at five locations within the nine filter cells. The resist may have a thickness of 500 nm. Figure 17 shows an excerpt of the sensor after five subsequent masking layers have been applied, resulting in five defined color filters such as red, green, blue, cyan, and magenta.

[0336] A resist containing a first pigment, identified as "A," is applied in a liquid state to the structure shown in Figure 16. Where a transparent pedestal is absent, the resist flows down to the sensor substrate, forming a 1,000 nm thick layer, for example, as indicated by the longer double arrow on the left in Figure 17. Where a transparent pedestal is present, the liquid resist does not flow to such depths, but rather settles on the pedestal, forming a 500 nm thick layer. This resist is masked and washed away wherever resist "A" is not desired. This results in a cell having filter layers of resist "A" of different thicknesses, namely a thin layer 181 where a transparent pedestal is present and a thick layer 182 where a transparent pedestal is absent. Such filters of different thicknesses have different filtering functions according to the Beer-Lambert law.

[0337] According to the Beer-Lambert law, if a first filter of an absorbent medium has a layer thickness L1 and a transmission function T1 (on a scale of 0 to 1), then a second filter of the same medium with a layer thickness L2 will have a transmission function T2 as follows: [ka]

[0338] The process detailed above is repeated for the second time using a second resist "B". Again, the "B" resist flows into the substrate where there is no transparent pedestal and settles on top of the transparent pedestal where it is present. The pigment layer is masked, leaving a region of pigment "B" with two thicknesses: a thin layer where the pigment settles on the transparent pedestal and a thick layer where the pigment settles on the substrate.

[0339] In the described embodiment, the process is repeated two more times using resists "C" and "D". For each color, a thick filter layer and a thin filter layer are formed, the latter located where a transparent pedestal is present. Finally, a fifth resist "E" is applied to the wafer and masked, creating a filter at the center of the color filter cell. Referring back to Figure 16, it can be seen that a transparent pedestal is present at this location. Therefore, the resist layer "E" does not extend to the photosensor substrate, but rather rests on the transparent pedestal, resulting in a layer of only 500 nm.

[0340] The array in Figure 17, therefore, contains nine different filter functions, but is achieved using only six masks (one to form a transparent pedestal and one for each of the five colored pigment layers).

[0341] In the illustrated embodiment, the thicker and thinner filter layers of the same color resist have a thickness ratio of 2:1 (i.e., 1,000 nm and 500 nm). However, this is not required. Such ratios can be 1.1:1 to 3:1 or 4:1 or greater. Generally, the ratio is 1.4:1 to 2.5:1, with ratios of 1.5:1 to 2:1 being more common.

[0342] Figure 18 shows an excerpt from a color filter cell in which a 400 nm thick green resist layer is formed on a 300 nm thick transparent pedestal. Any location within this color filter cell may be a green pigment layer extending to the level in which the transparent pedestal is formed with a thickness of 700 nm. The thickness / thinness ratio for these green pigment-containing filter layers is therefore 1.75:1 (i.e., 700:400).

[0343] In some exemplary embodiments, the pedestal has a height of 200–500 nm, and the resist is applied to a depth to achieve a thick-layer filter of 600–1,100 nm (without a pedestal). In one particular embodiment, all pedestals have a height of 200–300 nm. In the same or other embodiments, the resist is applied to form a thick-layer filter of 700–1,000 nm thickness (with a thinner filter where a pedestal is located).

[0344] In Figure 17, thin and thick layers of a given resist color are adjacent at their edges. However, this is not necessary. In some implementations, such thin and thick layers of the same resist color are adjacent at their corners or not adjacent at all. Some CFAs (or cells) have a combination of such relationships, where the thin and thick layers of a first color resist are adjacent at their corners, and the thin and thick layers of a second color are adjacent at their edges or not adjacent at all. Some CFAs or (cells) are characterized by all three relationships: adjacent at the corners with respect to the thin and thick layers of the first color, adjacent at the edges with respect to the thin and thick layers of the second color, and not adjacent with respect to the thin and thick layers of the third color.

[0345] The checkerboard pattern of transparent pedestals in Figure 16 can be reversed so that the four corner locations and the center location lack pedestals, and pedestals are formed instead at the other four locations. Instead of four or five pedestals, a cell can contain more or fewer than one, up to one less than the total number of filters in the cell. As long as the number of pedestals is less than the number of filter elements in the cell (i.e., some filter elements do not reside on the pedestals), the array of pedestals can be called "sparse," meaning that not all photosensors (or microlenses) are associated with pedestals.

[0346] One embodiment is an image sensor comprising a sparse array of transmissive pedestals, wherein the array of photosensors is positioned below the pedestals and a colored filter medium (e.g., pigment) is positioned above the pedestals. The sparse array may, but is not required to be, a checkerboard array. Such an array generally includes filter elements of thicker and thinner dimensions, with each of the thinner filter elements positioned above one of the transmissive pedestals.

[0347] Typically, checkerboard pedestals like those in Figure 16 have their corners touching, but this is not essential. A gapped checkerboard pattern can have such arrays of pedestals that do not touch the corners (for example, by reducing the size of each pedestal in Figure 16 in the horizontal dimension by 1% or more (e.g., 2%, 5%, 10%, 25%, or 50%)).

[0348] Figures 19A–19E show several such sparse patterns, where "T" indicates filter locations with transparent pedestals. These can each be inverted, and the transparent pedestals are formed in the unmarked locations of the "T" rather than the marked locations. As can be seen from the figures, transparent pedestal locations within a given cell can be adjacent by edges, adjacent by corners, non-adjacent, or any combination of these three (here, adjacency relationships are described in the context of a single cell, as in the earlier discussion of thick and thin layers of the same color; once cells are tiled with other cells, different adjacency relationships may arise).

[0349] One favorable array consists of a 3x3 filter cell formed using seven masking steps: the first step forms a transparent pedestal pattern, and the following six steps apply a colored resist, such as red, green, blue, cyan, magenta, and yellow, once each (sometimes the first three colors are referred to as primary colors, and the latter three as secondary colors).

[0350] Figure 20 shows a cell of this type containing three transparent pedestals, using the pedestal pattern from Figure 19E. The three locations with transparent pedestals (three of the four corners) result in a lower density color filter because such filters are physically thinner. These are indicated by thinner lines and letters. The locations lacking transparent pedestals result in a higher density color filter because such filters are physically thicker. These are indicated by thicker lines and letters.

[0351] In this embodiment, each of the three primary color filters appears twice within the color filter cell, i.e., once in the thinner layer and once in the thicker layer. Each secondary color filter appears only once within the cell, and all appear in the thicker layer (i.e., not formed on the transparent pedestal).

[0352] In other arrangements, a filter appearing twice within a cell may be a secondary color. In yet another embodiment, a filter appearing twice within a cell may include one or more primary colors and one or more secondary colors.

[0353] Naturally, it is not required that all three primary and all three secondary colors be contained within such a cell. Filters of other functions may also be included, including filters with desired ultraviolet (e.g., below 400 nm) and infrared (e.g., above 750 nm) properties, and various unconventional types of filters as detailed earlier. Each such filter may be contained once or twice within a cell, i.e., once in a thin layer and once in a thick layer. Naturally, it is not required that three transparent pedestals be present within a 3x3 cell; more or fewer may be present. In some 3x3 cells, six transparent pedestals are employed such that six of the filter layers are relatively thin and three of the filter layers are relatively thick. As in other embodiments detailed herein, some pixels may not be filtered by a color resist that is transparent in all applicable wavelengths (panchromatic).

[0354] Naturally, transparent pedestals to achieve thinner filter layers can be used in cells of different sizes than 3x3, such as 4x4, 5x5, and non-square cells.

[0355] Another exemplary implementation is shown in Figure 21, which is based on a 2x2 Bayer cell. However, here, the first masking operation defines a transparent pedestal at one of the four pixel locations (indicated by a thinner line and letter in the upper left). Three more masking operations follow, defining four color filters: one red, one blue, and two green. The upper left green filter formed on the transparent pedestal is thinner than the green filter formed in the lower right (the upper left green filter is also thinner than the red and blue filters). Because it is thinner, this thinner green filter allows more light to pass through than the thicker green filter (which, like the red and blue filters, is of conventional thickness). This increases the efficiency of the sensor. Also, because it is thinner, it broadens the spectral curve according to the Beer-Lambert law. This changes the slope and position of the filter tails, allowing for improvements in color accuracy.

[0356] Figure 22 shows illustrative spectral curves for the four filters in the cells of Figure 21, with the light green filter indicated by a thick solid line. This plot is for the case where the thin-layer filter is one-third the thickness of the other filters (the red, green, and blue curves are based on data from Table III).

[0357] The Bayer cell employs two green filters in its 2x2 pattern, taking into account the human visual system's (HVS) sensitivity to green. When the sensor serves machine vision purposes, the HVS-based rationale for dual green becomes invalid, and another color, namely red or blue, may be dual. Figure 23 shows a modified Bayer cell employing two diagonally adjacent blue filters, one thick and one thin. Figure 24 shows the transmission curve for such an array. The light blue filter curve is indicated by a thick solid line. Here again, the thin-layer filter is one-third the thickness of the other filters. Similar to the array in Figure 22, this modification increases sensor efficiency, diversifies the spectral curve, and allows for better color accuracy.

[0358] The cells do not need to be square. Since there are resists containing six readily available pigments (i.e., three primary colors: red, green, and blue, and three secondary colors: cyan, magenta, and yellow), such resists can be used to form six filters within a 2x3 pixel cell. Again, transparent pedestals can be formed first on those pixels, and then masked in those locations, so that the resist is thin on pixels that do not have pedestals.

[0359] Figure 25 shows such a cell. A transparent pedestal is formed under a secondary color filter, as indicated by a thin border and lighter lettering. The pedestal is missing under a primary color filter, as indicated by a thick border and bolder lettering.

[0360] The cells in Figure 25 can be paired with their associated cells, each with a filter color shifted one pixel to the left, but the original pedestal pattern is maintained. This is shown in Figure 26. The top two rows contain the cells from Figure 25. The bottom 2x3 pixel cells are identical except that the filters are each shifted one position to the left. The result is 4x3 pixel cells with 12 filters, containing 4 thin and thick filters out of 6 colors, along with two thin filters for a fifth color (cyan here) and two thick filters for a sixth color (red here). As in the previous example, the thin and thick filters for the common colors are formed in a single masking step, the difference being that a transparent pedestal is directly beneath the thin filters. Thus, despite containing 10 different filter functions, only 7 masking steps are used to produce the cells (and CFAs formed from multiple cells) in Figure 26. The variety of slopes provided by 10 different filter functions allows for improved color accuracy. Six thin-layer filters also play a role in providing increased efficiency.

[0361] Accordingly, one embodiment comprises an image sensor with a sparse (e.g., checkerboard) pattern of transparent pedestals extending across the sensor, the pattern defining two types of scattered locations, namely, relatively elevated locations and relatively descended locations. A continuous region of the sensor includes cyan, magenta, and yellow filters in one of the locations of the type (e.g., relatively descended, i.e., without pedestals) and red, green, and blue filters in the other location of the type (e.g., relatively elevated, i.e., with pedestals).

[0362] Another arrangement employing all six primary / secondary colors is shown in Figure 27. This is a 1x6 linear cell arrangement where every other filter element is formed on a transparent pedestal (in this embodiment, under secondary magenta, cyan, and yellow filters, but one or more primary colors can be substituted).

[0363] A second row can be formed by shifting the pedestals horizontally by one position so that each pedestal is angularly adjacent to another. This second row can be overlaid by a sequence of filters identical to that in Figure 27, but shifted two positions to the left. The 2x6 cells in Figure 28 then result. It will be recognized that these 12 filter cells provide 12 different filter functions, including thin and thick layer filters for each of the six colors. This large number of diverse filter functions allows for excellent color accuracy, while the numerous thin layer filters provide high efficiency. As with the cells in Figure 26, such cells can be processed using only seven masking operations: once for the transparent pedestals and once for each of the six colors.

[0364] The cells in Figure 28 can be duplicated in a tiled arrangement, with the same 2x6 cells positioned on the left, right, top, and bottom, repeated as needed until they span the area of ​​the photosensor. This results in a checkerboard arrangement of pixels with transparent pedestals. These transparent pedestals are defined in the first masking step. The resulting 3D checkerboard structure provides square wells that facilitate the creation of color filters at intervening pixel locations in subsequent masking steps (the arrangement in Figure 16 thus appears checkerboard, but when tiled using a similar structure, it is found that many of the transparent pedestals have edges adjacent to other pedestals, rather than only corners adjacent to other pedestals as in a checkerboard).

[0365] Figure 29 shows the group-normalized transmission function for a 6-element cell employing five resists. Two filters are formed using blue resist, one thick and one thin (e.g., approximately 1,000 nm to 500 nm). Three other thick filters (e.g., approximately 1,000 nm), namely red, green, and cyan, are present. Finally, one thin yellow filter (e.g., approximately 500 nm) is present. It should be understood that such an array allows the thin filter elements to be formed on a clear pedestal with a height of approximately 500 nm.

[0366] This is a type of more general color filter cell embodiment, comprising N>3 filter elements, two of which are formed from the same resist, but one having a thickness of 80% to 20% (preferably 66% to 33%, most preferably 60% to 40%) of the other.

[0367] This is also a type of color filter embodiment comprising N>3 filter elements, wherein two or more of the filter elements are relatively thin, having a thickness of 80% to 20% (preferably 66% to 33%, most preferably 60% to 40%) of the thickness of another filter element in the cell. In one such array, one of the relatively thin elements has a relatively thick corresponding element in the cell formed from the same material, while another of the relatively thin elements does not have a relatively thick corresponding element in the cell formed from the same material. In one such array, one of the thin-layer filters is a red, green, or blue pass-through filter, and another of the thin-layer filters is a red, green, or blue attenuation filter (i.e., cyan, magenta, or yellow filter), respectively.

[0368] Many further variations exist. For example, two masking operations can be used to overlap some on top of others to form two layers of transparent pedestals. For instance, a first masking operation can create six 500nm thick pedestals in a 3x3 cell. A second masking operation could then create three more pedestals, each, on top of one of the 500nm pedestals created in the first masking step, for example, with a thickness of 300nm. This results in a first set of three 500nm thick pedestals and a second set of three pedestals totaling 800nm ​​thick. Three other locations in the cell have no pedestals. If the latter locations are filled with color resist to a thickness of 1,000nm, the resist will form a 500nm thick layer on the first set of transparent pedestals and a 200nm thick layer on the second set of transparent pedestals. The color resist can therefore form filters of two or three different thicknesses in a single masking operation.

[0369] In one such embodiment, following two masking operations that form pedestals of different thicknesses, three more resists are successively applied and masked to create filters of three different colors (e.g., selected from red, green, blue, cyan, magenta, and yellow). Each of these resists can be masked at positions corresponding to three different thicknesses. Five masking operations thus result in nine different filter functions.

[0370] In another variation, the pedestal is not transparent (clear), but rather selectively spectrally absorbent, for example, by using a color resist. Such a sequence yields the same thickness modification results as discussed above, but with the addition of spectral modulation of the pedestal color. Each of the sequences described above can be implemented using spectrally absorbent pedestals (a "transparent" or "clear" pedestal has a transmittance of 98% or above 98% over 400–700 nm, while a spectrally absorbent pedestal has a lower transmittance at certain wavelengths. The general term "transparent" pedestal is used to encompass both transparent and spectrally absorbent pedestals).

[0371] In one embodiment, some or all of the pedestals are formed of a resist (e.g., a pigment-containing resist) that blocks infrared light. If one filter in the cell is formed on such an IR-blocking pedestal, and another filter is formed from the same resist but not on an IR-blocking pedestal (e.g., not formed on a pedestal, or formed on a pedestal that transmits infrared light), the resulting different responses of the two pixels in infrared light provide information about the scene content in infrared light. This can be useful, for example, in AI applications.

[0372] One particular such resist has an IR tapered panchromatic response. An IR tapered panchromatic response is essentially panchromatic through visible light wavelengths and has a spectral transmission function above 80%, 90%, or even 95+% over the 400–700 nm range, after which the sensitivity gradually decreases toward IR. In certain embodiments, the spectral transmission function of such a resist is below 50%, 20%, or 10% at some point in the 700–900 nm range, preferably at some point in the 700–780 nm range such as 720, 740, or 760 nm.

[0373] In further variations, the pedestals may have optical functions, for example, if their refractive indices are higher or lower than those of the overcoated photoresist.

[0374] Both positive and negative photoresists can be used in the detailed arrays. The choice of tonality can be based on practical considerations such as photosensitivity, resolution, sidewall tilt / aspect ratio, ion implantation stopping ability or etching resistance, and fluidity.

[0375] Resists with high solid content (such as pigment-loaded CFA resists) often work best in negative tones because they are easiest to remove when the so-called "gravel" (solid content) is not exposed by the lithography process and the resist matrix has the best possible solubility in the developer. When positive resists are selected, the volume to be removed first needs to be solubilized by exposure, which can lead to the formation of more residue. Also, due to the light absorption rate with depth, the sidewall profile tends to be more gradual, which is undesirable in continuous CFA arrays.

[0376] The creation of the pedestal adds a further degree of variability to the manufacturing process. This variability can be measured and recorded in the Chromabath process as detailed herein, similar to the other process variations above. The Gaussian variability in the thickness of the (thinner) layer formed on the pedestal is likely to be a percentage greater than the variability in the thickness of the filter that is not formed on the pedestal, and is another dimension of variability that can be characterized in the Chromabath process.

[0377] One embodiment according to one aspect of the present technology is a color filter cell including a first filter made of a first colored resist formed on a transparent pedestal and a second filter made of the same first colored resist not formed on the transparent pedestal, wherein the second filter has a thickness greater than that of the first filter.

[0378] One embodiment according to another aspect of the present technology is a photosensor including a checkerboard pattern of a transparent pedestal extending across the photosensor.

[0379] One embodiment according to another aspect of the present technology is a color filter cell having filter pixels with N different spectral transmission functions created using only M masks, where M < N. In some such embodiments, M = N - 1, and in other such embodiments, M = N - 2, or M = N - 3, M = N - 4, or M = N - 5.

[0380] Generating more of those profiles than the number of masks for generating pixel spectral profiles through the layer thickness mechanism is one way to achieve a set of diverse filter functions. In some embodiments, the filters are derived from only CMYRGB color resists.

[0381] While this section focuses on commercially available conventional R, G, B, C, M, Y, and infrared resists, it should be understood that the detailed teachings can be applied to any type of filter medium, including non-standard filters, and including filters (and filter cells) having the attributes discussed earlier herein.

[0382] (An embodiment employing near-infrared light) Image sensors that produce red, green, blue, and near-infrared channels in output data are known. The Canon 120MXS sensor is an example comprising an array of silicon pixels overlaid with a color filter array, each cell containing three visible light filters (red, green, and blue) and a near-infrared filter. The infrared output channel allows for discrimination between image features that would otherwise appear identical based only on red, green, and blue data.

[0383] According to one embodiment of this technology, the image sensor includes four pixels that are filtered to produce a maximum output (i.e., maximum sensitivity) in the 400–700 nm range. These are referred to as visible light pixels, in contrast to NIR filtered pixels used in image sensors such as the Canon 120MXS. At least two of these four pixels have a strong color response in wavelengths ranging from red (e.g., 650 nm) to near-infrared (i.e., above 700 nm). The sensitivity of these two or more visible light pixels to near-infrared allows for the generation of four channels of image data, at least one of which is influenced by the content of the infrared image.

[0384] Figure 30, taken from the Canon datasheet for the 120MXS sensor, shows the response of its red, green, and blue filtered pixels in the near-infrared range. The pixel response for the monochrome version of the Canon sensor is also shown in Figure 30 as a solid line. This represents the sensor's panchromatic response, i.e., without the overlaid color filter array. The shape of this panchromatic response curve is primarily due to the quantum efficiency of the silicon photosensor, but is also influenced by the sensor's microlens array and other factors.

[0385] The above refers to a filtered pixel having a "strong" response at a certain wavelength. As used herein, a strong response is greater than 50% of the sensor's panchromatic (unfiltered) response at that wavelength, preferably greater than 60%, 70%, or 80%. For example, the red filtered pixel in the sensor in Figure 30 has a strong response from 580nm to 800nm, with a response exceeding 60% at 590 to 800nm, exceeding 70% over the same range, and exceeding 80% at 600 to 660nm and 740 to 800nm.

[0386] The responses of red, green, and blue pixels within the Canon sensor are shown in Table XXIII as percentages of the sensor's panchromatic response over the 350–800 nm range. [Table 23-1] [Table 23-2]

[0387] The definition given above "strong" compares the filtered response of a pixel at a given wavelength with its panchromatic response at that wavelength, but sometimes the latter data is unavailable. In such cases, a preferred definition of "strong" is to compare the filtered response of a pixel at a given wavelength with its maximum response within the wavelength range of 400–700 nm. The peak response of a visible light pixel decreases with increasing wavelength, for example, to about 70% of the peak response at 650 nm, 50% at 700 nm, and 40% at 750 nm. If the filtered response of a pixel at one of these wavelengths (relative to its peak response) exceeds half of the percentage given above, the pixel is said to have a strong response at that wavelength. For example, if a red pixel has a peak response of 0.9 at 600 nm (on some arbitrary scale) and its response at 700 nm is 0.3 (i.e., 33% of the peak response), this is judged to be a strong response because 33% is more than half of the 50% figure referenced above in relation to 700 nm (again, a strong response of a pixel is preferably more than half of the figures given above, such as 60%, 70%, or 80% of the percentages given above).

[0388] Another method for determining a "strong" response is to refer to the spectral transmission function of the individual filters of the pixels. A pixel can be said to have a strong response at a given wavelength if the filter allows more than 50% or 50% of the illumination incident on the filter to pass through to the photosensor below at that wavelength.

[0389] Therefore, one embodiment comprises an image sensor including four pixels that are most sensitive in the 400-700 nm range, each pixel having a photosensor and an individual filter that causes the pixel to have a different color response from the other four pixels. At least two of the four pixels' filters allow at least 50% of the illumination incident on the filter to pass onto their individual photosensors at wavelengths greater than 650 nm-700 nm.

[0390] At wavelengths of approximately 500–780 nm, yellow-filtered pixels exhibit a strong response of over 70%, generally over 80%, of the panchromatic response at such wavelengths (the yellow filter is panchromatic except for blocking blue wavelengths below 500 nm). Between 640–780 nm, yellow-filtered pixels have a response very close to that of red-filtered pixels, as detailed in the table above. However, yellow pixels have greater efficiency than red pixels (for example, across the spectrum extending from 400–750 nm).

[0391] As described, the embodiment currently discussed comprises an image sensor including four pixels having a peak response within the visible light wavelength range. At least two of these four pixels have a strong color response in the wavelength range from red (e.g., 650 nm) to near-infrared.

[0392] In a first class of such embodiments, the four visible light pixels include exactly three that are filtered to become primary color pixels and one that is filtered to become a yellow or magenta color pixel. In a second class of such embodiments, the four visible light pixels include exactly two that are filtered to become primary color pixels.

[0393] In one embodiment of the first class of embodiments, four visible light pixels are filtered to become red, green, blue, and yellow pixels. The red and yellow pixels respond strongly to wavelengths in the near-infrared range (e.g., 700-800 nm). By using four channels of image data that respond differently to the imaged scene in the blue-to-near-infrared wavelength range, it is possible to extract four channels of output data. Three of these channels are red, green, and blue, and can represent image scene content as perceived by the receptors of the human eye. A fourth channel can be fabricated to vary according to near-infrared scene content (but does not need to vary exclusively according to near-infrared scene content).

[0394] In one embodiment of the second class of embodiments, four visible light pixels are filtered to become red, green, yellow, and IR tapered panchromatic pixels. IR tapered panchromatic pixels are essentially panchromatic throughout the visible range, but their response decreases in the near-infrared region. For example, such pixels may respond within 80%+ (preferably 90%+ or 95%+) of the unfiltered response of the sensor in the 400-700 nm range (i.e., their spectral transmission function is 80%, 90%, or 95+) but may exhibit lower responsiveness above 700 nm. In certain embodiments, these pixels are filtered so that their transmission function decreases to 50%, 20%, or 10% of the corresponding panchromatic level at some point in the 700-900 nm range, such as 700, 740, or 780 nm.

[0395] Again, in this representative example of the second class of embodiments, the red and yellow pixels respond strongly to wavelengths in the near-infrared range. The four channels of image data in this array do not include a channel sensed by the blue pixel, although the IR tapered panchromatic pixel is sensitive to blue. Here again, it is possible to extract four channels of output data by using four channels of image data that respond differently to scenes imaged in the blue-to-near-infrared wavelength range. As in the former embodiment, three of these channels are red, green, and blue and can represent image scene content as perceived by the receptors of the human eye, while the fourth channel can be fabricated to vary according to near-infrared scene content.

[0396] It should be understood that in the embodiments of this discussion, at least the red and yellow pixels (and in many embodiments, the blue and green filters as well) lack infrared blocking filters (sometimes referred to as hot mirror filters), which are commonly used with image sensors. Alternatively, IR attenuation filters may be used, but they may enable significant pixel responses in the near-infrared range, such as the response at 750 nm exceeding 5% or 5% of the peak response in the visible light range.

[0397] The embodiments described in this section can also be implemented by forming a filter on an IR filtering pedestal, as previously described.

[0398] The embodiments described may, of course, utilize filters and filter cells having the attributes detailed above. Unless explicitly stated, red, green, blue, cyan, magenta, and / or yellow filters are not required.

[0399] (A spatially variable color filter array) A color image sensor typically comprises an array of photosensors, overlaid with a corresponding array of color filters. The filters are carefully matched so that each filter corresponds to one or an integer of the photosensors.

[0400] According to certain aspects of this technology, this alignment constraint is not maintained. The color filter array may be positioned randomly or intentionally across the photosensor array such that a single filter covers a non-integer number of photosensors. Some photosensors may be overlaid by multiple filters. In some embodiments, the photosensors and filters have different dimensions, which contributes to this effect.

[0401] In one such embodiment, the side dimensions of the photosensor and filter have a non-integer ratio. One such array is shown in Figure 31, which shows an excerpt from a color image sensor, where the color filter 311 is depicted by thick-lined squares and the underlying photosensor 312 is depicted by thin-lined squares. This excerpt comprises an 8x8 color filter patch covering a 7x7 patch of photosensors. The photosensors are therefore larger than the filters. Each photosensor has a side dimension 8 / 7 times that of the color filter. Each photosensor has an area 64 / 49 times that of the color filter.

[0402] This non-integer relationship between the dimensions of the filter and the dimensions of the photosensor gradually shifts the position of the filter relative to the photosensor in a modulo manner. This is further illustrated in Figures 32 and 32A.

[0403] In retrospect, an image sensor color filter array generally consists of multiple tiled cells, each containing multiple filters. An example is a 2x2 cell array of color filters used in a classic Bayer filter (red, green, green, blue). In the particular array in Figure 32, each cell contains a 3x3 color filter array. Two such identical filter cells 21 and 22 are shown in Figure 32, with different shading to aid the illustration.

[0404] Figure 32A is an enlarged excerpt from Figure 32, which serves to illustrate that the filter in one embodiment of the present technology has a different location relative to the underlying photosensor.

[0405] The location of a filter cell can be established by any arbitrary feature of the cell. For illustrative purposes, consider the lower left corner of the filter cell to serve as a reference point for defining the cell's location (other corners or the center of the cell are also possible reference points). The location of the filter cell can then be defined by the spatial relationship between this reference point and the photosensor below it.

[0406] The left side of Figure 32A is annotated with two Cartesian axes, x and y, defining the coordinate system within the leftmost photosensor being depicted (indicated by a thin square), which is overlaid by the lower left corner of the filter cell (i.e., its reference point). Any point within the boundary of that leftmost photosensor may be defined here by coordinates along the x-axis, ranging from 0 to 100, and again by coordinates along the y-axis, also ranging from 0 to 100. The position of the filter cell 21, indicated by a light shade, is defined by such coordinate location of its lower left corner. This location is indicated by an arrow in Figure 32A and has coordinates 76.6 and 49.7.

[0407] The position of the adjacent filter cell 22, indicated by a darker shading, is different. This location is again defined by the coordinates of its lower left corner within the photosensor area covering it. Again, the arrow in Figure 32A indicates this location, with coordinates 38.7 and 49.7.

[0408] In a similar scheme, other filter cells have other locations relative to the underlying photosensor. Within a given color image sensor, there may be 5, 10, 20, 100, 1,000, or 1 million or more different locations of filter cells relative to the underlying photosensor.

[0409] In this particular embodiment, the y-coordinate of each filter cell is constant between cells within a single horizontal row (i.e., in the row containing lightly shaded and darkly shaded cells 21 and 22, 49.7). The x-coordinate varies. In other words, the reference point for different filter cells will differ from the distance between the nearest neighbor columns of the photosensor. In this embodiment, the distance between the reference point for filter cell 21 and the nearest neighbor column 23 of the photosensor is smaller than the distance between the reference point for filter cell 22 and the nearest neighbor column 24 of the photosensor.

[0410] Conversely, although not depicted, the x-coordinate of each filter cell in this embodiment is constant between cells within a single vertical column. The y-coordinate, however, varies. In other words, the reference point for different filter cells within a given column will differ from the distance between the nearest adjacent rows of the photosensor.

[0411] (Here, the conventional configuration where rows are longer than columns is used. Therefore, if a photosensor array has 3,000 photosensors in one direction and 4,000 photosensors in an orthogonal direction, the latter direction is the row direction.)

[0412] A color imaging device as described above is characterized in part by including a J-row photosensor overlaid by a color filter array containing K-row color filters, where neither J / K nor K / J is an integer. Such a device may also be characterized in addition, or alternatively, by including a P-row photosensor overlaid by a color filter array containing Q-row color filters, where neither P / Q nor Q / P is an integer.

[0413] Furthermore, note that multiple (in fact, most) of the photosensors in the array in Figure 31 are partially overlaid by the four filters. Similarly, multiple (again, most) of the filters in the array in Figure 31 cover the four photosensors (with the exception of the areas around the boundaries).

[0414] In other embodiments, each of the multiple photosensors is partially overlaid by nine or more filters. In yet another embodiment, each of the multiple filters covers nine or more photosensors.

[0415] In the array shown in Figure 31, if the side dimensions of the photosensor and the side dimensions of the filters are related by a ratio of 8 / 7, the spatial relationship between the photosensor and the individual filters will begin to repeat after 8 rows and columns of filters and after 7 rows and columns of photosensors. The location of the 3x3 filter cells relative to the photosensor will also repeat, but over longer intervals.

[0416] Such iterations can be performed at a lower frequency by selecting an appropriate ratio between the photosensor and filter side dimensions. In this embodiment and some other embodiments, the numerator and denominator of the ratio are selected to be disjoint (i.e., without any common factors other than 1).

[0417] In some embodiments, each filter cell has a different location relative to the photosensor. This can be achieved by selecting a ratio of relatively prime values ​​between the filter and photosensor side dimensions, where the two numbers defining the ratio are each greater than the maximum pixel dimension of the color imaging device. For example, if the device has a pixel dimension of 4,000 × 3,000, each filter cell will have a different location relative to the photosensor if its side dimensions are selected to have a ratio such as 4,001 / 9,949 (in this example, both the numerator and denominator are prime numbers).

[0418] In the array shown in Figure 31-32A, the photosensor is larger than the individual filters, but this is not required. The filters can be larger than the photosensor. In both cases, it is desirable that the ratio between their side dimensions is not an integer value, for example, not 2.

[0419] However, photosensors and filters whose side dimensions have integer ratios, including 2 (and 1), can also be used when they are overlaid in a distorted relationship where the rows of photosensors are not parallel to the rows of filters (and the same applies to the columns). Such an arrangement is shown in Figure 33, where the side ratio of the filters and photosensors is 1, i.e., they are the same size. Again, due to the distortion angle, different filter cells have different locations relative to the photosensor. Depending on the choice of distortion angle, 5, 10, 20 or more filter cells in a color image sensor may have different locations relative to the underlying photosensor. Again, in some embodiments, all filter cells may have different locations relative to the underlying photosensor.

[0420] The distortion angle between the color filter array and the photosensor array can be achieved intentionally, or it can be achieved in other ways, such as by relaxing manufacturing tolerances so that a certain degree of "slop" occurs in the alignment of the color filter array with respect to the photosensor array. Any degree of randomness in positioning (including machining) the color filter array across the photosensor array can also introduce such distortion.

[0421] Similar to the arrangement in Figure 31, the arrangement in Figure 33 also introduces a gradual shift in the location of the filter cells across the device relative to the underlying photosensor.

[0422] In yet another embodiment, the filter, comprising filter cells, may have a different shape from that of the photosensor. For example, the filter may be an elongated rectangle, while the photosensor may be square. Such an arrangement is shown in Figure 34. Many other different filter shapes (and photosensor shapes) can be devised, and they are not all quadrilaterals. Again, such an arrangement gives different filter cells different locations relative to the photosensor, and the locations are gradually shifted across the sensor.

[0423] The array combinations detailed above can also be employed. For example, the color filter arrays in Figure 31 or 33 can be positioned at a distorted angle on their corresponding photosensor arrays.

[0424] The different locations of the filter cells on the photosensor in the embodiments detailed above result in different pixel spectral filter functions. In a typical color imaging device using a tiled 3x3 color filter array of filter cells, each pixel has one of nine spectral filter functions. However, in the arrays detailed above, individual pixels are filtered at different rates (corresponding to the percentage area of ​​the pixel photosensor covered by each filter) using multiple different physical filters, resulting in hundreds, thousands, or even millions of different pixel filter functions across the device.

[0425] The illustrated embodiment typically employs a color filter, but relatively thick and relatively thin filters with different spectral transmission functions, such as non-standard filters described in detail earlier, can also be used.

[0426] According to a further aspect of this technology, a spectral filter function is characterized for each photosensor in the device. The applicant's Chromabath procedure can also be used. Associated data, recording the filter function for each photosensor, is stored in the device's memory. Similarly, data relating to a kernel, which thereby converts scalars output from individual photosensors in a neighborhood into color values ​​relating to a desired color channel for a pixel at the center of the neighborhood, is also stored in the device's memory (such neighborhoods may be, for example, 5x5 or 7x7 in size).

[0427] In this array, as in other arrays detailed herein, filters of two different spectral functions can be achieved by fabricating two filters of different thicknesses using the same medium (e.g., a pigment-containing resist), as detailed previously. Similarly, spatially varying filter arrays can also employ filters and filter cells having the attributes detailed previously.

[0428] A particular filter cell comprises individual filters of conventional red, green, blue, cyan, magenta, and / or yellow color resists, one or more of these colors being formed with different layer thicknesses. For example, such colors can be formed as thick layers (e.g., with thicknesses in the range of 0.8 to 1.5 microns) and as thin layers (e.g., with thicknesses in the range of 0.4 to 0.8 microns).

[0429] In some such embodiments, a clear resist is applied to a photosensor substrate to define a checkerboard pattern of clear elements. Then, 2x3 filter cells are formed, for example, from a colored resist. Half of these filters are on top of the clear resist elements (and therefore thin layers), and the other half extend between the clear resist elements (and therefore thick layers). All filter cells can have the same pattern, or two or more filter cells can be repeated in a tiled pattern.

[0430] One such array employs two different 2x3 filter cells in a tiled array. The first filter cell comprises R, G, B, c, m, and y filters, and the second filter cell comprises r, g, b, C, M, and Y filters, where uppercase letters indicate thick-layer filters and lowercase letters indicate thin-layer filters (each letter corresponds to one of red, green, blue, cyan, magenta, and yellow, respectively). Such cells can be tiled in a checkerboard arrangement as shown in Figure 35 (shading is added simply to make it easier to distinguish between two different filter cells). A color filter array having some or all of the attributes detailed above can be employed in any of the other embodiments detailed herein.

[0431] (Chromabath) As described, the Chromabath process optically characterizes pixels on a sensor. In one particular implementation, this produces a multi-parameter pixel behavior map of the entire sensor, i.e., a map of the optics, primarily chromaticity deviations, for a standard of the entire sensor. "Characterizing" means measuring and recording how each pixel responds to light. Classical properties such as "bias" and "gain" are known as panchromatic parameters. Measuring, storing, and utilizing these two classical parameters is included within the Chromabath process. The Chromabath process also handles color filter array image sensors, i.e., panchromatic characterization is an additional advantage. This can be used on all sensors employing CFA.

[0432] Figure 7 and Table III contain the sensitivity functions for red, green, and blue for representative Bayer sensor pixels. Clearly, not all pixels of any given color will precisely possess these functions; they will typically deviate from the global standard at a deviation level of less than 10%. Such deviations can be due to variations in filter thickness, crosstalk between different photosensors, filter contamination with pigment component residues from previous masking steps, layer matching errors (including filters and microlenses), etc. Data characterizing the resulting deviations in pixel performance is measured and stored as part of the Chromabath process, allowing for later correction of such pixel data and compensation for such error sources.

[0433] Figure 36 exaggerates the idea for illustrative purposes. In this figure, four specific regions of the spectrum are identified, and a given red pixel located at a certain point in the pixel group appears to deviate from the global mean red spectral function to a measurable degree. This is the spectral function deviation of this type that the Chromabath procedure measures and corrects. As previously described, the execution of this process can involve a test phase of sensor calibration using a multi-LED lighting system and a calibration phase using a monochromator.

[0434] As a single, independent word, "Chromabath" often refers to the entire procedure involved in its application to an actual sensor. Strictly speaking, the singular word refers to the prolonged light bathing of a sensor using light from a multi-LED lighting unit. This light bathing process involves hundreds, possibly thousands, of image captures from the sensor under test. This image data is typically processed offline, with the data collected and stored, and processing of that data not beginning until all data has been collected from the sensor.

[0435] In one illustrative array, spectral transmittance curves are measured for each pixel on the sensor. Each curve, for example, has 85 data points and can detail the transmittance in 5nm increments from 380 to 800nm. For each different type of filter on the sensor (e.g., light cyan), an 85-point global average curve is determined based solely on that type of filter. Each individual filter of that type is then characterized by its deviation from the mean of the corresponding type. Figure 37 shows a sample of such filter characterization curves for individual pixels. These curves may be used as signatures for individual pixels (in this example, Y-axis calibration is not important, and the figure serves only to illustrate the concept).

[0436] As will be recognized, the pixel signature shown in Figure 37A is a noisy spectral function plot. These types of noise waveforms are suitable for a wide range of "compression" approaches, which in this embodiment involves obtaining 85 floating-point values ​​(in 5nm increments from 380nm to 800nm) and converting them into 4-byte compressed values.

[0437] One way to do this is to employ principal component encoding. Principal component encoding starts with a very large set of sample pixel signatures (e.g., all pixel signatures for a light cyan type filter across a sensor) and then performs singular value decomposition of this set. This yields, for example, six significant principal component vectors (known as eigenvectors), which, when multiplied by a unique coefficient for each pixel signature, would "fit" that pixel signature to some acceptable level of accuracy, such as 0.1%. Each of these coefficients is quantized to one of 16 values, each represented by, for example, four binary bits. These 16 values ​​can be spaced uniformly (e.g., in increments of 1 from -7.5 to +7.5) or non-uniformly (i.e., to one of 16 histogram bins, each selected to have roughly the same number of counts).

[0438] Each pixel signature is represented by six coefficients (corresponding to six principal component vectors), totaling 24 bits when using a 4-bit array as described above. The remaining 8 bits (of the 4 bytes) can be allocated as 4 bits for pixel offset value data and 4 bits for gain data. The 4 bits for the pixel offset value can be used to represent 16 uniformly spaced or non-uniformly spaced values ​​that relate the offset value for that pixel to a global standard for the entire sensor. The same applies to the 4 bits for the gain data (the global standard for the entire sensor used for the offset and gain data can be of a similar type, e.g., for light cyan pixels or for all pixels on this device).

[0439] In the modified embodiment, the 24 bits allocated to represent the six principal component coefficients are not evenly distributed, with 4 bits for each coefficient. Instead, more bits (e.g., 6 bits) are used to represent the first-order component coefficients, and sequentially fewer bits are used to represent subsequent components. For example, the second and third-order components may be represented by 5 bits each, and the fourth and fifth-order components by 3 bits each. This leaves 2 bits to represent the sixth coefficient.

[0440] To store these 32 bits per pixel, the sensor is equipped with associated memory of an appropriate size for the task. Since the per-pixel signature data, offset value, and gain value are all relative to the overall sensor average, the memory also stores these average values. When image data is transferred from the sensor to the system for use, the pixel data is transferred along with the 4 bytes associated with each pixel and the global average. The receiving system can then correct, compensate for, or otherwise consider the pixel values ​​according to this correction data to obtain more accurate results.

[0441] There are at least three uses for Chromabath data. 1. Sensor manufacturing quality control and assurance 2. Creating a globally uniform color image 3. Pixel-specific input data for AI and machine learning image processing (classical pixel data is supplemented with signature data (Chromabath-generated) for a given pixel).

[0442] The use of such data in creating globally uniform images can increase the manufacturing yield of image sensors to a measurable level. The yield of high-quality large CMOS image sensors is particularly low, often below 90%. This means that 10% or more of the sensors manufactured are sent to the recycling bin, or in some cases, to the waste bin. Practitioners understand that the yield problem is more complex than this simple explanation suggests, and practices such as "bin classification" attempt to match economic demands with quality, but the main point remains: the yield of CMOS image sensors is generally poor.

[0443] In particular, with respect to Bayer / RGB CFA sensors, yield is primarily determined by the global uniformity quality specification. It is unacceptable for part of the sensor to measure true cyan for one scene patch, while another part of the sensor measures a greenish cyan for the same patch. This is the general version of the situation; the professional version is summarized in the optical and chromaticity uniformity specification.

[0444] The Chromabath procedure addresses these specifications. When a global color correction matrix is ​​applied to a conventionally produced sensor, and the resulting color image is out of tolerance, the sensor is typically considered junk. Instead, when the Chromabath process is applied to that sensor, thereby enabling local color adjustments on a pixel-by-pixel basis, the sensor's usefulness is not only preserved but even improved.

[0445] (Color correction) A "color correction matrix" is a well-known approach to transform a triplicate of raw red, green, and blue sensor data into better estimates of the X, Y, and Z chromaticity estimates defined within the CIE 1931 color standard. This color correction matrix is ​​"global" or at least "regional" in the sense that the same matrix is ​​applied to hundreds, thousands, and even millions of pixels. Aspects of this technique relate in part to manufacturing and / or sensor operation stage calibration procedures that first collect data, measure the degree to which pixel neighborhoods have some variation in their individual pixel spectral response functions, store such data for later use, and then use that data to compute a regional, local, neighborhood, or cell-level color correction kernel that modifies a classical color correction matrix, which is subsequently adjusted to a specific pixel within any given neighborhood pixel. Such calibration procedures can be equally applied to three-channel sensors such as Bayer RGB sensors, as well as 4- to N-channel sensors where N can range from several hundred.

[0446] A “matrix” is a simpler form of a more generally defined kernel. Kernel operations enable both linear and nonlinear local pixel value operations, while “matrices” are conventionally limited to linear operations. This specification teaches both linear, nonlinear, and machine learning-based kernel operations, which perform locally tuned color correction. Pixel behavior defects and non-uniformities can be mitigated and corrected primarily by this.

[0447] The manufacturing of color image sensors is difficult. Many sensors fail quality assurance tests and are destroyed. This consumes excess resources, incurs excess costs, and generates excess semiconductor waste.

[0448] Manufacturers of color image sensors apply numerous quality assurance (QA) standards to their finished devices. Some standards concern the uniformity of the pixel output signal in the presence of uniform illumination. It is not uncommon for some rows or columns of pixels to be "hotter" than others. Or, with respect to the pattern of pixels, for example, a checkerboard pattern will be hotter than others. CMOS imager manufacturers generally "bin-classify" individual sensors and thus categorize them into commercial grade systems. This grade system results in price inequality that can be imposed on any given sensor. Enormous amounts of research and development, engineering, and quality assurance budgets are allocated to increase the yield of higher quality level bins.

[0449] Some criteria relate to spectral sensitivity. Some "green" pixels may be more sensitive to red illumination than other "green" pixels, etc. The same applies to other color-filtered pixels.

[0450] These and other anomalies can be localized to small areas of the photosensor, or they can extend to the device, or they can extend to a large wafer in which dozens of photosensor devices are processed simultaneously. Spin coating, used to deposit a minute layer of material across the wafer for processing, is one of many different steps that can introduce non-uniformity.

[0451] According to this aspect of the technology, each pixel on an image sensor is associated with an individualized N-byte signature that represents the unique behavior of the pixel. This behavior includes, but is not limited to, the specific spectral radiometric sensing characteristics of the pixel. In an exemplary array, N can be 3 or 4 bytes. These signatures are generated during the Chromabath testing, calibration, and certification phases of the manufacturing / quality assurance process for both mass-produced and specialized image sensors.

[0452] In some embodiments, the Chromabath process illuminates a sensor (or a wafer with many sensors) using a very narrowband light source swept across the range of the electromagnetic spectrum to which the sensor is sensitive, where "very" can represent a monochromator optical output of 1 or 2 nanometer bandwidth. The response of each pixel is detected as a function of the illumination wavelength. The deviation of each pixel's attributes is determined and recorded as a function of the optical wavelength across the swept spectrum relative to the standard behavior of local or regional neighboring pixels, and / or the entire pixel sensor, and / or the entire pixel wafer. The sensitivity differences thus determined are encoded into N-byte compressed "signatures" which can be stored directly on a memory fabricated on the CMOS sensor substrate, or in some other (e.g., off-chip) manner, so that processes utilizing the image sensor output signal can have access to this N-byte signature data. During use of the sensor, pixel processing of the output image utilizes the information indicated by these N-byte signatures to improve the quality of the image output. For example, the output of "hot" pixels can be reduced, and the unique spectral response of a pixel can be taken into consideration when rendering or analyzing data from a sensor. Machine learning and AI applications can also use these N-byte signatures as an additional dimension of "feature vectors" employed during the training, testing, and use of neural networks and related applications.

[0453] A series of individually illuminated, medium-narrowband LEDs can also be used instead of a monochromator for the actual measurement of the N-byte signature of a pixel. The practical advantages of using a series of LEDs are that they are generally less expensive than a monochromator, and the so-called "shape factor" of placing the LEDs in close proximity to the wafer-scale sensor is better. As suggested by the commercial product Gamma Scientific's RS-7-4 Wafer Probe Illuminator, the group of LEDs can be placed directly above the wafer.

[0454] As shown, global sensor uniformity of radiometric behavior is one of several manufacturing tolerance criteria used to determine whether an individual sensor passes or fails quality assurance testing. Providing pixel-based N-byte signatures that quantify pixel-by-pixel variability and other pixel non-uniformity of radiometric behavior allows for relaxed manufacturing and quality assurance tolerances, as such non-uniformity can be quantified and considered in the use of sensor image data. Relaxing these tolerances increases manufacturing yield and reduces the cost per sensor. Sensors that were previously rejected and destroyed instead pass quality testing. Furthermore, such sensors yield better imaging results than prior art sensors, as the N-byte signature data can mitigate otherwise acceptable minute variability per pixel, thus passing stricter acceptance criteria.

[0455] In certain scenarios, sensor manufacturers identify the quality assurance criteria that most frequently result in failure. Some of these may be less mitigated by N-byte signature information, such as sensors that are completely non-functional or have some functions disabled, or sensors with internal short circuits or open circuits. However, the majority of QA failures stem from performance defects, such as output uniformity metrics or pixel spectral sensitivity metrics, exceeding the permitted threshold by a few percent. N-byte signature data is therefore used to convey the data (or an index to the data stored in any location), thereby potentially mitigating these peculiarities. Often, it may also be that connected pairs of pixels, connected triplets, etc., are either "non-functional" or otherwise deviate from specific performance parameters. Here, N-byte pixel characterization can also be a useful mitigating factor, transforming sensors classified into lower bins that trade at lower prices into sensors classified into higher bins that trade at higher market prices.

[0456] In a further aspect, one embodiment of the present technology employs a non-coordinated photosensor array and a color filter array, as detailed herein. With such an array, for example, with N=9 different color filtering elements, the image sensor has M>N pixels with different spectral response characteristics.

[0457] It should be understood that the embodiments described above, along with other features described in the discussion entitled “Embodiments employing near-infrared light,” can employ filters having a strong response above 650 nm. This makes it possible to obtain data from which information concerning both visible light and NIR can be extracted. Similarly, it should be understood that the embodiments described can utilize filters and filter cell arrays detailed further in this specification.

[0458] (General image solution for arbitrary spectral profile pixel neighborhoods) In some embodiments, the neighborhood of the pixel spectral function (profile) does not necessarily have to employ a fixed, repeating cell pattern such as a 2x2 Bayer (RGGB) cell. An example is a spatially variable color filter array, as detailed above. The following discussion addresses how the data generated by these non-repeating pixel neighborhoods can be transformed into an image solution.

[0459] In many operations that attempt to map pixel data to output data, it is desirable to first determine the luminance (luma) image values ​​corresponding to the pixel data. One way to do this is to use N × N linear kernel operators. For a 2 × 2 filter cell, N may be 6. For larger filter cells, such as the 3 × 3 cell detailed above, N may be 7.

[0460] A different kernel is defined for each pixel that is colored differently, along with a given neighborhood surrounding the pixel color. Thus, in the red, green, blue, and yellow filter cells discussed above, there are four different 6x6 kernels, each centered on a distinct pixel that is colored differently. In a cell employing nine different color filters, there would be nine different 7x7 kernels. Considering the latter case, when pixel data is generated, it is multiplied by all 49 kernel values ​​associated with its particular pixel color (or its specific neighborhood if the neighborhoods are not all the same), and the 49 results are stored in an accumulator array centered on the pixel and extending through a 7x7 pixel neighborhood centered on the target pixel. All pixels are processed in this way, and the accumulated values ​​constitute a complete lumen image.

[0461] To obtain a color mapping from pixel output values ​​to a desired output color (spectral function solution), the first step may be to parameterize the spectral response profile of each differently colored pixel. This parameterization is preferably accurate enough to "fit" the experimentally measured pixel profile within a few percent points of the pixel's peak response level in the spectrum. Figure 38 illustrates an example of a windowed Fourier series of the function defined here over intervals of 350–800 nm, which can fit both (a) the pixel spectral response profile and (b) the pixel spectral function solution. In some cases, the function can also be weighted by the quantum efficiency of the photosensor as a function of wavelength. Each term of the Fourier series is associated with a corresponding weighting coefficient, and the weighted function is then summed up, as is well known in other applications employing parameter fitting of functions.

[0462] Note that the Fourier series does not have a function that is not zero-mean. This indicates that the initial solution for the image should first determine the image brightness using the procedure described above, before determining the pixel spectral function solution. The brightness values ​​should be removed from the pixel values ​​before the spectral solution function in Figure 38 is used.

[0463] Here, there are two sets of data that should be transformed into a vector of Fourier coefficients. One is the spectral sensitivity profile of each pixel. This is primarily a function of the filter and photosensor quantum efficiency, but can also include lens effects, scattering, etc. The function defined by it is constrained by the raw spectral response of the photosensor (its irradiance profile). This is a well-known technique in function fitting, where coefficients are determined to best match the sum of weighted Fourier functions to the experimental measurement of the spectral sensitivity of each pixel.

[0464] The same method is applied to spectral profile solutions. True value data is used for this method; for example, a set of reference scene images is collected without sensor filtering, but illuminated with 46 narrow wavelengths of light in the 350, 360, 370...800 nm range, thereby determining the spectral scene reflectance for each image pixel at each of these wavelengths. The same scene is also imaged using a target color filter array.

[0465] Once the pixel spectral sensitivity profile function and the spectral profile solution function are parameterized, it is then possible to derive a set of linear equations that match the obtained pixel data to the solution, one such set of linear equations for each differently colored pixel.

[0466] In some embodiments, demosaicing is ignored, and the procedure determines nine different spectral function values ​​for each pixel. This can be done via interpolation. As a trade-off, the spatial sampling distance of each spectral channel is one-third more accurate in each direction than a sensor where all nine channels are actually present in each pixel.

[0467] In other embodiments, further demosaicing operations are employed. Such operations may follow the teachings described, for example, in Sadeghipoor, et al, A novel compressive sensing approach to simultaneously acquire color and near-infrared images on a single sensor, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1646-1650, Park, et al, Visible and near-infrared image separation from CMYG color filter array based sensor, 2016 IEEE International ELMAR Symposium, pp. 209-212, and Teranaka, et al, Single-sensor RGB and NIR image acquisition: toward optimal performance by taking account of CFA pattern, demosaicing, and color correction, Electronic Imaging, 2016(18), pp. 1-6. These documents are incorporated herein by reference as a whole.

[0468] Alternatively, a neural network approach using Fourier vectorization can be employed instead of the linear solution described above. This involves modeling millions of scenes and training a per-pixel neural network solution that will be associated with a unique pixel. The neural network typically yields a more compact solution than the linear approach, perhaps requiring only 10% of the data storage, for example.

[0469] While Fourier series are used in the aforementioned array, other polynomials such as Taylor polynomials, Chebyshev polynomials, and the Legendre polynomial family can also be used as substitutes.

[0470] (Nearby color distortion correction kernel) According to one embodiment, local color adjustment is performed by applying a neighborhood-specific color correction kernel (color correction matrix or CCM) to an array of pixel values ​​surrounding each target pixel.

[0471] The color correction matrix (kernel) can be traced back to the Bayer sensor. What began as a complex mathematical technique has now become mainstream, especially in hardware and firmware implementations.

[0472] In one known prior art, an image sensor is used to capture an image of a color test chart having multiple printed patches of known colors under known lighting. The captured image is stored in an m×n×3 array, where m is the number of rows in the sensor, n is the number of columns, and 3 represents the number of different output colors. Ideally, the captured image would be identical to another m×n×3 array containing reference data corresponding to the correct colors, but in reality, this is not the case. To transform the captured pixel array to more closely approximate the ideal reference pixel array, the captured image array is multiplied by a 3×3×3 color correction matrix, the coefficients of which are adjusted so that the product of such multiplications yields the best least squares fit to the reference array. Subsequently, when an arbitrary image is captured, its image data is similarly processed by multiplication using the same 3×3×3 color correction matrix to yield a color-corrected counterpart to the captured image.

[0473] According to this aspect of the technology, instead of applying a single color correction matrix to image pixels across the captured image, various locally fitted color correction matrices are employed. The mathematical version of this is the global color correction matrix, which can be simplified as follows: [ka] (In the formula, A is simply a general-purpose letter representing various mathematical formulas, some of which are accompanied by X, Y, and Z, others by R, G, and B, and there are also other hybrids of these.) Converting this to a localized adaptation form results in the following: [ka] In the formula, "4BN" maps to 4-byte data associated with a particular pixel neighborhood. These are Chromabath-derived pixel-map pixel signatures. While 4 bytes is a nominal number, other data sizes can, of course, be used. Neighborhoods can be of any size, such as 2x2 pixels, 4x4 pixels, or 6x6 pixels. Therefore, the locally fitted color correction matrix value is a function of many parameters, including its "index" of the location of the 3x3 color correction matrix itself (the local color correction data, like the 4-byte pixel signature data detailed earlier, is stored in memory on a suitable device for this purpose).

[0474] The formal definitions of these new functions f in Equation XX2 should be treated with the same level of care and attention as the definition and measurement of the 4-byte signature itself. In one embodiment, these functions f in Equation XX2 are derived experimentally using machine learning training, but their "classical" forms can be described herein.

[0475] One form of the 4-byte pixel signature assumes encoding a “mixture” (cross-contamination) of masked pigments, e.g., trace amounts of red and green pigments present in a nominal blue pixel, and the same situation for nominal red and nominal green. If we cite this “encoding scheme” as an example for a method to construct these f-functions for a locally adaptive color correction matrix, an additional transformation layer can be built upon it. Again, experimentally (and via simulation), the mapping can be solved by combining a “true” dataset, matched to millions of instances of 4-byte neighbor values ​​imaging the full color gamut of the colors, and learning the answers (the f-functions themselves). This is a brute-force approach, but it is acceptable in the sense that training these functions can be done using as much time and computational resources as needed. The final result yields linear multiplicative coefficients, which define each f-function. [ka] For every nine local adaptive color correction values, there are perhaps up to 20 or 30 c11 coefficients. This is also referred to as solving a system of L1 or L2 linear equations. Again, one method is to train these local adaptive color correction functions through machine learning and any number of choices, matching millions or billions of true instance examples to corresponding millions or billions of 4-byte neighboring pixel signature values, observing millions of color patches and color patterns across the entire color gamut.

[0476] When the sensor is subsequently used, the conventional global color correction step in image processing is replaced by this local adaptive deformation, in which case the additional data feed of the 4-byte pixel signature itself exists. Generally, these 4-byte pixel signatures are fixed and lead to similarly fixed local adaptive color correction matrices. From a mathematical standpoint, these new local adaptive matrices are fixed but spatially deformable. Therefore, they would employ a per-pixel memory allocation, as well as the required 4-byte / pixel signature. However, in this operation, what is computationally relevant is the color correction matrix map, and no longer the 4-byte signature itself. Therefore, computationally, there is also the option of not needing to prepare the 4-byte signature values ​​themselves that are available for image processing. If only color correction is required, then in fact, 4 bytes are not needed, but if AI / machine learning image processing is being performed, it would be undesirable to discard these informationally rich 4-byte signature values. Estimating the memory requirements for local adaptive color matrices is difficult and depends on many factors. However, it is not unreasonable to estimate 10 bytes / pixel as a general upper limit.

[0477] These matrices are then used in kernel-based processing of individual pixel outputs in the same manner as conventional kernel-based processing for deriving pixel data.

[0478] An illustrative excerpt from the local adaptive color correction matrix is ​​shown below. [Table 24]

[0479] These nine numbers vary locally as a function of the local 4-byte pixel signature. These numbers may vary slightly per pixel, or per kernel, in the context of the demosaicing algorithm. This data can also be compressed into 4-6 byte differential encodings, which will remain fixed until further calibration measurements may modify them.

[0480] Figure 39 goes further, revealing (and exaggerating) how these color correction matrices (which may be called CCMs) are functions of local position. The function that converts the 4-byte signature to these numbers is as discussed earlier. The example here has one CCM / 2×2 pixel Bayer cell, but other arrays are obviously also possible (with overlap, one CCM / each N×N region, and applying the CCM to pixels rather than cells). It can be seen that the CCMs are locally adjusted for slight defects in the per-pixel spectral function of the underlying pixel type. Regional and global scale corrections can be built into these local CCMs, for example, if a spin coating across the chip is at a unit thickness at one corner of the sensor and at 0.97 unit thickness at another corner, this global scale non-uniformity can still be corrected by gradually changing the local values ​​as appropriate.

[0481] (ShadowChrome) This section expands upon the previous section, "Image Data Restoration."

[0482] Prior-tech camera systems do not function well when discriminating colors at low light levels. According to aspects of this technology detailed below, the image sensor generates data that allows the chromaticity of pixels to be accurately determined even at very low light levels. Such technology is sometimes referred to as ShadowChrome.

[0483] The accepted theory of optical measurement using CMOS sensor photosites (pixels) is that incident light generates so-called photoelectrons at discrete pixels. The number of photoelectrons collected is discrete and integer, and does not have any numbers other than 0, 1, 2, 3, etc. For existing commercial-grade CMOS sensors, directly measuring these discrete numbers is impossible due to so-called read noise from the amplifier + analog / digital conversion array. There is also so-called shot noise, an industrial term used to explain the Poisson statistics of discrete (integer) measurement arrays. Furthermore, an additional factor that should often be considered is the pixel-by-pixel variation in the individual measurement behavior, which is often referred to as fixed pattern noise. ShadowChrome considers the following items, namely (a) the sparseness of photoelectrons, and (b) the typical approximately 2e for modern CMOS image sensors. - ~5e - This addresses all four issues: (c) reading noise, (d) significant shot noise due to rarity, and (c) pixel non-uniform measurement behavior.

[0484] ShadowChrome works well for typical brightness scene imaging, where pixels receive hundreds, possibly thousands, of photoelectrons generated within each image snap. However, it is actually designed for very low light levels where the so-called signal-to-noise ratio (SNR) drops to 10 or even below 1. Numerically, a signal-to-noise ratio of 1 for a given digital image sensor is simply defined as the average number of photoelectrons generated within a pixel being exactly equal to the read noise specification.

[0485] In fact, a signal-to-noise ratio (SNR) of 1 is precisely what guides many of the various parameter choices presented within the ShadowChrome solution framework. Commercial imaging requires "seeing color" even when the luminous shape of an object is barely visible at extremely low light levels, and this is a driving force behind the development of ShadowChrome. One example of hundreds of applications where dark-scene color is critical is ADAS cameras (Automated Driver Assistance Systems), where stop signs and lane markers (as well as roadside objects) under starlight are required to exhibit their reddish and yellowish hues, respectively, as soon as their shapes become apparent.

[0486] Prior art digital imaging systems define so-called "dark frames," in which every pixel has an average digital number (DN) value, typically recorded after hundreds or even thousands of "dark" images have been recorded. The values ​​resulting from these dark frames are most typically fractional rather than discrete integers, due to the many measurement frames used to derive the average.

[0487] In ShadowChrome, the explicit data structure serves as a scaffolding mechanism, which then derives these dark frames and logs the median DN point for each pixel, as opposed to their mean value. ShadowChrome reduces the low-light color measurement problem to a vast network of coin tosses or pseudo-50-50 decisions, which leads to chromaticity-hue-angle and saturation estimation. Attempts to achieve a "pseudo-50-50" decision begin by applying their principles to the non-light behavior of each pixel. These per-pixel median encoded dark level values ​​can, for convenience, use exactly the same fractional form as their predecessors, the "dark frames". In other words, a median fractional value, e.g., 20.17, indicates that slightly less than half of a pixel's dark-framed DN will be below DN 20, and slightly less than half of those values ​​will be above DN 20. The remaining values ​​would correspond to DN=20. Next, when the fractional value is lower than 0.5, i.e., 20.17, this indicates that the integrated histogram values ​​of 21 and above are slightly greater than the integrated values ​​of 19 and below. These minor details can become problematic when attempting to measure color at SNRs of 1 and even lower.

[0488] Therefore, a possible preliminary step for using ShadowChrome is the dark frame calibration of the sensor, which is done explicitly in the form of median pixel values. Chromabath (discussed elsewhere) has this dark level histogram creation built into its process. For brevity, therefore, ShadowChrome can utilize the Chromabath data results.

[0489] A second possible preliminary step for ShadowChrome involves the use of either a calibrated white patch, or, in situations where such a patch is unavailable, an equivalent "scene" where access to a "colorless" object exists. As a last resort when no scene is available at all, "theoretical" sensor specification data, such as sensor spectral sensitivity curves for each pixel type (e.g., nine types in a 3x3 cell array), can still be used. The aim of this second preliminary step is to track the so-called "gray gain" in the Chromabath scheme, either A) a given spectral type, e.g., all nine pixels, or B) the gray gain of each pixel. The latter is preferred to achieve maximum color measurement performance, but the former is equally acceptable, as CMOS sensors typically behave well with uniformity within 1% at "general-purpose gain". This 1% uniformity represents a diminishing return for the class as a whole, as it deals with very low light level imaging, often involving single-digit photoelectron counts. In other words, while pixel spectral types as a class have very different Gray gains when one type is compared to another, within a given spectral type, the Gray gains are virtually identical.

[0490] Therefore, measuring the gray gain of pixel spectral types can be useful to maximize ShadowChrome's color measurement capabilities. Chromaticity and color are measured exclusively by the signal ratio characteristics between pixels of different spectral types, and thus these signals preferably have some kind of zero-point reference with respect to color, which is what becomes these gray gain values. They are, in their raw form, "colorless" baseline signals. Again, the individual values ​​of these spectral types can be derived by taking images of tens to hundreds of white patches and removing the influence of the dark levels discussed above, and then the gray gain of the spectral type appears as the average value from these many images. While the gray gain values ​​themselves can be arbitrarily defined and then normalized with respect to each other, this disclosure uses the convention that the highest gray gain value belonging to only one of the spectral pixel types will be assigned a value of 1.0, and all others will be slightly lower than 1.0 but at an appropriate ratio. The so-called "white patch equalization" between pixel spectral types would be assumed to be avoided, preferably, if possible, if gray gain values ​​below 0.8 (these white patch and gray gain data would be recognized as indicators of pixel efficiency in some sense).

[0491] The use of dark median and, separately, these gray gains will become clearer in the further explanation of ShadowChrome below.

[0492] (Specific example) Consider an embodiment involving an image sensor comprising a 3x3 cell of nine pixels, some or all of which have different spectral responses (i.e., they are of different types). Filters and filter cells having the attributes detailed above are embodiments. These nine pixels can be labeled as first- to ninth pixels (or synonymously, pixel AI) according to some arbitrary mapping of such labels to the nine pixel positions.

[0493] Referring to Figure 40, useful information for indicating the chromaticity of a central pixel is generated by comparing a scene value associated with one pixel in the cell, referred to as the reference pixel, with scene values ​​associated with different pixels in the cell, referred to as the ornate pixels. (The term “scene value” is sometimes used to refer to a value associated with a pixel when the image sensor is illuminated by light from a scene. A pixel’s scene value can be its raw analog or digital output value, but other values ​​can be used as well. The term “digital number or DN” is also commonly used to represent the sensed data of scene brightness.) In Figure 40, pixel A is the reference pixel, pixel B is the ornate pixel, and the comparison is indicated by arrow 401 between these pixels.

[0494] This scene value comparison operation is performed between different pairs of pixels within a cell. For example, the comparison can be performed between pixel A and pixel C. This is also shown in Figure 40 by arrow 402, where pixel A still acts as the reference pixel and pixel C acts as the second coordinated pixel.

[0495] Query data is formed based on the results of these comparisons and provided to the color reconstruction module 411 of the image sensing system 410 as input data (Figure 41), from which such a module determines the chromaticity information to be assigned to the pixels in the cell (typically the central pixel of the cell). In some embodiments, the color reconstruction module operates only on the query data as input and does not act on the pixel data itself (e.g., the central pixel).

[0496] These two pixel-pair comparison data are useful in indicating the color (i.e., chromaticity) of the central pixel within a cell, despite the fact that at least one of the comparisons, in this case both, does not involve the central pixel (in this case, pixel E). That is, the reference pixel and the two ornate pixels do not necessarily have to include the central pixel.

[0497] Such pixel-to-pixel comparison data is generated by a hardware network module, preferably fabricated on the same semiconductor substrate as the image sensor, and a representation of such data (e.g., as a vector data structure) is output by such network as query data. This query data is applied to a subsequent process (typically implemented as an additional hardware network module, either on the same substrate or on an auxiliary chip), which in part assigns output color information about the central pixel based on such data (this module may be referred to as a demosaicing module or a color reconstruction module). Such a hardware array is shown in Figure 41, where the dashed lines indicate a common semiconductor substrate containing the modules described (for commercial applications, the raw data from the pixels can be communicated to either a camera module or a cloud-based processor to perform the functions of the color reconstruction module, as desired to keep the sensor cost to a minimum).

[0498] The quality of the output color information will ultimately depend on the richness of the query information. Therefore, query information based on only two pixel-to-pixel comparisons (reference and first ornate; reference and second ornate) is rarely used. In many embodiments, further comparison operations are performed between scene values ​​associated with the reference pixel and scene values ​​associated with yet other pixels in the cell, resulting in other pixel-to-pixel data. If the reference pixel is referred to as the first pixel, then eight pixel-to-pixel comparison data can be generated, accompanied by comparisons with the second to ninth (ornate) pixels. The first two pixel-to-pixel comparison data are produced as described above, namely by comparing the scene value associated with the first pixel with the scene data associated with the second pixel (i.e., a [1,2] comparison, where the former number indicates the reference pixel and the latter number indicates the ornate pixel), and by comparing the scene value associated with the first pixel with the scene data associated with the third pixel (i.e., a [1,3] comparison). Similar comparisons are similarly made by comparing the scene value associated with the first pixel with the scene data associated with the fourth-to-ninth pixels in the cell, respectively, resulting in [1,4]-[1,9] pixel-to-pixel data. Figure 42 illustrates these further comparisons, each with pixel A as the reference pixel. Again, all representations of such pixel-to-pixel data are output as query data by the hardware network.

[0499] The use of the word "compare" and its various forms, such as "comparing" and "comparison," is used for various mathematical options of how the comparison is performed precisely. One form of comparison is the sigmoid function comparison (see Wikipedia for details). In the case of the sigmoid function limit, it is a simple greater-than / less-than comparison of two distinct values. In the case of integers DN, the equality case is also a case that actually holds, often leading to a null result or the assignment of the value 0. The limit values ​​of the sigmoid are the numbers 1 and -1, both in this disclosure and more generally.

[0500] Up until now, the query data has been accompanied by only a single reference pixel. However, just as the first pixel can be compared to eight other pixels, i.e., the second through ninth pixels (or more precisely, the scene values ​​associated with such pixels are compared), similarly, the second pixel can be compared to seven other pixels, i.e., the third through ninth pixels. These comparisons are illustrated in Figure 43. Similarly, the third pixel can be compared to six other pixels, i.e., the fourth through ninth pixels. Likewise, the fourth pixel can be compared to five other pixels, i.e., the fifth (center) through ninth pixels. And so on, until the eighth pixel is compared to only one pixel, i.e., the ninth pixel.

[0501] In summary, these comparisons yield a total of 36 pixel-for-pixel data points. This value is the N-summatorial value, as mentioned above, where N is the number of pixels in the cell (i.e., 9). This value is also the (N-1)th triangular number. This can be calculated as N(N-1) / 2, which is equal to 36 for N=9.

[0502] In other words, if there are N pixels in a cell, the process described compares the scene value associated with the Q-th pixel in the cell with the scene value associated with the R-th pixel in the cell, and updates the Q-th to R-th ([Q,R]) pixel pair data for each Q from 1 to N-1 and for each R from Q+1 to N.

[0503] The comparison result, comprising pixel-for-pixel data, can take different forms in different embodiments. In one embodiment, the comparison result is a count, which is incremented if the reference pixel scene value is greater than the ornate pixel scene value and decremented if the reference pixel scene value is less than the ornate pixel scene value (if the reference and ornate values ​​are equal, the comparison yields a result of zero). In the embodiment given above, a 36 comparison thus yields a 36-element vector, each element of which is either -1, 0, or 1. This may be referred to as a high / low comparison.

[0504] In another embodiment, the comparison result is the arithmetic difference between the two scene values ​​being compared. For example, if the scene value of the reference pixel is 25 and the scene value of the ornate pixel is 70, the comparison result (pixel vs. data) is -45. In such a case, the 36-element vector consists of 36 integers or real numbers (depending on whether the scene values ​​are integers or real numbers). This may be called an analog or difference-preserving comparison. Similarly, there are various forms of nonlinear "weighting" that can be applied to these comparisons, which often apply to machine learning implementations that do not have complete knowledge of the ultimately correct choice, allowing the data to be trained on large-scale "true value" based images to make selections. Likewise, the parameters of the sigmoid function itself can be machine learning tuned.

[0505] As described, the quality of the output color information will ultimately depend on the richness of the query information. The array described above generates query data by comparison within a single color filter array cell, but richer query information can be obtained by extending such comparisons beyond that single cell to include the pixel fields.

[0506] In most image sensors, the color filter array comprises tiled cells. That is, referring to the single cell discussed above as the first cell, there exist multiple further tiled cells in the neighborhood surrounding the first cell. Such further cells are adjacent to the first cell, or adjacent to other further cells adjacent to the first cell, and so on. Each of these further cells may replicate the first cell in its pattern and orientation of pixels. In such cases, pixels found at the same spatial location within each of the two cells are sometimes referred to as spatial counterparts or spatially corresponding (for example, the first pixel found in the upper left of the first cell is the spatial counterpart of the first pixel found in the upper left of the further cell).

[0507] In other embodiments, some or all of these further cells may have the same pattern of pixel types as the first cell but be oriented differently, for example, rotated 90 degrees to the right. Alternatively, some or all of these further cells may have different patterned pixel types but contain one or more types of pixels found in the first cell. In such cases, pixels of the same type found in each of the two cells are sometimes referred to as color (or type) counterparts or color (or type) correspondings (for example, a blue pixel found in the first cell is a color counterpart of a blue pixel found in a further cell).

[0508] (In the case of a further cell that is a duplicate of the first cell, all spatially corresponding pixels between the two cells are also pixels whose color corresponds to the corresponding pixel.)

[0509] According to some embodiments, the scene value of a pixel in a first cell is compared to the scene value of a spatial or color-corresponding pixel in a further cell.

[0510] To provide a specific, non-limiting embodiment, the scene value associated with a first pixel in a first cell is compared not only with the scene value of a second pixel in the first cell (as described above), but also with the scene value associated with a second pixel in one of the further cells. The previously referenced first-to-second ([1,2]) pixel-to-pixel data reflects the result of this comparison. This operation is repeated one or more times using a second pixel in one or more of the other further cells.

[0511] Such an operation is shown in Figure 44, which shows the first cell (i.e., the nine pixels outlined by a thick line in the center) within the local neighborhood of the duplicated cell. Here, pixel A in the first cell is the reference pixel. As previously explained, this is compared to the second pixel (B) in the first cell, as indicated by the short arrow. This reference pixel is also compared to the second pixels in the cells to the left and right of the first cell, as indicated by the longer arrow. All three of these comparisons contribute to updating the [1,2] pixel-versus-data. That is, if high / low comparison is employed, the [1,2] count is incremented each time the reference pixel scene value is greater than the assorted pixel scene value, and decremented each time the inverse occurs. If analog comparison is employed, the difference in scene values ​​between the reference pixel and the assorted pixel is accumulated as a cumulative sum.

[0512] Similarly, the scene value associated with the first pixel in the first cell is compared not only within the first cell but also with the scene value associated with the third pixel in one of the further cells. The previously referenced first-third ([1,3]) pixel-to-pixel data is modified to reflect the result of this comparison. This operation is repeated one or more times using the third pixel in one or more of the other further cells.

[0513] Such an operation is shown in Figure 45, which corresponds to Figure 44, but concerns the case of [1,3] pixel pairs.

[0514] The richer [1,2] and [1,3] pixel-pair data thus generated forms part of the query data.

[0515] Similarly, the first pixel (A) in the first cell can be compared to two or more fourth pixels in further cells, resulting in richer [1,4] pixel-versus-pixel data.

[0516] If, as shown here, there are N=9 pixels in each cell, such a comparison can be similarly extended to comparing the first pixel in the first cell with the 5th to the Nth pixel in further cells, thereby expanding the [1,5]-[1,N] pixel-versus-pixel data with this further information.

[0517] In this embodiment, the comparison can be carried out similarly, as in the previous embodiment, where the first pixel is compared with eight others, then the second pixel with seven others, and so on, until the eighth pixel is compared with one others. That is, the second pixel (B) in the first cell can be compared with the third through ninth pixels in several further cells, thereby expanding the comparison data used in the query data. Similarly, the third pixel (C) in the first cell can be compared with the fourth through ninth pixels in further cells, and so on.

[0518] As explained above, the scene value associated with a pixel in the first cell is compared against the scene values ​​associated with a second pixel in two further cells, namely the one to the left and the one to the right. However, a larger set of further cells may also be used. For example, instead of just the left and right further cells, eight further cells may be used in this way, namely the adjacent cells to the left, right, top, and bottom, as well as the cells with four adjacent corners. [1,2] The pixel data is therefore compared against a total of nine comparisons, namely the second pixel in the first cell and the second pixels in the eight adjacent cells. That is, the first reference pixel in the first cell is compared against the second aligned pixel in each of the 3x3 tiles of the cell that have the first cell at its center.

[0519] In a similar scheme, the comparison can be extended to the ornate pixels within a 5x5 tile of cells, with the first cell at its center. Each pixel-to-pixel data, such as [1,2], is therefore based on a comparison of 25. When high / low comparisons are employed, each pixel-to-pixel data can have values ​​ranging from -25 to +25. In many embodiments, each such data is shifted by 25 to make a non-negative value. For example, if the reference pixel for the pixel pair [1,2] is associated with a scene value of 150, and the 25 ornate pixels compared to it are associated with scene values ​​of 40 to 60, then the [1,2] pair data would accumulate to 50 (because in the case of 25, the reference value exceeds the ornate value with a shift of 25).

[0520] If analog comparison is employed, each pixel-to-pixel data can have a large value, depending on the cumulative sum of the scene value differences. For example, in the embodiment given above, the [1,2] pixel-to-pixel data would accumulate to approximately 2,500 (because in the 25 example, the baseline value exceeds the ornate value by approximately 100).

[0521] The two detailed comparisons, namely high / low and analog, are merely illustrative. Many other comparison operations can also be used. For example, the arithmetic difference between a reference value and an ornate value can be weighted according to the spatial distance between the pixels being compared, with greater distances being weighted less. Many other arrays will be recalled by those skilled in the art, assuming this disclosure. Similarly, machine learning applied to large training sets of images, as described above, can induce neural network implementations / weightings of these comparisons.

[0522] The scene values ​​associated with each pair of reference and ornate pixels can each be raw pixel values, either analog or digital. Alternatively, they can be processed values, such as data output by an image signal processor module that performs hot pixel correction or other adjustments on the raw pixel data. Furthermore, better color measurement output will be produced if each pixel is "corrected" by its own unique dark median, as described above. Thus, any comparison of raw data of one pixel with raw data of another pixel will also be accompanied by a dark median correction value for each pixel. In addition, the individual gray gain or type class gray gain (described above) of individual pixels can also be used to "adjust the luminance level" of the compared values ​​prior to the comparison operation itself. If photoelectron generation in most pixels is only at single-digit levels, such gray gain adjustment may be a fractional DN value that is added to the value prior to the comparison operation, and the dark median calculation has already produced a fractional DN value. Therefore, both correction operations are involved in producing a comparison of two such corrected fractional DNs. At an SNR level of 1, these finely detailed fractional values ​​can be important for "seeing true color" instead of the randomized chaotic set of colors caused solely by sensor reading noise, when applied to what would typically be well over 100 comparison operations affecting the final chromaticity value output of any given pixel.

[0523] Furthermore, the scene value associated with the target pixel can also be the mean or median, calculated using all pixels of the same type within a 3x3 or 5x5 neighborhood centered on the target pixel (when forming the mean or median, pixels farther from the target pixel may be weighted less than nearby pixels).

[0524] The values ​​associated with a pixel should not be assumed to be raw values ​​unless otherwise stated.

[0525] In some embodiments, a reference pixel is associated with one type of scene value (e.g., mean value), while an ordained pixel is associated with another type of scene value (e.g., raw value).

[0526] The preceding discussion details the procedure for generating query data and determining color information for a single pixel within a cell of N pixels, i.e., the central pixel within the cell. To obtain color information for different pixels within the cell, the process is repeated. However, the cell boundaries are shifted, the cell is reconstructed, and this different pixel becomes the central pixel.

[0527] Assuming a field of millions of pixels within an image sensor, the boundaries of repeating tiled cells are arbitrary. For example, a Bayer cell, scanned from the top left and then downwards, could be considered a group of red / green / green / blue, or green / red / blue / green, or green / blue / red / green, or blue / green / green / red. In connection with this, the nine pixels of the illustrative cell in Figure 40 can be reframed in nine ways, as shown in Figure 47. Different sets of query data are generated for each of these frames based on different sets of comparison data and used to determine the color information for pixels E, F, D, H, I, G, B, C, and A.

[0528] When representing query data, the applicant employs a sorting convention that begins with the top-left pixel of the framed cell as the reference pixel, and compares it with the ornate pixels to the right and below (within the first cell, optionally, and in further neighboring cells), yielding the first eight elements of the 36 pixel-pairs data. The process continues by using the top-center pixel of the framed cell as the reference pixel, and comparing it with the ornate pixels to the right and below, yielding the next seven elements of the 36 pixel-pairs data. This continues through subsequent pixels. Thus, if the pixels in the framed cell are numbered 1-9, starting from the top-left (across and then down), the sets of pixel-pairs data are ordered as follows: [ka]

[0529] It will be recognized that these 36 pixel pairings represent only half of the ordered pairings that are possible. For example, "Pixel 2 compared to Pixel 1" does not exist.

[0530] When the raw value is used for both the reference and ornate pixels, comparing pixel 2 to pixel 1 does not provide any new information; that is, it is simply the negative of pixel 1 compared to pixel 2. However, if the value associated with the reference pixel is determined by mean or median calculation, and the value associated with the ornate pixel is the raw value (or vice versa), the comparison between pixel 2 and 1 may yield a different result than the comparison between pixel 1 and 2. In such cases, a vector of 72 elements may be used based on the comparison between any ordered pixel pairs that could be considered. However, such differences are usually insignificant, and therefore a smaller number of elements is typically used (i.e., 36) even when the reference and ornate scene values ​​are not determined in the same manner. Therefore, in practice, the applicant proposes combining somewhat independent low-high pairing results and high-low pairing results, keeping the pairing count itself at 36 instead of 72 (sometimes using the phrase “order-independent pixel pairing” to refer to arrays where order is not important; thus, the count for all order-independent pixel pairings in a 9-pixel cell is 36).

[0531] If a simple high / low form comparison is used (with a shift to yield a positive value), and the comparison is performed across a 5x5 array of cells (with only a few comparisons, indicated by arrows, as shown in Figure 46), then query data for a single pixel at the center of the framed cells may take the following forms: [ka]

[0532] Such a data structure would be perceived as having a multi-symbol code that represents the result of determining which of a pair of pixels is associated with a larger scene value.

[0533] This is 50 36 It is an extremely rich feature vector in the sense that it has a number of possible forms (i.e., a decimal number of the order of 61 digits). The universe of latent colors is significantly smaller. If a color is considered to be in the RGB space, and each of the three values ​​is represented by 16 bits of data, then 2 48 Only the possible colors (i.e., decimal numbers of the order of 14 digits) exist. In the 36-dimensional universe described by the query data, the universe of possible colors forms an extremely sparse mapping. Each color of light reflected from a certain scene and incident on a pixel E in the array of cells in Figure 46 falls into a volume in the 36-dimensional space of the query data that is unique to that color. Once the pixel's response to the incident light is characterized in this 36-dimensional space, the volume it occupies represents that color. The reference data serves to provide a mapping between the volume in the 36-dimensional space and the corresponding color of pixel E.

[0534] (Practically speaking, it is well known that the human visual system itself has limited ability to distinguish colors, and generally corresponds to a two-dimensional chromaticity surface, as exemplified by the 1931 CIE chromaticity diagram. Furthermore, it is accepted that the so-called Macadam ellipse can map the space of discriminable colors, rounding these mathematically astronomical numbers down to at most a few thousand as addressable "perceptible" colors. The important point is that this mathematical richness ultimately becomes merely a rough estimation of directionality in the hue angle as the sensor's signal-to-noise ratio approaches and then decreases.)

[0535] One method for generating reference data involves employing a sensor to image a color chart (e.g., a Gretag color chart) with patches of known colors under known illumination (e.g., D65), performing the comparison detailed above on the resulting pixel data, and obtaining a 36D reference data vector. That is, the reference comparison data is generated in the same format as the query data, but the scene color is known, not unknown. A given patch of reference scene color will generate various data vectors depending on various random factors involved, including random variations in patch color, random variations in illumination intensity, sensor shot noise, sensor reading noise, and variations in photosensor sensitivity between pixels. Such disturbances play a role in spreading the vector representation of known colors into a distribution of data vectors. The 36D volume containing such vectors defines a space associated with known colors.

[0536] Once such reference data vectors, associated with known colors, are captured, they are stored and can be used as a basis for comparison with 36D query data associated with target pixels E that capture light of unknown colors. The task is to find the 36D vector data within the reference data that best matches the query vector. The known colors associated with the best-matching reference vector are then assigned as output color information for that pixel.

[0537] This is a method in string matching (or pattern matching), and numerous techniques can be employed. One method calculates the Hamming distance between the query data and the reference data vector to determine the closest match. Another similar method employs the Levenshtein distance. Yet another performs a dot product operation between the query data and the reference data vector to determine the closest match. Many other suitable methods are detailed in the Approximate String Matching section of "Encyclopedia of Algorithms" Springer Link, 2016 (ISBN 9781493928637).

[0538] Instead of serving as a catalog of reference data from which the best match can be determined, a reference vector, labeled with a corresponding color, can be used to train a convolutional neural network. The network's parameters and weights are iteratively adjusted during training, for example, by an inverse gradient descent process, to configure the network to respond to input query vectors corresponding to pixels E by providing output data indicating the color for that pixel (such parameters / weights can then be stored as reference data).

[0539] In these and other implementations, colors can be defined within a desired color space. Most commonly, CIE coordinate data for X,Y is used, but other color spaces, including sRGB, L*a*b*, and hue angle (L*c*h), can also be used.

[0540] Color charts provide only a limited number of known colors. Another method for generating reference data is to employ reliable multispectral images. One preferred set of multispectral images is the so-called CAVE dataset, published by Columbia University. The set consists of 32 scenes, each represented by 16-bit reflectance data with full spectral resolution across 10 nm increments from 400 nm to 700 nm (a total of 31 bandwidths). This set of data is available at www <dot>cs <dot>columbia <dot>edu / CAVE / databases / multispectral / and also the corresponding web <dot>archive <dot>It is available on the org page.

[0541] This approach does not utilize the physical image sensor itself to sense the scene. However, the behavior of the image sensor can be modeled by measuring, for example, the spectral transmittance function of its different filtered pixels, its spectral transmittance variation between filters of the same type, its shot noise, its readout noise, its pixel amplitude variation, etc. Such parameters characterizing the sensor behavior are applied to the exposed image, and the image sensor can generate thousands or more sets of simulated pixel data that can be generated (and disturbed) from a given scene in the Monte Carlo scheme. Each such different frame of pixel data is analyzed to determine a 36D vector associated with each "E" pixel in the frame. The color of each such pixel is known (in terms of the exposed amplitude in each of the 31 spectral bands) and can be converted to a desired color space. This reference data, which associates the 36D reference vector with the known color, is then used in one of the forms detailed above to output color information in response to input query data.

[0542] The above discussion should be understood as relating to assigning color information to a single pixel E within a cell. The reference data discussed above is specific to that pixel E.

[0543] The same procedure is similarly performed to expand and utilize the reference data for the other eight pixels within the cell. Therefore, a total of nine sets of reference data are typically required.

[0544] As detailed in the illustrative embodiments, query data is nominally invariant to changes in brightness (i.e., as the scene becomes darker, all pixels should simultaneously produce smaller output signals, keeping the inter-pixel comparison results invariant). However, the applicant has found that this is not always the case. This is particularly noticeable at very dark brightness levels, for example, where the signal-to-noise ratio is less than 10:1, 4:1, or 2:1. Therefore, in some embodiments, the applicant generates multiple sets of reference data for every nine pixels in a cell, each corresponding to a different range of brightness levels (brightness can be determined on a local neighborhood basis, such as the average raw pixel value across a 5x5, 9x9, or 15x15 pixel field, or a 3x3, 5x5, or 10x10 pixel cell field).

[0545] Therefore, when assigning a color to a pixel, the first step is often to determine the brightness of the area surrounding the pixel, and then to select a set of reference data or neural network parameters / weights adjusted to that brightness (in the case of a neural network, the training data can consist of a triplicate of information: a pixel-for-data vector, local brightness, and known color. When used, the network is provided with the query data vector and the measured local brightness as input and outputs its estimate of the corresponding color information).

[0546] In some embodiments, there may be only two brightness ranges, e.g., dark and non-dark (depending on a signal-to-noise ratio less than or greater than 5:1, etc.). In other embodiments, there may be two, five, dozens or more different brightness ranges, each triggering a different mapping between query data and output color. One range may relate to a signal-to-noise ratio less than 1.5. Another may be used when the SNR is less than 3 but at least 1.5. Yet another may be used when the SNR is less than 5 but at least 3. And so on.

[0547] A vector of 36 elements is one of many possible representations of this pixel comparison information. Symmetric group theory in linear algebra provides many alternative representations. In particular, pixel pairings within a cell of nine pixels can be represented using an S8 algebra (i.e., a bijective group from a set of nine pixels). Any of these alternative representations can be stored in a corresponding data structure and used in embodiments of this technique.

[0548] One alternative representation focuses on color information output, directly defined by hue angles and saturation space. An independent mapping exists between the hue angles of points in a given scene and how those hue angles are mapped as single scalar values ​​in and out of a 36-dimensional pixel-pair comparison space. One approach to measuring these direct hue angles is to find a hyperplane in 36-dimensional space using cosine and sine functions acting on the hue angle, which optimizes the fit between angles in 36-dimensional space and x and y chromaticity hue angles in CIE chromaticity space (or a and b vectors in Lab color space, or some other color space where color is separated from luminance).

[0549] While the above disclosure focuses on 3x3 pixel cells, cells of other dimensions (and non-square shapes) can also be used. If a pixel cell has an even number of rows or columns, such as 2x2 or 2x3, an interpolation method can be employed to fit the above array (assuming a "center" pixel) into such a cell.

[0550] Alternatively, such cells can be reframed as larger cells, i.e., having a central pixel. An example is a classic Bayer cell. This cell can be reframed into a 3x3 cell, for example, as indicated by the thick outline in Figure 48. This pattern can therefore be seen as a tile of four different 3x3 cells. One cell (the bolded cell in the upper left) contains 5 green, 2 red, and 2 blue. Another cell (on the right) contains 4 green, 4 blue, and 1 red. A third cell (the bolded cell in the lower left) contains 4 green, 4 red, and 1 blue. And a fourth cell again contains 5 green, 2 red, and 2 blue. Thus, it can be seen that cells can contain two or more (sometimes four or more than four) pixels of the same type. Furthermore, it can be recognized that the cells are different, but their component colors are identical.

[0551] The method detailed above can be used here. For example, in each 3x3 pixel cell, a vector of 36 {-1, 0, +1} elements is formed and used to assign a color to the center pixel of the cell.

[0552] Other methods detailed above can also be adapted here. For example, comparisons that extend beyond a single cell to augment query data can also be used. One such method is detailed below.

[0553] Consider pixel B (with diagonal lines) at the center of the cell shown in bold in the lower left of Figure 48. In the upper left of this cell, the first pixel position is pixel R, which serves as the reference pixel against which the eight other pixels in the cell are compared as an ornate. The second pixel position (i.e., the first ornate) is pixel G. In addition to comparing the value of the first pixel (R) with the value of this second pixel (G) in the cell, the first pixel is also compared with the G pixel in the cell adjacent to the left, which is the nearest neighbor to the reference pixel. The comparison is also made with the G pixel in the cell adjacent to the right, which is the nearest neighbor to the reference pixel (these G pixels are underlined). This triples the richness of the [1,2] pixel-versus-data, i.e., it expands from a single comparison to three comparisons.

[0554] Similarly, this first reference pixel (R) can also be compared to the G pixels in adjacent cells above the target cell, and the nearest neighbor G pixels in adjacent cells below the target cell, although these are the reference pixels' nearest neighbors. Both of these pixels are indicated by an asterisk. This expands the [1,2] pixel-versus-pixel data to reflect five comparisons instead of just one.

[0555] Further comparisons with additional pixels outside the target cell can also be employed, depending on the needs of the specific application.

[0556] Sometimes, the "nearest neighbor" pixels in adjacent cells to the left / right / up / down are ambiguous because two such pixels of a defined type are equidistant within adjacent cells. In such cases, the upper pixels of two equidistant pixels in the left cell and the lower pixels of two equidistant pixels in the right cell can be selected for comparison. In relation to this, the leftmost pixels of two equidistant pixels in the up cell and the rightmost pixels of two equidistant pixels in the down cell can be selected for comparison.

[0557] In this embodiment of the technology, the first, second, and third pixels will be recognized as being of the first, second, and third types, respectively (indicated by the enlarged letters R, G, and R in Figure 48, respectively). Furthermore, the image sensor includes a plurality of further cells surrounding the first cell, each comprising a pixel of the type contained within the first cell. Such an embodiment includes the step of comparing a scene value associated with a first pixel in the first cell with a scene value associated with a second type (G) pixel in one of the further cells, and updating the first-second ([1,2]) pixel pair data based on the result of this comparison. This act is further repeated one or more times using a second type pixel in one of the other further cells.

[0558] This embodiment may further include the step of comparing a scene value associated with a first pixel in a first cell with a scene value associated with a third type (R) pixel in one of the further cells, and updating the first-third ([1,3]) pixel-pair data based on the result of this comparison. Again, this action can be repeated one or more times using third type pixels in other of the further cells. The query data is then formed to partially represent the [1,2] and [1,3] pixel-pair data, respectively.

[0559] In a similar manner, each of the 36 pixel-to-pixel data points can be augmented by performing other comparisons outside of the target cell.

[0560] In the given embodiment, where pixel cells with an even number of pixels are reframed to form pixel cells with an odd number of pixels, the query data is not broken down into color data by referencing one of nine sets of reference data, as in the previous case (the nine sets correspond to nine reframes of the cell to center each pixel, as shown in Figure 47). Instead, one of 36 sets of reference data is used (excluding further sets of reference data to account for brightness variations). That is, there are four different cell arrays, each with nine unique reframes.

[0561] The processing detailed herein can be performed by a general-purpose microprocessor, GPU, or other computing unit of a computer. However, more generally, some or all of the processing is performed by a special image signal processor (ISP). The ISP network (equipped with an array of transistor logic gates) can be integrated on the same substrate as the photosensor of the image sensor, which is usually silicon, or the ISP network can be provided on an auxiliary chip. In some embodiments, the ISP processing is distributed, some on the image sensor chip and others on the auxiliary chip.

[0562] An advantage is that many of the comparison operations used to generate query data, along with the associated cumulative operations, can be performed using simple logic circuits, such as addition and subtraction units. This contributes to a low gate count, which in turn reduces board area and device cost. Comparison operations can be performed without the use of multiplication or division operations, and query data can also be generated (multiplication may be required in other circuits, for example, for neural network execution).

[0563] Unless otherwise indicated, the term “pixel” as used herein includes a photosensor and may also include separate filters and / or microlenses.

[0564] The 36 pixel-pair data sets, represented by query data in some of the detailed embodiments, are merely illustrative, and it should be understood that more or fewer pixel-pair data sets can, of course, be used. For example, in one basic embodiment, two pixel-pair data sets are used. For example, scene values ​​associated with a pair of pixels in a first cell are compared, and the result is adopted in one pixel-pair data set. Scene values ​​associated with a second pair of pixels in the first cell are compared, and the result is adopted in the second pixel-pair data set (the two pixel pairs may have a common base or ornate pixel, for example, they may be pixel pairs [1,2] and [1,3]; or they may be accompanied by four pixels, for example, they may be pixel pairs [1,2] and [3,4]).

[0565] Furthermore, after these comparisons and their accumulation processes are complete, it can be understood that a mapping from these collected values ​​(e.g., 36 comparisons) and the x and y chromaticities of the scene points will then exist. The range of choices when performing such mappings is essentially broad, ranging from classical linear mappings to nonlinear mappings, and further to machine learning-trained mappings and AI processing in general. By using all of these mappings, the raw data as input is appropriately generalized to the term "feature vector," that is, this same term is commonly used in machine learning applications. This large set of comparisons between pixel data allows for the construction of increasingly richer feature vectors and enables measurements of color at lower light levels.

[0566] Furthermore, it should be understood that each pixel-to-data pair can be initialized to a value such as zero or 25. The comparison data then plays the role of updating such values.

[0567] Sometimes, the center or central pixel within a cell is referenced. A central pixel is a pixel that straddles the cell's geometric center point. Sometimes, a cell does not have a central pixel (for example, a 4x4 pixel cell). In such cases, the central pixel is the pixel whose distance from the cell's geometric center point is not greater than the distance from any other pixel to that geometric center point. Thus, a 4x4 pixel cell has four central pixels. A 3x3 pixel cell has a single central pixel, i.e., only one central pixel.

[0568] In one embodiment employing the detailed principle, an average chromaticity accuracy better than 0.03 (in XY space) can be achieved using an image of a standard Gretag24-panel color chart captured under dim lighting such that the signal-to-noise ratio is less than 3:1. Overall, the applicant has confirmed an improvement of at least 1 F-stop, possibly 2 F-stops, and even 3 F-stops, in a pseudo-side-by-side comparison between a classic Bayer sensor and one of the 3x3 9-channel variations (an F-stop is equivalent to a doubling in photographic terms).

[0569] The detailed ShadowChrome technique works in part due to the fact that the imaged scene is not a random pixel field; that is, the color in one pixel is not independent of the colors in adjacent pixels. Therefore, information about the relationship between pairs of pixels in a neighborhood, in particular their scene values, can derive color estimates for individual pixels. The size of the neighborhood can vary depending on the application requirements. Chromaticity MTF requirements will influence what size neighborhood is used to obtain what level of color accuracy. Lower spatial frequency colors are very good with ShadowChrome in the exemplary 3x3 cell embodiment, but in all embodiments, there is a chromaticity spatial frequency limitation where aliasing and moiré begin to appear. An unacceptable level of such artifacts can serve as a constraint, thereby training a neural network-based embodiment to achieve an implementation in which such artifacts are kept within the desired boundaries.

[0570] (Conclusion) Different embodiments of the applicant's technology achieve better performance in several key areas than image sensors using Bayer pattern color filter arrays and multispectral image sensors. These areas include sensitivity, extended spectral sensitivity, multiple independent channel confirmation, contrast (panchromatic MTF), color gamut, and color accuracy. In one 9-filter embodiment, for example, sensitivity is approximately twice that of Bayer CFA, and the color gamut is significantly extended. Refer to Figures 49A and 49B for a comparison of standard Bayer performance with that achieved by the third filter set described above (both using a Sony IMX428 sensor array).

[0571] While the principles of this technology have been described and illustrated with reference to illustrative embodiments, it should be noted that the present invention is not limited thereto.

[0572] For example, while many embodiments of this technology have been described with reference to a 3x3 color filter cell, embodiments employing both larger and smaller cells can also be used. Examples include 4x4, 5x5, 2x3, 2x2, 2x1, etc.

[0573] A color filter array can be fabricated from a photosensor (e.g., onto a glass plate) and then bonded to the sensor, but more commonly, the color filter array is fabricated using photolithography as an integrated part of the photosensor. In image sensors, the photosensor assembly also typically includes a microlens array and / or an anti-reflective film.

[0574] Some implementations of the detailed embodiments include pixels with sides of less than 10 microns. The majority include pixels with sides of less than 2 microns, less than 1.5 microns, or less than 1 micron.

[0575] In some embodiments, a non-standard filter comprises some or all of the filters. In other embodiments, a standard red, green, blue, cyan, magenta, yellow, or panchromatic filter comprises all of the filters.

[0576] In one embodiment, with four filter elements, filters can be derived from the first, second, or third exemplary filter sets detailed above. Such an embodiment is depicted in Figure 50. Filter element (i) can use any of the nine filters in the filter set. Element (ii) can use any of the remaining eight filters. Element (iii) can further use any of the remaining seven other filters. Element (iv) can use one of the six other filters that still remain. Alternatively, element (iv) can use the same filter selected with respect to element (i). Or, a standard (R, G, B, C, M, Y) filter such as the green filter in Table III can be used.

[0577] Except for the exceptions listed below, it should be understood that the limitations characterizing various embodiments can be combined to further characterize specific embodiments. This helps to increase the diversity between filters within the color filter cell. For example, one such more specifically characterized embodiment is the following color filter cell: The dot product calculated between the group-normalized transmission functions of two different filters within a cell at 10 nm intervals from 400 to 700 nm is less than 4. The dot product calculated between the group-normalized transmission functions of a first pair of different filters within a cell is 2 to 4 times the same dot product between a second pair of different filters within the cell. Multiple pairs of filter transmission curves, defined by samples at 10nm intervals between 400 and 700nm, intersect each other at least four times. The filter cell contains three or more different filters, each with an associated transmission curve, and the count of the intersections between all pairs of such curves in each of the 30 10 nm bands from 400 to 700 nm results in a vector of 30 count values, each containing at least 9 single count values. The cell contains one filter that has an efficiency at least 2.0 times or at least 2.5 times that of another filter within the cell. The correlation calculated between the transmission functions of two different filters within a cell at 10nm intervals from 400 to 700nm is negative.

[0578] The exception is when the limitations would be contradictory.

[0579] It will be noted that the specification sometimes refers to the filter output or filter value. This is generally an abbreviation and may more appropriately refer to the photosensor output or value when a photosensor with 100% quantum efficiency is overlaid with the identified filter.

[0580] While many detailed embodiments employ non-standard filters (i.e., not the usual red, green, blue, cyan, magenta, yellow, or panchromatic filters as defined above), it should be understood that color filter cells employing such ordinary filters can also be employed. In some embodiments, color filter cells can also be used in which a single color resist is applied in two or more different thicknesses to achieve two or more different spectral transmission functions.

[0581] In this specification, there are various references to “different filters.” In a sense, all filters are different due to manufacturing variations, even if they are intended to be the same. For the purposes of this specification, the term “slightly different” means that, when applied to two filters in a color filter cell, the mean squared error between the filters is greater than 0.0018 when the transmission functions of the two filters are normalized to each other (i.e., at least one reaches a maximum value of 1.0) and sampled at 10 nm intervals across the spectrum of interest. Unless otherwise stated, the spectrum of interest is 400–700 nm.

[0582] In contrast, for the purposes of this specification, the terms “moderately different” and “substantially different” mean that, when applied to two filters in a color filter cell, the mean squared errors between the filters are greater than 0.02 and 0.25, respectively, when the transmission functions of the two filters are normalized to each other and sampled at 10 nm intervals over the spectrum of interest (assumed here to be 400–700 nm).

[0583] When not used with the modifying adjectives "slightly," "moderately," or "substantially," "different" is intended to mean "slightly different." However, since "slightly different" has a mean squared error greater than 0.0018, such a term also encompasses "moderately different" and "substantially different" as a whole, and these latter two terms should also be understood as the intended kind of "different" when not used with any modifying adjective.

[0584] The mean squared error index described above (and also previously mentioned when determining filters that are "comparable" to the usual red, green, blue, etc.) involves the steps of determining the difference between each pair of transmission values ​​at each sample point in the spectrum of interest, squaring those differences (for example, 31 when 400-700 nm is used), summing those (31) values, and dividing by any number of sample points.

[0585] Sometimes, "transparent" pedestals, pigments, etc., are used as references. As used herein, "transparent" refers to exhibiting a spectral transmission function greater than 90%, preferably greater than 95% or 98%, across the spectrum of interest. If the image sensor produces an RGB or XYZ-based output, the spectrum of interest is the spectrum of human vision, here measured in the range of 400–700 nm.

[0586] A color filter array detailed herein is characterized in part by the fact that each filter in a cell has a spectral transmission function that is linearly independent of the transmission functions of any other different filters in the cell. Linear independence means that the transmission function of a filter cannot be achieved (within an error tolerance) by a linear combination of the transmission functions of the other filters in the cell. For the purposes of this specification, the error tolerance is the same 0.25 mean square error threshold that defines "substantially different" as detailed above.

[0587] To avoid misunderstanding, the standard deviations referenced herein are calculated using the STDEVP function in Microsoft Excel, which is understood to be the application of the following equation to a set of N values ​​x: [ka]

[0588] It should be noted that, unless otherwise stated, the spectral transmission functions discussed are measured by a spectrometer without near-infrared (NIR) or near-ultraviolet (NUV) filtering. Furthermore, such data will not reflect any spectral variations due to photosensor sensitivity. If the data is obtained in a situation involving NIR or NUV filtering, or with a photosensor, such effects may need to be addressed (e.g., removed) to ensure that any comparison is valid.

[0589] In this specification, the term “cell” generally refers to a geometric arrangement of multiple adjacent filters (or filtered pixels) repeated through a color filter array (or photosensor). A CFA may contain cells of different sizes or shapes in a tiled pattern. A block itself, containing two or more identical tiles, is also a cell. For example, a conventional Bayer cell is 2x2 pixels in size, with two pixels filtered green and the other pixels filtered red and blue. A group of 2x4 pixels (four green, two red, and two blue) with two such Bayer cells side by side is considered not a cell, but rather two cells.

[0590] In subsequent claims, it should be understood that numerical limitations are used in a non-restrictive sense unless otherwise indicated by modifiers such as "exactly." A classic embodiment is that a claim for a three-legged stool is also satisfied by a four-legged stool, since it includes (and provides more than) three of the four legs required. Similarly, for example, a claim for a filter cell with an average efficiency of 50% also includes a filter cell with an average efficiency of 55%.

[0591] Sometimes, primary and secondary colors are referred to. Red, green, and blue are generally called primary colors, while cyan, magenta, and yellow are generally called secondary colors. When the context indicates that primary or secondary colors are one of three options, these are included in the options.

[0592] However, more generally, primary and secondary colors can be characterized in terms of the composition of the optical spectrum with respect to the peak values ​​of the spectrum within the visible light wavelength range (400-700 nm). An optical spectrum is called primary if the majority of the visible light spectrum has magnitudes below 50% of the peak value, and the other wavelengths, which have magnitudes above 50%, comprise a continuous range of wavelengths. An optical spectrum is called secondary if the majority of the visible light spectrum has magnitudes above 50% of the peak value, and the other wavelengths, which have magnitudes below 50%, comprise one or two continuous spectra.

[0593] The spectral transmission function of a pixel is frequently referenced. Generally, such statements also apply to the spectral sensitivity function of the associated pixel. Alternative enumerations in each case are omitted for clarity of reading.

[0594] The reader is generally assumed to be familiar with image sensors and color filter arrays. A textbook on these technologies is "Image Sensors and Signal Processing for Digital Still Cameras" by Nakamura (CRC Press, ISBN 978-0-8493-3545-7, 2005). The fundamental technology is described in the original Bayer patent (U.S. Patent No. 3,971,065), which was transferred to Kodak. Patent documents U.S.20070230774 (Sony), U.S.8,314,866 (Omnivision), U.S.20150185380 (Samsung), and U.S.20150116554 (Fujifilm) illustrate other color filter arrays and image sensor arrays, the details of which can be incorporated into embodiments of this technology (e.g., including pixels of various areas, triangular pixels, non-square shaped cells, etc.). The use of interference filters within color filter arrays is detailed, for example, in U.S. Patent Publications 20220244104, 20170195586, 20050153219, and 6,638,668.

[0595] The fabrication processes for color filter arrays are well known to those skilled in the art. Examples are detailed in U.S. Patents 9,632,222, 8,853,717, 8,603,708, 7,914,957, and 7,763,401 (their disclosures are incorporated herein by reference). Spin coating is one of several techniques that can be employed to achieve photoresist layers of different thicknesses.

[0596] Formulations of pigment-containing or dyed resists to achieve spectral transmission functions as discussed herein are within the scope of the skill of those skilled in the art. An exemplary yellow resist contains CI pigment yellow 185 with a particle size of 0.01–0.1 microns, and the pigment particle content (by mass ratio) is equivalent to 30–60% of the resist (those skilled in the art will understand that controlling the size of the pigment particles plays a role in varying the color intensity and hue, while controlling the mass content plays a role in varying the saturation and maximum transmittance).

[0597] Additional pigments can be combined to adjust the spectral characteristics of yellow resists, detailed above CI pigments yellow 11, 24, 31, 53, 83, 93, 99, 108, 109, 110, 138, 139, 147, 150, 151, 154, 155, 167, 180, 199, etc., as well as pigments of other colors (e.g., red). Such yellow resists can be used to fabricate so-called yellow filters (such filters, of course, do not filter yellow light, but rather filter (attenuate) blue light, and thus yellow remains. Such use is also common with other filters).

[0598] Further details regarding the formulation of resists and the creation of color filters can be found, for example, in Japanese Patent Publication No. JP2006098684 and U.S. Patent Publications No. 20220043344, 20190332008, 20140349101, and 20110217636. The latter two publications teach resists employing both pigments and dyes.

[0599] The design of infrared attenuation filters is also well known to those skilled in the art. Exemplary arrangements are taught in U.S. Patent Publications 20150346404 and 20210079210.

[0600] Many of the sequences detailed herein can be implemented using ready-made resists available from suppliers such as Fujifilm, Toppan, Sumitomo Chemical, and Samsung SDI.

[0601] Additional information regarding color filter arrays, including the selection of filter transmission functions, is provided in the following papers (each incorporated herein by reference). The teachings from these references can be used in conjunction with the arrays disclosed by the applicant. ·Yako, M. et al, Video-rate hyperspectral camera based on a CMOS-compatible random array of Fabry-Perot filters. Nature Photonics, pp.1-6, January 23, 2023; ·Arad, B., et al, Filter selection for hyperspectral estimation. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3153-3161), 2017. ·Imai, FH, et al, Digital camera filter design for colorimetric and spectral accuracy. In Proc. of third international conference on multispectral color science (pp. 13-16), July, 2001. ·Sippel, F., et al, Optimal Filter Selection for Multispectral Object Classification Using Fast Binary Search. In 2022 IEEE 24 th International Workshop on Multimedia Signal Processing (MMSP) (pp. 1-5). ·Li, SX, Filter selection for optimizing the spectral sensitivity of broadband multispectral cameras based on maximum linear independence. Sensors, 18(5), p.1455, 17 pp., 2018; and ·Hardeberg, JY, Filter selection for multispectral color image acquisition. Journal of Imaging Science and Technology, 48(2), pp.105-110, 2004.

[0602] Employing the nine filters shown in Table I, when the sensor is exposed to a scene, each filtered pixel provides output data (e.g., 12 or 16 bits). The ensemble of nine values ​​from each 3x3 filter cell can be mapped to values ​​in a desired color space by multiplication with a linear transformation matrix. A common color space is the RGB space, which models the color receptors of the human eye. However, other color space data can be generated in the same way. The nine different filtered data from each filter cell can be mapped to a color space with more than three channels. While 4, 5, and 6-dimensional output color spaces are examples, 7, 8, and 9-dimensional output color spaces can also be used. Different applications can be optimized by using different color spaces. In some embodiments, multiple different transformation matrices are employed so that different filtered pixel data can be mapped to two or more different color spaces, such as human RGB and different color spaces characterized by Gaussian curves centered around 450, 500, 550, 600, and 650 nm.

[0603] When image data is provided to a neural network for analysis, for example, in autonomous vehicle applications or medical diagnostic systems, the resulting color space data, as described above, can be used. Alternatively, and often preferred, no mapping is performed, and untransformed pixel data is used as input to the neural network system. The system is trained using such data and learns transformations of the sensor data that best serve to reduce the error metrics used by the network.

[0604] The neural networks referenced herein can be implemented in various ways. Exemplary networks include AlexNet, VGG16, and GoogleNet (U.S. Patent No. 9,715,642). Preferred implementations are available from GitHub repositories and from cloud processing providers such as Google, Microsoft (Azure), and Amazon (AWS).

[0605] Some cameras employing this technology provide both types of output: data mapped to one or more different color spaces, and unconverted data.

[0606] The processes and arrays disclosed herein can be implemented as instructions for computing devices, including general-purpose processor instructions for various programmable processors such as microprocessors (e.g., Intel Atom, ARM A8, etc.), and these instructions can be implemented as software, firmware, etc. These instructions can also be implemented in various forms of processor networks, including programmable logic devices and field-programmable gate arrays.

[0607] In addition, or as an alternative, implementations may employ dedicated electronic networks, such as application-specific integrated circuits (ASICs), which are custom-designed and manufactured to perform some or all of the component operations.

[0608] Software instructions for implementing the detailed functionalities can be authored by those skilled in the art, without excessive experimentation, from the descriptions provided herein, and written in, for example, C, C++, Visual Basic, Java®, Python, Tcl, Perl, Scheme, Ruby, Matlab®, etc., together with the associated data.

[0609] Software and hardware configuration data / instructions are generally stored in one or more data structures, which are transmitted as instructions via tangible media such as magnetic or optical disks, memory cards, and volatile and non-volatile semiconductor memories.

[0610] This specification discusses various embodiments. It should be understood that methods, elements, and concepts detailed in conjunction with one embodiment can be combined with methods, elements, and concepts detailed in conjunction with other embodiments. Several such arrangements are described in particular, but many are not described due to the number of permutations and combinations. The applicant also recognizes and intends that the methods, elements, and concepts herein can be combined, substituted, and interchangeable not only within and between themselves, but also with those known from cited art. Furthermore, it will be recognized that the art described in detail can be incorporated together with other current and future art for advantageous effects. Implementations of such combinations will be readily conceivable to those skilled in the art from the teachings provided within this disclosure.

[0611] While this disclosure details specific sequences of actions and specific combinations of elements, it will be recognized that other methods to be considered may involve rearranging the actions (possibly omitting some and adding others), and other combinations to be considered may involve omitting some elements and adding others.

[0612] The complete system is disclosed, but secondary combinations of the detailed sequences are also considered separately (for example, various features of the complete system are omitted).

[0613] While certain aspects of this technology are described by reference to illustrative methods, it will be recognized that apparatuses configured to perform the actions of such methods are also considered as part of the study of the applicant's invention. Similarly, other aspects are described by reference to illustrative apparatuses, and methodologies performed by such apparatuses are also within the scope of this technology. Furthermore, tangible computer-readable media containing instructions for configuring a processor or other programmable system to perform such methods are also explicitly considered.

[0614] While providing a comprehensive disclosure, in order to comply with the patent law requirement of brevity, the applicant incorporates each of the documents referenced herein by reference (such documents are incorporated as a whole, even if they are cited above in relation to the specifics of their teachings). These references are intended to be incorporated into the array detailed herein by the applicant and disclose the technologies and teachings into which the technologies and teachings detailed herein are incorporated.

[0615] In light of the wide variety of embodiments to which the principles and features discussed above may apply, it should be clear that the detailed embodiments are merely illustrative and should not be considered as limitations on the scope of the invention.< / dot> < / dot> < / dot> < / dot> < / dot> < / dot>

Claims

1. An image sensor, wherein the image sensor includes a checkerboard pattern of a transparent pedestal (172) that extends across the image sensor.

2. The image sensor according to claim 1, wherein the checkerboard pattern of the transparent pedestal defines two types of scattered locations, namely, relatively elevated locations and relatively descended locations, and the continuous region of the sensor includes cyan, magenta, and yellow filters in one of the locations of the type and red, green, and blue filters in the other location of the type.

3. The image sensor according to claim 1, wherein a first filter, comprising a first colored resist, is formed on one of the transparent pedestals, and a second filter, comprising the first colored resist, is not formed on a transparent pedestal, and the second filter has a thickness greater than that of the first filter.

4. An image sensor including a color filter array, wherein the color filter array includes a first filter made of a first colored resist formed on a pedestal, and a second filter made of the same first colored resist not formed on a pedestal, the second filter having a thickness greater than that of the first filter.

5. The image sensor according to claim 4, wherein the pedestal is transparent.

6. The image sensor according to claim 4, wherein the pedestal has a spectral transmission function greater than 80% at 400-700 nm and less than 50%, 20%, or 10% at 720, 740, or 760 nm.

7. An image sensor comprising four pixels that are most sensitive at wavelengths of 400 to 700 nm, wherein each pixel comprises a photosensor and a separate filter that gives the pixel a different spectral color response sensitivity than the other four pixels, and at least two of the four pixels' filters allow at least 50% of the illumination incident on those separate photosensors at wavelengths of 650 nm to over 700 nm to pass through.

8. The image sensor according to claim 7, wherein the four pixels comprise a red-colored pixel, a green-colored pixel, a blue-colored pixel, and either a yellow or magenta-colored pixel.

9. The image sensor according to claim 7, wherein the four pixels comprise a yellow-colored pixel, a magenta-colored pixel, and two different pixels selected from red, green, and blue-colored pixels.

10. The image sensor according to claim 7, wherein the four pixels comprise a yellow-colored pixel, a red-colored pixel, a green-colored pixel, and a fourth pixel whose spectral transmission function exceeds 80% at 400-700 nm, but is below 50%, 20%, or 10% at 740 nm or 780 nm.

11. An image sensor comprising a photosensor array overlaid with a color filter array, wherein the filter cells (311) of the color filter array are located at different locations from the photosensors (312) of the photosensor array, and the locations are gradually shifted across the sensors.

12. An image sensor comprising a row of N photosensors overlaid by a row of M filters, wherein neither N / M nor M / N is an integer.

13. The image sensor according to claim 12, wherein M and N are relatively prime.

14. In an imaging system including an image sensor and a color reconstruction module, the image sensor includes a first cell, the first cell comprising a spatial group of N pixels, the first pixel having a first pixel at a first location in the spatial group, a second pixel at a second location in the spatial group, a third pixel at a third location in the spatial group, up to the Nth pixel at the Nth location in the spatial group, each of the N pixels having a distinct spectral response defining the type of the pixel, the first cell comprising at least two pixels of different types, the method being the following actions, namely, (a) Comparing the scene values ​​associated with the first pair of pixels in the first cell to obtain the first pair of pixel data (401), (b) Comparing the scene values ​​associated with the first pair of pixels with a second pair of pixels in the first cell that are different from the first pair, to obtain second pixel pair data (402), (c) Forming query data based on the first and second pixel pair data and providing the query data as input data to the color reconstruction module (411) Methods that include...

15. The method according to claim 14, wherein the first and second pairs of pixels comprise four pixels.

16. The method according to claim 14, wherein the first and second pairs of pixels comprise three pixels.

17. The method according to claim 14, comprising performing an additional comparison of scene values ​​associated with different pairs of pixels in the first cell to obtain a set of N(N-1) / 2 pixel-pair data, and forming the query data based on the set of pixel-pair data.

18. The method according to any one of claims 14-17, wherein the query data comprises each of the pixel-pair data.

19. The method according to any one of claims 14-18, further comprising the color reconstruction module assigning output color information for the central pixel in the first cell, in part, based on the query data.

20. The method according to claim 19, wherein the color reconstruction module assigns output color information relating to the central pixel without referring to a scene value associated with the central pixel.

21. The method according to claim 19 or claim 20, wherein the color information comprises chromaticity information.

22. The method according to any one of claims 19-21, wherein the central pixel is not included in the first pair of pixels and is not included in the second pair of pixels.

23. The first pair of pixels comprises the first and second pixels, forming a first-second ([1,2]) pixel pairing from which the first pixel pair data is obtained; the second pair of pixels comprises the first and third pixels, forming a first-third ([1,3]) pixel pairing from which the second pixel pair data is obtained; the first, second, and third pixels are of the first, second, and third types, respectively; the image sensor includes a plurality of further cells adjacent to the first cell; and the method performs the following actions, namely: (d) Comparing the scene value associated with the first pixel with the scene value associated with the pixel that is the spatial or color counterpart of the second pixel in one of the further cells, and updating the first pixel-pair data based on the result of the comparison, (e) Performing action (d) one or more times using the first pixel and one or more pixels that are spatial or color corresponding to the second pixel in the other of the further cell, (f) Comparing the scene value associated with the first pixel with the scene value associated with the pixel that is the spatial or color counterpart of the third pixel in one of the further cells, and updating the second pixel-pair data based on the result of the comparison, (g) Performing action (f) one or more times using the first pixel and one or more pixels that are spatial or color corresponding to the third pixel in the other of the further cell, (h) Forming the query data based on the first and second pixel pair data. The method according to claim 14, including the method described in claim 14.

24. The method according to claim 23, wherein the first, second, and third pixel types are all different.

25. The method according to claim 23 or 24, wherein each of the further cells replicates the first cell.

26. Comparing scene values ​​associated with each possible pairing of a reference pixel and an assorted pixel, wherein the reference pixel is one of the first to (N-1) pixels of the first cell, and the assorted pixel is the spatial or color correspondence of the second to Nth pixel of the first cell in one of the further cells, Based on the results of the above comparison, the corresponding pixel-to-pixel data is updated, To form the query data based on the aforementioned pixel-to-pixel data. The method according to any one of claims 23-25, including the following:

27. The method according to any one of claims 23-26, wherein the color reconstruction module includes assigning color information relating to a central pixel in the first cell, the central pixel being none of the first, second, or third types.

28. The method according to any one of claims 23-27, comprising determining the scene value associated with the first pixel in the first cell by determining a median or mean based on the output signal from the first pixel in the first cell and the output signals from pixels spatially or chromatically corresponding to the first pixel in one or more further cells.

29. The method according to any one of claims 14-28, wherein each type of pixel has an efficiency index associated with it, and the method comprises correcting the pixel scene value using the corresponding pixel efficiency index prior to performing the comparison operation.

30. The method according to any one of claims 14-29, which includes performing the comparison act using a logic network fabricated on a semiconductor substrate common to the image sensor.

31. The method according to claim 30, wherein the logic network performs the comparison operation without performing multiplication or division operations.

32. The color reconstruction module performs a pattern matching calculation, according to any one of claims 14-31.

33. The method according to any one of claims 14-31, wherein the color reconstruction module comprises a neural network that is pre-trained to assign output color information in response to input query data.

34. The method according to any one of claims 14-31, wherein the color reconstruction module includes identifying stored reference data that matches the query data within the reference data structure and outputting color information associated with the identified reference data.

35. The color reconstruction module calculates the Hamming or Levenshtein distance, according to any one of claims 14-31.

36. The color reconstruction module calculates the dot product, according to any one of claims 14-31.

37. The method according to any one of claims 14-36, comprising converting incident light received from a scene by the image sensor into an electrical signal by the pixel's photosensor, and generating the scene value associated with the pixel from the electrical signal.

38. An imaging system comprising an image sensor having a semiconductor substrate processed to define a plurality of pixels, wherein the image sensor includes a first cell, the first cell comprising a spatial group of N pixels, the first pixel having a first pixel at a first location in the spatial group, a second pixel at a second location in the spatial group, a third pixel at a third location in the spatial group, up to the Nth pixel at the Nth location in the spatial group, each of the N pixels having a distinct spectral response, the first cell comprising at least two pixels having different spectral responses, and the semiconductor substrate further comprises (a) Comparing the scene values ​​associated with the first pair of pixels in the first cell, and obtaining the first pixel pair data, (b) Comparing the scene values ​​associated with the first pair of pixels and a second pair of pixels in the first cell that are different from the first pair, to obtain second pixel pair data, (c) Forming query data based on the first and second pixel pair data. To be processed to define a hardware circuit network configured to perform An imaging system characterized by the following:

39. The imaging system according to claim 38, further comprising a color reconstruction module having an input for receiving the query data and configured to assign color information to pixels in the cell based on the query data.

40. The imaging system according to claim 39, wherein the color reconstruction module assigns color information to the central pixel in the cell without referring to the scene value associated with the central pixel, based on the query data.

Images