Color image sensor, method and system

Abstract

The noise performance of the image sensor is improved by using hundreds of pixel-to-pixel comparisons that all contribute to information about a given pixel. This allows accurate determination of image chromaticity even at extremely low signal-to-noise ratios. In other embodiments, filters of a non-conventional spectral transmittance function are employed to reduce metamerism, enabling resolution of scene information that is not visible to the human eye. Yet other embodiments relate to manufacturing a sparse matrix of transparent pedestals on an array of image photosensors. When a color resist is applied thereafter, these pedestals produce a change in thickness of the resulting resist layer, resulting in the pixels having a different spectral response despite the use of the same color resist. Yet other embodiments relate to image sensor arrangements that enable generation of red, green, blue, and NIR output data using only four conventional (red, green, blue, cyan, magenta, yellow) color resists. Numerous other novel features and arrangements are also detailed.

Description

Technical Field

[0001] In the United States, this application is a continuation of international application PCT / US2023 / 073352, filed September 1, 2023, which claims priority to the following U.S. provisional applications: 63 / 478,527, entitled "Color Filter Arrays and Methods," filed January 5, 2023; 63 / 478,728, entitled "Color Filter Arrays Employing Filters with Non-Normative Transmission Functions," filed January 6, 2023; 63 / 479,572, entitled "Hue Recovery from Noisy Color Photosensor Data," filed January 12, 2023; 63 / 481,390, entitled "Color Filter Arrays Having Improved Modulation Transfer Functions," filed January 25, 2023; and 63 / 487,941, entitled "Color Image Sensors With N-Byte Pixel Signature Mechanism to Increase Manufacturing." Yields, filed March 2, 2023; 63 / 500,089, entitled Color Image Sensors Employing Neighborhood-Variant Color Correction Kernels, filed May 4, 2023; and 63 / 515,577, entitled Color IntoInfrared Image Sensing, filed July 25, 2023. These applications are incorporated herein by reference.

[0002] This application also relates to: U.S. Application 18 / 056,704, filed November 17, 2022, and its priority applications (i.e., Application 63 / 280,898), filed November 18, 2021; 63 / 267,892, filed February 11, 2022; 63 / 269,194, filed March 11, 2022; 63 / 362,508, filed April 5, 2022; 63 / 365,482, filed May 27, 2022; 63 / 367,999, filed July 8, 2022; and 63 / 381,639, filed October 31, 2022. These applications are also incorporated herein by reference. Background Technology

[0003] Digital imaging systems have proliferated in recent years, including in consumer, medical, agricultural, automotive, and other machine vision applications. Many systems rely on Bayer pattern color filter arrays, which were first developed in the 1970s.

[0004] The rise and increasing popularity of neural networks have expanded the capabilities of digital imaging systems and created new demands on technology.

[0005] Despite the advancements in digital imaging systems since the 1970s, many shortcomings remain. Various aspects of this technology aim to address these shortcomings.

[0006] One drawback involves the noise level of digital imaging systems in low-light environments. Current imaging systems perform poorly when the signal-to-noise ratio (SNR) drops to 10:1, 3:1, and lower, especially in distinguishing scene chromaticity. This jeopardizes human appreciation of the image and impairs the machine vision use of the data.

[0007] Based on certain aspects of the technology, a digital image sensor that provides excellent low-light performance is provided.

[0008] One specific embodiment is an image sensor including a semiconductor substrate fabricated to define a plurality of pixels, the plurality of pixels comprising a pixel unit of N pixels. The unit includes a first pixel at a first location in a spatial group, a second pixel at a second location, a third pixel at a third location, and so on up to an Nth pixel at an Nth location. Each pixel has a corresponding spectral response, and at least two pixels in the unit have different spectral responses.

[0009] The semiconductor substrate is further fabricated to define a hardware circuit system configured to: (a) compare scene values ​​associated with a first pair of pixels in the cell to obtain first pixel pair data; (b) compare scene values ​​associated with a second pair of pixels in the cell that are different from the first pair of pixels to obtain second pixel pair data; and (c) form query data based on the first and second pixel pair data. In some embodiments, such comparisons are performed between all pairs of pixels in the cell. In other embodiments, such comparisons are further extended to pairs between the target cell and surrounding cells. The resulting query data (which in some embodiments may be based on tens or hundreds of pixel pair values) is provided as input data to a color reconstruction module, which identifies color information about the pixels in the cell based on the query data.

[0010] The foregoing and many other features of this technology will become more apparent from the following detailed description, which continues with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1 and Figure 1A Two embodiments are shown.

[0012] Figure 2 and Figure 3 A detailed description of the spectral transmittance curves of existing technologies is provided.

[0013] Figure 4 Another embodiment is shown.

[0014] Figure 5 Indicates the arrangement of pixel markers.

[0015] Figures 6A-6I The spectral transmittance curves used for illustrative filters are described in detail.

[0016] Figure 7 The spectral transmittance curves of existing technologies are described in detail.

[0017] Figure 8 Compare Figure 6A and Figure 6B The curve.

[0018] Figure 9 Identify the characteristics of the spectral transmittance curve of the filter.

[0019] Figure 10- Figure 10I Spectral transmittance profiles for more illustrative filters are described in detail.

[0020] Figure 11 , Figure 11A , Figure 11B and Figure 11C A pixel field and its use are shown in one embodiment.

[0021] Figure 12 The variation of the spectral transmittance function due to filter thickness is shown.

[0022] Figure 13 The spectral transmittance functions for illustrative red, green, blue, cyan, magenta, and yellow filters with a thickness of one micrometer are shown.

[0023] Figure 14 This demonstrates how filters of different thicknesses (even when using the same color photoresist) can contribute to the diversity of the spectral transmittance function.

[0024] Figure 15 Depicting Figure 14 The first derivative of the function.

[0025] Figure 16 This illustrates a sparse array of transparent substrates (e.g., transparent photoresist) fabricated on a photoelectric sensor array.

[0026] Figure 17 Showing the result after applying the photoresist layer Figure 16 By arranging the photoresist layers, we obtained photoresist layers of different thicknesses.

[0027] Figure 18 Depict the green filter on top of the transparent base.

[0028] Figures 19A-19E Different sparse substrate arrays are shown on the photoelectric sensor array.

[0029] Figure 20 and Figure 21 The filter unit is shown to employ both relatively thick and relatively thin filters.

[0030] Figure 22 Showing the use of Figure 21 The spectral transmittance curve of the filter.

[0031] Figure 23 The diagram shows an additional filter unit that uses both relatively thick and relatively thin filters.

[0032] Figure 24 Showing the use of Figure 23 The spectral transmittance curve of the filter.

[0033] Figures 25-28 Other filter units employing both relatively thick and relatively thin filters are shown.

[0034] Figure 29 The spectral transmittance curves are shown for a six-filter unit that uses both relatively thick and relatively thin filters.

[0035] Figure 30 The diagram shows spectral transmittance curves for a prior art color image sensor and spectral transmittance curves for a monochrome version of the same sensor.

[0036] Figure 31 , Figure 32 , Figure 32A , Figure 33 and Figure 34 An exemplary arrangement is described in detail that allows the photoelectric sensor and color filter to have a spatial relationship that varies on the image sensor.

[0037] Figure 35 The diagram shows two different color filter arrays that use 2x3 filter units, each composed of relatively thick and relatively thin filters.

[0038] Figure 36This illustrates how the spectral transmittance function of a red filter can deviate from its nominal value (e.g., the average spectral transmittance function of all red filters on an image sensor).

[0039] Figure 37 This section details how deviations in the nominal spectral transmittance function used for red filters can vary between pixels in an image sensor.

[0040] Figure 38 This shows how the spectral transmittance function can be parameterized using the basis functions.

[0041] Figure 39 This illustrates how the color correction matrix can vary depending on the local position of the filter unit (or filter).

[0042] Figure 40 This demonstrates how to compare the base pixel (A) with two ordinate pixels (B and C) to obtain paired data for the two pixels.

[0043] Figure 41 This is a block diagram of an embodiment of an image sensor.

[0044] Figure 42 This shows how a base pixel can be compared with all other pixels in the cell.

[0045] Figure 43 This shows that each pixel in a cell can act as a base pixel for comparison with one or more other pixels in the cell.

[0046] Figures 44-46 The comparison with the base pixel shows that the filter unit can be extended beyond the base pixel.

[0047] Figure 47 Nine reconstructions are shown, each placing a different pixel in the center of a 3x3 pixel unit.

[0048] Figure 48 This illustrates an embodiment in which a 2x2 Bayer filter unit has been reconstructed into a 3x3 unit.

[0049] Figure 49A and Figure 49B Compare the performance of the embodiments with that of the prior art.

[0050] Figure 50 The positions of the filters within the 2x2 unit are shown. Detailed Implementation

[0051] This specification begins with a detailed description of a first embodiment, which introduces certain aspects of the technique that are then further discussed in later embodiments. Such aspects include improvements in manufacturing, calibration, and image processing processes, for example, to reduce manufacturing tolerance constraints on the color filter array of an image sensor while maintaining or improving colorimetric, contrast, and / or machine vision performance.

[0052] This first embodiment relates to a sensor comprising a color filter array (CFA) organized as 3x3 pixel units (tiles), but other embodiments may naturally employ CFAs with other configurations. This embodiment uses a pigment-based CFA, but other filtering materials (e.g., dyes, quantum dots, etc.) can also be used. Fujifilm and Toppan are well-known suppliers of suitable pigments. We refer to all these materials as “pigment,” “ink,” “photoresist,” or “color photoresist,” regardless of their technology type.

[0053] Reference Figure 1 The 3x3 CFA of the first embodiment uses four different commercially available color photoresist products (e.g., FujiFilm Color Mosaics) drawn in the depicted pattern. ® (Branded product). Using six separate photomasks, each requiring a sequence of process steps (e.g., cleaning; photoresist coating / baking; exposure in the photolithography tool; development; hard baking; associated with metrology and inspection operations, as familiar to the technician). For the sake of specific examples, this color filter array can cover a 1200x1600 or 3000x4000 pixel image sensor, with each pixel outputting a 16-bit brightness value. Pixels on one side can be less than 10 micrometers, less than 2 micrometers, or less than 1 micrometer.

[0054] Figure 1A Alternative embodiments are shown.

[0055] The first layer to be processed is Figure 1 The three magenta pixels are arranged diagonally as S7, S5, and S3. The specification for this first layer aims for a 0.5-micron-thick layer of magenta (M). After the development and hard baking stages, metrology and testing specifications will allow for more lenient (larger) tolerances compared to contemporary commercial-grade specifications for CFA-based CMOS image sensors. Such lenient tolerances include, for example, a higher degree of mixing of the physical materials of the photoresist material beyond the nominally specified color photoresist material for any given pixel. That is, for example, instead of tolerating only 2% residual "magenta" photoresist in another "red" pixel, the manufacturer can tolerate 10%. These figures are used here only to illustrate aspects implied by the lenient tolerances. Such lenient tolerances enable lower-cost and more environmentally friendly chemical choices than have become possible within existing tolerance specifications.

[0056] The second layer is specified as a photomask corresponding to two other corners S1 and S9 of the 3x3 pixel unit. These two pixel units will use the same CFA magenta pigment M, for example, from the same bottle and chemical delivery system, but this second layer will be specified as 1 micrometer thick, compared to 0.5 micrometers for layer 1. The second layer photomask will be fabricated such that, for example, very small physical contacts will be allowed at the corners of the two units in the second layer, as they contact the three units of the first layer. For example, on a sensor with a pixel size of 2 square micrometers, a small 100nm “overlap” can exist, where the layer 2 pixel material covers the layer 1 material. During metrology and detection steps after the second layer is completed, these contacts between the layer 1 and layer 2 units can be quantified, for example, by the effective nanometers of overlap. As with layer 1, the tolerances for layer 2 will be more lenient compared to contemporary specifications. This leniency stems from the same reasons stated for layer 1. For example, current specifications might recognize a tolerance of 15% standard deviation for the thickness of the color photoresist and only 3% cross-material residue. Embodiments of this technology increase one or both of these figures by 50%, 100%, 200% or more.

[0057] The third layer will be designated as a common green (G) because it is used in a Bayer filter CFA sensor and will only apply to a single cell S2 in a 3x3 cell array. There will be two modest differences from current greening practices: A) the specified thickness of this G layer will be on the order of 10 to 40% thinner than typical green layers in Bayer RGB sensors; B) the tolerances for metrology and sensing will be as lenient as they are for layers 1 and 2, exceeding contemporary practices. For this particular design, 0.7 micrometers is used as the specified thickness for this third layer. The key point is: a thinner thickness for greening is chosen than that selected in modern Bayer CFA designs.

[0058] The fourth layer uses commercially available cyan (C) to fill the left middle divot (i.e., pixel position S4) of the 3x3 unit structure. The thickness specification for this fourth layer using cyan is 1 micrometer. As with the previous three layers, loose tolerances are used. The fifth layer uses the same cyan, this time filling the right middle pixel unit divot S6 of the 3x3 unit pattern. This makes it adjacent to the cyan pixel deposited in the fourth layer adjacent to the right 3x3 unit. The thickness of this fifth layer is 0.5 micrometers, instead of the 1 micrometer used for the fourth layer. In some embodiments, the physical mask used for layer 4 is rotated and used as a mask for layer 5.

[0059] Finally, the sixth layer is the yellow (Y) mask and color photoresist layer at pixel location S8. The thickness specification for this sixth layer is 1 micrometer, with the previous lenient tolerances.

[0060] Figure 2 and Figure 3 The spectral curves associated with these color photoresists are shown. These curves are based on published information from Fujifilm depicting the characteristics of the color mosaic lines of their pigments. Figure 2 The transmission of cyan, magenta, and yellow pigments at different layer thicknesses is shown, where solid lines indicate 0.9 micrometers (nominal), dashed lines 0.7 micrometers, and dotted lines 1.1 micrometers. Note that the curves do not simply shift up and down along the same contour with different thicknesses. Instead, their shape changes. For example, the width of the depressions changes, as does the slope of the curves and the shape of their blocking bands. Figure 3 The transmission of red, green, and blue pigment filters at nominal layer thickness is shown.

[0061] After these process stages, the final result of the sensor (or a large wafer containing many sensors) is something that can be called a very faintly wrinkled mosaic of color photoresist cells, where each cell is very faintly different from all its neighboring cells and, in fact, from all the 3x3 cells in the entire sensor. All cells share a stable, coarsely defined pattern at the 3x3 pattern level, but are all different to measurable amounts at the nanometer level measured on the surface. Furthermore, statistically nontrivial amounts (trace amounts) of material in each layer will exist in the discrete pixels of cells assigned to one or more other layers (and in some cases, cells assigned to all other layers). Considering the three diagonal cells of layer 1 as an example, electron microscopy can reveal that if we normalize the atomic volume of the expected material inside the expected cell of layer 1 to a value of 1.0, we will find that the volume of material in layers 2, 3, 4, 5, and 6 also occupies that cell in values ​​that are perhaps in the range of 0.01 to 0.05 and sometimes in the range of 0.005 to 0.1 or higher. To illustrate, the material from all layers will be found in all pixels at some measurable level. These trace amounts can also be expected to exhibit typical "layer physicochemicals," where the physicochemical properties can be anticipated to mimic the layering sequence.

[0062] Figure 2 and Figure 3 The curves show the visible range of the spectral axis from 400 to 700 nanometers. In all designs and applications where the expectation is not merely "excellent color images for the human eye to see," extension to near-infrared (NIR) and near-ultraviolet (NUV) is encouraged. As taught in previous publications, a balance is encouraged: optimize the quality of the color image while maintaining the specified quality of the multi-channel information useful for machine vision applications (or vice versa).

[0063] In the embodiments detailed above, all six layers of filters can have at least some transmittance in both NUV and NIR. This allows for the estimation of the NUV channel optical signal and the NIR channel optical signal. This differs from, for example, achieving two separate NIR optical signal estimations (e.g., bands 700nm to 750nm together with bands 750nm to 800nm), although this can be accomplished in other embodiments. That is, we here consider the far-infrared and NIR bands from approximately 690nm to approximately 780nm as a single channel, and the deep blue to NUV bands from approximately 360nm to approximately 410nm as a single channel. The above six-layer filtering implementation is used for the diverse filtering behavior of these two new bands, which we refer to as NIR and NUV for simplicity.

[0064] As blue light moves into the NUV and as far-red light moves into the NIR, the fundamental quantum efficiency (QE) of the silicon detector declines towards lower levels. Therefore, in both cases, the fundamental behavior of the sensor is to shift the photoelectron signal level downwards. On the NUV side of the spectrum, we have additional attenuation introduced by the glass, both from typical lenses used in general applications and from any glass used as a cover for the sensor itself. Thus, for NUV, we can use this naturally occurring attenuation to provide a full-pixel NUV cutoff frequency in the wavelength range of approximately 350 nm to 360 nm. Similarly, for NIR, the quantum efficiency of silicon rolls off with increasing wavelength, and a full-pixel NIR cutoff can be fabricated either by pigment deposition or by explicitly defining a glass surface or other means. The first embodiment employs a full-pixel NIR cutoff somewhere between approximately 750 nm and approximately 800 nm.

[0065] Given the full pixel cutoff established as described above for both NIR and NUV, we generally want the spectral transmittance characteristics of the six layers to differ slightly between the six layers in the NUV and NIR bands. Commercially available color photoresists of the four M, C, Y, and R bands essentially do this. As long as practitioners cannot find commercially available color photoresists that achieve this diversity of transmittance in both the NIR and NUV bands, the option remains: add a small amount of pigment to existing color photoresists that are extremely transparent through the approximately 400 nm to approximately 700 nm visible range, but add some opacity to one or both of the NUV and / or NIR bands.

[0066] Another embodiment is shown in Figure 4The 3×3 color filter array comprises three conventional red, green, and blue filters plus two of cyan, yellow, and magenta filters. Each of these latter filters is manufactured with two different thicknesses—thin and thick (in the figure, narrow is denoted as "N" and thick as "T")—to obtain two different spectral transmission filter profiles for each of the three colors. The thin filter can be, for example, less than 1.0 micrometer (e.g., 0.9, 0.8, or 0.7 micrometers), while the thick filter can be, for example, greater than 0.8 micrometers (e.g., 0.9, 1, 1.2, or 1.5 micrometers) (appropriately paired with the thin filter for that color to maintain a thin / thickness relationship).

[0067] The embodiments detailed above are, of course, merely exemplary. A basic concept is that a color filter array can include elements formed of the same color photoresist, but with different thicknesses to achieve diversity in filtering action. In one described embodiment, one magenta filter layer is 0.5 micrometers, while another magenta filter layer is 1 micrometer. In one embodiment, there are three different magenta filter layers with thicknesses of 0.4, 0.6, and 0.8 micrometers. Such ratios of thickness are merely exemplary. In different embodiments, one layer may be only 10%, 20%, or 50% thicker than another layer of the same color. Or one layer may be 100%, 200%, or more thick than another layer of the same color. One embodiment uses different thicknesses for only one color, while other embodiments use different thicknesses for multiple colors. As indicated, some embodiments deposit a single photoresist at more than two different thicknesses.

[0068] Later sections detail how to use a base to vary the filter thickness. The embodiments detailed in this discussion can be achieved by directly fabricating filter layers of different thicknesses without using a base.

[0069] Tolerance compensation, calibration, and algorithm implementation: pixel-by-pixel correction

[0070] Using the previously described first embodiment, four separate color photoresists (C, M, Y, G) are specified, and each photosite will contain a finitely measurable amount of each of the four pigments (i.e., the color it assigns and trace amounts of the other three). Six different nominal surface thicknesses have been specified for these four color photoresists. Each photosite has a nominal surface thickness value ranging from a few tenths of a micrometer to more than one micrometer; we refer to this as the nominal thickness of the color photoresist layer.

[0071] Due to the looseness of traditional manufacturing tolerances, a "mixing" of the other three colors of photoresist pigments is expected at all its photosensitive points for a given sensor. This looseness speeds up sensor processing and reduces costs.

[0072] We define the normalized pigment concentration level of the pigment assigned to the photosensitive point as the sensor global mean of the pigment after the sensor has been manufactured and packaged. This global sensor mean is normalized or set to 1.0 (i.e., we do not discuss micrometers here).

[0073] For each photosensitive point, detailed measurements will show a deviation from the sensor's global mean of 1.0, where the deviation will be partly a function of the degree of looseness in the tolerances in the sensor's manufacturing. This deviation can be very low, around 0.03 or less of the mean, for cases where higher (stricter) tolerances have been implemented in manufacturing, all the way up to significantly higher standard deviations of 0.1 or greater in the "push-to-the-edge" cases where experiments can be designed to tolerate and still function correctly. Let us call this standard deviation of the assigned pigment the "normalization spillover (slop)".

[0074] For each photosensitive point in the first embodiment, there are three (contamination) pigments that are different from the pigment assigned to the photosensitive point. Each of these three pigments will have a certain global mean of the sensor measured throughout the entire manufactured and packaged sensor in the unit where it is not the assigned pigment. This global mean for each of the three different pigments may be referred to as the "mixed mean". All three pigments will have a unique mixed mean, where for higher tolerance manufacturing practices, it is a value in a normalized unit of a few percent (e.g., 0.015, 0.03, or 0.05), while for pusing-the-envelope experiments, it is an even higher value (e.g., 0.06, 0.1, 0.15, or higher normalized units).

[0075] Similarly, the mixed mean of these unassigned pigments will have its own standard deviation relative to the standard deviation of the nominally assigned pigments, referred to as "mixture spillover". (Experimental practice is expected to show that for many sensor designs, the mixture mean and mixture spillover values ​​will be correlated via a simple square root relationship; however, this disclosure reserves these values ​​as independent values.)

[0076] Recognizing that we have four pigments and six specified thicknesses across 3x3 CFAs for those pigments (where two thicknesses are used for two of the pigments, and one thickness is used for the other two pigments, giving a total of six types), we find six normalized 1.0 levels for the six photosensitive point types, each with its own normalized overflow, and thus three more mixed means for each of the six photosensitive point types, where each of these mixed means has an associated mixed overflow value. Therefore, for the baseline design, we have:

[0077] Six normalized overflow values; eighteen normalized mixed means; and eighteen normalized mixed overflow values. This is a total of forty-two calibration parameters applicable to any given manufactured sensor. This set of values ​​can be measured and then stored in memory on the sensor chip at the per-pixel or per-cell level. One illustrative approach is to measure the global mean and values, fit a histogram to the individual pixel behavior, and then digitize the histogram values, where the individual behavior of a given pixel falls into one of these encoded histogram bins.

[0078] During normal operation of the sensor, where all photosensitive points generate "digital values" during a given exposure to represent the detected photoelectron count, the task is to correctly interpret these digital values ​​within the context of their calibration values. This context includes several elements, including:

[0079] 1) Unique characteristics of the photosensitive point relative to the global average;

[0080] 2) Regarding the inference of the mixing ratio of the four pigments used for each photosensitive point;

[0081] 3) Absolute irradiance level, as a function of the spectrum, produces a given global mean, thus relating the normalized 1.0 value to the sensor irradiance level;

[0082] 4) The characteristics of a photosensitive point relative to its immediate neighbors of eight surrounding photosensitive points; essentially, it is surrounded by 3x3 CFA cells; [Note: There is an alternative definition of the CFA pattern in which each photosensitive point can claim ownership of the "center pixel," thereby placing its surrounding neighbors in the place where they would be; this is a useful algorithmic definition different from the definition used in the CFA introduced in the baseline design; we can call it the photosensitive point centering CFA];

[0083] 5) The detailed relationship between a certain photosensitive point centered CFA and the eight 3x3 photosensitive points centered CFA around the central CFA.

[0084] The list of five is not exhaustive, but it is sufficient for further description of pixel-wise correction (PWC).

[0085] One embodiment of the technology is thus a color imaging sensor with multiple pixels, wherein each pixel is associated with a multi-byte memory, and the memory stores data relating to one or more parameters of the pixel, such parameters being used for similar pixels across the sensor.

[0086] Each photosensitive point has its own unique signature relative to these 42 calibration parameters, which raises questions about the measurement and use of these parameters.

[0087] The first problem was solved by something called Chromabath and FuGaI illumination. The second problem was solved by pixel-by-pixel correction.

[0088] Chromabath and FuGaI

[0089] This disclosure builds upon the Chromabath technique previously taught in applications 63 / 267,892 and 18 / 056,704, replacing the monochromator used therein with a multi-LED (e.g., ten or twelve LED) illumination system called a full-gamut illuminator (FuGaI). However, the monochromator arrangement remains relevant because it is used to train and calibrate the use of FuGaI on a scaled-up manufacturing line.

[0090] We can add further parameters (e.g., dark value) to the parameters mentioned above. Thus, each photosensitive point can be characterized by parameters (some of which depend on the type of photosensitive layer) including: 1) its dark value in a numerical value; 2) its nominal equivalent white light gain in a numerical value, which is then related to irradiance (the light level falling on the photosensitive point); and then 3) through 5) are the CYMG mixing ratios, where the sum of the ratios is 1.0, where only three parameters are needed to fully specify those ratios.

[0091] We begin with the spectral profiles of the C, Y, M, and G photoreceptors; these profiles interact with the quantum efficiency profiles of the underlying silicon sensor photoreceptors. These spectral profiles also reflect the Beers-Lambert behavior of the photoreceptors, allowing for more refined spectral estimates based on thickness and mixing ratio knowledge. We then characterize five of the aforementioned spectrometer parameters for each photoreceptor.

[0092] Dark value measurements are derived from the astronomical characterization of "dark frames." This involves preventing light from falling onto the sensor; capturing multiple frames of data; and storing a long-term global average of frames that are sometimes associated with metadata indicating the sensor's temperature. Such data is then used to correct later measurements, for example, by subtracting the dark frame data pixel-by-pixel. Many existing CMOS image sensors have some form of dark level adjustment and / or correction.

[0093] In some embodiments, the applicant uses the median of a series of dark frames instead of the mean for correction. This is believed to facilitate certain operations that employ comparisons of neighboring sensor points.

[0094] The equivalent white light gain value for the sensor's photosensitive points is typically measured after corrections have been applied for the dark value of each photosensitive point. A "flat field" imaging process can be used to measure these gain values.

[0095] Further details involve measuring the CMYG mixing ratio. Various methods can be employed. The illustrative methods detailed below are suitable for low-cost, high-volume manufacturing designed for applications with annual unit volume levels of millions of sensors.

[0096] Four “color-calibrated sensors” are fabricated using only one of the C, M, Y, and G color photoresists. These sensors undergo all the steps required to fabricate a final CFA-based CMOS imaging sensor, except that only one stage of the CFA color photoresist process is applied: color photoresist coating. However, the thickness of the color photoresist actively varies in different regions of the sensor, ranging from a nominal thickness of a few tenths of a micrometer to sometimes 1 or 2 micrometers. Spatial patterns (e.g., sine and square) or photosensitive dot-level masking can be applied. The resulting color-calibrated sensors will be used to characterize the sensor’s global spectrometer properties: how the specific selection of C, M, Y, and G, with respect to the photosensitive dot size (spacing) of that particular class, all interact with the silicon-driven quantum efficiency (QE) sensitivity spectral profile.

[0097] Why have four calibration sensors, each for four C, M, Y, and G? We don't want any cross-mixing of pigments. A complete Beers-Lambert model of a single, isolated color photoresist can be measured in-situ in the same photoresist structure designed / built in the final product. If cost constraints are dominant, the color photoresist can be modeled based on the supplier's spectral data to obtain approximately 90% good results, but it is preferable to apply the color photoresist to the exact photoresist structure being used, thinning those photoresist layers over a wide range of thickness values, and then measuring the digital output.

[0098] The next step is to run each of the color calibration sensors through (preferably) a monochromator-based Chromabath. As noted, "Chromabath" is a coined term from a previously provisional application. Once the designer has selected the full effective spectral range (e.g., 350 nm to 800 nm) for the image sensor array, Chromabath is the process of immersing the entire sensor in monochromatic light moving from one end of the spectral range to the other. In the case of a monochromator-based Chromabath, using four color calibration sensors, a λ step size of 1 nm can be used, giving 451 wavelength steps for each acquired image. Following good practice, it is assumed that the illumination field is uniform (e.g., within a low single-digit percentage; ideally less than 1%). The same applies to the absolute irradiance values ​​at all wavelengths between 350 and 800 nm. Multiple measurement sessions can collect data over several hours.

[0099] The result of this data collection will be a Beers-Lambert law-like spectral curve covering a selected wavelength range. Figure 2 The diagram illustrates the variation in spectral response revealed by different thicknesses of different photosensitive points on a color-calibrated sensor. These curves will, of course, be isolated into one each of C, Y, M, and G, and will reveal the properties of the photosensitive points, typically including the quantum efficiency effect of silicon response to photosensitive point variations. Again, a lower-cost approach is to use modeling instead of measurement, but since measurement is a one-time pre-production laboratory step, the cost is well amortized across even low-volume production.

[0100] Having obtained these pseudo-Beers-Lambert spectral curves for four color calibration sensors, we will next take N different production samples from the CFA CMOS image sensor for FuGaI training.

[0101] These N sample production sensors (where N can be, for example, five) act as proxies for the production run of the sensors and will serve as what we call pigment-mixed true calibration sensors (as opposed to single-photoresist color calibration sensors). Similar to the color calibration sensors, we begin by measuring these sensors to determine the median dark value and median white balance gain value for each photosensitive point on each sensor.

[0102] We also determine the molar or thickness-equivalent concentration values ​​of C, Y, M, and G for each photoresist point. This is done by performing Chromabath measurements on each sensor, just as with the color calibration sensors. The goal is to determine the “thickness-equivalent” concentration levels (c, m, y, and g) of all four color photoresist pigments C, Y, M, and G for each photoresist point. We sometimes refer to these as the four true vectors. The nonlinear least-squares fit for each photoresist point can be based on the following equation:

[0103] The spectral curve measured by the photosensitive point = c*Cbl(c) + m*Mbl(m) + y*Ybl(y) + g*Gbl(g) (1)

[0104] Here, lowercase bl indicates that these are either modeled (lower-cost scenarios) or empirical measurements of the corresponding pigments, specifically pseudo-Beers-Lambert curves. "Pseudo" merely acknowledges that empiricism prevails over theory.

[0105] For any given photoreceptor within a baseline 3x3 CFA, there exists a nominal thickness of the color photoresist to which it is assigned. This nominal value should be approximately the average of calibration measurements taken across the entire sensor for that photoreceptor type. Thus, the other three values ​​will typically be less than 10% of this nominal value.

[0106] Based on monochromator measurements and nonlinear least-squares fitting in the Chromabath process, we have thickness-equivalent pigment values ​​for each photosensitive point in our N pigment-mixing true calibration sensors. We want comparable data for every sensor in our production runs, but without such a lengthy process. This is where FuGaI comes in.

[0107] FuGaI is again an abbreviation for full-gamut illuminators. An illustrative illuminator comprises ten or twelve narrowband LED emitters, each with a bandwidth typically ranging from 20 to 50 nanometers across the full width at half maximum (FWHM). A center wavelength is chosen such that all but two of them are spaced across the visible spectrum, with the remaining two positioned within the NIR spectrum. These LEDs are expected to be tested to ensure they are time-stable at their center wavelengths and in their luminance. Wavelength stability within single-digit nanometer ranges is expected. Luminance variations in the low single digits or even less than 1% are expected.

[0108] At an illumination distance of, for example, 40 cm, and with a frosted white coating over the LED illuminators, the individual LEDs of the FuGaI system are turned on sequentially to illuminate N pigment mixing real calibration sensors one or all at a time. Numerous images (e.g., hundreds or thousands) are captured for each LED state using the pigment mixing real calibration sensors. The median value of each “pixel” is recorded across these 100 to 1000 images per LED state.

[0109] The terms "pixel" and "photosensor" are used somewhat interchangeably. "Pixel" is sometimes preferred to refer to the numerical value associated with a photosensor, while "photosensor" is sometimes preferred when discussing the physical properties of a photosensitive site. Note that photosensors generate raw digital values ​​(DN), and each photosensor can be associated with calibration information that can be used to adjust the raw photosensor DN. The resulting adjusted value (i.e., the digital value plus the calibration data) can be referred to as DN+Cal. Therefore, each pixel will generate a DN value within the Cal context. The FuGaI Chromabath process is used to generate this Cal information for each pixel, trained from mixed real data attached to each of N pigment mixing real sensors.

[0110] Each pigment mixing real sensor produces 12 images with median-DN values ​​during the FuGaI Chromabath process. This yields a 12-dimensional vector for each pixel in each of the N sample sensors, which we call "R12" for true 12D. (The optical field inhomogeneities of the FuGaI cell itself will affect the measurement of the flat white gain value, but those inhomogeneities have a minor impact on the mixing coefficients (c, y, m, and g) to be measured from the set of 12 images.)

[0111] Each pixel in each pigment-mixing real sensor also has an associated 4D vector (“R4”, including the values ​​c_truth, m_truth, y_truth, and g_truth) generated by the least-squares curve fitting process detailed above based on measurements from a 451-point monochromator Chromabath.

[0112] The task then becomes establishing a mapping: transforming values ​​in the 12D vector space (R12) to values ​​in the 4D vector space (R4). The consideration for this type of mapping problem is specifically applied to the so-called "one-to-one" mapping problem of mapping R4 singularities back to R12 space: will both separate points in R4 space be mapped to singularities in R12 space? Also relevant is the issue of common noise: if the R12 measurements are too noisy, will there be unacceptable smearing of the R4 solutions? For example, in a given green pixel, will there be excessive errors in the measurement of cyan, magenta, and yellow traces?

[0113] Empirical evidence suggests these are merely theoretical concerns; in practice, trace amounts of the three unassigned pigments can be measured as a ratio to the assigned pigments. Numerically speaking, let's take a cyan pixel with a nominal thickness of 0.5 micrometers of color photoresist as an example. Can we actually measure the equivalent thickness in the 10 to 100 nanometer range used for magenta, yellow, and green using the FuGaI Chromabath process? Tests indicate the answer is yes.

[0114] This thickness equivalence measure for the three unassigned pigments is intuitively a good choice in describing the mixing of pigments. While it does not technically describe the nanoscale physical reality of the photosensitive dots, it serves our purpose. In an exemplary embodiment, we employ this thickness equivalence calibration method to obtain a thickness equivalence measure, measured in nanometers, that is correlated with the nominal thickness of the assigned color photoresist, measured in micrometers.

[0115] Moving on to the final steps of the FuGaI Chromabath process applied to pigment-mixing real sensors, recall that the first embodiment included six pixel types within a 3x3 CFA. Let us call these:

[0116] Layer 1 – M0.5 (magenta, 0.5 micrometer thickness)

[0117] Layer 2 – M1 (magenta, 1 micrometer thickness)

[0118] Layer 3 – G0.7 (green, 0.7 micrometers thick)

[0119] Layer 4 – C1 (cyan, 1 micrometer thick)

[0120] Layer 5 – C0.5 (cyan, 0.5 micrometers thick)

[0121] Layer 6 – Y1 (yellow, 1 micrometer thickness)

[0122] For all pixels on all N pigment-mixing real sensors, we now empirically solve for the mapping function between the R12 FuGaI vector and the R4 real vector to determine trace mixing. Each of the six pixel types mentioned above requires its own mapping function. Flat white gain and dark values ​​can also be measured in the Chromabath process. These will be measured during production line use in the FuGaI Chromabath process, so we can do the same here.

[0123] One objective of the FuGaI Chromabath process on hybrid real sensors is to calibrate the mixing ratio measurement capabilities of the FuGaI Chromabath process itself for use in large-scale production. The applicant prefers FuGaI-based Chromabath testing of sensors rather than monochromator-based Chromabath testing for reasons including cost, simplicity, scalability, and integration considerations within existing sensor testing systems.

[0124] Solving equation (1) on all pixels and on all N sensors yields six 4-row by 12-column arrays of mapping function coefficients. These matrices can then be used to estimate the trace pigment mixing ratio for each pixel in each of the sensors used in mass production.

[0125] Specifically, testing of each mass-produced sensor involves performing a FuGaI Chromabath process to obtain an R12 vector for each pixel. For each pixel of type X (layers 1 through 6), the R12 vector measured during post-production testing is multiplied by the Xth 4x12 matrix to calculate the trace pigment mixing ratio for that pixel. Just as each individual pixel in contemporary imaging sensors has associated dark offset and gain parameters, each pixel can also have its own associated pigment mixing ratio. This data is written to non-volatile memory on the sensor chip. Although sometimes considered a defect in the manufacturing process, this "overflow" within the pixel-to-pixel manufacturing process is utilized to correct various downstream processing steps, of which de-mosaicing is an illustrative downstream step.

[0126] Pixel-by-pixel correction

[0127] This disclosure then turns to the use of pigment mixing ratio data (i.e., pixel-by-pixel correction).

[0128] This calibration uses calibration data stored on the sensor chip. The detailed arrangement achieves efficient use of memory storage while providing enhanced imaging (e.g., contrast, color accuracy, color gamut, machine learning channel richness, etc.).

[0129] In one embodiment, a 3-byte per-pixel calibration storage scheme is used. Four bits of one byte are reserved for the dark value of the pixel, and four bits of the same byte are dedicated to the white gain correction value. These two stored values ​​indicate the difference relative to the global mean for each of the six pixel types (the layers mentioned above). These values ​​correspond to small intervals in a histogram representing the range of positive and negative deviations from the global pixel type mean. (The data for the six means and the small intervals for each of the six histograms are stored separately on the sensor chip.) Sixteen values ​​are generally suitable for a histogram with relatively tightly defined coverage values.

[0130] The remaining two bytes indicate the pigment blending value for a specific pixel among the six pixel types. Alternatively, a 4-bit histogram algorithm based on the global mean can be used here. Each trace ratio has a global mean, and the histogram describes how the population of individual pixels of that pixel type varies around that mean. The specific 4-bit value indicates one of the histogram intervals and the corresponding calibration value accessed from memory and applied to adjust the DN value. There are four such 4-bit values ​​associated with a given pixel: one for magenta trace ratio, one for cyan, one for green, and one for yellow. (Here and elsewhere, we refer to four “trace” ratios. However, one of these colors is the assigned photosensitive point color, and therefore not just “trace.” All are treated similarly in terms of histogram representation.) Different histograms are associated with different types among the six layer types.

[0131] Of course, the arrangement just described is merely exemplary. Different applications have different considerations, including cost, physical space, complexity, power consumption, etc. The 3 bytes per pixel are illustrative, as is the use of the histogram indicating the difference from the corresponding mean. It is recommended to use one of the three bytes for on-the-fly offset / gain correction. The other 2 bytes, plus the offset / gain correction DN value, can be used in the demosaicing stage, which uses algebraic algorithms or AI / ML / CNN algorithms to derive the demosaicing color data for each pixel. The static 2-byte trace ratio value will simply be added to the static metadata of the demosaicing algorithm as part of its normal operation.

[0132] First exemplary filter set

[0133] According to certain other aspects of the technology, a novel set of different filters is used to select portions of a color filter array (CFA) to form a photodetector array and to filter light aligned to pixels of the photodetector array. As indicated, a color filter array typically comprises square cells of multiple filters, which are repeated across the photodetector and other such cells. In other configurations, cells with two or more different filter patterns can be laid out in a tessellation manner. In some arrangements, each filter in a cell may have a different spectral transmittance function T (sometimes referred to as a spectral profile or transmittance function) than the other filters in the cell. In other arrangements, certain filter types may be repeated within a cell (e.g., two green filters within a 2×2 Bayer cell). Sometimes non-square cells are used, including rectangular, triangular, and hexagonal cells.

[0134] Figure 5 Examples of color filter units with all nine different filters labeled A, B, C, D, E, F, G, H, and I are shown. They can be selected using, for example, the process described in application 18 / 056,704 filed November 17, 2022, but other selection processes may be used. The transmittance function for filter AI is shown in... Figures 6A-6I The following Table I details the transmittance function of the filters at wavelengths spaced 10 nm apart:

[0135]

[0136] Table I

[0137] The transmittance function data was measured without NIR or NUV filtering as found in some embodiments.

[0138] It should be noted that the maximum transmittance value in Table I is 0.9643 (i.e., at 690 nm in filter C). To provide a metric that can be compared across a dataset, it is sometimes helpful to normalize the filter transmittance values ​​to the maximum transmittance value in the set of filters. In this example, we divide each value in Table I by 0.9643 to obtain the data in Table II. We refer to this type of data as “group normalization.” The transmittance value for filter C at 690 nm is now 1.0, and all other data are proportionally larger. This group normalization is well known to practitioners, where, in teaching the practical operation of sensors and the use of these curves, for example, in color correction matrices, these normalizations revert to their unnormalized form. Since most of the following discussion involves wavelengths between 400 nm and 700 nm, we also limit the tables to this data (the extensions to wavelengths below 400 nm or above 700 nm are omitted to simplify this section of this disclosure):

[0139]

[0140]

[0141] Table II

[0142] The filters just described differ in their transmittance functions from those commonly encountered in the prior art (i.e., red, green, blue, cyan, magenta, and yellow filters). Fujifilm is a seller of these prior art filters. The transmittance functions of the 5000 series "color mosaic" line used for their red, green, and blue filters, and the 4000 series "color mosaic" line used for their cyan, magenta, and yellow filters, are considered normative. The published curves have been digitized, and the resulting transmittance function data for these six filters are detailed in Table III below. (As in the previous table, this data does not include infrared or ultraviolet filtering.)

[0143]

[0144]

[0145] Table III

[0146] Any filter with a transmittance function value equivalent to the values ​​given in column “R” of Table III is considered a regular or normal red filter. “Equivalent” here means that two arrays of transmittance values ​​from 400–700 nm—when each array is normalized to have a peak of 1.0—have a mean square error between them of less than 0.05. We similarly define filters with transmittance function values ​​equivalent to the values ​​given in columns G, B, C, M, and Y of Table III as normal (conventional) green, blue, cyan, magenta, and yellow filters.

[0147] Furthermore, the equivalent transmittance function is the pure filter transmittance value, without near-infrared (NIR) or near-ultraviolet (NUV) filtering or silicon quantum efficiency shaping. Generally, curves depicting the filter response of a camera are published. Of course, these curves are shaped by the spectral response of the camera sensor (typically silicon), and can also be shaped by NIR or NUV filtering. Figure 7(Based on data published by Canon for its 35mm FHDXS series camera sensors) The effects are shown. The red, green, and blue (R, G, B) filter curves are factored from the panchromatic (P) camera response curve (i.e., silicon efficiency). In the published data, the panchromatic curve is generally omitted. A clue to the factorization of the color filter curves from the panchromatic sensor response is that the green peak is higher than the red and / or blue peaks—attributable to the roll-off of silicon sensitivity at the red and blue ends of the visible spectrum. (The same consideration applies to the published cyan, magenta, and yellow curves.)

[0148] Filters that are substantially flat in the 400-700nm spectral range (i.e., their mean transmittance values ​​vary by less than + / -3% across this spectral range) are considered normal (panchromatic) filters in this paper.

[0149] Color filter units and arrays embodying aspects of this technology may include normal red, green, blue, cyan, magenta, yellow, and / or panchromatic filters. Any filter that is not a normal red, green, blue, cyan, magenta, yellow, or panchromatic filter is referred to as a "non-canonical" filter. Each of the filters described in Table I is a non-canonical filter.

[0150] In one embodiment, some or all of the filters are selected to achieve diversification. Diversification takes many forms and can be characterized using a variety of factors. The specific form of diversification preferred by the applicant is detailed below.

[0151] dot product

[0152] One factor that characterizes the diversity of spectral response curves is the dot product metric (also known as the inner product). The dot product metric is calculated by multiplying corresponding pairs of transmittance function values ​​taken from the two filters at each of several wavelengths spaced, for example, 10 nm apart, and then summing the results. Applicants prefer to calculate the dot product metric across the 400-700 nm range at 10 nm intervals, but other intervals and ranges can be used. Group-normalized data as shown in Table II is used. In this example, the dot product metric between filters A and B is calculated by summing the product of their corresponding transmittance functions at 400 nm with the product of their corresponding transmittance functions at 410 nm, and so on, up to 700 nm. That is:

[0153] The dot product measure (A,B) = (.6521)(.5620) + (.6445)(.4670) + … + (.0490)(.5556) = 11.2017.

[0154] The dot product is generally expressed in the form of a non-normalized dot product and a normalized dot product. This publication discusses both; we will use the non-normalized dot product for the purpose of discussion below.

[0155] Among the nine filters, there are 36 different filter pairs, from which 36 dot product measures can be calculated. For the nine filters detailed in Table II, the 36 dot products are shown in Table IV below:

[0156]

[0157]

[0158] Table IV

[0159] In any given set of filters, there will exist some pairs of transmittance functions that are more or less similar to the others. This is evident from the variation of the dot products in Table IV. For example, these dot products range from a minimum of 3.4899 to a maximum of 15.1875. The maximum value is 4.35 times larger than the minimum. The average of all 36 dot products is 8.24; their standard deviation is 2.99.

[0160] Some embodiments include color filter units, characterized in that: the dot product calculated between the normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm ranges from a minimum value to a maximum value less than 5 times or less than 4.5 times the minimum value.

[0161] Some embodiments include color filter units, characterized in that: the dot product calculated between the normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm ranges from a minimum value to a maximum value that is at least 3 times, at least 4 times, or at least 4.3 times the minimum value.

[0162] Some embodiments include color filter units, characterized in that the dot product calculated between the group normalized transmittance functions of two different filters in units spaced 10 nm apart from 400-700 nm is less than 5, less than 4, or less than 3.5.

[0163] Some embodiments include color filter units, characterized in that the dot product calculated between the group normalized transmittance functions of two different filters in units spaced 10 nm apart from 400-700 nm is at least 10, at least 12, or at least 15.

[0164] Some embodiments include color filter units, characterized in that the maximum dot product calculated between the group normalized transmittance functions of all different filter pairs in units spaced 10 nm from 400-700 nm is less than 17, less than 16, or less than 15.5.

[0165] Some embodiments include color filter units, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and at least 10% of the values ​​are less than 5.

[0166] Some embodiments include color filter units, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and 20% or more or 25% or more of these values ​​are less than 6.

[0167] Some embodiments include color filter units, characterized in that: the set of values ​​is obtained by calculating the dot product between the normalized transmittance functions of all possible pairs of different filters in the 400-700nm units spaced in 10nm intervals, and 40% or more of these values ​​are less than 7.

[0168] Some embodiments include color filter units, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and at least 20% or 25% or more of these values ​​are 10.

[0169] Some embodiments include color filter units, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in units spaced 10 nm apart from 400-700 nm has an average value of at least 6, at least 7, or at least 8.

[0170] Some embodiments include color filter units, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in units spaced 10 nm from 400-700 nm has an average value of less than 10, less than 9, or less than 8.5.

[0171] Some embodiments include color filter units, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in units spaced 10 nm apart from 400-700 nm has a standard deviation of at least 2.6 or at least 2.9.

[0172] Some embodiments include color filter units, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in units spaced 10 nm apart from 400-700 nm has a standard deviation of less than 3.5 or less than 3.

[0173] Each of the embodiments just described may consist in part or in whole of a non-standard filter.

[0174] Top code

[0175] Another metric useful in characterizing filter diversity is the so-called top code. A top code is an array of values ​​indicating which of two filters has a larger transmittance value at each wavelength in a series of uniformly increasing wavelengths. An exemplary top code is a binary sequence where "1" indicates that the first of the two filters has a larger transmittance value at a particular wavelength, and "0" indicates that the second of the two filters has a larger transmittance value at that wavelength. Again, we specifically consider a transmittance function sampled in 10nm increments between 400-700nm.

[0176] Regarding the term "code" and its variants (e.g., "coding," "codebins," etc.), this term forms a clear connection between signal measurement represented by pixel data and classical "coding theory." Coding theory provides a powerful and helpful tool for dealing with the general situation of using low-light, high-noise measurement systems (e.g., normal-view cameras) in extremely dark and twilight-like environments, including places where the signal-to-noise ratio is close to 1:1 or even lower.

[0177] Referring to Table I, we can see that filter A has a transmittance value of .5500 at 380nm, and filter B has a transmittance value of .7069. Therefore, the first bit of the top code (AB) starting at 380nm is "0". Filter A has a transmittance value of .5886 at 390nm, and filter B has a transmittance value of .6174, so the second bit of the top code (AB) is also "0". At 400nm, filter A switches to a transmittance value higher than filter B (i.e., .6288 vs .5420), so the third bit of the top code (AB) is "1". Continuing in this manner to all 41 wavelength samples from 380nm to 780nm, we obtain the complete top code (A,B) for this range:

[0178] Top-code(A,B) = 00111111111000000000000000000000000000000

[0179] Table V shows all 36 paired top code values ​​for the 9 filters in Table I in the 380-780nm wavelength range:

[0180]

[0181] Table V

[0182] We can narrow the 380-780 nm spectral range of the top codes in Table V by discarding bits at the beginning or end. For example, to generate top codes for the 400-700 nm spectral range, we discard only the first two bits and the last eight bits of each top code, changing the code length from 41 bits to 31 bits. The top codes from 400-700 nm for the filter set in Table I are shown in Table VI:

[0183]

[0184] Table VI

[0185] Crossing code

[0186] Within any top code string (vector), a transition between "1" and "0" indicates that one of the two transmittance function curves intersects with the other. For example, in the top code (A,B) from Table VI, there is a transition from "1" to "0" at the tenth position corresponding to 490 nm. This indicates that the transmittance function value of filter A drops below the transmittance function value of filter B somewhere between 480 and 490 nm. This can be seen in... Figure 8 From this, we can see that Figure 8 Presented in a superimposed manner, covering the range of 400-700nm (in Figure 6A and Figure 6B (Shown separately) Transmittance functions for filters A and B. The transition from "0" to "1" is represented by a signal indicating that the first curve has risen above the second curve.

[0187] By stepping through the top 31 bits of the code string to find the transition, we can derive the 30-bit string representing the curve intersection as a signal, which can be called the cross code. For each consecutive pair of bits in the top code string that have the same value ("1" or "0"), the cross code has a value of "0". When a transition occurs in the top code string, the cross code has a value of "1".

[0188] Taking the top code string (A, B) of element 31 as an example, there exists a single bit transition from "1" to "0" at the tenth position. Therefore, the corresponding cross code of element 30 is all "0" except for the "1" at the ninth position, i.e.: 000000001000000000000000000000

[0190] The cross codes corresponding to the top codes in Table VI are presented in Table VII:

[0191]

[0192] Table VII

[0193] The number of "1"s in each string indicates the number of times the two curves intersect. For example, the cross code (A, G) contains four "1"s, indicating that the curves intersect each other four times. The same applies to curves H and I.

[0194] Some embodiments include a color filter unit, characterized in that: the spectral transmittance curves of a plurality of paired filters defined by the sample from 400-700 nm at 10 nm intervals intersect each other at least four times.

[0195] Some embodiments include a color filter unit, characterized in that: the transmittance curves of a plurality of paired filters defined by samples from 400-700 nm at 10 nm intervals intersect each other exactly once or exactly zero times.

[0196] As previously described, each of the embodiments detailed may consist in part or in whole of a non-standard filter.

[0197] Cross code histogram

[0198] By summing the corresponding cross-code position values ​​for all 36 cross codes, we can determine the number of curve crosses in each wavelength band. For example, in the first position, all values ​​in all 36 cross codes are "0". Therefore, in the 400-410nm band, the filter transmittance function curves do not cross with any other filter transmittance function curve. The 36 cross codes in the second position (410-420nm) have a total of three "1"s, indicating the crosses between three pairs of curves (i.e., the (B,G), (C,F), and (D,E) curves). The complete set of this data is shown in Table VIII:

[0199]

[0200] Table VIII

[0201] This vector or set of data elements serves as a histogram of curve intersections for 30 wavelength bands. It can be called a cross histogram. Within this set of data elements in the cross histogram, the mean is 2.17 and the standard deviation is 2.05. The cross histogram shows zero counts for adjacent 10nm wavelength bands where no curves intersect. That is, within each 20nm range identified in Table VIII, the transmittance function curves for at least one pair of filters intersect each other.

[0202] Some embodiments include a color filter unit comprising three or more different filters, each of which has an associated transmittance curve, wherein the filter is characterized in that the count of the intersections between all pairs of said curves yields a vector of 30 count values ​​with an average value of at least 2 in each of thirty 10nm bands between 400-700nm.

[0203] Some embodiments include a color filter unit comprising three or more different filters, each of which has an associated transmittance curve, wherein the filter is characterized in that the count of the intersections between all pairs of said curves yields a vector of 30 count values ​​with a standard deviation of at least 2 in each of thirty 10 nm bands between 400-700 nm.

[0204] Some embodiments include a color filter unit comprising three or more different filters, each of which has an associated transmittance curve. The filter is characterized in that the count of the intersections between all pairs of said curves yields a vector of 30 count values ​​in each of thirty 10nm bands between 400-700nm, and one or more of said count values ​​have a value of at least 6, at least 8, or at least 9.

[0205] Some embodiments include a color filter unit comprising three or more different filters, each of which has an associated transmittance curve. The filter is characterized in that the count of the intersections between all pairs of said curves yields a vector of 30 count values ​​in each of thirty 10nm bands between 400-700nm, and two consecutive count values ​​in said vector are not both equal to zero.

[0206] As previously described, each of the embodiments detailed may consist in part or in whole of a non-standard filter.

[0207] Hamming distance

[0208] The difference between two cross codes can be indicated by the so-called Hamming distance between their bit strings. The Hamming distance between two strings of equal length is the number of positions where corresponding bits differ. Although Hamming distance is used in various disciplines, the applicant is unaware of its use in conjunction with the component filters of a color filter array in a color image sensor.

[0209] The Hamming distance between cross codes (A,B) and (A,C) is determined by comparing their string unions and counting the number of distinct bit positions. From Table VII:

[0210] Cross code (A,B) = 0000000010000000000000000000000

[0211] Cross code (A,C) = 000100000000000000100000000000

[0212] It can be seen that the two strings differ at three bit positions. Therefore, the Hamming distance between the cross code (A,B) and the cross code (A,C) is 3.

[0213] Returning to reference Table VII, the nine different filters can be paired in 36 different ways to obtain this set of 36 cross codes. That is, filter A can be compared with 8 other filters (BI), filter B can be compared with 7 other filters (CI), and filter C can be compared with 6 other filters (DI), and so on, until filter H can only be compared with 1 other filter (I). The number of such pairwise combinations among the nine items is sometimes called the 9-sum (i.e., 8 + 7 + 6... + 1 = 36).

[0214] In calculating the Hamming distance, the 36 cross keys in Table VII can be paired using a 36-sum method. That is, there are 630 Hamming distances between the 36 cross keys in Table VII. There are too many 630 values ​​to list here. Simply put, the values ​​range from 0 to 8, with a mean of 3.29 and a standard deviation of 1.23.

[0215] There are actually three Hamming distances of 0 in the set of 630 values: between the cross codes [(A,D) and (E,G)]; between the cross codes [(A,H) and (G,H)]; and between the cross codes [(B,H) and (D,F)]. Checking Table VII shows that each of these paired 30-bit cross code strings is identical. The latter of these pairs can be called trivial—both cross codes are zero. However, the other two pairs are non-trivial. Specifically, the first pair includes three "1"s in each cross code, while the second pair includes one "1" in each cross code.

[0216] The Hamming distance for 8 lies between the cross codes (A,G) and (H,I). There are two Hamming distances for 7 among the 630 values.

[0217] Some embodiments include a color filter unit, characterized in that: multiple Hamming distances between all possible cross codes defined between different filters in the unit have a value of zero. One or more of these zero Hamming distances may relate to cross codes that are not all zero. At least one of these zero Hamming distances may relate to a cross code that includes at least three "1"s.

[0218] Some embodiments include color filter units, characterized in that: multiple Hamming distances between all possible cross codes defined between different filters in the unit have values ​​of 5 or greater or 7 or greater.

[0219] Some embodiments include color filter units, characterized in that the average Hamming distance between all possible cross codes defined between different filters in the unit is at least 3.

[0220] Some embodiments include color filter units, characterized in that the standard deviation of all Hamming distances between all possible cross codes defined between different filters in the unit is at least 1.2.

[0221] Some embodiments include color filter units, characterized in that the standard deviation of all Hamming distances between all possible cross codes defined between different filters in the unit is less than 1.25.

[0222] As previously described, each of the embodiments detailed may consist in part or in whole of a non-standard filter.

[0223] efficiency

[0224] Another metric related to filter diversity is efficiency. For a given spectrum, the efficiency of a filter can be approximated as the average of the transmittance function values ​​spaced uniformly across that spectrum. Taking filter “A” in Table I as an example, the sum of 31 transmittance function values ​​in the 400-700 nm range (i.e., .6288 + .6214 + … + .0473) divided by 31 indicates an efficiency of 0.43 or 43%. The efficiencies of the nine filters “A”-“I” detailed in Table I are given in Table IX:

[0225]

[0226] Table IX

[0227] Some embodiments include a color filter unit, characterized in that the average efficiency across all non-canonical filters in the unit is at least 40%. In some such embodiments, the average efficiency across all non-canonical filters is at least 50%, or at least 60%, or at least 70%.

[0228] The efficiency of individual filters within a unit can vary significantly. In Table IX, efficiencies range from less than 25% to greater than 65%. That is, one filter can have an efficiency that is more than 2.65 times higher than the second filter in the unit.

[0229] Some embodiments include a color filter unit, characterized in that it includes a first non-standard filter having an efficiency that is at least 2.0 times or at least 2.5 times that of a second non-standard filter in the unit.

[0230] Some embodiments include a color filter unit, characterized in that: at least the third of a plurality of different non-normalized filters in the unit has an efficiency of at least 50%.

[0231] There is something to be done about balancing efficiency and versatility. High efficiency can be achieved by each of the filters having a transmittance function that remains near 100%. This will give you near 100% efficiency, but will provide relatively little versatility. On the other hand, high versatility can be achieved by filter curves that oscillate across a full range of possible values, sometimes running above 0.9 and sometimes falling below 0.1. However, these latter filters tend to have relatively lower efficiency, reducing the overall imager sensitivity.

[0232] Some embodiments include a color filter unit, characterized in that it includes at least one non-normalized filter having a group normalized transmittance function maintained above 0.4 in the wavelength range of 400-700 nm.

[0233] Some embodiments include a color filter unit characterized by comprising or having one or more non-normalized filters having a set normalized transmittance function maintained above 0.2 in the 400-700 nm wavelength range.

[0234] Some embodiments include a color filter unit, characterized in that it includes at least one filter having a group-normalized transmittance function maintained below 0.7 from 400-700 nm.

[0235] Some embodiments include a color filter unit, characterized in that it comprises a plurality of filters having a group-normalized transmittance function maintained below 0.75 from 400-700 nm.

[0236] Some embodiments include a color filter unit, characterized by comprising three filters having a group-normalized transmittance function maintained below 0.8 from 400-700 nm.

[0237] As previously described, each of the embodiments detailed may consist in part or in whole of a non-standard filter.

[0238] Correlation

[0239] Another useful metric for characterizing filter diversity is the sample correlation coefficient. Given two arrays x and y of sample values ​​of the transmittance function of n filters (e.g., the 31 values ​​for filters A and B detailed in Table I), the sample correlation coefficient r (hereinafter referred to as "correlation") is calculated as follows:

[0240]

[0241] The correlations between different pairs of filters AI spanning the 400-700nm wavelength range are detailed in Table X:

[0242]

[0243]

[0244] Table X

[0245] Some embodiments include color filter units, characterized in that the correlation calculated between the transmittance functions of two different filters in at least one of the non-canonical units, which are in the 400-700 nm range and spaced 10 nm apart, is negative.

[0246] Some embodiments include color filter units, characterized in that the correlation calculated between the transmittance functions of two different filters in units spaced 10 nm apart from 400-700 nm is at least 0.8, at least 0.9, or at least 0.95.

[0247] Some embodiments include color filter units, characterized in that: a set of values ​​is obtained by calculating the correlation between the transmittance functions of all possible pairings of different filters in the unit at 10 nm intervals from 400-700 nm, and at least one-quarter of these values ​​are at least 0.75. In another embodiment, this condition holds for all possible pairings of different non-canonical filters in the unit.

[0248] The mean of the correlation values ​​in Table X is 0.5596. The standard deviation is 0.2885. Eleven of the 36 items have values ​​within one standard deviation below the mean (i.e., between 0.2712 and 0.5596). Fourteen have values ​​within one standard deviation above the mean (i.e., between 0.5596 and 0.8308).

[0249] Some embodiments include color filter units, characterized in that: a set of values ​​is obtained by calculating the correlation between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and a first count of correlation values ​​within the mean of all values ​​in the set is greater than 25% of the smaller of the first and second counts, which is within one standard deviation above the mean of all values ​​in the set.

[0250] Slope and extreme values ​​of transmittance curve

[0251] Other measures useful in characterizing filter diversity involve the shape of local maxima and minima (extremes) in the transmittance function, indicated by the slope of the transmittance curves entering and leaving such extrema. As a general principle, the slope in an individual spectral response function causes color / spectral differentiation in a higher-dimensional spectral data space (e.g., the 9-dimensional data space in a 3x3, nine-channel sensor). Each value derived from the sensor defines one dimension in that 9-dimensional space. Thus, when one of the pixel's data values ​​changes rapidly as we change the spectral content being sensed, this also causes a slope in that 9-dimensional space. When one pixel is moving in one direction in that 9-dimensional space, while another pixel is moving in the opposite direction given the same movement in the sensed spectrum, the result is a spectral differentiation produced by that pair of pixels. Therefore, the "opposite slope" for each pair of pixels also becomes a useful measure of diversity.

[0252] The qualifier “local” indicates an extremum of the spectral transmittance function within a threshold neighborhood of the wavelength. An exemplary neighborhood spans 60 nm (i.e., from the center wavelength plus or minus 30 nm). For clarity, we sometimes refer to local maxima or minima as, for example, local maxima or minima spanning 60 nm.

[0253] To avoid small noise artifacts in the measured transmittance function being mistaken for local maxima, in the following discussion, we consider a local maximum only if the group-normalized value of a feature in the transmittance function curve of the filter is 0.05 higher than another transmittance function value in a 60 nm neighborhood centered on that feature. Similarly, for a minimum value—it must have a value 0.05 lower than another transmittance function value in a 60 nm neighborhood. If the transmittance function is at either end of the curve (indeed, for example, at...),... Figure 6A The left edge of the value is either high or low, and we do not know anything outside of it, so it is not referred to as a local maximum or minimum in this discussion.

[0254] If the transmittance function falls less than 25% below its maximum value within a 40nm spectrum centered at the maximum wavelength (sampled at 10nm intervals), then we consider the local maximum to be "wide." That is, the maximum is broadly topped. Specifically, for depressions; if the transmittance function value at the bottom of the depression falls less than 25% below the maximum transmittance function value within a 40nm spectrum centered at the depression wavelength, then we consider the depression to be "wide."

[0255] The antonym of "wide extrema" is "narrow extrema," and this also applies to both local maxima and local minima. That is, if the transmittance function drops by more than 25% from its highest value within a 40nm spectrum (intervals of 10nm) centered on the wavelength of the maximum, then we consider the local maxima to be "narrow." In other words, the maximum value is narrowly topped. Correspondingly, for local minima; if the transmittance function value at the depression is greater than 25% below the maximum value within a 40nm spectrum centered on the depression wavelength, then we consider the minimum to be "narrow."

[0256] Table I and Figures 6A-6I The curves provide an example. In filter A, a wide local maximum is found at 490 nm. Within ±20 nm of the top wavelength, its maximum value drops by only 5.5% (i.e., from .767 at the top to .725 at 470 nm). A total of seven wide local maximums exist across the nine filters AI. No narrow local maximums are found in these filters.

[0257] In filter D ( Figure 6D In the study, a wide local minimum was found at 590 nm. Its dip is only 19% below the maximum value within 20 nm (i.e., the transmittance function at 590 nm is 0.400, while the maximum value in the 40 nm window is 0.493 at 610 nm). This is the only wide local minimum among the nine filters detailed. Other transmittance functions that appear wider at first glance do not technically fall within the given definition because the steep edges are within the central 40 nm window and raise the transmittance function value high enough that the minimum is below 25%. See, for example... Figure 6E The minimum value is at 640 nm. Under the given definition, this is not a wide depression because its value (.07060) is more than 25% lower than the .1827 value found at 660 nm.

[0258] The above emphasizes the fact that the defined narrow / wide extrema distinction largely relates to the slope of the filter curve near the extrema. A 60nm span extrema adjacent to a steep slope is considered a narrow extrema, while a 60nm span extrema adjacent only to a gentle slope is considered a wide extrema.

[0259] A narrow local minimum is found at 450 nm in filter B. Its depression is 61.2% lower than another transmittance function value within the central 40 nm window (i.e., .203 vs .524). Seven narrow local minima exist in the detailed filter set (including...). Figure 6E (The minimum value just discussed in the text).

[0260] Some embodiments include a color filter unit, characterized in that the count of local minima across a narrow 60nm span exceeds the count of local maxima across a narrow 60nm span. Some such embodiments are characterized in that the count of local minima across a narrow 60nm span is at least 150%, at least 200%, at least 300%, or at least 400% of the count of local maximum values ​​across a 60nm span.

[0261] Some embodiments include a color filter unit, characterized in that the count of local minima spanning a narrow 60 nm band exceeds the count of local minima spanning a wide 60 nm band. Some such embodiments are characterized in that the count of local minima spanning a narrow 60 nm band is at least 150%, at least 200%, at least 300%, or at least 400% of the count of local minima spanning a wide 60 nm band. Some such embodiments are characterized in that the count of local minima spanning a narrow 60 nm band is at least seven times the count of local minima spanning a wide 60 nm band.

[0262] Some embodiments include a color filter unit, characterized in that the count of a local maximum spanning 60 nm wide exceeds the count of a local maximum spanning 60 nm narrow. Some such embodiments are characterized in that the count of a local maximum spanning 60 nm wide is at least 150%, at least 200%, at least 300%, or at least 400% of the count of a local maximum spanning 60 nm narrow.

[0263] Some embodiments include a color filter unit, characterized in that the count of local maxima spanning 60 nm exceeds the count of local minima spanning 60 nm. Some such embodiments are characterized in that the count of local maxima spanning 60 nm is at least 150%, at least 200%, at least 300%, or at least 400% of the count of local minima spanning 60 nm. Some such embodiments are characterized in that the count of local maxima spanning 60 nm is at least seven times the count of local minima spanning 60 nm.

[0264] In some other embodiments, the criteria for the first four paragraphs were not met.

[0265] Some embodiments include a color filter unit, characterized in that: one or more non-normal filters in the unit include a 60 nm span local maximum between 430 and 670 nm, and are wide-topped (i.e., extending over ±20 nm from the 60 nm span local maximum with a transmittance function drop of 25% or less from the 60 nm span local maximum).

[0266] Some embodiments include a color filter unit, characterized in that: one or more non-canonical filters in the unit include a local maximum spanning 60 nm between 430 and 670 nm, and most of the N filters are wide-topped.

[0267] Some embodiments include a color filter unit, characterized in that: a plurality of non-normal filters in the unit include a local maximum spanning 60 nm between 430 and 670 nm, and all of the plurality of filters have a transmittance function drop of less than 50% relative to the transmittance value at the local maximum within ±20 nm of the maximum.

[0268] Some embodiments include a color filter unit, characterized in that one or more non-normal filters in the unit have a transmittance function including a local maximum spanning exactly 60 nm.

[0269] Some embodiments include a color filter unit, characterized in that one or more non-normal filters in the unit have a transmittance function that does not include a local maximum spanning 60 nm.

[0270] Some embodiments include a color filter unit, characterized in that one or more non-normal filters in the unit have a transmittance function including a local minimum spanning exactly 60 nm.

[0271] Some embodiments include a color filter unit, characterized in that one or more non-normal filters in the unit have a transmittance function that does not include a local minimum spanning 60 nm.

[0272] Some embodiments include a color filter unit, characterized in that: the transmittance function of one or more non-canonical filters in the unit has a transmittance function that includes exactly one local minimum over a 60 nm span and does not include any local maximum over a 60 nm span.

[0273] Some embodiments include a color filter unit, characterized in that: the transmittance function of one or more non-canonical filters in the unit has a transmittance function that includes exactly one local maximum value over a 60 nm span and does not include a local minimum value over a 60 nm span.

[0274] Some embodiments include a color filter unit, characterized in that: the transmittance function of one or more non-canonical filters in the unit has a transmittance function including exactly one local maximum and one local minimum over a 60 nm span.

[0275] Some embodiments include a color filter unit with multiple different non-standard filters, characterized in that: the count of the maximum width among the non-standard filters is greater than the count of the minimum width among the non-standard filters.

[0276] Some embodiments include a color filter unit with multiple different non-standard filters, characterized in that: the count of the narrowest values ​​among the non-standard filters is greater than the count of the narrowest values ​​among the non-standard filters.

[0277] As mentioned, the slope of the filter curve connected to the extrema can vary. This diversity can be aided by the diversity of the slope of the transmittance curve.

[0278] We define the slope of the curve as the change in group-normalized transmittance over a span of 10 nm (i.e., from 400 nm to 410 nm, 410 nm to 420 nm, etc.). Although the 10 nm interval is defined, the slope is expressed in units of nanometers. For example, between 690 nm and 700 nm, the group-normalized transmittance value of filter A changes from .0403 to .0490 or .0087 over a span of 10 nm. It therefore has a slope of .00087 / nm. Depending on whether the curve rises or falls with increasing wavelength, the slope can be positive or negative.

[0279] Table XI gives the slope of the filter “A”–“I” described in Table II:

[0280]

[0281]

[0282] Table XI

[0283] Some embodiments include a color filter unit comprising at least one non-normalized filter, characterized in that: the slope of the normalized filter transmittance function of all groups is calculated to obtain a set of values ​​when calculated over 10 nm intervals between 400-700 nm, and at least 60% of the values ​​are negative.

[0284] The filter curve can also be partially characterized by the absolute value of the slope.

[0285] Some embodiments include a color filter unit comprising at least one non-normalized filter, characterized in that: the slope of the absolute value of the normalized filter transmittance function of all groups of non-normalized filters is obtained when calculated over 10 nm intervals between 400-700 nm, and at least 50% of the values ​​are less than 0.01 / nm or less than 0.005 / nm.

[0286] Some embodiments include a color filter unit comprising at least one non-normalized filter, characterized in that: the slope of the absolute value of the normalized filter transmittance function of all groups of non-normalized filters is obtained when calculated over 10 nm intervals between 400-700 nm, and at least 20% of the values ​​are less than 0.001 / nm.

[0287] In characterizing the diversity of a filter set, it is sometimes useful to classify the state of the transmittance curve in a particular region into one of the following three categories: (1) in the neighborhood of a peak, (2) in the neighborhood of a trough, or (3) between neighborhoods.

[0288] Peaks and troughs can be the previously defined local maxima and minima spanning 60 nm. The neighborhood of a peak can include those points whose transmittance values ​​are within 0.15 of the local maximum (sampled at 10 nm intervals). The neighborhood of a trough can include those points whose transmittance values ​​are within 0.15 of the local minimum (sampled at 10 nm intervals).

[0289] Figure 9 The transmittance function curve of filter A after group normalization is shown. (and...) Figure 6A The comparison shows the same curve shape, but Figure 9 The values ​​in the table have been scaled so that one of the filters in the set (in this case, filter C) reaches a peak of 1.0. According to Table II, there is a trough at 450 nm with a value of 0.5497. The associated trough neighborhood includes the transmittance function values ​​of filter A at 400, 410, 420, 430, 440, 450, and 460 nm, because each of these values ​​is within 0.15 of 0.5497.

[0290] Similarly, a peak exists at 490 nm with a value of .7952. The associated peak neighborhood includes filter A transmittance function values ​​at 470, 480, 490, 500, 510, 520, and 530 nm, as their values ​​are within 0.15 of .7952. It is understandable that peak and trough neighborhoods may overlap in some instances.

[0291] Recall that local extrema spanning 60 nm are defined by referencing a 60 nm wide neighborhood (i.e., 30 nm or less from the center wavelength). Since spectral transmittance function data beyond 400–700 nm are sometimes unavailable, local extrema are typically defined starting at 430 nm and ending at 670 nm.

[0292] Figure 9A second trough and a second trough neighborhood exist on the right. The filter curve shows a local minimum of transmittance function at 690 nm with a value of .0403. Since it is unknown whether the curve continues to decline, for example, at 710 nm or 720 nm, it is unknown whether this value meets the definition of a local minimum (i.e., a trough) across a 60 nm span. Nevertheless, 620, 630, 640, 650, 660, 670, 680, 690, and 700 nm can all be identified as falling within a trough neighborhood because they all have group normalized values ​​below 0.15. That is, it is known that such values ​​will be at least 0 regardless of the transmittance function value at 710 nm or 720 nm, therefore the aforementioned wavelengths with values ​​of 0.15 or less must necessarily be within 0.15 of the minimum transmittance value. Thus, sometimes a trough neighborhood can be identified even when a specific trough (i.e., a minimum across a 60 nm span) cannot be identified. The same applies to crest neighborhoods. In other words, any transmittance function sample with a normalization value of 0.85 or higher will necessarily be within the peak neighborhood.

[0293] By applying the definitions of peak and trough neighborhoods to the group-adjusted filter transmission data in Table II, we can identify which filter functions fall into which type of neighborhood at which wavelengths. This information is represented by the symbols "MAX" and "MIN" in Table XII.

[0294]

[0295] Table XII

[0296] The blank blocks in the table above correspond to regions of the filter curve that are neither close to the peaks nor the troughs. (As used in this paper, "close to" means within 0.15 based on the group-normalized filter transmittance value). As mentioned, these regions between the peak and trough neighborhoods constitute the third type of transmittance function region. In these intermediate regions, each "third type" 10nm wavelength span is characterized by a slope value, which can be positive or negative as detailed in Table XI.

[0297] In a nine-filter embodiment comprising more than three different filters, at each of at least 16 of the 24 10nm wavelength bands between 430 and 670 nm, the first group of filters 1-5 are all at or near local extrema, the second group of filters 1-5 are all positively sloped, and the third group of filters 1-5 are all negatively sloped.

[0298] In the second and third groups, the magnitudes of the slopes are expected to include a variety of values, for example, often in the range of 0.001 / nm to 0.1 / nm. For example, in the band between 470-480 nm, the filters in the second group have slopes of 0.019 / nm and 0.033 / nm (i.e., filters B and I), and the filters in the third group have slopes of -0.0022 / nm, -0.0064, and -0.0089 / nm (i.e., filters E, F, and C). It should be understood that the nine different filters fall into different groups in different wavelength bands.

[0299] Examining Tables XI and XII reveals that among the nine filters and 25 wavelengths sampled at 10nm intervals from 430-670nm (i.e., 9*25 or 225 filter-wavelengths), 88 filter-wavelengths lie in the trough neighborhood and 64 in the peak neighborhood. The remaining 73 filter-wavelengths (i.e., the blank blocks in Table XII) define the endpoints of third-class 10nm spans with positive or negative slopes. (Some of these spans begin at the last member of the peak or trough neighborhood of the filter-wavelength). Specifically, 28 of these third-class spans have positive slopes (ranging from .0005 / nm to .0837 / nm), of which more than one-third have values ​​greater than .01 and more than one-third have values ​​less than .01. There are 71 of these third-class spans with negative slopes (ranging from -.00009 / nm to -.0134 / nm), of which more than one-third have values ​​above -.0005 and more than one-third have values ​​below -.0005.

[0300] The examination further revealed that for each of the 25 sampled filter values ​​from 430-670 nm, there exists at least one filter whose transmittance value lies in the peak neighborhood. Similarly, for each of the 25 sampled filter values, there exists at least one filter whose transmittance value lies in the trough neighborhood. (In fact, for each of the 25 sampled filter values, there are at least four filter functions that lie in the extreme neighborhood, with one span (i.e., 660 nm) having eight filter functions that lie in the extreme neighborhood). For each of the 25 sampled filter values, there is also at least one filter that lies neither in the peak neighborhood nor in the trough neighborhood but rather in the "third category". (At one wavelength (i.e., 550 nm), there are five filter functions that lie in this third category).

[0301] Some embodiments of the technology include color filter units comprising N different filters (where N can be three or more, four or more, nine or more, or 16 or more filters), and including one or more non-canonical filters. Each filter is characterized by a set of normalized transmittance functions comprising values ​​sampled at 10 nm wavelength intervals from 400-700 nm, wherein some sampled values ​​are within 0.15 of a local maximum value for the corresponding filter across a 60 nm span and are referred to as members of a peak neighborhood, while some sampled values ​​are within 0.15 of a local minimum value for the corresponding filter across a 60 nm span and are referred to as members of a trough neighborhood. Some filter functions for 24 different 10 nm wavelength spans extending between 430-670 nm include 10 nm spans that are neither entirely within a peak neighborhood nor entirely within a trough neighborhood. Some of the 10nm spans exhibit a positive slope with increasing wavelength, while others exhibit a negative slope with increasing wavelength. These embodiments are characterized by:

[0302] For each of the 25 wavelengths sampled at 10nm intervals from 430-670nm, there exists at least one filter whose transmittance function lies in the neighborhood of the peak, and / or

[0303] For each of the 25 wavelengths sampled at 10nm intervals from 430-670nm, there exists at least one filter whose transmittance function lies in the neighborhood of a trough, and / or

[0304] For each of at least 20 of the 24 different 10nm spans between 430nm and 670nm, there exists at least one filter whose transmittance function is neither entirely within the peak region nor entirely within the trough neighborhood, and which also has a negative slope value, and / or

[0305] For each of the 24 different 10nm spans between 430nm and 670nm, there exists at least one filter with a negative slope whose transmittance function is neither entirely within the peak region nor entirely within the trough neighborhood, and / or

[0306] On N different filters (each with transmittance function values ​​sampled at 10nm intervals from 430-670nm, thus defining 25N filter-wavelengths), at least 25% of the 25N filter-wavelengths are neither in the peak region nor in the trough neighborhood, and / or

[0307] 25N filters - at least 65% of the wavelength is in the neighborhood of a peak or trough, and / or

[0308] One or more of the positive slopes have a value less than .001 / nm, and / or

[0309] One or more of the positive slopes have a value of at least .05 / nm, and / or

[0310] One-third or more of the positive slope has a value less than .01 / nm, and / or

[0311] One-third or more of the positive slope has a value of at least .01 / nm, and / or

[0312] One or more of the negative slopes have values ​​between -0001 / nm and zero, and / or

[0313] One or more of the negative slopes have values ​​between -.01 / nm and -.1 / nm, and / or

[0314] One-third or more of the negative slope has a value between -0.005 / nm and zero, and / or

[0315] One-third or more of the negative slopes have values ​​between -005 / nm and -1 / nm.

[0316] The filters detailed in Table I are readily available and used for proof-of-concept testing. In practice, filters with more complex transmittance profiles can be used. An “M”-shaped profile—like that of a Bactrian camel—is an example of a more complex filter transmittance profile. Such a profile rises from a low value in the blue or ultraviolet spectrum, reaches a first peak at a first wavelength, then falls to a trough, rises again to a second peak at a second wavelength, and finally falls again in the infrared or red spectrum. Another exemplary profile is “W”-shaped—starting at a value in the blue or ultraviolet spectrum, then falling to a first trough, then rising to a peak, then falling to a second trough, and then rising again towards the infrared or red wavelength.

[0317] In one embodiment employing a non-normal M-shaped transmittance function filter, two local peaks have corresponding transmittance values ​​within 0.25 of each other.

[0318] In other embodiments, one or more filter curves are still more complex—including three local maxima or minima spanning 60 nm (e.g., a three-humped camel).

[0319] While complex transmittance functions offer benefits, the applicant has found that overly complex transmittance functions can introduce problems. For example, metamerism becomes more prevalent if one or more filters in a color filter unit of N different filters have a relatively large number (as defined above) of local maxima and minima. Such metamerism can interfere with accurate color measurements. The applicant has found that this problem arises when the combined count of local minima and maxima in the transmittance function of the filters has a value of (N-2) or greater. This is the aforementioned "relatively large number." Therefore, in a color filter unit of nine different filters, the problem becomes concerning when one or more filters have seven or more local extrema. In a color filter unit of six different filters, the problem becomes concerning when one or more filters have four or more local extrema. And so on. Therefore, while not necessary, the applicant generally uses filters with N-3 or fewer local extrema in each of the N different color filter units.

[0320] Figures 6A-6I The curves given in the figure and detailed in Table I are for dyed polyester filters commercially sold from Rosco Laboratories. Specifically, the filters are:

[0321] 4390CalColor 90 cyan

[0322] 3304 Tough Plus Green

[0323] 343 Neon Pink

[0324] 55 Pale Lilac

[0325] 65 Sunlight Blue

[0326] 365Tharon Delft Blue

[0327] 370 Italian Blue

[0328] 386 leaves green

[0329] 389 shades of green

[0330] While dyed polyester filters can be used in photoelectric sensors, the pixels will generally be relatively large. Pigmented filters are more commonly used in photoelectric sensor arrays. Filters based on interference filters, dichroic mirrors, and quantum dots (nanoparticles) can also be used. Some embodiments are achieved by mixing pigment powder / paste or nanoparticles in a (negative) photoresist carrier. Different pigments and nanoparticles absorb at different wavelengths, resulting in dips in the resulting transmission spectrum. The higher the concentration, the deeper the dip.

[0331] Second exemplary filter set

[0332] One of the applicant's objectives in selecting filters was to obtain a diverse set of filters with optimized modulation transfer functions (MTFs) compared to other alternatives. This led to the development of the first set of filters described above. It also led to a second set of nine filters detailed below. Again, these nine filters are selected from the Rosco catalog:

[0333] AA: Dark straw color (#15)

[0334] BB: True Pink (#337)

[0335] CC: Cal30 Green (#4430)

[0336] DD: Lapis Lazuli (#92)

[0337] EE: Gallo Gold (#316)

[0338] FF: Cal60 Pink (#4860)

[0339] GG: GaslightGrn (#388)

[0340] HH: Cal60 Cyan (#4360)

[0341] II: Sky Blue (#72)

[0342] Transmission data for this second filter set was plotted on Figures 10A-10I This is further detailed in Table XIII:

[0343]

[0344] Table XIII

[0345] When normalized as described above, this second filter set has a transmittance function in the wavelength range of 400-700 nm, as detailed in Table XIV:

[0346]

[0347] Table XIV

[0348] Many characteristics characterizing the second filter set are similar to or identical to those of the first filter set. Some of the characteristics characterizing the second filter set are discussed below. Other characteristics can be determined directly from the provided data using the methods previously taught.

[0349] The dot product values ​​among the nine different filters in the second set range from a minimum of 6.33 to a maximum of 17.14. The maximum value is 2.7 times the minimum value. The average of the 36 dot product values ​​is 11.11, and the standard deviation is 2.72. The complete set of dot products is shown in Table XV:

[0350]

[0351] Table XV

[0352] Some embodiments include color filter units composed entirely or partially of non-normalized filters, characterized in that: the dot product calculated between the normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm ranges from a minimum value to a maximum value less than 3 times or less than 2.75 times the minimum value.

[0353] Some embodiments include color filter units consisting entirely or partially of non-normalized filters, characterized in that: the dot product calculated between the normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm ranges from a minimum value to a maximum value that is at least twice or at least 2.5 times the minimum value.

[0354] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that the dot product calculated between the group normalized transmittance functions of two different filters in units spaced 10 nm apart from 400-700 nm is less than 7 or less than 6.5.

[0355] Some embodiments include color filter units composed of some or all non-normalized filters, characterized in that the dot product calculated between the group normalized transmittance functions of two different filters in units spaced 10 nm apart from 400-700 nm is at least 14, at least 16, or at least 17.

[0356] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and the average value of the set of values ​​is at least 9, at least 10, or at least 11.

[0357] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and the average value of the set of values ​​is less than 13, less than 12, or less than 11.5.

[0358] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in the unit spaced 10 nm from 400-700 nm has a standard deviation of at least 2 or at least 2.5.

[0359] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in units spaced 10 nm apart from 400-700 nm has a standard deviation of less than 2.75 or less than 3.

[0360] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and at least 20% of the values ​​are less than 9.

[0361] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in units spaced 10 nm from 400-700 nm, and at least 10% of the values ​​are at least 15.

[0362] The top code, cross code, and cross histogram for the second filter set can be determined using the methods detailed previously. The cross histogram for the second filter set is shown in Table XVI:

[0363]

[0364] Table XVI

[0365] The mean value in this histogram is 1.97, and the standard deviation is 1.60.

[0366] Some embodiments include a color filter unit comprising three or more different filters, each of which has an associated transmittance curve, the filters being wholly or partially composed of non-normal filters, characterized in that: the count of the intersections between all pairs of the curves yields a vector of 30 count values ​​with an average value less than 2 in each of thirty 10nm bands between 400-700nm.

[0367] Some embodiments include a color filter unit comprising three or more different filters, each of which has an associated transmittance curve, the filters being wholly or partially composed of non-normal filters, characterized in that: the count of the intersections between all pairs of the curves yields a vector of 30 count values ​​with a standard deviation of less than 1.7 in each of thirty 10nm bands between 400-700nm.

[0368] As previously mentioned, the difference between two cross codes can be indicated by the Hamming distance. Similar to the first filter set (Table I), the 36 cross codes associated with the second filter set can be paired in 630 ways. In the case of the second filter set, the Hamming distance ranges from 0 to 7, with a mean of 3.065 and a standard deviation of 1.24. There are 11 Hamming distances of 0 in the set of 630 values. There are 4 Hamming distances of 7.

[0369] Some embodiments include color filter units, characterized in that the average Hamming distance between all possible cross codes defined between different filters in the unit is less than 3.1.

[0370] The efficiency of the filters in the second set can be calculated as detailed above. The results are shown in Table XVII:

[0371]

[0372] Table XVII

[0373] It can be seen that, in the 400-700nm range, the efficiency of the filters in this second set varies from less than 40% to close to 70%. The average value is 48.5%.

[0374] Some embodiments include a color filter unit, characterized in that at least 85% of the plurality of different non-standard filters in the unit have an efficiency of at least 40%.

[0375] Some embodiments include a color filter unit, characterized in that at least one non-normal filter in the unit has an efficiency of more than 66%.

[0376] Some embodiments include a color filter unit consisting of some or all of non-standard filters, characterized in that the average efficiency calculated on all different filters in the unit exceeds 48%.

[0377] The diversity of the second filter set can be partly characterized by its narrow and wide 60 nm span extremes. There are nine 60 nm span minimums, six of which are wide and three are narrow (the latter are at 450 nm in filter CC, 450 nm in filter EE, and 640 nm in filter II). There are six 60 nm span maximums, four of which are wide and two are narrow (the latter are at 490 nm in filter FF and 500 nm in filter EE).

[0378] Some embodiments include a color filter unit with one or more non-canonical filters, characterized in that the count of local minima spanning 60 nm wide exceeds the count of local minima spanning 60 nm narrow. Some such embodiments are characterized in that the count of local minima spanning 60 nm wide is at least 150% or at least 200% of the count of local minima spanning 60 nm narrow.

[0379] Some embodiments include a color filter unit with one or more non-canonical filters, characterized in that the count of local minima spanning 60 nm exceeds the count of local maxima spanning 60 nm. Some such embodiments are characterized in that the count of local minima spanning 60 nm is at least 150% of the count of local maxima spanning 60 nm.

[0380] Some embodiments include color filter units with one or more non-canonical filters, characterized in that the count of filters having extreme values ​​across a wide 60 nm span exceeds the count of filters having extreme values ​​across a narrow 60 nm span. Some such embodiments are characterized in that the count of local extreme values ​​across a wide 60 nm span is at least 150% or at least 200% of the count of local extreme values ​​across a narrow 60 nm span.

[0381] Third exemplary filter set

[0382] Transmission data for the third filter set used in the wavelength range of 400-700 nm are detailed in Table XVIII:

[0383]

[0384] Table XVIII

[0385] When normalized as described above, this third filter set has a transmittance function as detailed in Table XIX:

[0386]

[0387] Table XIX

[0388] The filters in this third set are again primarily stained filters from Rosco. The exception is the last one, filter III, which comes from Lee Filters USA.

[0389] AAA: Neon Pink (#343)

[0390] BBB: Cal90 Yellow (#4590)

[0391] CCC: Sky Blue (#72)

[0392] DDD: True Pink (#337)

[0393] EEE: Dark straw color (#15)

[0394] FFF: Cal30 Green (#4430)

[0395] GGG: Cal30 Cyan (#4330)

[0396] HHH: Cal60 Cyan (#4360)

[0397] III: Medium amber (Lee #20)

[0398] Five of these filters are the same as the second set discussed above. The table data for the second set (Table XIII) was scanned from a printed data sheet, while the table data for the third set (Table XVIII) was measured using a spectrometer. A certain variation will be noted.

[0399] Many characteristics characterizing this third filter set are similar to or identical to those of the first and / or second filter sets. Some of the characteristics characterizing this third filter set are discussed below. Other characteristics can be determined directly from the provided data in a manner previously taught.

[0400] The dot product values ​​for the nine different filters in the third set range from a minimum of 7.79 to a maximum of 21.26. The maximum value is again 2.7 times the minimum. The average of the 36 dot product values ​​is 13.8, and the standard deviation is 3.61. The complete set of dot products is shown in Table XX:

[0401]

[0402] Table XX

[0403] Some embodiments include color filter units composed of some or all non-normalized filters, characterized in that the dot product calculated between the group normalized transmittance functions of two different filters in units spaced 10 nm from 400-700 nm is at least 18, at least 20, or at least 21.

[0404] Some embodiments include color filter units composed of some or all non-normalized filters, characterized in that the dot product calculated between the group normalized transmittance functions of two different filters in units spaced 10 nm apart from 400-700 nm is less than 10, less than 9, or less than 8.

[0405] Some embodiments include color filter units composed of some or all non-normalized filters, characterized in that the minimum dot product calculated between the group normalized transmittance functions of all different filter pairs in units spaced 10 nm from 400-700 nm is at least 6, at least 7, or at least 7.5.

[0406] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in the unit spaced 10 nm from 400-700 nm has an average value of at least 10, at least 12, or at least 13.

[0407] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in units spaced 10 nm from 400-700 nm has an average value of less than 17, less than 15, or less than 14.

[0408] Some embodiments include color filter units composed of some or all non-normalized filters, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in the unit spaced 10 nm from 400-700 nm has a standard deviation of at least 3 or at least 3.5.

[0409] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that the set of dot products between the group normalized transmittance functions of all different filters in the unit spaced 10 nm from 400-700 nm has a standard deviation of less than 4 or less than 3.7.

[0410] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in the 400-700 nm range at 10 nm intervals, and at least 10% of the values ​​are less than 8.5.

[0411] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in the 400-700 nm range at 10 nm intervals, and at least 25% of the values ​​are less than 11.

[0412] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in the 400-700 nm range at 10 nm intervals, and at least 20% of the values ​​are greater than 16.

[0413] Some embodiments include color filter units composed partly or entirely of non-normalized filters, characterized in that: a set of values ​​is obtained by calculating the dot product between normalized transmittance functions of all possible pairs of different filters in the 400-700 nm range at 10 nm intervals, and at least 10% of the values ​​are greater than 19.

[0414] Forming good images from problematic data

[0415] Noise can impair the usability of a color image. A familiar example is in underexposed images (e.g., captured in low light or with short exposure intervals), where colors are low in saturation and the image has blotches due to pixels of incorrect colors. (Readout noise is generally a problem in low light conditions. In other cases, shot noise may dominate when the supply of photons is insufficient.)

[0416] This technology offers high-confidence techniques for accurately and less noisyly recovering color from increasingly noisy imaging. The "color orientation" of a hue represents one aspect of color measurement. In extremely dark scenes where the signal-to-noise ratio of the pixel data itself is close to 1:1, classic processing of RGB data is challenging, even in measuring the principal orientation of color along the red-green and yellow-blue axes. The method described below accomplishes this task admirably in increasingly lower light levels while still determining the hue angle and principal orientation.

[0417] First, consider a noise-free image. Then, consider a region within the imaged scene with a spectral hue of 400 nm. Each pixel imaged in this region will produce an output signal that depends on the transmittance function of its filter at 400 nm. Refer to Table I, and specifically consider pixels covered by filters A and B (e.g., ...). Figure 6A and Figure 6B As shown in the diagram, pixels filtered by filter A will produce relatively high values, while pixels filtered by filter B will produce relatively low values. This is because the former filter function allows more energy to pass through at 400nm than the latter (i.e., transmittance values ​​of .6288 vs .5420). However, due to the transmittance value of .6288 vs .7063, pixels covered by filter A will produce relatively lower values ​​than pixels covered by filter C.

[0418] In a similar manner, depending on whether the transmittance value in Table I used for filter A at 400 nm is higher or lower than the transmittance values ​​used for each of the other filters, filter A will allow more or less light to pass through than each of the other six filters DI used to image the 400 nm scene area.

[0419] Similarly, depending on their respective transmittance values ​​at that wavelength, filter B will allow more or less light to pass through than each of the other seven filters CI that image the 400nm scene area. And, depending on their transmittance values ​​at that wavelength, filter C will allow more or less light to pass through than each of the other six filters DI that image the 400nm scene area. And so on.

[0420] Therefore, in a correctly exposed scene, pixels subjected to different filters will produce different output signals from the 400nm scene region based on their corresponding transmittance values ​​at that wavelength. The output signals from pixels subjected to different filters can be compared based on the transmittance values ​​in Table I and arranged in order of magnitude, and will be found to be ordered at 400nm: EDCFABGHI. There are different orders (i.e., 36) that can be used to arrange the filtered pixels.

[0421] The arrangement of the filters will differ at different wavelengths of the spectrum. At 410 nm, it is the same as at 400 nm (i.e., EDCFABGHI). However, at 420 nm, it switches to DEFCAGBHI. It is the same at 430 nm, but switches to DEFACGBHI again at 440 nm. Each segment of the spectrum is associated with a unique arrangement of the filter's output signal. There are 26 unique arrangements of the filters across 31 sampling wavelengths in the range of 400–700 nm. Repetitions are expected to be adjacent. The complete set of arrangements is given in Table XXI. These nine-letter arrangements can be referred to as the spectral reference string.

[0422]

[0423] Table XXI

[0424] If the scene lighting becomes dim, or if the camera exposure becomes very short, the output signals from the pixels will become noisy. Some will be higher than they should be, while others will be lower. Some remnants of the nine wavelength-based sorted outputs from the different filters will remain. (Consider the 400nm sorting again. At the two ends of the sorted sequence of filters, filter E has a transmittance value of .8075 at 400nm, while filter I has a transmittance value of .0426 at 400nm. That is, in a 400nm scene, a pixel filtered with filter E would normally produce an output signal nearly 20 times larger than a pixel filtered with filter I. Even in extreme noise, it would be impossible for the latter pixel to produce a larger output signal than the former.)

[0425] In noise recovery mode, the output signals of pixels in a 3×3 pixel region from the photoelectric sensor can be arranged by amplitude to indicate the corresponding order of their filter transmittance values ​​at an unknown wavelength of incoming light. This ordering of the filters will be slightly disturbed by noise effects, but the ordering will correspond more closely to some spectral reference strings in Table XXI than others. The closest match in Table XXI can be used to indicate the hue of the incoming light.

[0426] Various known string matching algorithms can be used. One is based on Hamming distance. First, the order of the outputs from nine differently filtered pixels in a low-light scene is determined. This nine-element sequence is called the query string. Then, the query string is compared with each of the 36 spectral reference strings in Table XXI. Within each spectral reference string, the number of positions where the query vector has different letters at a given string position is counted. The smaller this count, the better the query string matches the spectral reference string. The spectral reference string that best matches the query string (i.e., the string with the smallest count of letter differences from the query string) indicates the hue at that region in the photosensitive sensor.

[0427] If a query string most closely matches two spectral reference strings, then the query string can be considered to match the wavelength between the two wavelengths indicated by the two spectral reference strings. For example, if the query string most closely matches the spectral reference string EDCFABGHI, and that reference string is found in table XXI for both 400nm and 410nm, then the query string can be associated with the hue at 405nm.

[0428] The method described above does not use the 31 nine-letter strings as shown in Table XXI. Instead, it uses a sequence of 36 31-symbol topcodes from Table VI.

[0429] Recall that each top code in Table VI represents a pair of filters that have a higher transmittance value at 31 wavelengths from 400 to 700 nm. The first entry in Table VI compares filters A and B. The second entry in Table VI compares filters A and C. And so on, until all 9-summated (36) possibilities are found.

[0430] Within each top code, the first digit indicates which of the paired filters has a higher transmittance value at 400 nm. The second digit indicates which of the paired filters has a higher transmittance value at 410 nm. And so on, up to all 31 sampling wavelengths.

[0431] If we read the top code data in Table VI by vertical columns instead of horizontal rows, we get a 36-digit sequence for each wavelength. Each of the 36 digits indicates which of the two filters in the corresponding pair has the maximum output at the target wavelength. We call each of these 36-digit sequences a reference tone code. Completing this exercise yields the reference tone codes for Table XXII:

[0432]

[0433] Table XXII

[0434] Referring to the reference hue code for 400nm, the initial symbol "1" indicates that filter A has a transmittance value greater than that of filter B. The second symbol "0" indicates that filter A has a transmittance value less than that of filter C. The next three "0"s in the reference hue code indicate that filter A has a transmittance value less than that of each of filters D, E, and F. A subsequent symbol "1" indicates that filter A has a transmittance value greater than that of filter G. The next two symbols are both "0", indicating that filter A has a transmittance value greater than that of filters H and I. The subsequent symbols for the 400nm reference hue code continue in this manner, comparing filter B with all the remaining filters, then filter C, and so on.

[0435] The binary reference hue codes just described in Table XXII are equivalent to the spectral reference strings in Table XXI. Each hue code corresponds to a specific spectral wavelength and serves as a reference for comparing codes derived from the noisy image data to assign a spectral hue to each pixel in the noisy image data. Again, the Hamming distance can be used to compare the reference hue codes with the query codes derived from the noisy image data with respect to a specific pixel to determine the best match (i.e., the minimum Hamming distance). The best-matching reference hue code indicates the most likely hue for the query pixel.

[0436] Considering a 3x3 pixel unit, image a uniformly tinted scene region in low light using the filters detailed in Table I. Pixels will produce different output signals (data) – varying both due to their different filters and due to underexposure-related noise. Treat the nine output data as 36 pairs to identify which filter produces the larger output signal for each pair. If a pixel filtered by A produces an output signal greater than pixels filtered by B, C, and D but less than pixels filtered by E, F, G, H, and I, the lookup code will begin with the number 11100000… The remaining bits of the lookup code are generated by comparing the output signal from the pixel filtered by B with others, then the pixel filtered by C, and so on. This produces a 36-bit lookup code. Comparing the lookup code with each of the reference hue codes in Table XXII reveals that the lookup code is closest to the reference hue code used for 430nm (i.e., 1100001100000110000111111111111111111). Therefore, the pixel values ​​for this region in noisy image frames will be replaced with the RGB (or CMY) pixel values ​​corresponding to the 430nm hue.

[0437] In one embodiment, a lookup table in memory stores the corresponding red, green, and blue (RGB) values ​​for each hue code that define the color of a pixel. This mapping from hue to RGB values ​​can be achieved by first identifying the CIE XYZ chromaticity coordinates for each hue of interest. These XYZ coordinates are then transformed to the desired RGB space by a 3x3 transformation matrix. A suitable matrix corresponding to the sRGB standard with D65 illumination is:

[0438]

[0439] This, along with other suitable transformation matrices, can be found on the website brucelindbloom. <dot>You can find it at com / index.html?Eqn_RGB_XYZ_Matrix.html.

[0440] Therefore, one embodiment of the technology involves receiving values ​​corresponding to output signals from a plurality of photoelectric sensors within a local neighborhood, wherein the photoelectric sensors include photoelectric sensors covered by filters having different spectral transmittance functions. Then, based on the received values, the method includes: providing, for example, a set of multiple color values ​​(e.g., R / G / B or XYZ) from a memory for use with a target photoelectric sensor within the neighborhood. The method may also include: for each of the plurality of photoelectric sensors, determining whether the output signal corresponding to that photoelectric sensor is greater than or less than the output signal corresponding to another of the photoelectric sensors. Generally, this involves: for each of a plurality of paired photoelectric sensors within the neighborhood, determining which of the two has a larger received value. In many embodiments, the provided plurality of color values ​​all correspond to a specific hue, and such color values ​​are only available for discrete sampling of the hue, lacking other hues between the discrete samplings of the hue.

[0441] In the aforementioned arrangement, both binary hue codes and a previously referenced spectral reference string are used, assuming the underexposed scene comprises regions of a specific color, such that each of the nine differently filtered pixels is noisily sampled for the same color using the transmittance function of its corresponding filter. In most imaging scenarios, this is not a bad approximation because most adjacent pixels will be similar in chromaticity. Therefore, in some embodiments, each pixel is considered to be at the center of the cell, and its hue is determined based on a comparison of its value with the values ​​of other pixels in that cell. (If the cell does not have a central pixel, a near-central pixel can be used).

[0442] Other arrangements can be used as alternatives. Consider using arrangements that all have... Figure 5 The photoelectric sensor is arranged in an array of 3x3 color filter units. The pixel of interest (i.e., the target pixel) is determined by one of the nine filters (e.g., ...). Figure 11 The filter "A" is shown in bold in the image. Different filters are used to filter the other pixels surrounding the target pixel, depending on the cell and the tile pattern.

[0443] To perform one of the above methods without assuming a constant or near-constant color region (based on a binary hue code or spectral reference string), we interpolate the values ​​of pixels filtered by B, C, D, E, F, G, H, and I to the position of the subject pixel filtered by "A". We then employ the previously described method: in the case of the ninth actual filtered pixel value at the target pixel, we use eight such interpolated filtered pixel values ​​at the target pixel. Various interpolation techniques can be used.

[0444] A simple technique is linear interpolation. To project the B-filtered value onto the location of the target pixel, we look at the values ​​of the two nearest B-filtered pixels (i.e., those in the same row). Figure 11A The image shows two pixels identified as B1 and B2. Their values ​​are weighted according to the inverse of their distances to the target pixel, where the weights sum to 1. In this case, the interpolated "B" value at the target pixel location is:

[0445] B1 / 3 + 2B2 / 3

[0446] Similarly, the linear interpolation value is calculated for a photoelectric sensor with other filters (i.e., filters C, D, and G) found in the same row and column as the target pixel. Filtered pixels located on the same diagonal as the target pixel (e.g., Figure 11B The same applies to pixels E1 and E2 in the image.

[0447] For filters that are not in the same row or column as the target pixel, different interpolation methods can be used (e.g., bilinear or bicubic interpolation). Figure 11C An example is shown to project F-filtered pixel values ​​at a target location. Bilinear interpolation is illustrated, involving the performance of three linear interpolations. First, the values ​​of the two upper "F" pixels (F1 and F2) are combined with a 2 / 3, 1 / 3 weighted combination to obtain the F-filtered pixel value for linear interpolation at the location indicated by the pair of upper opposing arrows. The values ​​of the two lower "F" pixels (F3 and F4) are combined in the same manner to obtain the F-filtered pixel value for linear interpolation at the location indicated by the pair of lower opposing arrows. These upper and lower interpolation values ​​are then combined with a 1 / 3, 2 / 3 ratio to obtain the interpolated value for the "F" pixel at the target location.

[0448] At the target location, we then have the pixel value measured by the photoelectric sensor and filtered by the "A" filter, along with eight interpolated pixel values ​​from the other eight filters. These nine values ​​are then compared to each other to generate a 9-symbol ordered lookup string or a 36-bit binary lookup code. This string / code is compared to a reference spectral string or hue code to find the best match, and thus determine which hue should be assigned to the target pixel. This process is similarly performed for all pixels in the photoelectric sensor array (or only underexposed pixels).

[0449] It should be recognized that the process just described for assigning colors to pixels / photodetectors is derived from a limited, discrete sampling of hues. For example, in one embodiment detailed earlier, there are 26 unique reference sequences (i.e., reference spectral strings) for the filters. This corresponds to 26 hues. In contrast, a photodetector providing 8 bits of red, green, and blue output data (i.e., a 24-bit output color space) has 2^24 possible hues. Therefore, the available samples are very sparse—less than 1 out of a total of 2^19 available colors. Of course, depending on the implementation, the sparsity can vary between different embodiments. However, many implementations will still be able to produce less than 1% or less than .001% or less than .000001% of the hues that can be represented in the output color space.

[0450] The conclusion drawn from the foregoing is that a many-to-one mapping exists between the photodetector values ​​in the neighborhood surrounding the target photodetector (pixel) and the indicated hue value (and corresponding RGB output value). The arrangement of photodetector output values ​​and the results of comparisons between pairs of pixels are insensitive to certain changes in the photodetector values. For example, a photodetector covered with filter A will have a larger output signal than a photodetector covered with filter B, regardless of whether the former has an output value of 20 or 200, as long as the latter has an output value of 10.

[0451] In addition to the noise reduction capabilities provided by the current arrangement, many embodiments can be implemented without complex arithmetic operations. For example, determining whether the output value for one photodetector is greater than the output value for another photodetector is a simple operation that can be performed by an analog comparator (if done before the accumulated photoelectron charge in the photodetector is converted into digital form) or a digital comparator (if done afterward). Such operations can be implemented with hardware circuitry simpler than the arithmetic operations commonly used in image denoising (which may include multiplication or division).

[0452] The above arrangement provides hue output data but is silent on luminance. Luminance can be estimated based on the magnitude of the photodetector signal at the target pixel. Alternatively, a weighted average of nearby photodetector signals can be used, where the target pixel is typically given a greater weight than other pixels. Non-linear weighting can also be used to reduce the influence of outlier signal values. If the average transmittance values ​​of the various filters differ, the photodetector signals can be normalized accordingly so that, for example, filters that allow a small portion of full-color light to pass through—in estimating local image luminance—count more filters than filters that allow a relatively large portion of full-color light to pass through. In yet another embodiment, local luminance in the imaged region can be estimated based on the statistical distribution of the (normalized) values, since low-light images exhibit greater bias. Different RGB values ​​can be stored in a lookup table memory for different combinations of hue and luminance. Alternatively, a single set of RGB values ​​can be stored for each hue, and the values ​​can then be scaled up or down based on the estimated luminance.

[0453] Therefore, according to one embodiment, a value associated with a first pixel having a first spectral response function is compared with values ​​associated with a plurality of other pixels in a neighborhood having a spectral response function different from the first spectral response function, and a color or hue is assigned to the first pixel based on the result of the comparison. In some embodiments, the comparison is performed without any multiplication or division operations. In some embodiments, the comparison is used to determine the order of the pixels. Some embodiments include assigning color or hue based on Hamming distance or based on the result of a string matching operation.

[0454] Naturally, the above teachings can be practiced using filters and filter units having the properties described in detail above.

[0455] CMYRGB filter arrangement

[0456] Suppose that the color filter array designer has only six choices of color photoresist at their disposal (i.e., red R, green G, blue B, cyan C, magenta M, and yellow Y). These colors are generally available from commercial sellers. They are therefore economical, and the manufacturing processes using such photoresist are well understood. Table III provides an exemplary set of transmittance functions for such filters.

[0457] The transmittance function of a filter using this type of photoresist can be varied according to the Beer-Lambert law by changing its thickness. Such variations can include changing the width at the maximum and minimum values, as well as changing the slope of the filter. Figure 12 This is exemplary and illustrates the transmittance function of a magenta photoresist at thicknesses of 450 nm, 850 nm, and 1.0 μm at various wavelengths. By using the same color photoresist but fabricating multiple filters within a filter unit with different thicknesses, different filtering functions can be achieved, thereby expanding the filter palette from an initial set of, for example, six photoresists.

[0458] Figure 13 Exemplary spectral response functions for six colors (CMYRGB) are shown for applications all at a thickness of one micrometer. (Units are arbitrary, where 1000 represents complete transparency and 0 represents complete opacity.) Note that the slope for yellow between 450 and 500 nm is almost identical to the slope for green in the same wavelength range. From an informational standpoint, this is undesirable. Similar redundancy occurs for other filter pairings.

[0459] Figure 14 This illustrates how varying thicknesses can induce significantly different linearly independent spectral filter functions. The two blue, two cyan, and two red filters are all doubled, each accompanied by a curve depicting the transmittance function for photoresist thicknesses at 800 nm and 350 nm. This set of nine curves is generated using only six color photoresists. Therefore, the arrangement described elsewhere in this disclosure, employing nine diverse filter functions, can thus be achieved using only six photoresists.

[0460] Figure 15 Slightly more abstract, but only depicts Figure 14 The first derivative of the curve (i.e., the slope). It is from the interaction of slopes that the spectral information is differentiated, and here we see the beautiful diversity of slopes caused by coarse / fine bifurcation. For example, we can see that the peak in the coarse red "slope" is offset by approximately 10 nm from the peak in the fine red; this diversity is a major driving factor behind color accuracy. There are various physical and manufacturing methods that can be utilized to produce monochromatic photoresists with this coarse / fine bifurcation (or, for that matter, trifurcation).

[0461] One approach is to form an uncolored, transparent, stabilized (cured) layer of positive or negative photoresist at each of certain filter locations within the chip. This creates a raised, clear substrate (platform) on which subsequent layers of photoresist can be coated, serving to thin the subsequent photoresist layer at locations lacking transparent photoresist.

[0462] Figure 16 The concept is illustrated. In this example, a transparent photoresist is coated on a photodetector substrate 171 and exposed, developed, and cleaned using a mask (sometimes collectively referred to as "masking") to form transparent substrates 172 at five locations in the nine-filter unit. The photoresist may have a thickness of 500 nm. Figure 17 This section of the sensor is shown after five subsequent mask layers have defined five color filters (e.g., red, green, blue, cyan, and magenta).

[0463] exist Figure 16 In the liquid state of the structure, a first colored photoresist, marked "A," is applied. Where the transparent substrate is missing, the photoresist accumulates downwards onto the sensor substrate, forming a layer, for example, 1000 nm thick. Figure 17 As indicated by the longer double-headed arrow on the left. Where the transparent substrate is present, the liquid photoresist does not accumulate downwards to this depth; instead, it resides on top of the substrate, forming a 500nm thick layer. This photoresist is masked and washed away from any location other than the desired photoresist "A". This results in units of filter layers with different thicknesses of photoresist "A"—a thin layer 181 present in the transparent substrate and a thick layer 182 absent in the transparent substrate. These filter layers of different thicknesses have different filtering capabilities according to Beer-Lambert's law.

[0464] According to Beer-Lambert's law, if a first filter with an absorbing medium has a layer thickness L1 and a transmittance T1 (on a scale of 0-1), then a second filter with the same medium and a layer thickness L2 will have a transmittance function T2 as follows:

[0465]

[0466] The process described above is repeated a second time using a second photoresist "B". Again, photoresist "B" flows down to the areas where the transparent substrate is missing and accumulates on top of the areas where the transparent substrate is present. The pigment layer is masked to leave two thicknesses—thin where the pigment resides on the transparent substrate and thick where the pigment resides on the substrate—areas of pigment "B".

[0467] In the depicted example, the process is repeated twice more with photoresists "C" and "D". For each color, a thick filter layer and a thin filter layer are formed—the latter located at the position with the transparent substrate. Finally, a fifth layer of photoresist "E" is coated onto the wafer and masked to create a filter at the center of the color filter unit. Reference Figure 16 As can be seen, a transparent substrate exists at this location. Therefore, the photoresist layer "E" does not extend downwards to the photoelectric sensor substrate, but instead resides only on top of the transparent substrate in the 500nm layer.

[0468] Figure 17 The arrangement thus includes nine different filtering functions, but this is achieved using only six masks (a mask for forming the transparent base and a mask for each of the five color pigment layers).

[0469] In the example shown, the thicker and thinner filter layers of the same color photoresist have a thickness ratio of 2:1 (i.e., 1000 nm and 500 nm). However, this is not mandatory. Such ratios can range from 1.1:1 to 3:1 or 4:1 or larger. Common ratios are between 1.4:1 and 2.5:1, with ratios between 1.5:1 and 2:1 being more prevalent.

[0470] Figure 18 The diagram shows a segment from a color filter unit, in which a 400 nm thick green photoresist layer is formed on top of a 300 nm thick transparent substrate. Elsewhere in this color filter unit, there may be green pigment layers extending downwards to the level forming the transparent substrate with a thickness of 700 nm. The thickness ratio of these green-dyed filter layers is therefore 1.75:1 (i.e., 700:400).

[0471] In some exemplary embodiments, the substrate has a height between 200 and 500 nm, and photoresist is coated to a depth to achieve a thick filter of 600 to 1100 nm (where the substrate is not located). In one particular embodiment, the entire substrate has a height of 200-300 nm. In the same or other embodiments, photoresist is coated to form a thick filter of 700-1000 nm (a thinner filter where the substrate is located).

[0472] exist Figure 17 In this implementation, thin and thick filters of a given photoresist color are edge-to-edge adjacent to each other. This is not necessary. In some implementations, such thin and thick filters of the same photoresist color are corner-to-corner adjacent to each other, or not adjacent at all. Some CFAs (or units) have combinations of such relationships, where thin and thick filters of the first color photoresist are corner-to-corner adjacent to each other, while thin and thick filters of the second color are edge-to-edge adjacent to each other, or not adjacent at all. Some CFAs (or units) are characterized by all three relationships: corner-to-corner adjacent for thin and thick filters of the first color, edge-to-edge adjacent for thin and thick filters of the second color, and non-adjacent for thin and thick filters of the third color.

[0473] Figure 16 The checkerboard pattern of the transparent base can be reversed, with the four corner positions and the center position lacking a base, and bases additionally formed at the other four positions. Instead of four or five bases, the unit can include a number ranging from 1 to more or less than 1 less than the total number of filters in the unit. As long as the number of bases is less than the number of filter elements in the unit (i.e., some filter elements do not reside on the base), the array of bases can be called "sparse." That is, not every photodetector (or microlens) is associated with a base.

[0474] One embodiment is thus an image sensor comprising a sparse array of transmissive substrates, wherein a photoelectric sensor array is deployed below the substrates and a colored filter medium (e.g., pigment) is deployed above the substrates. The sparse array may be a checkerboard array, but it does not have to be. This arrangement generally includes filter elements of both thicker and thinner dimensions, with the thinner filter elements all deployed above one of the transmissive substrates.

[0475] Although usually, the base of the chessboard (e.g., Figure 16 (The two bases) collide at their corners, but this is not necessary. A chessboard pattern with gaps can include an array of such bases without colliding at the corners (e.g., by horizontally dividing the bases). Figure 16 Each of them is reduced in size by 1% or more (e.g., 2%, 5%, 10%, 25% or 50%).

[0476] Figures 19A-19E Several such sparse patterns are shown, where "T" indicates the location of a filter with a transparent base. Each of these can then be reversed, where the transparent base is formed in an unmarked location instead of the location marked with "T". It can be seen that the transparent base location can be edge-adjacent, corner-adjacent, or non-adjacent, or any combination of these three within a given cell. (Here, as in the previous discussion of thick and thin filters of the same color, adjacency is declared in the context of a single cell. Different adjacencies may arise once the cell is laid out with other cells.)

[0477] An advantageous arrangement includes a 3x3 filter unit formed by seven masking steps (a step for forming a pattern of a transparent base and a step for each of six subsequently coated coloring photoresists (e.g., red, green, blue, cyan, magenta, and yellow)). (Sometimes the first three colors are called primary colors, and the latter three colors are called secondary colors).

[0478] Figure 20 Showing the use Figure 19E This type of unit, with its base pattern, comprises three transparent bases. The three locations with transparent bases (three of the four corners) have less dense color filters because such filters are physically thinner. These are indicated by lighter lines and letters. The locations lacking transparent bases have denser color filters because such filters are physically thicker. These are indicated by darker lines and letters.

[0479] In this example, each of the three primary color filters appears twice in the filter unit—once in the thinner layer and once in the thicker layer. Each of the secondary color filters appears only once in the unit—each time in the thicker layer (i.e., not formed on top of the transparent base).

[0480] In other arrangements, the filter appearing twice in the unit may be a secondary color. In yet another embodiment, the filter appearing twice in the unit may include one or more primary colors and one or more secondary colors.

[0481] Naturally, it is not required that this unit include all three primary colors and all three secondary colors. Other functional filters may be included—including filters with desired ultraviolet (e.g., below 400 nm) and infrared (e.g., above 750 nm) properties, and the various non-traditional types of filters previously detailed. Each of these filters may be included once in the unit, or may be included twice—once thin and once thick. Of course, the presence of three transparent substrates is not required in a 3×3 unit; more or fewer may be present. In some 3×3 units, six transparent substrates are used, such that six of the filter layers are relatively thin, while the three filter layers are relatively thick. As in other embodiments detailed in this specification, some pixels may not be filtered by a color photoresist that is transparent at all wavelengths involved (panchromatic).

[0482] Of course, a transparent base for achieving a thinner filter layer can be used in cells of different sizes than 3×3 (e.g., 4×4, 5×5 cells and non-square cells).

[0483] Another exemplary implementation is shown in Figure 21 In this implementation, a 2x2 Bayer cell is used. However, the first masking operation defines a transparent base at one of the four pixel locations (in the upper left, indicated by lighter lines and letters). Three additional masking operations follow, defining four color filters: one for red, one for blue, and two for green. The green filter formed on top of the transparent base, in the upper left, is thinner than the green filter formed in the lower right. (The upper left green filter is also thinner than the red and blue filters.) Due to its thinner thickness, this thinner green filter allows more light to pass through than the thicker green filters (which, like the red and blue filters, are of conventional thickness). This increases the sensor's efficiency. According to Beer-Lambert's law, the thinner filter also broadens the spectral profile. This alters the slope and position of the filter edges, resulting in improved color accuracy.

[0484] Figure 22 Showing the use of Figure 21 The graph shows exemplary spectral curves for the four filters in the unit, with the thin green filter shown as a thick solid line. This graph is for cases where the thin filter is one-third the thickness of the other filters. (The red, green, and blue curves are based on data from Table III).

[0485] Given the sensitivity of the human visual system (HVS) to green, the Bayer unit employs two green filters in its 2x2 pattern. If the sensor is used for machine vision purposes, the HVS-based rationale for dual green is irrelevant, and the other color can be doubled (i.e., red or blue). Figure 23 This illustrates a modified Bayer cell employing two diagonally adjacent blue filters (one thick and one thin). Figure 24 The transmission curves used for this arrangement are shown. The curve for the thin blue filter is shown by a thick solid line. Again, the thin filter is one-third the thickness of the other filters. Figure 22 With the same setup, this modification increases sensor efficiency and diversifies the spectral curves—achieving better color accuracy.

[0486] The cells do not need to be square. Since six readily available colored photoresists (i.e., the three primary colors red, green, and blue, and the three secondary colors cyan, magenta, and yellow) are available, such photoresists can be used to form six filters within a 2×3 pixel cell. Again, a transparent substrate can be formed first on some of these pixels so that the photoresist later masked at such locations is thinner relative to pixels lacking a substrate.

[0487] Figure 25 This unit is shown. A transparent base is formed beneath an intermediate-colored filter, as indicated by a fine border and thinner lettering. A base beneath a primary-colored filter is absent, as indicated by a coarse border and thicker lettering.

[0488] Figure 25 The unit can be paired with a related unit whose filter color is shifted one pixel to the left while the former's base pattern remains unchanged. This is shown in Figure 26 The top two lines include... Figure 25 The units are identical except that all filters are shifted one position to the left. The result is a 4×3 pixel unit with 12 filters, containing thin and thick filters for four of the six colors, along with two thin filters for the fifth color (here, cyan) and two thick filters for the sixth color (here, red). As in the previous example, thin and thick filters for the common color are formed in one masking step—the difference being the transparent base beneath the thin filters. Thus, despite including ten different filter functions, only seven masking steps are used to produce... Figure 26 The unit (and CFA formed by multiple units). Improved color accuracy is achieved through the diversity of slopes provided by ten different filter functions. Six thin filters also contribute to increased efficiency.

[0489] Therefore, one embodiment includes an image sensor with a sparse (e.g., checkerboard) pattern spanning a transparent base of the sensor, wherein the pattern defines two types of scattered positions: relatively raised positions and relatively lowered positions. A continuous region of the sensor includes cyan, magenta, and yellow filters at positions of one type (e.g., relatively lowered (i.e., without a base)) and red, green, and blue filters at positions of the other type (e.g., relatively raised (i.e., with a base)).

[0490] exist Figure 27 The image shows another arrangement using all six primary / secondary colors. This is a 1x6 linear unit, in which every other filter element is formed on a transparent base (in this embodiment, the secondary colors magenta, cyan, and yellow are below the filters, although one or more primary colors may be replaced).

[0491] A second row of bases can be formed, with each base positioned horizontally by one location, such that each base is adjacent to another base at a corner. This second row can be formed by... Figure 27 The same filter sequence is covered, but shifted two places to the left. Figure 28 The 2x6 unit was thus created. It will be recognized that this 12-filter element unit includes thin and thick filters for each of the six colors, providing 12 different filtering functions. This large number of diverse filter functions achieves excellent color accuracy, while the large number of thin filters provides high efficiency. Figure 26 Similar to the unit, this unit can be manufactured using only seven masking operations (an operation for the transparent base and an operation for each of the six colors).

[0492] It can be replicated through the laid layout. Figure 28 The cells, where identical 2x6 cells are positioned on the left, right, top, and bottom, are repeated as needed to span the blocks of the photoelectric sensor. This results in a checkerboard-like arrangement of pixels with transparent bases. These transparent bases are defined in the initial masking step. The resulting 3D checkerboard structure provides square wells, which facilitate the creation of color filters at the intermediate pixel positions in the subsequent masking step. Figure 16 The layout appears as a chessboard, but when laid with similar structures, it is found that many transparent base edges are adjacent to other bases, rather than only at the corners—as is the case in the chessboard.

[0493] Figure 29 The diagram illustrates the group-normalized transmittance function for a six-element unit employing five photoresists. Two filters are formed with blue photoresist—one thick and one thin (e.g., approximately 1000 nm and approximately 500 nm). Three other thick filters are present (e.g., approximately 1000 nm): red, green, and cyan. Finally, a thin yellow filter is present (e.g., approximately 500 nm). It should be understood that this arrangement can form a thin filter element on top of a clear substrate with a height of approximately 500 nm.

[0494] This is a more general type of color filter unit, characterized by comprising N>3 filter elements, two of which are formed of the same photoresist, but one has a thickness between 80% and 20% (and preferably, between 66% and 33%, and most preferably, between 60% and 40%) of the thickness of the other.

[0495] This is also a type of color filter embodiment, characterized by comprising N>3 filter elements, wherein two or more of the filter elements are relatively thin, having a thickness between 20% and 80% (and preferably, between 33% and 66% of, and most preferably, between 40%) of the thickness of another filter element in the unit. In one such arrangement, one of the relatively thin elements has a relatively thick equivalent element in the unit formed of the same material, while the other of the relatively thin elements does not have a relatively thick equivalent element in the unit formed of the same material. In one such arrangement, one of the thin filters is a red, green, or blue pass-through filter, while the other of the thin filters is a red, green, or blue attenuation filter (i.e., a cyan, magenta, or yellow filter).

[0496] Many further variations exist. For example, two masking operations can be used to form two layers of transparent substrates, some on top of others. For instance, the first masking operation can create six 500nm thick substrates at a location in a 3 x 3 cell. The second masking operation can form three additional substrates (e.g., 300nm thick) – all on top of one of the 500nm thick substrates created in the first masking step. This results in a first set of three substrates with a thickness of 500nm and a second set of three substrates with a total thickness of 800nm. The three other locations in the cell have no substrates. When the latter locations are filled with a colored photoresist to a thickness of 1000nm, the photoresist will form a 500nm thick layer on top of the transparent substrates of the first set and a 200nm thick layer on top of the transparent substrates of the second set. The colored photoresist can thus form two or three filters of different thicknesses in a single masking operation.

[0497] In one embodiment, after two masking operations to form substrates of different thicknesses, three further photoresists are successively coated and masked to create three different colored filters (e.g., selected from red, green, blue, cyan, magenta, and yellow). Each of these photoresists can be masked at a location corresponding to the three different thicknesses. Five masking operations then yield nine different filtering functions.

[0498] In another variation, the substrate is not transparent (clear), but rather selectively absorbs light through the spectral means, for example, by using a colored photoresist. This arrangement yields the same thickness modification results discussed above, but with the added spectral modulation of the substrate color. Each of the above arrangements can be practiced using a spectrally absorbing substrate. (While "transparent" or "clear" substrates have 98% or more transmittance across the 400-700 nm range, spectrally absorbing substrates have less transmittance at certain wavelengths. We use the general term "transmissive" substrate to encompass both transparent and spectrally absorbing substrates.)

[0499] In some embodiments, the substrate is partially or entirely formed with a photoresist that cuts infrared light (e.g., a stained photoresist). If one filter in the cell is formed on this IR-cut substrate, while another filter is formed on the same photoresist but not on the IR-cut substrate (e.g., it is not formed on the substrate, or it is formed on a substrate that transmits infrared light), the resulting different responses of the two pixels at infrared light provide information about the scene content at infrared light. This can be useful, for example, in AI applications.

[0500] One particular photoresist exhibits an IR-narrowed panchromatic response. An IR-narrowed panchromatic response is a substantially panchromatic response across visible light wavelengths, possessing a spectral transmittance function greater than 80%, 90%, or even 95% in the 400-700 nm range, but then narrowing down to a less IR-responsive state. In a particular embodiment, the spectral transmittance function of this photoresist is below 50%, 20%, or 10% at some point in the 700-900 nm range, more preferably at some point in the 700-780 nm range (e.g., at 720 nm, 740 nm, or 760 nm).

[0501] In other variations, for example, if the refractive index of the substrate is greater than or less than the refractive index of the photoresist being coated, they can have optical functions.

[0502] Both positive and negative photoresists can be used in the detailed arrangement. The choice of tone can be based on practical considerations (e.g., light speed, resolution, sidewall slope / aspect ratio, implantation stopping capability or etch resistance, and flow properties).

[0503] Photoresists with high solids content (e.g., pigment-loaded CFA photoresists) generally work best in negative frequency modulation because the so-called "pebbles" (solids content) are most easily removed if the photoresist matrix not exposed by the photolithography process has the best possible solubility in the developer. If a positive photoresist is chosen, the volume to be removed first needs to be dissolved by exposure, which may result in more residue formation. Furthermore, due to absorption with depth, the sidewall profile will tend to be flatter, which is undesirable in continuous CFA arrays.

[0504] The creation of the substrate adds a further degree of variability to the manufacturing process. This variability can be measured and recorded in the Chromabath process, as detailed in this paper, just like other process variations. The Gaussian variability of the thickness of the (thinner) layer formed on top of the substrate will likely be greater by a percentage than the variability of the thickness of the filter not formed on top of the substrate—another dimension of variability that can be characterized in the Chromabath process.

[0505] An embodiment of one aspect of the technology is a color filter array comprising a first filter formed on a transparent substrate and composed of a first colored photoresist, and a second filter not formed on the transparent substrate and composed of the same first colored photoresist, wherein the second filter has a thickness greater than that of the first filter.

[0506] According to another embodiment of the technology, a photoelectric sensor is included, the photoelectric sensor including a checkerboard pattern spanning a transparent base of the photoelectric sensor.

[0507] According to another embodiment of the technology, a color filter unit has filtered pixels with N different spectral transmittance functions, created using only M masks, where M < N. In some such embodiments, M = N-1; in other such embodiments, M = N-2, or M = N-3, M = N-4, or M = N-5.

[0508] Generating pixel spectral profiles through layer thickness, rather than using a mask to produce those profiles, is one way to achieve a diverse set of filter functions. In some embodiments, the filters are extracted solely from CMYRGB color photoresists.

[0509] While this section has focused on commercially available conventional R, G, B, C, M, Y and infrared photoresists, it should be understood that the detailed techniques can be applied to any kind of filter medium, including filters (and filter units) having the properties previously discussed in this specification, including non-standard filters.

[0510] Examples using near-infrared radiation

[0511] Image sensors that generate red, green, blue, and near-infrared channels for output data are known. The Canon 120MXS sensor is exemplary and includes an array of silicon pixels covered with a color filter array, each unit of which includes three visible light filters (red, green, and blue) and a near-infrared filter. The near-infrared output channel enables differentiation between image features that otherwise appear as the same, based solely on red, green, and blue data.

[0512] According to certain embodiments of this technology, an image sensor includes four pixels that are filtered to obtain maximum output (i.e., exhibit maximum sensitivity) between 400 and 700 nm. These are called visible light pixels, contrasted with NIR-filtered pixels used in image sensors (e.g., Canon 120MXS). At least two of these four pixels have a strong color response at wavelengths extending through red (e.g., 650 nm) and into the near-infrared (i.e., above 700 nm). The sensitivity of these two or more visible light pixels into the near-infrared range enables four channels for generating image data, at least one of which is influenced by the infrared image content.

[0513] Figure 30 Taken from a Canon datasheet for the Canon 120MXS sensor, and showing the response of its filtered red, green, and blue pixels in the near-infrared range. Figure 30 The solid line also shows the pixel response in the monochrome version of the Canon sensor. This is the sensor's panchromatic response (i.e., without the covered color filter array). The shape of this panchromatic response curve is primarily attributed to the quantum efficiency of the silicon photoelectric sensor, but is also influenced by the sensor's microlens array and other factors.

[0514] The above references filtered pixels that exhibit a "strong" response at certain wavelengths. As used herein, a strong response is a response that is 50%, preferably 60%, 70%, or 80% or more of the sensor's full-color (unfiltered) response at that wavelength. For example, Figure 30 The red-filtered pixels in the sensor have a strong response from 580nm up to 800nm, with a response of over 60% from 590-800nm, over 70% across the same range, and over 80% from 600-660nm and 740-800nm.

[0515] The responses of the red, green, and blue pixels in the Canon sensor are shown as percentages of the full color response of the sensor across the 350-800nm ​​range in Table XXIII:

[0516]

[0517]

[0518] Table XXIII

[0519] While the definition of "strong" given above compares the filtered response of a pixel at a wavelength to its full-color response at that wavelength, sometimes the latter data is unavailable. In such cases, a suitable definition of "strong" compares the filtered response of a pixel at a wavelength to its maximum response within the wavelength range of 400-700 nm. The peak response of a visible light pixel decreases with increasing wavelength to, for example, approximately 70% of the peak response at 650 nm, 50% at 700 nm, and 40% at 750 nm. If the filtered response of a pixel at one of these wavelengths (relative to its peak response) is greater than half of the percentage just given, the pixel is considered to have a strong response at that wavelength. For example, if a red pixel has a peak response of 0.9 (on some arbitrary scale) at 600 nm and its response at 700 nm is 0.3 (i.e., 33% of the peak response), this is judged to be a strong response because 33% is greater than half of the 50% figure above combined with the 700 nm reference. (Again, the strong response of a pixel is preferably more than half of the figures given above (e.g., 60%, 70% or 80% of the percentage just given).

[0520] Another way to determine a "strong" response is to use the spectral transmittance function of the corresponding filter for the reference pixel. If the filter allows 50% or more of the illuminance incident on it to pass through to the photoelectric sensor below at a given wavelength, then the pixel can be considered to have a strong response at that wavelength.

[0521] Therefore, one embodiment includes an image sensor comprising four pixels most sensitive between 400 and 700 nm, wherein each pixel has a photodetector and a corresponding filter, the corresponding filter causing the pixel to have a color response different from the other pixels among the four pixels. The filters of at least two of the four pixels allow at least 50% of the incident illuminance on the filters to pass through to their respective photodetectors at wavelengths from 650 nm to above 700 nm.

[0522] At wavelengths between approximately 500 and 780 nm, yellow-filtered pixels exhibit a strong response of over 70% and generally over 80% across the full spectrum at these wavelengths (the yellow filter is full-spectrum except for blocking blue wavelengths below 500 nm). Between 640 and 780 nm, yellow-filtered pixels have a response very close to that of the red-filtered pixels detailed in the table above. However, yellow pixels have greater efficiency than red pixels (e.g., across a spectrum extending between 400 and 750 nm).

[0523] Note that the embodiments discussed here include an image sensor comprising four pixels having a peak response in the visible light wavelength range. At least two of these four pixels have a strong color response at wavelengths extending through red (e.g., 650 nm) and into the near-infrared range.

[0524] In a first type of such embodiment, the four visible light pixels comprise exactly three pixels filtered to become primary color pixels and one pixel filtered to become yellow or magenta pixels. In a second type of such embodiment, the four visible light pixels comprise exactly two pixels filtered to become primary color pixels.

[0525] In the example of the first type of embodiment, four visible light pixels are filtered into red, green, blue, and yellow pixels. The red and yellow pixels respond strongly at near-infrared wavelengths (e.g., 700-800 nm). Given four channels of image data that respond differently to the imaged scene at wavelengths from blue to near-infrared, it is possible to extract four channels of output data. Three of these channels can be red, green, and blue—representing the image scene content perceived by the receptors of the human eye. The fourth channel can be varied according to the near-infrared scene content (but does not need to be specifically varied based on the near-infrared scene content).

[0526] In an example of the second type of embodiment, four visible light pixels are filtered to become red, green, yellow, and IR-narrowing panchromatic pixels. IR-narrowing panchromatic pixels are essentially panchromatic through visible light, but their response drops off in the near-infrared. For example, such pixels can exhibit a response from 400 to 700 nm within 80%+ (preferably 90%+ or 95%+) of the sensor's unfiltered response (i.e., their spectral transmittance function is 80%, 90%, or 95%+ from 400 to 700 nm), but a lower response somewhere above 700 nm. In a particular embodiment, these pixels are filtered so that their transmittance function drops below 50%, 20%, or 10% of the corresponding panchromatic level at some point in the 700 to 900 nm range (e.g., 700 nm, 740 nm, or 780 nm).

[0527] Furthermore, in this example of the second type of embodiment, the red and yellow pixels respond strongly to wavelengths from near-infrared to near-infrared. The four channels of the image data in this arrangement do not include the channel sensed by the blue pixels, but the IR-narrowing panchromatic pixels are sensitive to blue. Again, in the case of four channels of image data that respond differently to the imaged scene at wavelengths from blue to near-infrared, it is possible to extract four channels of output data. As in the previous example, three of these channels could be red, green, and blue—representing the image scene content perceived by the receptors of the human eye—while the fourth channel could vary according to the near-infrared scene content.

[0528] It should be understood that at least the red and yellow pixels in the embodiments discussed herein (and in many other embodiments, the same applies to the blue and green filters) lack the infrared blocking filters (sometimes called thermal mirror filters) commonly used with image sensors. Alternatively, IR attenuation filters can be used, but may allow for significant pixel responses in the near-infrared (e.g., a response at 750 nm or more, representing 5% or more of the peak response in the visible light range).

[0529] The embodiments described in this section can also be achieved by forming certain filters on the IR filter base, as previously stated.

[0530] The described embodiments can naturally use filters and filter units having the properties previously detailed. Red, green, blue, cyan, magenta, and / or yellow filters are not required unless explicitly stated otherwise.

[0531] Spatial variation color filter array

[0532] Color image sensors typically include an array of photoelectric sensors, which is covered by an array of color filters. The filters are carefully aligned such that each filter corresponds to one or an integer number of photoelectric sensors.

[0533] Depending on certain aspects of the technology, this alignment constraint is not maintained. The color filter array can be positioned arbitrarily or intentionally on the photodetector array such that a single filter covers a non-integer number of photodetectors. Some photodetectors may be covered by multiple filters. In some embodiments, the photodetectors and filters have different dimensions to contribute to this effect.

[0534] In one embodiment, the side dimensions of the photoelectric sensor and the filter have a non-integer ratio. One arrangement of this is shown in Figure 31 middle, Figure 31 A segment of a color image sensor is shown, comprising a color filter 311 depicted by thick square lines and a lower photoelectric sensor 312 depicted by thin square lines. This segment includes an 8×8 color filter panel that covers a 7×7 photoelectric sensor panel. The photoelectric sensor is therefore larger than the filter. Each photoelectric sensor has 8 / 7 of the side dimension of the color filter. Each photoelectric sensor has 64 / 49 of the area of ​​the color filter.

[0535] This non-integer relationship between the size of the filter and the size of the photodetector causes the filter's position relative to the photodetector to gradually shift in a modular fashion. This is in Figure 32 and Figure 32A Further details will be provided later.

[0536] A review of color filter arrays in image sensors reveals that they typically consist of multiple cells, each containing multiple filters. An example is a 2x2 cell of color filters (red, green, blue) used in classic Bayer filters. Figure 32 In this specific arrangement, each unit comprises a 3x3 unit containing a color filter. Figure 32 Two such identical filter units 21 and 22 are shown in the diagram, with different shades to aid illustration.

[0537] Figure 32A It comes from Figure 32 The magnified section, and in some embodiments of the illustrated technology, the filter has a different position relative to the photoelectric sensor below.

[0538] The position of the filter unit can be established by any arbitrary feature of the unit. For illustrative purposes, we consider the lower left corner of the filter unit as a reference point for specifying the unit's position. (Other corner points or the center of the unit are other possible reference points). The position of the filter unit can then be specified by the spatial relationship between this reference point and the photoelectric sensor below it.

[0539] Figure 32A The left side is labeled with two Cartesian axes, x and y, which define a coordinate system within the leftmost region of the photodetector, covered by the lower left corner (i.e., its reference point) of the filter unit (indicated by the thin-lined square). Any point within the boundary of this leftmost photodetector can be specified by coordinates along the x-axis, ranging from 0 to 100, and again along the y-axis, also ranging from 0 to 100. The position of the filter unit 21, shown in light shading, is defined by the coordinate position of its lower left corner. This position is determined by... Figure 32A The arrows in the diagram indicate that it has coordinates of 76.6 and 49.7.

[0540] The position of the adjacent filter unit 22, indicated by the darker shadow, is different. This position is again defined by the coordinates of its lower left corner within the image sensor block it covers. Again, Figure 32A The arrow in the image indicates the location, and its coordinates are 38.7 and 49.7.

[0541] In the same manner, other filter units have different positions relative to the image sensor below. Within a given color image sensor, there can be 5, 10, 20, 100, 1000, or a million or more different positions of the filter units relative to the image sensor below.

[0542] In this particular example, the y-coordinate of each filter unit is constant across units in a single horizontal row (i.e., 49.7 in the row including light and dark shaded units 21 and 22). The x-coordinate varies. In other words, the reference point for different filter units will be at different distances from the nearest neighbor column of the photodetector. In this example, the distance between the reference point for filter unit 21 and the nearest neighbor column 23 of the photodetector is less than the distance between the reference point for filter unit 22 and the nearest neighbor column 24 of the photodetector.

[0543] Conversely, although not depicted, the x-coordinate of each filter unit in this example is constant across units in a single vertical column, while the y-coordinate varies. In other words, the reference point for different filter units in a given column will be at different distances from the nearest neighboring column of the photodetector.

[0544] (We adopt the convention that rows are longer than columns. Therefore, if a photoelectric sensor array has 3,000 photoelectric sensors in one direction and 4,000 photoelectric sensors in an orthogonal direction, then the latter direction is the row direction.)

[0545] The color imaging device just described is characterized in that it includes J columns of photoelectric sensors covered by a color filter array comprising K columns of color filters, wherein J / K or K / J are not integers. Such a device may additionally or alternatively be characterized in that it includes P rows of photoelectric sensors covered by a color filter array comprising Q rows of color filters, wherein P / Q or Q / P are not integers.

[0546] We will pay further attention to this. Figure 31 The multiple (in fact, most) photoelectric sensor sections in the arrangement are covered by four filters. Similarly, Figure 31 Multiple (again, majority) filters are arranged to cover the four photoelectric sensors. (Exceptions occur around the boundaries).

[0547] In other embodiments, multiple photoelectric sensors are partially covered by nine or more filters. In yet another embodiment, multiple filters cover nine or more photoelectric sensors.

[0548] exist Figure 31 In the arrangement, when the side dimensions of the photodetector are related to the side dimensions of the filter by a ratio of 8 / 7, the spatial relationship between the photodetector and the individual filter will repeat after 8 rows and 8 columns of filters and after 7 rows and 7 columns of photodetectors. The position of the 3x3 filter unit relative to the photodetector will also repeat—albeit at longer intervals.

[0549] By appropriately selecting the ratio between the photoelectric sensor and the filter side dimension, such repetitions can be reduced. In this embodiment and some other embodiments, the numerator and denominator of the ratio are chosen to be coprime (i.e., having no common factors other than 1).

[0550] In some embodiments, each filter unit has a different position relative to the photosensor. This can be achieved by selecting a coprime ratio of the side dimensions of the filter and the photosensor, wherein each of the two numbers defining the ratio is greater than the largest pixel size of the color imaging device. For example, if the device has a pixel size of 4000x3000, then if the side dimensions are selected with a ratio such as 4001 / 9949, then each filter unit will have a different position relative to the photosensor. (In this case, both the numerator and denominator are prime numbers).

[0551] Although Figures 31-32A In this arrangement, the photodetector is larger than the individual filter, but this is not always necessary. The filter can be larger than the photodetector. In both cases, it is desirable that the ratio between their side dimensions is not an integer (e.g., not 2).

[0552] However, photoelectric sensors and filters with side dimensions having integer ratios (including 2 (and 1)) can be used when they are covered at an angle, that is, when the rows of photoelectric sensors are not parallel to the rows of filters (and similarly for columns). Figure 33 This arrangement is illustrated. Here, the side ratio of the filter and the photodetector is 1; they are the same size. Again, due to the tilt angle, the different filter units have different positions relative to the photodetector. Depending on the choice of tilt angle, 5, 10, 20 or more filter units in the color image sensor may have different positions relative to the photodetector below. Again, in some embodiments, each filter unit may have a different position relative to the photodetector below.

[0553] While the tilt angle between the color filter array and the photodetector array can be intentionally achieved, it can also be achieved in other ways (e.g., by relaxing manufacturing tolerances) so that some kind of "overflow" appears in the alignment of the color filter array relative to the photodetector array. Any degree of randomness in the positioning (including manufacturing) of the color filter array on the photodetector array may introduce this tilt.

[0554] and Figure 31 The layout is the same. Figure 33 The arrangement of the filter unit on the device introduces a gradual shift in its position relative to the photoelectric sensor below.

[0555] In other embodiments, the filter including the filter unit may have a shape different from that of the photoelectric sensor. For example, the filter may be an elongated rectangle, while the photoelectric sensor may be a square. Figure 34 This arrangement is illustrated in the figure. Many other different filter shapes (and photosensitive sensor shapes) can be envisioned, not all of them are quadrilaterals. Again, this arrangement allows the different filter units to have different positions relative to the photosensitive sensor, wherein the positions gradually shift on the sensor.

[0556] A combination of the arrangements just described can be used. For example, Figure 31 or Figure 33 The color filter array can be positioned at an angle on top of its corresponding photoelectric sensor array.

[0557] In the embodiments detailed above, the filter units at different positions relative to the photoelectric sensor achieve different pixel spectral filtering functions. Typically, in a color imaging device employing a color filter array with a 3x3 filter unit array, each pixel has one of nine spectral filtering functions. However, in the detailed arrangement, individual pixels are filtered with multiple different physical filters at different proportions (corresponding to the percentage area of ​​the pixel photoelectric sensor covered by each filter), resulting in hundreds, thousands, or millions of different pixel filtering functions on the device.

[0558] Although the illustrated embodiment uses a normal color filter, relatively thick and relatively thin filters with different spectral transmittance functions (e.g., the non-normal filters described in detail above) can be used.

[0559] According to a further aspect of this technology, the spectral filtering function for each photodetector in the device is characterized. The applicant's Chromabath procedure can be used. Associated data recording the filtering function for each photodetector is stored in the device's memory. Similarly, data for the kernel used to transform the scalar output from individual photodetectors in the neighborhood for a pixel at the center of the neighborhood into a color value for the desired color channel is also stored in the device's memory. (Such a neighborhood can be, for example, 5×5 or 7×7 in size.)

[0560] In this arrangement—as in the other arrangements detailed herein—two filters with different spectral functions can be achieved using the same medium (e.g., a dyed photoresist) by fabricating two filters of different thicknesses, as previously described. Similarly, spatially varied filtering arrangements can also employ filters and filter units having the properties previously detailed.

[0561] A particular filter unit comprises individual filters of conventional red, green, blue, cyan, magenta, and / or yellow photoresists, wherein one or more of these colors are formed with different layer thicknesses. For example, this type of color can be formed as a thick layer (e.g., with a thickness in the range of 0.8 to 1.5 micrometers) and can also be formed as a thin layer (e.g., with a thickness in the range of 0.4 to 0.8 micrometers).

[0562] In some such embodiments, a clear photoresist is coated onto the photosensitive sensor substrate to define a checkerboard pattern of clear elements. Then, for example, 2x3 filter units are formed using a colored photoresist. Half of these filters are on top of the clear photoresist elements (and are therefore thinner), while the other half extend downwards between the clear photoresist elements (and are therefore thicker). The filter units can all be the same pattern, or two or more filter units can be repeating the laid pattern.

[0563] One arrangement employs two distinct 2x3 filter units in the array. The first filter unit includes filters of type R, G, B, c, m, and y, while the second filter unit includes filters of type r, g, b, C, M, and Y, where uppercase letters indicate thicker filters and lowercase letters indicate thinner filters (each letter corresponds to one of red, green, blue, cyan, magenta, and yellow, respectively). These units can be laid out in a checkerboard pattern, such as... Figure 35 As shown. (Simply add shading to make the two different filter units easier to distinguish). In any other embodiments detailed herein, a color filter array having some or all of the properties just detailed may be employed.

[0564] Chromabath

[0565] Note that the Chromabath process optically characterizes pixels on a sensor. In a particular implementation, it produces a complete sensor, multi-parameter pixel behavior map: a mapping of certain sensor-wide specified optical (primarily chromatic) biases. "Characteristic" means measuring and recording how each pixel responds to light. Classical properties as panchromatic parameters (e.g., "bias" and "gain") are known. Measuring, storing, and using these two classical parameters are included in the Chromabath process. The Chromabath process additionally handles color filter array image sensors; panchromatic characterization is a bonus. It can be utilized on all sensors employing CFA.

[0566] Figure 7 Table III contains the red, green, and blue sensitivity functions for representative Bayer sensor pixels. It is clear that not every pixel of a given color will have these functions precisely; they will deviate from the global specification by a typical deviation level of less than 10%. Such deviations can be attributed to variations in filter thickness, crosstalk between different photodetectors, filter contamination by pigment residue from previous masking steps, layer alignment errors (including those involving filters and microlenses), etc. The resulting deviations characterizing pixel performance are measured and stored as part of the Chromabath process—implementing later corrections to this pixel data to compensate for these error sources.

[0567] Figure 36 For the sake of illustration, the concept is exaggerated. In this figure, we find four specific regions of the spectrum, where a given red pixel—located somewhere in a sea of ​​pixels—happens to be measurably deviated from the global average red spectral function. It is this spectral function deviation that the Chromabath process measures and corrects. As mentioned earlier, performing this process can involve calibration at the sensor-under-test level using a multi-LED illumination system, as well as calibration at the calibrator level using a monochromator.

[0568] The term "Chromabath," as a single, isolated word, generally refers to the entire process involved in its application to real sensors. Strictly speaking, the singular term refers to the light bathing of a sensor using light from multiple LED illumination units. This light bathing process involves hundreds or even thousands of image captures from the sensor under test. This image data is typically processed offline, where data is collected and stored, and processing of this data does not begin until all data from the sensor has been collected.

[0569] In one illustrative arrangement, a spectral transmittance profile is measured for each pixel on the sensor. Each profile may include, for example, 85 data points detailing transmittance in 5 nm increments from 380 nm to 800 nm. For each different type of filter on the sensor (e.g., thin cyan), an 85-point global average profile is determined based solely on that type of filter. Each individual filter of that type is then characterized by its deviation from the corresponding type's average. Figure 37 The diagram shows sampled characteristic curves of such filters for individual pixels. These curves can be used as a signature for the corresponding pixel. (Y-axis calibration is meaningless in this illustration; the diagram serves only to illustrate the concept.)

[0570] As will be recognized, Figure 37 The pixel signature shown in A is a spectral function graph that is susceptible to noise. These types of noisy waveforms are compliant with a wide range of "compression" methods, which in this example (380nm to 800nm, in 5nm steps) take 85 floating-point values ​​and convert them into 4-byte compressed values.

[0571] One approach to doing this is principal component encoding. With principal component encoding, we begin with a very large set of sample pixel signatures (e.g., all pixel signatures for a thin cyan-type filter on a sensor) and then perform singular value decomposition on that set. This produces, for example, six salient principal vectors (also called eigenvectors) that, when multiplied by unique coefficients for each pixel signature, will "fit" that pixel signature to an acceptable level of accuracy (e.g., 0.1%). These coefficients are each quantized, for example, into one of 16 values ​​represented by four binary bits. These 16 values ​​can be uniformly spaced (e.g., in steps of 1, from -7.5 to +7.5) or non-uniformly spaced (i.e., entering one of 16 histogram intervals selected such that they all have approximately the same number of counts).

[0572] If each pixel feature is represented by six coefficients (corresponding to six principal vectors), then the 4-bit arrangement described above is used, totaling 24 bits. The remaining 8 bits (four bytes) can be allocated as 4 bits for pixel offset value data and 4 bits for gain data. The 4 bits for pixel offset values ​​can be used to represent the offset value for the pixel relative to 16 uniform or non-uniform intervals of the sensor-wide global specification. Similarly, for the 4 bits for gain data. (The sensor-wide global specification for offset and gain data can be used for the same type of pixel (e.g., light cyan) or for all pixels on the device).

[0573] In a variant embodiment, the 24 bits allocated to represent the six principal component coefficients are not evenly distributed, with 4 bits used for each coefficient. Instead, more bits are used to represent the principal component coefficients (e.g., 6 bits), with subsequently fewer bits used to represent subsequent components. For example, the second and third components could each be represented by 5 bits, and the fourth and fifth components could each be represented by 3 bits. This leaves two bits for representing the sixth coefficient.

[0574] To store the 32 bits for each pixel, the sensor is equipped with an associated memory of sufficient size for the task. Since the signature data, offset value, and gain value for each pixel are all relative to the sensor's wide average value, the memory also stores these average values. When image data is transferred from the sensor to the system for use, the pixel data is transmitted along with the associated 4 bytes for each pixel and the global average value. The receiving system can then correct, compensate, or otherwise consider the pixel values ​​based on this correction data to obtain more accurate results.

[0575] There are at least three uses for Chromabath data:

[0576] 1. Sensor manufacturing quality control and assurance;

[0577] 2. Create a globally uniform color image; and

[0578] 3. Pixel-specific input data for AI and machine learning image processing, where classic pixel data is supplemented with signature (Chromabath-generated) data for a given pixel.

[0579] Using such data to create globally uniform images can significantly increase the manufacturing yield of image sensors. In particular, the yield of high-quality, large CMOS image sensors is surprisingly low, typically below 90%. This means that 10% or more of the sensors manufactured would ideally end up in the recycling bin, but in some cases, they do end up in the trash. Industry practitioners understand that the yield problem is more complex than this simple description, with practices such as "sorting" attempting to match economic needs with quality, but the main issue remains: the yield of CMOS image sensors is generally poor.

[0580] For Bayer / RGB CFA sensors, yield is primarily constrained by global uniformity quality specifications. You can't have one part of a sensor measuring true cyan for a patch of a scene and another part measuring greenish cyan for the same patch. This is the case for the layman's version; the professional version comes down to optical and chromaticity uniformity specifications.

[0581] These specifications are addressed by the Chromabath process. If a global color correction matrix is ​​applied to a conventionally manufactured sensor, and the resulting color image exceeds tolerances, the sensor is essentially rendered unusable. Conversely, if the Chromabath process is applied to the sensor, enabling localized color tuning on a per-pixel basis, the sensor's performance is preserved—or even enhanced.

[0582] Color Correction

[0583] A "color correction matrix" is a familiar method for transforming raw red, green, and blue sensor data triplets into superior estimates of X, Y, and Z chromaticity as defined in the CIE 1931 color specification. A color correction matrix is ​​"global" or at least "regional" in the sense that the same matrix applies to hundreds, thousands, or even millions of pixels. This technique involves, in part, a calibration process during the manufacturing and / or sensor operation phases: first, data is collected to measure minute variations in how pixel neighborhoods have their individual pixel spectral response functions; this data is stored for later use; then, this data is used to calculate a region-, local, neighborhood, or cell-level color correction kernel, which then modifies the classical color correction matrix tuned for a specific pixel in any given neighborhood of a pixel. This calibration process can be equivalently applied to three-channel sensors (e.g., Bayer RGB sensors) as well as four- to N-channel sensors, where N can be hundreds.

[0584] "Matrix" is a simpler form of the more broadly defined kernel. Kernel operations allow both linear and non-linear manipulation of local pixel values, while "matrix" is traditionally limited to linear operations. This manual teaches linear and non-linear, as well as machine learning-based, kernel operations for performing this localized color correction. By doing so, defects and non-uniformities in pixel behavior can be significantly mitigated and corrected.

[0585] Manufacturing color image sensors is difficult. Many sensors fail quality assurance tests and are destroyed. This consumes additional resources, incurs additional costs, and generates additional semiconductor processing waste.

[0586] Color image sensor manufacturers apply numerous quality assurance (QA) guidelines to their finished products. Some of these guidelines concern the uniformity of the pixel output signal in the presence of uniform illumination. It is not uncommon for some rows or columns of pixels to be "hotter" than others. Or, for example, a checkerboard pattern might have a pattern that is "hotter" than others. CMOS imager manufacturers typically "box" individual sensors and thus categorize them into commercial tiers. This tiering system results in price differences for any given sensor. Significant R&D, engineering, and QA budgets are allocated to increasing the yield of higher-quality packages.

[0587] Some criteria relate to spectral sensitivity. Some "green" pixels may be more sensitive to red illumination than other "green" pixels, and so on. The same applies to other color-filtered pixels.

[0588] These and other anomalies can be localized to small areas of a photoelectric sensor, or they can span across the device, or they can span a large wafer where dozens of photoelectric sensor devices are fabricated simultaneously. Spin coating, used to deposit micromaterial layers on the wafer for processing, is one of dozens of different steps that can introduce inhomogeneities.

[0589] From a technical perspective, each pixel on an image sensor is associated with a unique N-byte signature describing the pixel's unique behavior. This behavior includes, but is not limited to, specific spectral radiometric sensing characteristics of the pixel. In an exemplary configuration, N can be 3 or 4 bytes. These signatures are generated during the Chromabath testing, calibration, and certification phases of the manufacturing / quality assurance process for use in both mass-produced and specialized image sensors.

[0590] In some embodiments, the Chromabath process is used to illuminate the sensor (or a wafer with multiple sensors) with an extremely narrow band light source scanning over an electromagnetic spectrum range to which the sensor is sensitive, where "extreme" can indicate the light output of a monochromator with a bandwidth of one or two nanometers. The response of each pixel is detected as a function of the illumination wavelength. Deviations in the properties of each pixel relative to the canonical behavior of local or regional neighboring pixels, and / or pixel sensor width and / or pixel wafer width are determined and recorded as a function of the light wavelength across the scanned spectrum. Sensitivity differences identified in this way are encoded as an N-byte compressed "signature," which can be stored directly in memory fabricated on the CMOS sensor substrate, or in some other (e.g., off-chip) manner that allows access to the N-byte signature data using a process that utilizes the image sensor output signal. During use of the sensor, pixels are processed into output images using information indicated by these N-byte features to enhance the quality of the image output. For example, in rendering or analyzing data from the sensor, the output of "hot" pixels can be reduced, and the unique spectral response of the pixels can be taken into account. Machine learning and AI applications can also use these N-byte signatures as a further dimension of the "feature vector" used during the training, testing, and use of neural networks and related applications.

[0591] For this measurement of the N-byte signature of a pixel, instead of a monochromator, a series of medium-narrow-band LEDs with individual illumination can also be used. The practical advantage of using a series of LEDs is that it is generally cheaper than a monochromator, and the so-called "shape factor" of placing LEDs near the wafer-level sensor is superior: LEDs as a bank can sit exactly above the wafer, as inferred in Gamma Scientific's commercial products, their RS-7-4 wafer probe illuminators.

[0592] As indicated, the global sensor uniformity of radiation behavior is one of several manufacturing tolerance criteria used to determine whether an individual sensor passes or fails quality assurance testing. Pixel-based N-byte signatures, which quantify pixel-by-pixel variations in radiation behavior and other pixel non-uniformities, enable relaxed manufacturing and quality assurance tolerances because such non-uniformities can be quantified and taken into account in the use of sensor image data. Relaxing these tolerances increases manufacturing yield and reduces cost per sensor. Sensors that would previously have been rejected and destroyed instead pass quality testing. Furthermore, such sensors achieve superior imaging results compared to existing technology sensors that may have already passed more stringent acceptance criteria, because the N-byte signature data mitigates what would otherwise be acceptable slight variations from one pixel to the next.

[0593] In specific scenarios, sensor manufacturers identify the most frequent QA failures. A small number of these failures may be mitigated by N-byte signature information (e.g., simple failure sensors, or sensors with internal short-circuit or open-circuit circuits that disable certain functions). However, the vast majority of QA failures are attributed to performance defects (e.g., output uniformity metrics or pixel spectral sensitivity metrics exceeding permissible thresholds by a few percentage points). The N-byte signature data is then used to convey data that can improve these behaviors (or an index to data stored elsewhere). Generally, the pixels in connected pixel pairs, connected triplet pairs, etc., may also be either "failed" or otherwise outside of certain performance parameters. Here, N-byte pixel characterization can also become a useful mitigation factor in transforming a lower-priced, lower-package sensor into a higher-priced, higher-package sensor.

[0594] According to a further aspect, some embodiments of this technology employ uncoordinated arrays of photoelectric sensors and color filter arrays, as detailed herein. With this arrangement, an image sensor with, for example, N=9 color filter elements of different colors has pixels with M>N different spectral response characteristics.

[0595] It should be understood that the embodiments described above may employ filters with a strong response at 650 nm and above, wherein other features are described in the discussion entitled "Embodiments Employing Near-Infrared." This allows for the acquisition of data from which information about both visible light and NIR can be extracted. It should also be understood that the described embodiments may use the filters and filter unit arrangements previously detailed in this specification.

[0596] A universal image solution for pixel neighborhoods of arbitrary spectral contours

[0597] In some embodiments, the neighborhood of a pixel spectral function (profile) may not employ a fixed, repeating cell pattern (e.g., 2x2 Bayer (RGGB) cells). An example is the spatially varied color filter array detailed above. The following discussion addresses how data generated from non-repeating pixel neighborhoods can be transformed into an image solution.

[0598] In many operations that seek to map pixel data to output data, it is desirable to first determine the values ​​of the luminance (luma) image corresponding to the pixel data. One way to accomplish this is to use an NxN linear kernel operator. For a 2x2 filter unit, N can be 6. For larger filter units (e.g., the 3x3 unit detailed above), N can be 7.

[0599] Given a neighborhood of surrounding pixel colors, a different kernel is defined for each pixel with a different color. Thus, in the aforementioned red, green, blue, and yellow filter units, there will be four distinct 6x6 kernels—each centered on the corresponding pixel in its respective color. In a unit employing nine different color filters, there will be nine distinct 7x7 kernels. Considering the latter case, when generating pixel data, it is multiplied by all 49 kernel values ​​associated with that particular pixel color (and, if the neighborhood is not perfectly similar, that particular pixel neighborhood), and these 49 results are fed into an accumulator array centered on the pixel and extending through a 7x7 pixel neighborhood centered on the target pixel. Each pixel is processed in this way, and the accumulated values ​​include the complete brightness image.

[0600] To obtain a color map (spectral function solution) from pixel output values ​​to the desired output color, the first step could be to parameterize the spectral response profile of each pixel with different tint. This parameterization is expected to be accurate enough to "fit" empirically measured pixel profiles to a few percent of the peak response levels of the pixels in the spectrum. Figure 38 An example is shown of a windowed Fourier series that can be fitted to both (a) the pixel spectral response profile and (b) the solution to the pixel spectral function, defined here in the interval 350–800 nm. In some cases, the function can also be weighted by the quantum efficiency of the photoelectric sensor as a function of wavelength. Each term of the Fourier series is associated with a corresponding weighting coefficient, and these weighted functions are then summed, as is familiar in other applications employing parameterized fitting of functions.

[0601] Note that Fourier series do not have functions with non-zero means. This indicates that the initial solution for an image should first determine the image brightness, for example, through the process just described, and then determine the pixel spectral function solution. When utilizing... Figure 38 Before the spectral solution function, the brightness value should be removed from the pixel value.

[0602] Here we have two pieces of data that should be converted into a vector of Fourier coefficients. The first is the spectral sensitivity profile for each pixel. This is primarily a function of the quantum efficiency of the filter and the photoelectric sensor, but can also include lensing effects, scattering, etc. The function thus defined is constrained by the raw spectral response of the photoelectric sensor (its irradiance profile). This is a familiar activity in function fitting—determining the coefficients of an empirically measured spectral sensitivity that best matches the sum of the weighted Fourier functions to each pixel.

[0603] The same activity is performed for the spectral profile solution. This activity uses real data (e.g., a collection of reference scene images collected without sensor filtering but illuminated at 46 narrow wavelengths of light at 350, 360, 370…800 nm) to determine the spectral scene reflectance at each of these wavelengths for each image pixel. The same scene is also imaged using a target color filter arrangement.

[0604] Once the pixel spectral sensitivity profile function is parameterized and the spectral profile solution function is parameterized, we can now derive a system of linear equations that matches the acquired pixel data with the solution (the solution to this system of linear equations for each pixel with a different color).

[0605] In some embodiments, demosaicing is ignored, where the process determines nine distinct spectral function values ​​for each pixel. This can be done via interpolation. The trade-off is that the spatial sampling distance for each spectral channel is reduced by a factor of three in each direction, relative to a sensor that actually has all nine channels at each pixel.

[0606] In other embodiments, further demosaicing is employed. This process may follow teachings set forth below, for example: Sadeghipoor et al., A novel compressive sensing approach to simultaneously acquire color and near-infrared images on a single sensor, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1646-1650; Park et al., Visible and near-infrared image separation from CMYG colorfilter array based sensor, 2016 IEEE International ELMAR Symposium, pp. 209-212; and Teranaka et al., single-sensor RGB and NIR image acquisition: toward optimal performance by taking account of CFA pattern, demosaicing, and color correction, Electronic Imaging, 2016(18), pp. 1-6. These documents are incorporated herein by reference in their entirety.

[0607] A neural network approach, also employing Fourier vectorization, can be used as an alternative to the linear solution described above. We model millions of scenes and train pixel-by-pixel neural network solutions that associate unique pixels. The neural network will typically produce more compact solutions than the linear approach, requiring perhaps only 10% of the data storage.

[0608] Although Fourier series are used in the aforementioned arrangement, they can be replaced by others (e.g., Taylor polynomials, Chebyshev and Legendre polynomial families).

[0609] Neighborhood variation color correction kernel

[0610] According to some embodiments, local color adjustments are performed by applying a neighborhood-specific color correction kernel (color correction matrix, or CCM) to an array of pixel values ​​surrounding each target pixel.

[0611] The color correction matrix (kernel) traces back to the Bayer sensor. Originally a method of calculation based on esoteric mathematics, it has now become mainstream, especially when considering hardware and firmware implementations.

[0612] In a known prior art, an image sensor is used to capture an image of a color test map containing multiple known-color printed patches under known lighting conditions. The captured image is stored in an m x n x 3 array, where m is the number of rows in the sensor, n is the number of columns, and 3 represents the number of different output colors. Ideally, the captured image would be identical to another m x n x 3 array containing reference data corresponding to the correct colors, but this is not the case. To transform the captured pixel array more closely to the ideal reference pixel array, the captured image array is multiplied by a 3 x 3 x 3 color correction matrix, the coefficients of which are adjusted so that the product of this multiplication produces a least-squares fit to the reference array. Subsequently, when any image is captured, its image data is similarly processed by multiplying it by this 3 x 3 x 3 color correction matrix to obtain a color-corrected equivalent of the captured image.

[0613] According to one aspect of this technology, instead of applying a single color correction matrix to all pixels in the captured image, multiple locally adaptive color correction matrices are employed. A formulaic version of this is to use a global color correction matrix (which can be simplified to:

[0614] [ A11 A12 A13 Formula XX1

[0615] A21 A22 A23

[0616] A31 A32 A33]

[0617] (Where, A is just a general letter representing various formulas, some involving X, Y, and Z, some involving R, G, and B, and some being a mixture of these.) Transformed into a locally adaptive form:

[0618] [ f(4BN,1,1) f(4BN,1,2) f(4BN,1,3) Formula XX2

[0619] f(4BN,2,1) f(4BN,2,2) f(4BN,2,3)

[0620] f(4BN,3,1) f(4BN,3,2) f(4BN,3,3)]

[0621] Here, "4BN" is mapped to 4 bytes of data associated with a specific pixel neighborhood. These are the pixel signatures of the pixel map derived by Chromabath. The 4 bytes are the nominal value; other sizes of data can naturally be used. The neighborhood can be of any size, such as 2x2 pixels, 4x4 pixels, 6x6 pixels, etc. Therefore, the locally adaptive color correction matrix value is a function of many parameters, including its own "index" within the 3x3 color correction matrix. (As with the four-byte pixel signature data detailed earlier, the local color correction data is stored in sufficient device memory for this purpose.)

[0622] The definitions of these new functions f in Formula XX2, like the definition and measurement of the 4-byte signature itself, deserve our equal attention. In one embodiment, these functions f in Formula XX2 are empirically derived using machine learning training, but their "classical" form can be described here.

[0623] One form of 4-byte pixel signature assumes encoding some kind of "mixture" (cross-contamination) of the masked pigments; for example, in a nominal blue pixel, trace amounts of red and green pigments are present, and the same applies to nominal red and nominal green. If we take this "encoding scheme" as an example to illustrate how to construct these f-functions for a local adaptive color correction matrix, then we can construct this additional transformation layer. Similarly, empirically (and through simulation), the mapping (the f-functions themselves) can be solved by putting together a "real" dataset matching millions of instances of 4-byte neighborhood values, imaging the entire color gamut, and learning the answers. This is a brute-force approach, but acceptable in the sense that training these functions can utilize as much time and computational power as needed. The end result produces linear product coefficients, which become the definition of each f-function:

[0624] f(4BN,1,1) = c11_1 * 4BN1 + c11_2 * 4BN2 + c11_3 * 4BN3,,, etc.

[0625] For each of the nine locally adaptive color correction values, 20 or 30 c11 coefficients may be required. This is also known as solving a system of L1 or L2 linear equations. Alternatively, one approach is to train these locally adaptive color correction functions using machine learning and an arbitrary number of selections that look at millions of patches and color patterns across the entire color gamut, matching millions or billions of real-world examples with corresponding millions or billions of 4-byte neighboring pixel signature values.

[0626] When the sensor is used subsequently, the traditional global color correction stage of image processing is replaced by this local adaptive variant, where an additional data feed of 4 bytes of pixel signatures is provided. Typically, these 4-byte pixel signatures are fixed, resulting in a similarly fixed local adaptive color correction matrix. Mathematically, these new local adaptive matrices are fixed, but spatially variable. Therefore, they will employ per-pixel memory allocation, just as the required 4-byte signature per pixel. However, in this operational case, it becomes an operationally relevant color correction matrix graph, rather than the 4-byte signature itself. Therefore, operationally, it is possible to choose not to use the 4-byte signature value itself for image processing. If only color correction is needed, then the 4 bytes are indeed unnecessary, but if AI / machine learning image processing is being performed, it is undesirable to discard these information-rich 4-byte signature values. The order of magnitude of the memory requirements for the local adaptive color matrix is ​​difficult to determine because it depends on many factors. However, estimating 10 bytes per pixel as a common upper limit is not unreasonable.

[0627] These matrices will then be used for kernel-based processing of the individual pixel outputs, in the same way that traditional kernel-based processing derives pixel data.

[0628] The following is an excerpt of an example of a local adaptive color correction matrix:

[0629]

[0630] These nine numbers will vary locally based on local 4-byte pixel features. These numbers may vary slightly by pixel, or, in the demosaicing algorithm, slightly by kernel. This data can also be compressed into 4 to 6 bytes of differential encoding, and they will remain fixed until further calibration measurements may change them.

[0631] Figure 39 This further clarifies (and exaggerates) how these color correction matrices (also known as CCMs) function as a function of local location. The conversion of 4-byte features into functions of these numbers has been discussed previously. The example here has one CCM per 2x2 pixel Bayer unit; other arrangements are certainly possible (including one CCM per N x N region with overlap; and applying CCMs to pixels instead of units). We can see that the CCM locally adjusts for minute imperfections in the inter-pixel spectral function of the underlying pixel type. These local CCMs can have built-in regional and global corrections, where, for example, if a spin-coated layer on the chip is unit thickness at one corner of the sensor and 0.97 unit thickness at another, this global non-uniformity can still be corrected by slowly varying the local values.

[0632] ShadowChrome

[0633] This section expands upon the previous part titled "Image Data Recovery".

[0634] Existing camera systems have poor color discrimination capabilities in low-light conditions. According to aspects of this technology detailed below, image sensor data can accurately distinguish pixel chromaticity even in extremely low light conditions. This technology is sometimes referred to as ShadowChrome.

[0635] The accepted photometric theory for CMOS sensor photosensitive points (pixels) is that incident light generates so-called photoelectrons at discrete pixels. The number of collected photoelectrons is a discrete non-negative integer, with no other numbers besides 0, 1, 2, 3, and above. For current commercial-grade CMOS sensors, these discrete values ​​cannot be directly measured due to so-called readout noise from the amplifier plus analog-to-digital converter arrangement. Furthermore, there is so-called shot noise, an industry term used to describe the Poissonian statistics of discrete (non-negative integer) measurement arrangements. However, another factor that should often be considered is the inter-pixel variation in individual measurement behavior, a phenomenon commonly referred to as fixed-pattern noise. ShadowChrome addresses all four of these problems: (a) the scarcity of photoelectrons; (b) readout noise, typically on the order of 2e- to 5e- for modern CMOS image sensors; (c) significant shot noise due to scarcity; and (d) non-uniform pixel measurement behavior.

[0636] While ShadowChrome performs well in imaging scenes with normal brightness—that is, when each pixel has hundreds, if not thousands, of photoelectrons per image capture—it is actually designed for extremely low light levels, where the so-called signal-to-noise ratio (SNR) is 10 or even as low as 1, or even less than 1. Numerically speaking, we define a SNR of 1 for a given digital image sensor simply as the average number of photoelectrons generated within a pixel being exactly equal to the readout noise specification.

[0637] In fact, it was the SNR of 1 that guided many of the various parameter choices proposed in the ShadowChrome solution framework. Commercial imaging's desire to "see color"—even the emission patterns of objects barely visible in extremely low light levels—has driven the development of ShadowChrome. One example of how color is crucial in hundreds of dim scenes is perhaps ADAS (Advanced Driver Assistance Systems) cameras: stop signs and lane lines (and roadside objects) under starlight should be rendered separately, for example, in red and yellow, once they become visible.

[0638] Existing digital imaging systems define so-called "dark frames," where the average digital (DN) value of all pixels is typically recorded after hundreds or even thousands of "dark" images have been captured. The values ​​obtained from such dark frames are usually fractions rather than discrete non-negative integers, primarily because of the large number of measurement frames used to derive the average.

[0639] With ShadowChrome, the explicit data structure becomes a scaffolding mechanism that then derives these dark frames, logging the median of the DN points for each pixel, rather than their mean. ShadowChrome simplifies the low-light color measurement problem to a massive coin-flipping or pseudo-50-50 decision network that ultimately results in hue and saturation estimates, and this effort at "pseudo-50-50" decision-making begins with applying these principles to the dark behavior of each pixel. These pixel-wise median encoded dark values ​​can conveniently use the exact same fractional value form as the prior art "dark frame" predecessor. In other words, for example, a fractional median of 20.17 indicates that nearly half of the pixels in the dark frame will have a DN below 20, and nearly half of these values ​​will have a DN above 20. The remaining values ​​will be DN = 20 itself. Then, when the fractional value is below 0.5 (e.g., for 20.17), this indicates that there are slightly more histogram values ​​(integrals) of 21 and above, as there are histogram values ​​(integrals) of 19 and below. These small details can be important when trying to measure colors with an SNR of 1 or even lower.

[0640] Therefore, a possible initial step in using ShadowChrome is to perform dark frame calibration on the sensor, explicitly projecting the data as median pixel values. Chromabath (discussed elsewhere) has this dark level histogram creation functionality built into its process. Therefore, for simplicity, ShadowChrome can use the data results from Chromabath.

[0641] The second possible initial phase of ShadowChrome involves using calibrated white tiles, or, if no such tiles are available, some equivalent "scene" that provides access to "colorless" objects. As a final backup option where no scene is available at all, "theoretical" sensor specification data can still be used, such as the sensor spectral sensitivity curve for each pixel type (e.g., 9 types in a 3x3 cell arrangement). The goal of this second preliminary phase is to track the so-called "grayscale gain," A) all pixels of a given spectral type (e.g., 9); or B) the grayscale gain per pixel in a Chromabath manner. The latter is preferred for achieving optimal color measurement performance, but the former is also acceptable because the "universal gain" of CMOS sensors typically exhibits good uniformity within 1%. Since we are dealing with imaging at extremely low light levels, often involving single-digit photoelectron counts, this 1% uniformity is a case of diminishing returns. In other words: pixel spectral types have very different grayscale gains as a class, one type compared to another, but within a given spectral type, the grayscale gain is practically the same.

[0642] Therefore, measuring the grayscale gain of pixel spectral types can help maximize ShadowChrome's color measurement capabilities. Chromaticity and color are entirely measured by the signal-to-weight ratio characteristics between pixels of different spectral types, so these signals are expected to have a zero-point reference for some color, and these grayscale gain values ​​are precisely that. They can be considered "colorless" baseline signals. Similarly, these individual values ​​for spectral types can be derived by taking images of tens to hundreds of white patches, removing the aforementioned dark level effects, and then the grayscale gain of the spectral type represents the mean of these numerous images. The grayscale gain values ​​themselves can be arbitrarily defined and then normalized to each other, but in this disclosure, the convention we use is that the highest grayscale gain value belonging only to one spectral pixel type will be assigned a value of 1.0, while the grayscale gain values ​​for all other spectral pixel types will be slightly below 1.0, but proportionally. The so-called "white patch equalization" between pixel spectral types will assume that grayscale gain values ​​below 0.8 are preferably avoided, if possible. (It can be understood that these white patch and grayscale gain data are, in a sense, a measure of pixel efficiency.)

[0643] These data, now known as dark values, and the individual use of these grayscale gains, will be discussed in a further description of ShadowChrome below.

[0644] A concrete example

[0645] Consider an embodiment involving an image sensor comprising nine pixels in a 3x3 unit, some or all of which have different spectral responses (i.e., they are of different types). Filters and filter units having the aforementioned properties are exemplary. Based on an arbitrary mapping of these labels to the nine pixel locations, these nine pixels can be labeled as first through ninth pixels (or, interchangeably, labeled as pixel AI).

[0646] Reference Figure 40 By comparing the scene value associated with one pixel in the cell (called the base pixel) with the scene value associated with another pixel in the cell (called the ordinate pixel), information useful for indicating the chromaticity of the center pixel is generated. (The term "scene value" is sometimes used to refer to the value associated with a pixel when the image sensor is illuminated by scene light. A pixel's scene value can be its raw analog or digital output value, but other values ​​can also be used. The term "digital value" or "DN" is also commonly used to represent sensed scene brightness data.) Figure 40 In the diagram, pixel A is the base pixel, and pixel B is the pixel with the vertical coordinate; arrow 401 indicates a comparison between these pixels.

[0647] This scene value comparison operation is performed between different pairs of pixels within a cell. For example, a comparison can be performed between pixel A and pixel C. This is in Figure 40 Arrow 402 is also used to represent this, where pixel A is still used as the base pixel and pixel C is used as the second vertical coordinate pixel.

[0648] Based on these comparison results, query data is generated and provided as input data to the color reconstruction module 411 of the image sensing system 410. Figure 41 This module determines the chromaticity information to be assigned to a pixel in the cell (typically the center pixel of the cell). In some embodiments, the color reconstruction module operates only with the query data as input; the color reconstruction module does not operate on the pixel data itself (e.g., the data of the center pixel).

[0649] The comparison data of these two pixel pairs is useful for indicating the color (i.e., chromaticity) of the center pixel in the cell, although at least one comparison (two comparisons in this case) does not involve the center pixel (pixel E in this case). That is, the base pixel and the two y-coordinate pixels may not include the center pixel.

[0650] Such pixel-to-pixel comparison data is expected to be generated by a hardware circuit module fabricated on the same semiconductor substrate as the image sensor, and this circuit outputs a representation of such data (e.g., a vector data structure) as query data. This query data is used in subsequent processes (typically implemented as an additional hardware circuit module located on the same substrate or a companion chip), which in part assigns color information to the center pixel based on this data. (This module may be called a demosaicing module or a color reconstruction module.) Such hardware arrangement is as follows: Figure 41 As shown, the dashed line represents the common semiconductor substrate containing the aforementioned modules. (For commercial applications aiming to minimize sensor costs, raw pixel data can be passed to a camera module or cloud processor to perform the functions of the color reconstruction module.)

[0651] The quality of the output color information will ultimately depend on the richness of the query information. Therefore, query information based solely on two inter-pixel comparisons (base pixel and first y-coordinate; base pixel and second y-coordinate) is rarely used. In many embodiments, further comparisons are made between the scene value associated with the base pixel and the scene values ​​associated with other pixels in the cell to obtain additional pixel pair data. If the base pixel is referred to as the first pixel, eight pixel pair comparison data can be generated, involving comparisons with the second through ninth (y-coordinate) pixels. The first two inter-pixel comparison data are generated as described above, i.e., by comparing the scene value associated with the first pixel and the scene data associated with the second pixel (i.e., [1,2] comparison, where the first number represents the base pixel and the second number represents the y-coordinate pixel), and by comparing the scene value associated with the first pixel and the scene data associated with the third pixel (i.e., [1,3] comparison). A similar comparison also compares the scene value associated with the first pixel and the scene data associated with the fourth through ninth pixels in the cell respectively, to obtain pixel pair data from [1,4] to [1,9]. Figure 42 These further comparisons are shown—each comparison using pixel A as the base pixel. Similarly, the representation of all these pixel pairs of data is output as query data by hardware circuitry.

[0652] The use of the word "compare" and its various forms, such as "comparing" and "comparison," is to denote various mathematical choices to precisely describe how the comparison is performed. One form of comparison is the sigmoid function comparison (see Wikipedia for details). The limit of the sigmoid function becomes a simple greater than or less than comparison of two individual values. For non-negative integers DN, the equal to case also becomes an implementation, often resulting in a null or zero value. In this disclosure and more generally, the limit values ​​of the sigmoid function are the numbers 1 and -1.

[0653] So far, the query data has only involved a single base pixel. However, just as the first pixel can be compared (or more precisely, compared with the scene values ​​associated with these pixels) to the other eight pixels (i.e., the second through ninth pixels), similarly, the second pixel can be compared with the other seven pixels (i.e., the third through ninth pixels). These comparisons are as follows: Figure 43 As shown. Similarly, the third pixel can be compared with the other six pixels (i.e., pixels four through nine). Likewise, the fourth pixel can be compared with the other five pixels (i.e., pixels five (center) through nine). And so on, until the eighth pixel is compared with only one pixel (i.e., the ninth pixel).

[0654] These comparisons produce a total of 36 pixel pairs of data. This value is the N-summatorial number, where N is the number of pixels in the cell (i.e., 9), as described above. This value is also the (N-1)th triangle number. It can be calculated as N(N-1) / 2, which equals 36 for N=9.

[0655] In other words, assuming there are N pixels in a cell, the specific process is to compare the scene value associated with the Q-th pixel in the cell with the scene value associated with the R-th pixel in the cell to update the Q-R ([Q,R]) pixel pair data, where each Q is between 1 and N-1, and each R is between Q+1 and N.

[0656] The comparison results containing pixel-pair data can take different forms in different embodiments. In one embodiment, the comparison result is a count, which increments when the base pixel scene value is greater than the ordinate pixel scene value and decrements when the base pixel scene value is less than the ordinate pixel scene value. (If the base value and the ordinate value are equal, the comparison results in zero.) In the example given above, 36 comparisons thus produce a 36-element vector, where each element is -1, 0, or 1. This can be referred to as a high / low comparison.

[0657] In another embodiment, the comparison result is the arithmetic difference between two compared scene values. For example, if the scene value of the base pixel is 25 and the scene value of the ordinate pixel is 70, the comparison result (pixel-to-data) is -45. In this case, the 36-element vector consists of 36 integers or real numbers (depending on whether the scene value is an integer or a real number). This can be called a simulated comparison or a difference-preserving comparison. Various forms of non-linear "weighting" can also be applied to these comparisons, which is common in machine learning implementations because the final correct choice is not fully understood: allowing the data to make choices based on a large amount of training on "real" images. Similarly, the parameters of the sigmoid function itself can also be tuned through machine learning.

[0658] As mentioned above, the quality of the output color information will ultimately depend on the richness of the query information. Although the arrangement described above generates query data by comparing within a single color filter array cell, richer query information can be obtained by extending this comparison to pixel areas beyond that single cell.

[0659] In most image sensors, color filter arrays are tiled by multiple units. That is, let's call the single unit just discussed the first unit, and then there are multiple other units tiled around the first unit in its neighborhood. These other units are adjacent to the first unit, or to other units adjacent to the first unit, and so on. These other units can each replicate the first unit in terms of pixel type pattern and orientation. In this case, we sometimes refer to pixels located at the same spatial position in each of two units as spatial counterparts, or spatial correspondences (for example, the first pixel found at the top left corner of the first unit is a spatial counterpart to the first pixel found at the top left corner of another unit).

[0660] In other embodiments, some or all of these additional units may have the same pixel type pattern as the first unit, but with a different orientation, such as rotated 90 degrees to the right. Alternatively, some or all of these additional units may have different pixel type patterns, but include one or more types of pixels found in the first unit. In this case, we sometimes refer to pixels of the same type found in each of the two units as color (or type) counterparts, or color (or type) correspondents (e.g., a blue pixel found in the first unit is a color counterpart of a blue pixel found in another unit).

[0661] (If the other cell is a copy of the first cell, then all spatial pixels between the two cells are also color-corresponding pixels.)

[0662] According to some embodiments, the scene value of a pixel in the first unit is compared with the scene value of its spatial or color counterpart in another unit.

[0663] To give a specific, non-limiting example, the scene value associated with the first pixel in the first unit is compared not only with the scene value of the second pixel in the first unit (as described above), but also with the scene value associated with the second pixel in one of the other units. The aforementioned first-second ([1,2]) pixel pair data reflects the result of this comparison. This operation is repeated once or more with the second pixels in one or more other units.

[0664] Figure 44 This operation is illustrated, showing the first cell within the local neighborhood of the copied cell (i.e., the nine pixels outlined in bold in the center). Here, pixel A of the first cell is the base pixel. As previously stated, it is compared with the second pixel (B) in the first cell, as indicated by the short arrow. The base pixel is also compared with the second pixels in the cells to the left and right of the first cell, as indicated by the long arrow. All three comparisons contribute to (updating) the [1,2] pixel pair data. That is, if a high / low comparison is used, the [1,2] count is incremented whenever the base pixel scene value is greater than the ordinate pixel scene value, and decremented otherwise. If a simulated comparison is used, the scene value difference between the base pixel and each ordinate pixel will accumulate to a cumulative total.

[0665] Similarly, the scene value associated with the first pixel in the first cell is compared with the scene value associated with the third pixel not only in the first cell but also in one of the other cells. The previously referenced first-third ([1,3]) pixel pair data is modified to reflect the result of this comparison. This operation is repeated one or more additional times with the third pixel in one or more other cells.

[0666] This operation is like Figure 45 As shown, it is similar to Figure 44 Similarly, but for the case of [1,3] pixel pairs.

[0667] The resulting richer [1,2] and [1,3] pixel pairs form part of the query data.

[0668] Similarly, the first pixel (A) in the first unit can be compared with two or more fourth pixels in other units to obtain richer [1,4] pixel pair data.

[0669] In the case where there are N=9 pixels in each cell, as shown here, this comparison can be similarly extended to comparing the first pixel in the first cell with the 5th to Nth pixels in the other cells, thereby enriching the [1,5] to [1,N] pixel pair data with this additional information.

[0670] Just as the previous embodiment compared the first pixel with the other eight pixels, then the second pixel with the other seven pixels, and so on, until the eighth pixel was compared with another pixel, this embodiment can also do so. That is, the second pixel (B) in the first unit can be compared with the third to ninth pixels in multiple other units to enrich the comparison data used in the query data. Similarly, the third pixel (C) in the first unit can be compared with the fourth to ninth pixels in other units, and so on.

[0671] As described above, the scene value associated with the pixel in the first cell is compared with the scene value associated with the second pixel in each of the two additional cells (one on the left and one on the right). However, a larger set of additional cells can be used. For example, instead of only the left and right additional cells, eight additional cells can be used in this way: the cells adjacent to the left, right, top, and bottom, plus the cells adjacent to the four corners. Thus, the [1,2] pixel data is based on a total of nine comparisons, namely, comparisons with the second pixel in the first cell and the second pixel in each of the eight adjacent cells. That is, the first (base) pixel in the first cell is compared with the second (vertical) pixel in each of the 3x3 tiled cells centered on the first cell.

[0672] Similarly, the comparisons can be extended to the ordinate pixels in a 5x5 tiled cell centered on the first cell. Therefore, each pixel pair of data (e.g., [1,2]) is based on 25 comparisons. If a high / low comparison is used, the value range for each pixel pair of data is -25 to +25. In many embodiments, each such data is offset by 25 to make the value non-negative. For example, if the base pixel of pixel pair [1,2] is associated with a scene value of 150, and the 25 ordinate pixels compared to it are associated with scene values ​​between 40 and 60, then the [1,2] pair of data will accumulate to 50 (because in 25 instances, the base value is greater than the ordinate value, and it is offset by 25).

[0673] If a simulated comparison is used, each pixel pair of data may have a large value, depending on the cumulative sum of the scene value differences. For example, in the example given above, the [1,2] pixel pair of data will accumulate to about 2500 (because the base value exceeds the y-axis value by about 100 in 25 instances).

[0674] These two detailed comparisons, high / low and simulation, are merely examples. Many other comparison operations can be used. For example, the arithmetic difference between the base value and each ordinate value can be weighted according to the spatial distance between the compared pixels, with larger distances resulting in smaller weights. Given this disclosure, those skilled in the art will conceive of many other arrangements. Similarly, as previously stated, machine learning applied to large image training sets can guide the neural network implementation / weighting of these comparisons.

[0675] The scene values ​​associated with each pair of base values ​​and ordinate pixels can each be raw pixel values—analog or digital. Alternatively, they can be processed values, such as data output by an image signal processor module that performs thermal pixel correction or other adjustments on the raw pixel data. Furthermore, if each pixel has already been "corrected" with its own unique dark value, as described above, a better color measurement output will be produced. Therefore, any operation comparing the raw data of one pixel with the raw data of another pixel will involve the dark value correction value for each pixel. Additionally, before the comparison operation itself, the comparison value can be "luminance level adjusted" using the individual grayscale gain or type-category grayscale gain of each pixel (as described above). In cases where the photoelectron generation of most pixels is only at the single-digit level, this grayscale gain adjustment is equivalent to adding the fractional DN value to the value before the comparison operation, where the dark value operation has already produced the fractional DN value. Therefore, when comparing two such corrected fractional DN values, two correction operations are involved. At an SNR level of 1, these tiny detail values ​​– when applied to comparison operations that typically exceed 100 – affect the final chroma value output for any given pixel, which is crucial for “seeing true colors” rather than just a chaotic random color swarm produced by sensor reading noise.

[0676] In addition, the scene value associated with the target pixel can also be the mean or median calculated using all pixels of the same type within a 3x3 or 5x5 neighborhood centered on the target pixel. (When forming the mean or median, pixels farther from the target pixel can be weighted lower than those closer to it).

[0677] Values ​​associated with pixels should not be assumed to be original values ​​unless otherwise stated.

[0678] In some embodiments, the base pixel is associated with one type of scene value (e.g., mean), while the ordinate pixel is associated with another type of scene value (e.g., raw).

[0679] The above discussion details the process used to generate query data to determine the color information of a single pixel (i.e., the center pixel within the cell) within a unit of N pixels. This process is repeated to obtain color information for different pixels within the cell. However, the cell boundaries are shifted, and the cell is reframed so that the different pixel becomes the center pixel.

[0680] Given a field of millions of pixels in an image sensor, the boundaries of the repeatedly tiled cells are arbitrary. For example, scanning from the top left and then down, Bayer cells can be viewed as groups of the following: red / green / green / blue. Or green / red / blue / green. Or green / blue / red / green. Or blue / green / green / red. Repeatedly, the illustrative can be reframed in nine ways. Figure 40 The nine pixels of the unit, such as Figure 47 As shown. Different query datasets are generated for each of these frames based on different comparison datasets and used to determine color information about pixels E, F, D, H, I, G, B, C, and A.

[0681] When presenting the query data, the applicant has adopted the following convention: The pixel starting at the top left of the framed cell is used as the base pixel, and a comparison is performed with the pixels to the right and then downwards (within the first cell, and optionally, in further adjacent cells) to obtain the first 8 elements of the 36 pixel pairs. The process continues by using the pixel at the top center of the framed cell as the base pixel, and performing a comparison with the subsequent pixels to the right and downwards to obtain the next 7 elements of the 36 pixel pairs. This process continues to the subsequent pixels. Therefore, if the pixels in the framed cell start at the top left (horizontally and then downwards) and are numbered 1 to 9, the set of pixel pairs will be ordered as follows:

[0682] {[1,2]、[1,3]、[1,4]、[1,5]、[1,6]、[1,7]、[1,8]、[1,9]、[2,3]、[2,4]、[2,5]、[2,6]、[2,7]、[2,8]、[2,9]、[3,4]、[3,5]、[3,6]、[3,7]、[3,8]、[3,9]、[4,5]、[4,6]、[4,7]、[4,8]、[4,9]、[5,6]、[5,7]、[5,8]、[5,9]、[6,7]、[6,8]、[6,9]、[7,8]、[7,9]、[8,9]}

[0683] It will be recognized that these 36 pixel pairs only include half of the possible ordered pairs. For example, there is no "pixel 2 compared to pixel 1".

[0684] If the original values ​​are used for both the base and y-coordinate pixels, pixel 2 compared to pixel 1 provides no new information; it is simply the negative of pixel 1 compared to pixel 2. However, if the value associated with the base pixel is determined by a mean or median operation, and the value associated with the y-coordinate pixel is the original value (or vice versa), then the comparison between pixel 2 and 1 yields a different result than the comparison between pixel 1 and 2. In this case, a vector of 72 elements could be used based on comparisons between all possible ordered pixel pairs. However, this difference is usually not significant, so a smaller number of elements (i.e., 36) is typically used even if the base and y-coordinate scene values ​​are not determined in the same way. Therefore, in practice, the applicant suggests combining slightly independent low-high pairwise results with high-low pairwise results, keeping the pairwise count itself at 36 instead of 72. (We sometimes use the phrase "non-ordered pixel pairings" to refer to an arrangement where order is not critical. Thus, the count of all non-ordered pixel pairs in a nine-pixel unit is 36.)

[0685] If a simple high / low form comparison is used (by shifting to obtain a positive value), and the comparison is performed on a 5x5 array of cells (e.g., ...), Figure 46 If the text indicates that there are only a few comparisons indicated by the arrows, then the query data for a single pixel at the center of the framed cell can be obtained in the following form, for example:

[0686] {23, 21, 48, 8, 32, 42, 24, 22, 10, 4, 1, 46, 38, 31, 15, 49, 34, 0,34, 13, 13, 25, 2, 23, 28, 26, 25, 11, 35, 5, 4, 1, 10, 4, 0, 12}

[0687] This data structure will be identified as including multi-symbol codes, which represent the results of determining which pairs of pixels are associated with larger scene values.

[0688] In it has 50 36 This is an extremely rich feature vector in the sense of 2^35 possible forms (i.e., on the order of a 61-bit decimal number). The total number of potential colors is fundamentally smaller. If color is considered to be in RGB space, and each of the three values ​​is represented by 16 bits of data, then only 2^35 possible forms exist. 48 There are 100 possible colors (i.e., on the order of a 14-digit decimal number). Within the 36-dimensional population described by the query data, the total number of potential colors forms an extremely sparse mapping. Reflected from the scene and... Figure 46 Each color of light incident on a pixel E in the cell array falls into a volume within a 36-dimensional space of unique query data for that color. Once a pixel's response to the incident light is characterized in this 36-dimensional space, the volume it falls into indicates its color. The reference data serves to provide a mapping between volumes in the 36-dimensional space and the corresponding colors of pixel E.

[0689] (Regarding this practicality, it is known that the human visual system itself has a limited ability to distinguish colors, usually conventionally represented by a two-dimensional chromaticity surface illustrated by the 1931 CIE chromaticity diagram. It is further accepted that the so-called Macadam ellipse can arrange a space of discernible colors, which pulls down these mathematically astronomical numbers to a few thousand (if even more) solvable "perceived" colors. The point is that as the sensor's SNR approaches 1 and even becomes lower, this mathematical richness will eventually at most return to a good basic directional estimate in terms of hue angle.)

[0690] One method for generating reference data is to use a sensor to image a color chart (e.g., a Gretag color chart) containing patches of known colors under known illumination (e.g., D65), and perform the comparisons detailed above on the resulting pixel data to obtain a 36D reference data vector. That is, reference comparison data is generated in the same way as the query data, but the scene colors are known rather than unknown. A given patch of reference scene colors will produce various data vectors depending on various random factors involved, including random variations in patch color, random variations in illumination intensity, sensor shot noise, sensor readout noise, variations in photoelectric sensor sensitivity between pixels, etc. Such perturbations serve to expand the vector representation of known colors into a distribution of data vectors. The 36D space containing such vectors defines the space associated with the known colors.

[0691] Once such reference data vectors associated with known colors are captured, they can be stored and used as the basis for comparison with 36-D query data associated with the target pixel E of the unknown color of light. The task becomes finding the 36-D vector data in the reference data that best matches the query vector. The known color associated with the best-matching reference vector is then assigned as output color information for that pixel.

[0692] This is an activity in string matching (or pattern matching) and can employ numerous techniques. One is to calculate the Hamming distance between the vectors of the query data and the reference data to determine the closest match. Another similar method uses the Levenshtein distance. Yet another is to perform a dot product operation between the vectors of the query data and the reference data to determine the closest match. Many other suitable methods are detailed in the Approximate String Matching section of "Encyclopedia of Algorithms," SpringerLink, 2016 (ISBN 9781493928637).

[0693] Instead of a directory serving as reference data to determine the best match, reference vectors labeled with their corresponding colors can be used to train a convolutional neural network. The network's parameters and weights are iteratively adjusted during training, for example, via an inverse gradient descent process, to configure the network such that it responds to the input query vector corresponding to pixel E by providing output data indicating the color for that pixel. (These parameters / weights can then be stored as reference data).

[0694] In these and other implementations, colors can be defined in the desired color space. Most commonly, X,YCIE coordinate data is used, but other color spaces (including sRGB, L*a*b*, hue angles (L*c*h), etc.) can be used.

[0695] Color charts only provide a limited number of known colors. Another method for generating reference data is to use reliable multispectral images. A suitable set of multispectral images is the so-called CAVE dataset published by Columbia University. The dataset includes 32 scenes, each represented by 16-bit reflectance data at full spectral resolution from 400nm to 700nm (31 bands in total) in 10nm steps. This dataset is available at www. <dot>cs <dot>columbia <dot>edu / CAVE / databases / multispectral / and also on the corresponding web <dot>archive <dot>Available on the org page.

[0696] This method does not utilize the physical image sensor itself to sense the scene. However, the behavior of the image sensor can be modeled, for example, by measuring the spectral transmittance function of its pixels after different filtering, the spectral transmittance variation between filters of the same type, its shot noise, its readout noise, its pixel amplitude variation, etc. Such parameters characterizing the sensor behavior can be applied to published imaging methods to generate a set of one thousand or more simulated pixel data that can be generated (and perturbed) from a given scene by the image sensor in a Monte Carlo manner. Each of these different frames of pixel data is analyzed to determine the 36D vector associated with each "E" pixel in the frame. The color of each of these pixels is known (in terms of the published amplitude at each of the 31 spectral bands) and can be converted to the desired color space. This reference data, which associates the 36D reference vector with the known color, is then used in one of the ways detailed above to output color information in response to input query data.

[0697] It should be understood that the foregoing discussion has already addressed the assignment of color information to individual pixels E within the cell. The reference data just discussed is specific to that pixel E.

[0698] The same process is performed to develop and utilize reference data for the other eight pixels in the cell. Thus, a total of nine sets of reference data are typically required.

[0699] While the query data detailed in the illustrated embodiments is nominally invariant to changes in brightness (i.e., if the scene becomes darker, all pixels should consistently produce smaller output signals so that the inter-pixel comparison results do not change), the applicant has found that this is not so reliable. This is particularly evident at very low brightness levels (e.g., where the signal-to-noise ratio is less than 10:1, 4:1, or 2:1). Therefore, in some embodiments, the applicant generates multiple sets of reference data for each of the nine pixels in a cell, each corresponding to a different range of brightness levels. (Brightness can be determined based on a local neighborhood (e.g., the average original pixel value over 5x5, 9x9, or 15x15 pixels, or the field of a 3x3, 5x5, or 10x10 pixel cell).

[0700] Therefore, when assigning color to pixels, the first step is generally to determine the brightness of the area surrounding the pixel, and then select a set of reference data or the parameters / weights of a neural network adapted for that brightness. (In the case of a neural network, the training data can include triplets of information: a vector of paired pixel data, local brightness, and known color. In use, the network is provided with the vector of the query data and the measured local brightness as input, and outputs an estimate of its corresponding color information.)

[0701] In some embodiments, only two brightness ranges may exist (e.g., dark and not dark (e.g., with a signal-to-noise ratio of less than 5:1 or greater)). In other embodiments, two, five, twelve, or more distinct brightness ranges may exist, each triggering a different mapping from query data to output color. One range may be used for a signal-to-noise ratio of less than 1.5. Another range may be used when the signal-to-noise ratio is less than 3 but at least 1.5. Another range may be used when the signal-to-noise ratio is less than 5 but at least 3. Another range may be used when the signal-to-noise ratio is less than 10 but at least 5, and so on.

[0702] A 36-element vector is one of many possible representations of the pixel comparison information. The symmetric group theory of linear algebra provides many alternative representations. Specifically, pixel pairings within a cell of nine pixels can be described using an S8 algebra (i.e., a bijective group from a set of nine pixels). Any of these alternative representations can be stored in a corresponding data structure and used in implementations of the technology.

[0703] An alternative representation focuses on directly specifying color information output in hue angle and saturation space. There is an independent mapping between the hue angles of points in a given scene and how those hue angles, as single scalar values, map into and out of the 36-dimensional pixel pairwise comparison space. One approach to measuring these direct hue angles is to utilize cosine and sine functions on the hue angles to find a hyperplane in 36-dimensional space that optimizes the fit between the angles in 36-dimensional space and the x and y chromaticity hue angles in the CIE chromaticity space (or the a and b vectors in the Lab color space, or several other color spaces that separate color from luminance).

[0704] While the above disclosure has focused on 3x3 pixel units, other sizes (and non-square shapes) of units can also be used. If a pixel unit has an even number of rows or columns (e.g., 2x2 or 2x3), interpolation methods can be used to make the above arrangement (which is based on the "center" pixel) suitable for such units.

[0705] Alternatively, this cell can be reframed into a larger cell—one with a center pixel. An example is the classic Bayer cell. This cell can be reframed into, for example, a 3x3 cell, such as... Figure 48 The pattern is shown in medium-bold outline. It can thus be viewed as a layout of four distinct 3x3 units. In one unit (the bold unit in the upper left), there are five green, two red, and two blue. In another unit (on the right), there are four green, four blue, and one red. In the third unit (the bold unit in the lower left), there are four green, four red, and one blue. And in the fourth unit, again, there are five green, two red, and two blue. Therefore, it will be seen that a unit can include two or more (and sometimes four or more) pixels of the same type. It will be further recognized that although the units differ, the component colors are the same in each unit.

[0706] The methods detailed above can be used here. For example, within each 3x3 pixel unit, a vector of 36 {-1, 0, +1} elements can be formed and used to assign color to the center pixel of the unit.

[0707] The other methods detailed above can also be applied here. For example, to enrich the query data, comparisons extending beyond a single cell can be used. One such method will be detailed below.

[0708] Consider (in the cross-shading) Figure 48 The pixel B is located at the center of the bolded cell in the lower left of the cell. At the upper left of this cell, the first pixel position is the R pixel, which acts as the base pixel for comparison with the eight other pixel positions in the cell as their ordinates. The second pixel position (i.e., the first ordinate) is the G pixel. In addition to comparing the first pixel (R) value with this second pixel (G) value within the cell, the first pixel is also compared with the G pixel in the nearest adjacent cell to the base pixel but on the left. It is also compared with the G pixel in the nearest adjacent cell to the base pixel but on the right. (These G pixels are underlined). This triples the richness of the pairwise data for the [1,2] pixels—expanding it from a single comparison to three comparisons.

[0709] Similarly, the base pixel (R) can also be compared with the nearest G pixel in the adjacent cell above the target cell and the nearest G pixel in the adjacent cell below the target cell. Both of these pixels are represented by an asterisk. This enriches the data for the [1,2] pixel pair to reflect five comparisons instead of one.

[0710] Depending on the specific application requirements, a further comparison with additional pixels outside the target cell may also be used.

[0711] Sometimes, the "nearest" pixel in adjacent cells on the left / right / above / below is blurry because two such pixels of a specified type are equidistant within the adjacent cells. In this case, the upper pixel of two equidistant pixels in the left cell and the lower pixel of two equidistant pixels in the right cell can be selected for comparison. Similarly, the left pixel of two equidistant pixels in the upper cell and the rightmost pixel of two equidistant pixels in the lower cell can be selected for comparison.

[0712] It will be recognized that, in this embodiment of the technology, the first, second, and third pixels are of the first, second, and third types, respectively (in...) Figure 48 (These are indicated by enlarged letters R, G, and R respectively). Furthermore, the image sensor includes a plurality of additional units surrounding the first unit, each of which includes pixels of the type included in the first unit. This embodiment includes: comparing a scene value of a first pixel associated with the first unit with a scene value of a second type pixel (G) associated with one of the additional units, and updating the first-second ([1,2]) pixel pair data based on the result of the comparison. This action is repeated one or more additional times with the second type pixels in the other additional units.

[0713] This embodiment may further include: comparing the scene value of a first pixel associated with a first pixel in a first unit with the scene value of a third type pixel (R) associated with one of the other units, and updating the first-third ([1,3]) pixel pair data based on the result of the comparison. Again, this action can be repeated one or more additional times with third type pixels in other units. Query data that partially represents the [1,2] and [1,3] pixel pair data is then formed.

[0714] In a similar manner, each of the other 36 pixel pairs of data can be processed outside the target cell.

[0715] In the given example of reframing pixel units with an even number of pixels to form pixel units with an odd number of pixels, the query data is not resolved into color data by referring to one of nine sets of reference data, as in the previous case (the nine sets correspond to nine reframings used to place each pixel in the center, according to...). Figure 47 Instead, one of 36 sets of reference data is used (ignoring further sets of reference data to account for brightness variations). That is, there are four distinct cell arrangements, and nine reframings unique to each arrangement.

[0716] The processing described in detail herein can be performed by a general-purpose microprocessor, GPU, or other computing unit of a computer. More generally, however, some or all of the processing is performed by a dedicated image signal processor (ISP). The ISP circuitry (including arrays of transistor logic gates) can be integrated on the same substrate (typically silicon) as the photoelectric sensor of the image sensor, or the ISP circuitry can be provided on an accompanying chip. In some embodiments, ISP processing is distributed: part on the image sensor chip and others on an accompanying chip.

[0717] Advantageously, all the comparison operations used to generate query data, along with associated cumulative operations, can be performed using simple logic circuits (e.g., addition and subtraction units). This helps reduce gate count, resulting in a reduction in substrate area and device cost. Comparison operations can be performed, and query data can be generated without the use of multiplication or division operations. (Multiplication may be required in other circuits, for example, for neural network execution.)

[0718] Unless otherwise indicated, the term "pixel" as used herein includes photoelectric sensors and may also include corresponding filters and / or microlenses.

[0719] It should be understood that the 36 pixel pairs of data represented by the query data are merely exemplary in some detailed embodiments; naturally, more or fewer pixel pairs of data can be used. For example, in a basic embodiment, two pixel pairs of data are used. For example, scene values ​​associated with one pair of pixels in the first unit are compared, and the result is taken in one pixel pair of data. Scene values ​​associated with a second pair of pixels in the first unit are compared, and the result is taken in the second pixel pair of data. (The two pixel pairs may have a common base pixel or ordinate pixel; for example, they may be pixel pairs [1,2] and [1,3]. Or they may involve four pixels; for example, they may be pixel pairs [1,2] and [3,4]).

[0720] It can also be understood that after these comparisons and their accumulation process have concluded, there will be a mapping from these collected values ​​(e.g., 36 comparisons) to the x and y chromaticities of scene points. The range of choices for performing such mappings is inherently broad, ranging from classical linear mappings to nonlinear mappings, often extending to mappings used in machine learning training and AI processing. In all these mapping cases, the raw data as input generalizes quite well to the term "feature vector"; this same term is commonly used within machine learning applications. This large set of comparisons between pixel data enables the construction of increasingly richer feature vectors, allowing for increasingly lower light measurements of color.

[0721] To understand this further, each pixel in the data can be initialized to a value such as 0 or 25. Comparing the data then serves to update such values.

[0722] Sometimes it refers to the center point or center pixel within a cell. A center pixel is the pixel that spans the geometric center of the cell. Sometimes a cell does not have a center pixel (e.g., a 4x4 pixel cell). In such cases, the center pixel is a pixel whose distance from the geometric center of the cell is no greater than the distance of any other pixel from that geometric center. Thus, a 4x4 pixel cell has four center pixels. A 3x3 pixel cell has only a single center pixel (i.e., a central pixel).

[0723] In some embodiments employing detailed principles, imaging performance (in XY space) of a standard Gretag 24 panel color chart captured under such dark illumination with a signal-to-noise ratio of less than 3:1 can achieve an average chromaticity accuracy (between 0.03). Generally, the applicant has seen improvements in color measurement capabilities of at least one F-stop, even 2 F-stops and even 3 F-stops, in pseudo-side-by-side comparisons between classic Bayer sensors and 3x3, 9-channel variants. (F-stop is equivalent to a factor of 2 in photographic terminology.)

[0724] The detailed ShadowChrome technique works in part due to the fact that the imaged scene is not a random pixel field; the color at a pixel is not independent of the colors at its neighboring pixels. Therefore, information about the relationships between pairs of pixels within a neighborhood, and especially their scene values, can guide color estimation for individual pixels. The size of the neighborhood can vary depending on application requirements. Chroma MTF requirements will influence how large a neighborhood is used to achieve what level of color accuracy. Lower spatial frequency colors perform well for ShadowChrome in the exemplary 3x3 cell embodiment, but there are chroma spatial frequency limitations in every embodiment where aliasing and moiré patterns begin to appear. Specifications for unacceptable levels of such artifacts can serve as constraints used to train neural network-based embodiments to achieve an implementation where such artifacts remain within desired limits.

[0725] Conclusion

[0726] Different embodiments of the applicant's technology achieve superior performance in several key areas compared to image sensors and multispectral image sensors using Bayer patterned color filter arrays. These areas include sensitivity, extended spectral sensitivity, multiple independent channel confirmation, contrast (panchromatic MTF), color gamut, and color accuracy. In some nine-filter embodiments, for example, sensitivity is approximately twice that of a Bayer CFA, and the color gamut is also significantly extended. See also Figure 49A and 49B It compares the performance of the standard Bayer filter with that achieved by the third set of filters mentioned above—both using the Sony IMX428 sensor array.

[0727] The principles of the technology have been described and illustrated above with reference to illustrative embodiments, but it should be understood that the invention is not limited thereto.

[0728] For example, while many embodiments of this technology are described with reference to a 3x3 color filter unit, embodiments with larger and smaller units may also be used. Examples include 4x4, 5x5, 2x3, 2x2, 2x1, etc.

[0729] While color filter arrays can be fabricated separately from the photoelectric sensor (e.g., on a glass plate) and then bonded to the sensor, it is more common to fabricate them as an integral part of the photoelectric sensor using photolithography. Photoelectric sensor components used in image sensors typically also include microlens arrays and / or anti-reflective coatings.

[0730] Some implementations of specific embodiments include pixel side lengths of less than 10 micrometers. Most embodiments include pixel side lengths of less than 2 micrometers, less than 1.5 micrometers, or less than 1 micrometer.

[0731] In some embodiments, a non-standard filter includes some or all of the filters. In other embodiments, a standard red, green, blue, cyan, magenta, yellow, or panchromatic filter includes all the filters.

[0732] An embodiment having four filter elements may select filters from the first, second, or third exemplary filter groups detailed above. Figure 50 Such an embodiment is described. Filter element (i) can use any one of the nine filters in the filter group. Element (ii) can use any one of the remaining eight filters. Element (iii) can use any one of the remaining seven filters. Element (iv) can use any one of the remaining six filters. Alternatively, element (iv) can use the same filter selected for element (i). Alternatively, it can use a common (R, G, B, C, M, Y) filter, such as the green filter in Table III.

[0733] Except as described below, it should be understood that the limiting conditions characterizing the various embodiments can be combined to characterize more specific embodiments. Doing so helps to increase the diversity among the filters in the color filter unit. For example, one such more specific characterization is a color filter unit in which:

[0734] The dot product between the group normalized transmittance functions of two different filters in the unit, calculated at 10nm intervals in the range of 400-700nm, is less than 4;

[0735] The dot product between the group normalized transmittance functions of the first pair of different filters in the unit is 2 to 4 times that between the dot product of the second pair of different filters in the unit.

[0736] The transmittance curves of multiple pairs of filters defined by samples with 10nm intervals in the 400-700nm range intersect each other at least four times;

[0737] The filter unit contains three or more different filters, each filter having an associated transmittance profile, wherein, in 30 10nm bands between 400-700nm, the cross-counting between all the above profile pairs produces a vector containing 30 count values, one of which is at least 9;

[0738] The unit includes a filter whose efficiency is at least 2.0 times or at least 2.5 times that of the other filter in the unit; and

[0739] The correlation between the transmittance functions of two different filters in the unit was calculated in 10nm increments within the 400-700nm range, and the result was negative.

[0740] An exception is when there is a conflict between the constraints.

[0741] It is important to note that instruction manuals may sometimes refer to filter output or filter value. This is usually an abbreviation; more precisely, it refers to the output or value of the photoelectric sensor when a photoelectric sensor with 100% quantum efficiency is superimposed on a specified filter.

[0742] Although many detailed embodiments employ non-standard filters (i.e., not the standard red, green, blue, cyan, magenta, yellow, or panchromatic filters as described above), it is understood that color filter units employing such standard filters can also be used. In some embodiments, color filter units can be used to achieve two or more different spectral transmittance functions by coating a single color resist with two or more different thicknesses.

[0743] The term "different filters" is used repeatedly in this specification. In a sense, all filters are different, even if they were originally intended to be identical, due to manufacturing variations. For the purposes of this specification, the term "slightly different" when applied to two filters in a color filter unit indicates that, when the transmittance functions of the two filters are mutually normalized (i.e., such that at least one filter reaches a maximum value of 1.0) and sampled at 10 nm intervals on the spectrum of interest, the mean square error between the two filters is greater than 0.0018. Unless otherwise stated, the spectrum of interest is 400–700 nm.

[0744] Conversely, for the purposes of this specification, the terms "moderately different" and "significantly different" when applied to two filters in a color filter unit indicate that, when the transmittance functions of the two filters are normalized to each other and sampled at 10 nm intervals on the spectrum of interest (here assumed to be 400-700 nm), the mean square errors between the two filters are greater than 0.02 and 0.25, respectively.

[0745] When the word "different" is used without the qualifiers "slightly," "moderately," or "significantly," it is intended to mean "slightly different." However, since "slightly different" implies a mean squared error greater than 0.0018, the term also refers to "moderately different" and "significantly different," and when "different" is used without any qualifiers, the latter two terms should be understood as the expected kind of "different."

[0746] The mean square error metric mentioned earlier (and also previously mentioned when determining "comparable" filters to standard red, green, and blue filters) involves: determining the difference between each pair of transmittance values ​​at each sampling point in the spectrum of interest; squaring these differences (e.g., 31 if using 400-700nm); summing these (31) values; and dividing by the number of sampling points.

[0747] Sometimes, terms like "transparent" substrate or pigment are also used. As used in this article, "transparent" means that the spectral transmittance function is greater than 90%, preferably greater than 95% or 98%, within the spectral range of interest. If the image sensor produces an RGB or XYZ-based output, the spectral range of interest is the spectrum of human vision, which is taken here as 400-700 nm.

[0748] A key feature of certain color filter arrangements detailed herein is that the spectral transmittance function of each filter in the unit is linearly independent of the transmittance functions of all other different filters in the unit. Linear independence means that the transmittance function of a filter cannot be achieved by a linear combination of the transmittance functions of the other filters in the unit (within an error margin). For the purposes of this specification, the error margin is the same as the 0.25 mean square error threshold defined as "significantly different," as stated above.

[0749] To avoid any ambiguity, the standard deviation cited in this article is calculated using the Microsoft Excel STDEVP function, which should be understood as applying the following formula to a set of N... x value:

[0750]

[0751] It should be noted that, unless otherwise stated, the spectral transmittance functions discussed herein were measured using a spectrometer without near-infrared (NIR) or near-ultraviolet (NUV) filtering. Furthermore, such data do not reflect any spectral variations caused by the sensitivity of photoelectric sensors. If the data were obtained involving near-infrared (NIR) or near-ultraviolet (NUV) filtering or involving photoelectric sensors, it may be necessary to process (e.g., eliminate) such effects to ensure that any comparisons are valid.

[0752] This specification typically uses the term "cell" to refer to a geometric arrangement of multiple adjacent filters (or filtered pixels) repeating within a color filter array (or photosensitive sensor). A CFA can contain cells of different sizes or shapes arranged in a tile-like pattern. A block containing two or more identical tiles is not itself a cell. For example, a typical Bayer cell is 2x2 pixels in size, where two pixels are filtered for green and the other pixels are filtered for red and blue. A set of 2x4 pixels, comprising two such Bayer cells side-by-side (four green, two red, and two blue), is not considered a single cell, but rather two cells.

[0753] In the following claims, it should be understood that numerical limitations are used in an open-ended manner unless indicated by a qualifier such as "precise". A typical example is that a claim describes a three-legged stool, while a four-legged stool also satisfies the requirement because four legs encompass the necessary three legs (and provide more). Similarly, for example, a claim describes a filter unit with an average efficiency of 50%, which encompasses a filter unit with an average efficiency of 55%.

[0754] Primary and secondary colors are sometimes mentioned. Red, green, and blue are often referred to as primary colors, while cyan, magenta, and yellow are often referred to as secondary colors. These three choices are used when the context indicates that a primary or secondary color is one of them.

[0755] More generally, however, we can characterize primary and secondary colors based on the spectral composition of the spectrum relative to the peaks of the visible light spectrum (400–700 nm). If most of the visible spectrum has an amplitude below 50% of the peak, while other wavelengths (amplitude above 50%) comprise a continuous range, we call this spectrum a primary color. If most of the visible spectrum has an amplitude above 50% of the peak, while other wavelengths (intensity below 50%) comprise one or two continuous wavelengths, we call this spectrum a secondary color.

[0756] The spectral transmittance function of a pixel is frequently mentioned. Typically, this same formulation also applies to the spectral sensitivity function of associated pixels. For readability, alternative formulations for each case have been omitted.

[0757] It is assumed that the reader is generally familiar with image sensors and color filter arrays. Nakamura's book, *ImageSensors and Signal Processing for Digital Still Cameras* (CRC Press, ISBN 978-0-8493-3545-7, 2005), is a paper on this subject. Bayer's original patent (assigned to Kodak, US3,971,065) provides a good foundation. Patent documents US20070230774 (Sony), US8,314,866 (Omnivision), US20150185380 (Samsung), and US20150116554 (Fujifilm) illustrate other color filter arrays and image sensor arrangements—details of which can be incorporated into embodiments of this technology (including, for example, pixels of different areas, triangular pixels, non-square units, etc.). For example, the use of interference filters in color filter arrays is described in detail in U.S. Patent Publications 20220244104, 20170195586, 20050153219 and 6,638,668.

[0758] The fabrication process of color filter arrays is well known to those skilled in the art. Examples of this are detailed in U.S. Patents 9,632,222, 8,853,717, 8,603,708, 7,914,957, and 7,763,401, the disclosure of which is incorporated herein by reference. Spin coating is one of several techniques that can be used to achieve photoresist layers of varying thicknesses.

[0759] Those skilled in the art are familiar with mixing or dyeing photoresists to achieve the spectral transmittance function described herein. Exemplary yellow photoresists include CI Pigment Yellow 185 with a particle size of 0.01 to 0.1 micrometers, and the pigment particles constitute 30% to 60% of the photoresist by mass. (Those skilled in the art will understand that controlling the size of the pigment particles is used to change the color intensity and hue, while controlling the mass content of the pigment is used to change the saturation and maximum transmittance.)

[0760] Other pigments can be combined to customize the spectral characteristics of the yellow photoresist just detailed, such as CI pigments Yellow 11, 24, 31, 53, 83, 93, 99, 108, 109, 110, 138, 139, 147, 150, 151, 154, 155, 167, 180, 199, and other pigments (e.g., red). This yellow photoresist can be used to make so-called yellow filters. (Of course, these filters do not filter out yellow light, but rather filter out (attenuate) blue light, so the yellow is retained. This usage is also common in other filters.)

[0761] For further details on mixing photoresists and fabricating color filters, see, for example, Japanese Patent Publication JP2006098684 and US Patent Publications 20220043344, 20190332008, 20140349101, and 20110217636. The latter two patent publications teach photoresists that simultaneously employ pigments and dyes.

[0762] The design of infrared attenuation filters is also familiar to those skilled in the art. Some exemplary arrangements are taught in U.S. Patent Publications 20150346404 and 20210079210.

[0763] Many of the arrangements detailed in this article can be achieved using readily available photoresists such as those from FujiFilm, Toppan, Sumitomo Chemical, and Samsung SDI.

[0764] The following papers provide further information on color filter arrays, including the selection of filter transmittance functions, and are all incorporated herein by reference. The teachings in these references can be used in conjunction with the applicant's disclosed layout.

[0765] ·Yako, M. et al, Video-rate hyperspectral camera based on a CMOS-compatible random array of Fabry–Pérot filters. Nature Photonics, pp.1-6, January 23, 2023;

[0766] ·Arad, B., et al, Filter selection for hyperspectral estimation. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3153-3161), 2017.<000​​​​​​​​​​​​​

[0771] When a sensor using the nine filters in Table 1 is exposed to a scene, each filtered pixel provides output data (e.g., 12-bit or 16-bit). The nine values ​​of each 3x3 filter unit can be mapped to values ​​in the desired color space through multiplication with a linear transformation matrix. A common color space is RGB, which mimics the color receptors of the human eye. However, other color space data can also be generated. The nine different filtered data of each filter unit can be mapped to a color space with more than three channels. 4-dimensional, 5-dimensional, and 6-dimensional output color spaces are exemplary, and 7-dimensional, 8-dimensional, and 9-dimensional output color spaces can also be used. Different applications can achieve optimal results by using different color spaces. In some embodiments, multiple different transformation matrices can be used, thereby mapping, for example, the differently filtered pixel data to two or more different color spaces, such as human RGB and different color spaces characterized by Gaussian curves centered at 450, 500, 550, 600, and 650 nm.

[0772] When image data is fed to a neural network for analysis (e.g., in autonomous vehicle applications or medical diagnostic expert systems), color space data generated as described above can be used. Alternatively, and often preferred, no mapping is performed; the untransformed pixel data is used as input to the neural network system. The system uses this data for training and learns which transformations of the sensor data best reduce the error metric used by the network.

[0773] The neural networks cited in this article can be implemented in a variety of ways. Exemplary networks include AlexNet, VGG16, and GoogleNet (US Patent 9,715,642). Suitable implementations are available from GitHub repositories and from cloud processing providers such as Google, Microsoft (Azure), and Amazon (AWS).

[0774] Some cameras using this technology provide two types of output: data that has been mapped to one or more different color spaces, and untransformed data.

[0775] The processes and arrangements disclosed in this specification can be implemented as instructions for computing devices, including general-purpose processor instructions for various programmable processors, such as microprocessors (e.g., Intel Atom, ARM A8, etc.). These instructions can be implemented as software, firmware, etc. These instructions can also be implemented in various forms of processor circuitry, including programmable logic devices and field-programmable gate arrays.

[0776] Alternatively, the implementation may employ dedicated electronic circuits that have been custom-designed and manufactured as application-specific integrated circuits (ASICs) to perform some or all of the component actions.

[0777] Technicians can write software instructions to implement detailed functions based on the descriptions provided in this article without extensive experimentation. For example, they can use languages ​​such as C, C++, Visual Basic, Java, Python, Tcl, Perl, Scheme, Ruby, and Matlab, combined with relevant data.

[0778] Software and hardware configuration data / instructions are typically stored as instructions in one or more data structures that are transmitted via tangible media (e.g., disks or optical discs, memory cards, volatile and non-volatile semiconductor memories, etc.).

[0779] This specification discusses various embodiments. It should be understood that the methods, elements, and concepts detailed in one embodiment can be combined with the methods, elements, and concepts detailed in other embodiments. While some such arrangements have been specifically described, many arrangements have not been described due to the vast number of permutations and combinations. The applicant also recognizes and anticipates that the methods, elements, and concepts in this specification can be combined, substituted, and interchanged—not only among and between them, but also with methods, elements, and concepts known in the prior art. Furthermore, it should be recognized that the detailed techniques can be combined with other techniques, including current and emerging ones, to produce advantageous effects. Based on the teachings provided in this disclosure, the implementation of such combinations is readily apparent to those skilled in the art.

[0780] Although this disclosure describes in detail the specific order of actions and the specific combination of elements, it should be recognized that other conceived methods may rearrange the actions (possibly omitting some actions and adding others), and other conceived combinations may omit some elements and add others, and so on.

[0781] Although it is presented as a complete system, the detailed arrangement of sub-combinations is also conceived separately (e.g., various features of the complete system are omitted).

[0782] While certain aspects of this technology have been described with reference to illustrative methods, it should be understood that means configured to perform actions of such methods should also be considered part of the applicant's inventive work. Similarly, other aspects have been described with reference to illustrative means, and methods performed by such means are also within the scope of this technology. Furthermore, tangible computer-readable media containing instructions for configuring a processor or other programmable system to perform such methods are explicitly envisioned.

[0783] In order to provide a comprehensive disclosure and to comply with the conciseness requirements of the Patent Act, the applicant has incorporated every reference cited herein by way of citation. (Even if a reference is cited above only for its specific teaching, its entirety shall be incorporated herein.) These references disclose the techniques and teachings that the applicant intends to incorporate into the arrangements detailed herein, and the techniques and teachings currently detailed herein shall also be incorporated therein.

[0784] Given that the above principles and features can be applied to a wide variety of embodiments, it is obvious that these detailed embodiments are merely illustrative and should not be considered as limiting the scope of the invention.< / dot> < / dot> < / dot> < / dot> < / dot> < / dot>

Claims

1. An image sensor including a transparent base (172) with a chessboard pattern spanning the image sensor.

2. The image sensor as claimed in claim 1, wherein, The transparent base of the chessboard pattern defines two types of scattering positions: relatively rising positions and relatively falling positions, wherein the continuous area of ​​the sensor includes cyan, magenta, and yellow filters at one type of position, and red, green, and blue filters at the other type of position.

3. The image sensor as claimed in claim 1, wherein, A first filter composed of a first colored photoresist is formed on one of the transparent substrates, while a second filter composed of the first colored photoresist is not formed on the transparent substrate, wherein the second filter has a thickness greater than that of the first filter.

4. An image sensor, comprising a color filter array, the color filter array including a first filter formed on a substrate and composed of a first colored photoresist, and a second filter not formed on the substrate and composed of the same first colored photoresist, wherein, The second filter has a thickness greater than that of the first filter.

5. The image sensor as claimed in claim 4, wherein, The base is transparent.

6. The image sensor as claimed in claim 4, wherein, The base has a spectral transmittance function of more than 80% in the range of 400-700 nm and less than 50%, 20% or 10% at 720, 740 or 760 nm.

7. An image sensor comprising four pixels, the four pixels being most sensitive at wavelengths between 400 and 700 nm, each pixel comprising a photoelectric sensor and a corresponding filter, the filter giving the pixel a different spectral color response sensitivity than the other pixels among the four pixels, wherein the filters of at least two of the four pixels transfer at least 50% of the incident illuminance to their respective photoelectric sensors at wavelengths above 650 nm to wavelengths above 700 nm.

8. The image sensor as claimed in claim 7, wherein, The four pixels include a red-colored pixel, a green-colored pixel, a blue-colored pixel, and a yellow or magenta-colored pixel.

9. The image sensor as claimed in claim 7, wherein, The four pixels include a yellow-colored pixel, a magenta-colored pixel, and two different pixels selected from red, green, and blue-colored pixels.

10. The image sensor of claim 7, wherein, The four pixels include a yellow-colored pixel, a red-colored pixel, a green-colored pixel, and a fourth pixel. The spectral transmittance function of the fourth pixel exceeds 80% in the range of 400 to 700 nm, but is less than 50%, 20%, or 10% at 740 nm or 780 nm, respectively.

11. An image sensor, comprising a photoelectric sensor array, wherein a color filter array is stacked on top of the photoelectric sensor array, wherein, The filter unit (311) in the color filter array has a different position relative to the photoelectric sensor (312) in the photoelectric sensor array, and the position is gradually shifted on the sensor.

12. An image sensor comprising a row of N photoelectric sensors and a row of M filters stacked thereon, wherein, Neither N / M nor M / N are integers.

13. The image sensor of claim 12, wherein, M and N are coprime.

14. In an imaging system including an image sensor and a color reconstruction module, the image sensor includes: A first unit, comprising a spatial group of N pixels, wherein a first pixel is located at a first position in the spatial group, a second pixel is located at a second position in the spatial group, a third pixel is located at a third position in the spatial group, and so on up to an Nth pixel located at an Nth position in the spatial group, each of the N pixels having a corresponding spectral response defining the type of that pixel, the first unit comprising at least two pixels of different types, a method comprising the following actions: (a) Compare (401) with the scene value associated with the first pair of pixels in the first unit to obtain the first pixel pair data; (b) Compare (402) with the scene values ​​associated with a second pair of pixels in the first unit that are different from the first pair of pixels to obtain second pixel pair data; as well as (c) Query data is formed based on the first pixel pair data and the second pixel pair data, and the query data is provided as input data to the color reconstruction module (411).

15. The method of claim 14, wherein, The first pair of pixels and the second pair of pixels each consist of four pixels.

16. The method of claim 14, wherein, The first pair of pixels and the second pair of pixels each consist of three pixels.

17. The method of claim 14, comprising: Perform additional comparisons with scene values ​​associated with different pixel pairs in the first unit to obtain a set of N(N-1) / 2 pixel pair data, and form the query data based on the set of pixel pair data.

18. The method according to any one of claims 14-17, wherein, The query data includes each of the pixel pairs in the data.

19. The method of any one of claims 14-18, further comprising: The color reconstruction module assigns color information to the center pixel in the first unit based in part on the query data.

20. The method of claim 19, wherein, The color reconstruction module assigns the output color information to the center pixel without referring to the scene value associated with the center pixel.

21. The method of claim 19 or 20, wherein, The color information includes chromaticity information.

22. The method according to any one of claims 19 to 21, wherein, The center pixel is not included in the first pair of pixels, nor is it included in the second pair of pixels.

23. The method of claim 14, wherein, The first pair of pixels includes the first pixel and the second pixel, and a first-second ([1,2]) pixel pair from which the first pixel pair data is obtained is established; and the second pair of pixels includes the first pixel and the third pixel, and a first-third ([1,3]) pixel pair from which the second pixel pair data is obtained is established, wherein the first pixel, the second pixel, and the third pixel are of a first type, a second type, and a third type, respectively; the image sensor includes a plurality of additional units adjacent to the first unit; and the method includes the following actions: (d) Compare the scene value associated with the first pixel with the scene value associated with the pixel in one of the other units that is a spatial or color counterpart of the second pixel, and update the first pixel pair data based on the comparison result; (e) Perform the action (d) one or more additional times using the first pixel and one or more pixels in other units of the other unit that are spatial or color counterparts of the second pixel; (f) The scene value associated with the first pixel is compared with the scene value associated with the pixel in one of the other units that is the spatial or color counterpart of the third pixel, and the second pixel pair data is updated based on the comparison result; (g) Performing action (f) one or more additional times using the first pixel and one or more pixels in other units of the additional units that are spatially or colorally equivalent to the third pixel; and (h) The query data is formed based on the first pixel pair data and the second pixel pair data.

24. The method of claim 23, wherein, The first pixel, the second pixel, and the third pixel are all of different types.

25. The method of claim 23 or 24, wherein, Each of the other units replicates the first unit.

26. The method of any one of claims 23 to 25, comprising: The scene value associated with each possible pair of base pixels and ordinate pixels is compared, wherein the base pixel is one of the first to (N-1)th pixels of the first unit, and the ordinate pixel is the spatial or color equivalent of the second to Nth pixels of the first unit in one of the other units; Based on the comparison results, update the corresponding pixel pair data; and The query data is formed based on the pixel pair data.

27. The method of any one of claims 23 to 26, comprising: The color reconstruction module assigns color information to the center pixel in the first unit. Wherein, the center pixel is not of the first type, the second type, or the third type.

28. The method of any one of claims 23 to 27, comprising: The scene value associated with the first pixel in the first unit is determined by determining the median or mean based on the output signal from the first pixel in the first unit and the output signals from one or more pixels in the other units that correspond to the first pixel space or color.

29. The method according to any one of claims 14-28, wherein, Each type of pixel has an associated efficiency metric, and The method includes: correcting the pixel scene value using the corresponding pixel efficiency metric before performing the comparison operation.

30. The method of any one of claims 14-29, comprising: The comparison operation is performed using logic circuitry fabricated on a common semiconductor substrate together with the image sensor.

31. The method of claim 30, wherein, The logic circuit performs the comparison operation without performing multiplication or division operations.

32. The method according to any one of claims 14-31, wherein, The color reconstruction module performs a pattern matching operation.

33. The method according to any one of claims 14-31, wherein, The color reconstruction module includes a neural network that has been previously trained to distribute output color information in response to input query data.

34. The method according to any one of claims 14-31, comprising: The color reconstruction module identifies the stored reference data that matches the query data in the reference data structure and outputs color information associated with the identified reference data.

35. The method according to any one of claims 14-31, wherein, The color reconstruction module calculates the Hamming distance or Levenshtein distance.

36. The method according to any one of claims 14-31, wherein, The color reconstruction module calculates the dot product.

37. The method of any one of claims 14-36, comprising: The photoelectric sensor of the pixel converts the incident light received from the scene by the image sensor into an electrical signal, and generates a scene value associated with the pixel from the electrical signal.

38. An imaging system, comprising: An image sensor includes a semiconductor substrate fabricated to define a plurality of pixels. The image sensor includes a first unit comprising a spatial group of N pixels. The N pixels have a first pixel at a first position in the spatial group, a second pixel at a second position in the spatial group, a third pixel at a third position in the spatial group, and so on up to an Nth pixel at an Nth position in the spatial group. Each of the N pixels has a corresponding spectral response. The first unit includes at least two pixels with different spectral responses. The image sensor is characterized in that the semiconductor substrate is further fabricated to define hardware circuitry configured to: (a) Compare the scene value associated with the first pair of pixels in the first unit to obtain the first pixel pair data; (b) Compare the scene values ​​in the first unit associated with a second pair of pixels that are different from the first pair of pixels to obtain second pixel pair data; and (c) Based on the first pixel pair data and the second pixel pair data, query data is formed.

39. The imaging system of claim 38, further comprising a color reconstruction module, the color reconstruction module having input for receiving the query data and configured to: assign color information to pixels in the unit based on the query data.

40. The imaging system of claim 39, wherein, The color reconstruction module assigns color information to the center pixel based on the query data and without referring to the scene value associated with the center pixel in the unit.

Images