Method and system for capturing color images

Abstract

A system for capturing color images that has an array of multiple colored light-sensitive photosites. Each color has its own spectral sensitivity. Each color is assigned a high density of pixels in the array of photosites, resulting in a high color separation. The colors that are captured are assigned a lower pixel density in the photosite array. Digital Image Signal The processor generates a raw mosaic image from the image sensor when light strikes the image sensor. and adapted to reconstruct a full color image from the raw image data. The image data is demosaiced, a saturation denoiser is applied to the image data, and the image data is The image is converted to the specified color space, and the saturation denoiser is applied to the specified color space. The conversion to can be performed in any order.

Description

[Technical field]

[0001] Related Patents

[0001] This application is based on U.S. Application No. 63273054, filed on October 28, 2021. and claims the benefit of that priority. [Background technology]

[0002] 2. Background of the Invention

[0002] The present disclosure relates to an image sensor with improved color accuracy and sensitivity.

[0003] In colorimetric image capture using the three primary colors, the spectral sensitivity is calculated based on the human eye's It is necessary to make the spectral sensitivity of the membrane cones closer to that of the L and M cones. This is particularly problematic because the signals R and G are not accurate. Therefore, the signals must be differentiated, resulting in amplification of the noise.

[0004] In the current technical system, the spectral sensitivity characteristics of L, M, and S are subtracted from each other, and the The problem of noise amplification is overcome by bringing the spectral sensitivity characteristics of R, G, and B closer together. The spectral sensitivity of S is simple, but the corresponding R, G, B spectra obtained by subtraction are Sensitivity is complex, with some having low or negative sensitivity in certain sections of the electromagnetic spectrum. As a result, compromised R, G, B filters are used, resulting in poor color accuracy. It gets worse.

[0005] Therefore, improved techniques for capturing color images are needed. Summary of the Invention

[0003] Summary of the Invention A system for capturing color images that uses multiple color-sensitive photosites. The image sensor has an array of 1000 pixels, each with its own spectral sensitivity. Colors with low separation are assigned a high density in pixels within the array of photosites. Colors with high color separation are assigned a low density of pixels in the photosite array. The digital image signal processor detects the front when the image sensor is struck by light. The mosaic-processed raw image is received from the image sensor, and a full-color image is then generated. The raw image data is demosaiced and then processed with a chroma denoiser ( The image data is converted to the specified color space and the The application of the LoMa denoiser and the conversion to the specified color space may be performed in any order. [Brief description of the drawings]

[0004]

[0006] FIG. 1 is a schematic diagram of an exemplary color imaging system in accordance with an embodiment of the present invention. It is.

[0007] Figure 2 is a plot of the normalized spectral sensitivities of S, M, and L cones in the human retina.

[0008] FIG. 3 shows a minimum repeat unit of a color filter array pattern according to an embodiment of the present invention. In this example, one pixel out of nine pixels is color S, and the remaining pixels are colors L and M.

[0009] FIG. 4 shows a minimum repeat of a color filter array pattern according to an embodiment of the present invention. It is a unit, with one pixel out of nine being color S and the remaining pixels being colors L and M.

[0010] FIG. 5 shows a minimum repeat unit of a color filter array pattern according to an embodiment of the present invention. In this example, one pixel in eight is color S, and the remaining pixels are colors L and M.

[0011] FIG. 6 shows a minimum repeating unit of a color filter array pattern according to an embodiment of the present invention. where one pixel out of every eight pixels is color S, and the remaining pixels are colors L and M.

[0012] FIG. 7 illustrates a color filter array pattern capable of binning according to an embodiment of the present invention. In this case, one pixel out of eight pixels is color S, and the remaining pixels are colors L and M. It is.

[0013] FIG. 8 illustrates a color filter array pattern capable of binning according to an embodiment of the present invention. is the smallest repeating unit of the 8-pixel color space, with one pixel being color S and the remaining pixels being colors L and M. It is.

[0014] FIG. 9 illustrates a color filter array pattern capable of binning according to an embodiment of the present invention. This is the smallest repeating unit of the 36 pixels, of which 5 are color S and the rest are color L and M.

[0015] FIG. 10 illustrates a color filter array pattern capable of binning according to an embodiment of the present invention. is the smallest repeating unit of the 9 pixels, one of which is color S and the remaining pixels are colors L and M. It is.

[0016] FIG. 11 illustrates a color filter array pattern capable of binning according to an embodiment of the present invention. It is the minimum repeating unit of a pixel, and one pixel out of eight pixels is color S, and the remaining pixels are colors L and M. It is.

[0017] FIG. 12 shows a graph of the output from the so-called raw image sensor data according to an embodiment of the present invention. 1 is a flow chart of the image signal processing steps required to reconstruct a force image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0018] FIG. 1 shows a system for capturing a color image according to an embodiment of the present invention. The system may be implemented, for example, as part of a digital camera. As shown in FIG. The optical image of the scene 110 passes through and is focused by the lens 120. The image is filtered by a color filter array (CFA) 130 and projected onto an image sensor 140. The image sensor 140 may be a charge-coupled device sensor (CCD image sensor) or an In either case, the image Image sensor 140 consists of an array of discrete light-sensitive areas (or photosites). Each photosite converts the light energy that strikes it into a numerical value. The output of the image sensor 140 is a collection of data consisting of the numerical values ​​generated by each photosite. This "raw" data output of the image sensor 140 is fed to a digital image signal processor. The data is sent to the Internet Service Provider (ISP) 150, which receives, stores, processes, and An electronic image of the scene is reconstructed from the acquired image sensor data. The reconstructed electronic image is It can be stored, copied, reproduced, transmitted and / or displayed. For example, the reconstructed image can be It can be printed on paper or other media and can be displayed on the digital display of a mobile phone. do.

[0019] The image processor of the present invention is, for example, an ASIC (application specific integrated circuit) or the like. It can be implemented as customized hardware, or it can be implemented as generic or specific A software program that runs on a hardware computing platform for All such implementations can be implemented in cameras, phones, personal computers, etc. The present invention may be implemented in a computer, or in any other suitable hardware device. do.

[0020] Here, the image sensor, "raw pixel", "pixel" and "photosite" are discrete It refers to a photosensitive area that detects the electrical charge produced by the sensor and a circuit that digitizes the measurement. The image sensor considered in this invention is one that senses only one color. Spectral sensitivity is the efficiency of a photosite in detecting light as a function of the wavelength of light incident on the photosite. In this context, a "pixel" refers to a specific location in an image, not a In a black-and-white image, one value is associated with the image, whereas in a color or multispectral image, multiple values ​​are associated with the image. It is something that a value is associated with.

[0021] As shown in Figure 2, the pixels of an image sensor are composed of the L cones, M cones, and S cones of the human retina. Each pixel has a set of spectral sensitivities L, M, and S based on its spectral sensitivity. Sensitivity is approximately equal to the spectral sensitivities of the L, M, and S cones, or a linear combination of the L, M, and S cones. In the latter case, the spectral sensitivity characteristics are determined to minimize manufacturing difficulties and deviations from the Luther condition. This should be possible.

[0022] The spectral sensitivity of a pixel is determined by the quantum efficiency of the photoelectric conversion element, the spectral sensitivity of the color filter, It is a function of the lens and other factors used in the sensor. If the color cast of the optical components is known, it should also be taken into account when determining the spectral sensitivity of the pixel. It is necessary to.

[0023] The color filter arrangement considered in the present invention is a repeating pattern of color filters. Here, the CFAs are arranged horizontally, vertically, or in a combination of both. The edge can be defined as a combination of pixels that are aligned along the edges of the image, followed by an operation to trim the edges. The smallest repeating unit is a repeating unit that cannot itself be broken down into smaller repeating units. The minimum repeating unit of m pixels in width and n pixels in height is each m × n cell of the CFA. Note that this is not unique since the sections form the "topology" of the CFA. A CFA with a minimum repeating unit of n pixels, or n pixels in height, has m × n distinct phases.

[0024] The image signal processor (ISP) 150 first demosaices the sensor data and After reconstructing each color plane of ,M, and S, and applying chroma denoiser, color space conversion is performed to obtain RG The color conversion is performed by using the L and M color planes rather than the S color planes because the spectral sensitivities of the L and M colors overlap significantly. Noise in the L and M color planes is amplified by the high transparency of the L and M color filters. In conjunction with the transmittance and high density in the color filter array (CFA), the chroma denoiser To be controlled.

[0025] The L color filter has a higher transmittance than the red (R) color filter. This is because the passband of the L pixel is wide, and the SNR of the L pixel is therefore higher than that of the R pixel. Color filter array pattern

[0026] The color filter arrangement pattern used is more favorable for L and M pixels than for S pixels. One class of patterns is dense. The sampling grid is decomposed into two quintile grids, The CFA can be generated by assigning L to one quintile and M to the other quintile. This is accomplished by replacing a small number of pixels, either M or S, with S pixels.

[0027] Figure 3 shows an example of a CFA in which one pixel out of nine is S color and the remaining pixels are L and M colors. The CFA performance is calculated by perturbing S pixels by a distance of 1 pixel to 4 pixels in a 6x6 tile of pixels. It is possible to generate irregular deformations of turns. If you shift the S pixel diagonally by 1 pixel, This is more resistant to aliasing and false colors than shifting one pixel horizontally or vertically. An example of such an irregular pattern is shown in Figure 4.

[0028] Figure 5 shows an example of a CFA in which one pixel out of eight is S color and the remaining pixels are L and M colors. The CFA pattern is created by perturbing S pixels by a distance of 1 pixel per 4 pixels of a 4x8 pixel tile. It is possible to generate random deformations of the line. Shifting S by one pixel horizontally or vertically produces a This method is more resistant to aliasing and false colors than the method of shifting the image diagonally by one pixel. An example of a rule pattern is shown in FIG.

[0029] Figure 7 shows a CFA with 2:1 binning. The pattern is split into diagonal and anti-diagonal lines. The pixels can be divided into tiles of 2x2 pixels that are binned along the The binning is always performed on pairs of pixels of the same color. Figure 8 shows a variant of the CFA that allows 2:1 binning, where some S pixels are aligned diagonally. , and the other S pixels are placed along the opposite corners of the tile.

[0030] Figure 9 shows a CFA capable of binning, where the pattern is divided into tiles of 3x3 pixels. Two quintic grids of four or five pixels in each tile are The images are then binned to produce one raw image value per color for each image file. Binning is usually The CFA is performed for pixels of the same color.

[0031] Figure 11 shows a CFA that can be binned, where the pattern is split into 4x4 pixel tiles. Each tile is made up of 8 pixels and can be divided into two quintuplets. The children are binned to produce one value of raw image data per color per tile. The filtering is always performed on pixels of the same color.

[0032] Similar patterns include reflection, rotation, lateral flip, translation, phase change, L,M pixel color It can be generated by swapping Image Signal Processor

[0033] The process 200 performed by the image signal processor 150 (FIG. 1) is shown in the flow chart of FIG. Generally speaking, five pixels with pixel positions (x,y) where x+y is an odd number are The quintet contains all L pixels, and the other quintet consists of pixel locations (x,y) where x+y is an even number. Assume that the grid contains all M pixels.

[0034] The demosaic process starts with detecting the edge direction 202. Calculate the average of the absolute value of the horizontal gradient of the This is done by choosing the direction with the lower average absolute value of the gradient compared to the average. is the pixel (x x ) whose four horizontally and vertically adjacent pixels are all color S. For (x, y), d(x, y) exists if at least one pixel adjacent to (x, y) is color S. do not have TIFF2025503805000002.tif690 Next, all pixels in a small neighborhood of about 5x5 pixels centered on the pixel location (x, y) are For a predefined threshold τ, the edge direction E is given by is determined as follows: TIFF2025503805000003.tif1432

[0035] Following demosaicing, the L and M values ​​are interpolated at S pixel locations 204. If we have a cell and x+y is odd, then the L,M values ​​are interpolated. TIFF2025503805000004.tif2195 If location (x, y) has a pixel of color S, and x + y is an even number. TIFF2025503805000005.tif2199 where L is available for all pixels (x,y) in the quincunx grid where x+y is odd, and M is available for all pixels in other quintic lattices where x+y is even.

[0036] The next step is to interpolate the L,M color plane 206. Many of the algorithms used to reconstruct color planes are well known to those skilled in the art. We can reconstruct L and M by using the Laplacian, which is a common interpolation algorithm. Then, we can interpolate L at pixel location (x,y), where x+y is an even number: TIFF2025503805000006.tif15105When x + y is an odd number TIFF2025503805000007.tif16112

[0037] The guide image G is generated as a linear combination of L and M. Then, G is defined as the guide image 208. Then, S color planes are reconstructed by guided upsampling of S pixels. Upsampling techniques are known to those skilled in the art. See, for example, J. Kopf, M. F. Cohen, and D. L. Ischinski, and M. Uyttendaele, "Joint bilateral upsampling" "ACM Transactions on Graphics, vol. 26(3), no. 96, 2007, and , He, Kaiming, Jian Sun, and Xiaoou Tang. "Guided image filtering." "Image Filtering" IEEE Transactions on Pattern Analysis and Machine Learning telligence 35.6 (2012): 1397-1409.

[0038] The chroma denoiser is applied to the demosaiced image. The image is first Converted to C1C2 color space, where Y is the luminance and C1, C2 are the two chrominances as follows: TIFF2025503805000008.tif931a11, a12, and a13 are selected to maximize the average SNR of Y on the gray image. Illumination, read noise limited, exposure, or high illumination, shot noise limited, exposure, or both The options are a11+a12+a13=1, a21+a22+ The constraints are a23=0 and a31+a32+a33=0.

[0039] Otherwise, the image is converted to the 210, Y CLCM CS color space, where Y is the luminance. where CL, CM, and CS are the three color difference signals as follows: TIFF2025503805000009.tif2041 a11, a12, and a13 are selected to maximize the SNR of Y on the gray image. , read noise limited, exposure, or high illumination, shot noise limited, exposure, or both can be maximized by making a compromise between the two. Also, the constraint is a1 + a2 + a3 = 1.

[0040] Chroma Denoiser 212, which leverages the high SNR of Y, reduces the chrominance depending on the color space used. It is used to remove noise from the nonce signals C1, C2, or CL, CM, and CS. The implementation of the chroma denoiser is , a bilateration that is adapted to use the Y and chrominance channels in its edge stopping function. The linear filter, or sigma filter, is e.g., Tomasi, Carlo, and Roberto Mandu Chi's "Bilateral Filtering for Gray and Color Images" Sixth International onal conference on computer vision (IEEE Cat. No. 98CH36271). IEEE, 1998 and Lee , Jong-Sen. “Digital image smoothing and the sigma filter.” Computer Vision, Graphics, and Image Processing 24.2 (1983): 255-269. Also see Buades, Antoni, Bartomeu Coll, and Jean-Michel Morel's ``Non-local means den "Nonlocal Means Noise Removal." Image Processing On Line 1 (2011): 208-212 Wavelet and other dictionary denoisers determine the shrinkage or other adaptation of the transform coefficients. It is also possible to adapt the method to include the Y and chrominance channels when converting the image. "Image denoising using scale mixtures of Gaussians in the wavelet domain." Por tilla, Javier, et al. IEEE Transactions on Image Processing 12.11 (2003): 1338- 1351 and Elad, Michael, and Michal Aharon. “Image denoising via sparse and redu "Sparse redundancy representations over learned dictionaries." Image Denoising by the Current State. IEEE Transactions on Image Processing 15.12 (2006): 3 See 736-3745 for more information. Also, local affine color models such as guide image filters The filter based on the gaussian function acts as a chroma noise remover for images in any color space. For example, He, Kaiming, Jian Sun, and and Xiaoou Tang, “Guided image filtering,” IEEE transactions on pattern analysis. Please refer to, for example, sis and machine intelligence 35.6 (2012): 1397-1409. Such chroma noise reduction adaptation is within the capabilities of those skilled in the art.

[0041] The chroma denoiser decomposes the image into a Laplacian pyramid and Can be used in multi-scale settings by applying chroma denoiser at each level can be done.

[0042] The denoised Y C1C2 or Y CLCM CS images are then stored for further processing or output. Then, it is converted to RGB or another color space. generalization

[0043] The L, M, and S CFAs generalize to CFAs with high and low pixel density colors. The associated image signal processing device first demosaices the high density colors and then linearly We take the combination to obtain the guide image, and then demote the low-density colors by guided upsampling. The ISP then applies a chroma denoiser and converts it to the desired color space.

[0044] Colors with substantially low color separation are assigned substantially higher density pixels in the CFA. If the color space conversion performed by the image signal processor is linear, it can be expressed as R: TIFF2025503805000010.tif1639This time JPEG2025503805000011.jpg1794 In this case, the color space of Y is not luminance or chrominance, but Note that the color components X should be defined in terms of the color primaries. j , 1Ͱ 4; The "color separations" for j ≤ N are defined as follows: TIFF2025503805000012.tif725

[0045] When using a general nonlinear color space conversion method, the color space of the image sensor is The metric "color separation" of a specific color component in It is defined as the reciprocal of the absolute value of the maximum additive or subtractive effect of a particular color component in As with linear color space conversion, colors with substantially lower separation have substantially higher separation in the CFA. Density pixels are assigned.

[0046] In accordance with the present invention, colors with high color separation are generally assigned low pixel density, while colors Low separation colors are generally assigned a high pixel density. This effectively means that This is true for colors that have a substantially lower color separation, but not necessarily for colors that have a substantially lower color separation. In other words, the pixel density allocation may not make a substantial difference to the color separation. For light colors, the color separation is not necessarily monotonous. For example, there is no substantial difference between L and M colors. However, the color separation of M is slightly higher than that of L. Therefore, in one embodiment of the present invention, , color M may be assigned a higher pixel density than color L, despite having a slightly higher color separation. Conversely, color S has a much higher color separation than colors L or M, so it is more sensitive to the It is preferable that much fewer pixels are allocated. Therefore, in this example, Colors with relatively low color separation can be used to achieve substantially higher pixel density in a photosite array. whereas colors with substantially high color separation are substantially is assigned a lower pixel density.

Claims

1. 1. A method for capturing a color image, comprising the steps of: Providing an image sensor having an array of multiple color sensitive photosites. Thus, each color has a spectral sensitivity, and the colors with substantially lower color separation are The colors with a substantially higher pixel density and substantially higher color separation are assigned to the photosite art. A substantially lower density of pixels within the ray is assigned. The image sensor is used to generate a raw image by impinging light on the image sensor. generating data; A digital image signal processor is used to reconstruct a full color image from the raw image data. 。

2. 2. The method of claim 1, wherein at least one color has a substantially higher color separation than the other colors. is assigned a substantially higher pixel density than other colors.

3. 2. The method of claim 1, wherein the reconstruction of the full color image comprises: of. performing demosaic processing on the raw image data to generate demosaic processed image data; Steps A chroma noise removal technique is applied to the demosaiced image data. converting the processed image data into a specified color space; The application of the subtraction techniques and the conversion to the specified color space may be done in any order.

4. 2. The method according to claim 1, comprising: The two-color photosites labeled L and M have poor color separation. The third color photosites labeled S are present in lower density than the L and M color photosites. It has high color separation.

5. The method according to claim 4, wherein the spectral sensitivities of the colors L, M, and S are close to the spectral sensitivities of the L, M, and S cones of the human retina. It is characterized as an approximation of the spectral sensitivity of the L, M, and S cones in the human retina, or a linear combination of these. To

6. In the method according to claim 4, the L color filter is more transparent than the red color filter. Something with a high chance of success.

7. 6. The method of claim 5, wherein the image sensor is made up of two or more photosite arrays. The eyes are divided into arrays, with all the photosites of color L on one, fifth array. and further characterized in that all photosites of color M are on another fifth array. 。

8. 4. The method of claim 3, wherein the full color image is reconstructed by: Completely demosaicing a color plane having a high density of light sites. A specific linear combination of the demosaiced dense color planes is defined as the guide image. Steps to define A guide demosaic camera is used to reconstruct a color plane having low density light sites. Use the la.

9. 4. The method of claim 3, wherein noise is suppressed by: The process of converting the full-color image into a luminance-chrominance color space. Top applying said chroma noise removal technique to the image in luminance-chrominance color space. Here, luminance is either a weighted sum of the L, M, and S color planes to achieve a high average SNR, or an average SNR is a highly nonlinear function of the L, M, and S color planes.

10. 10. The method of claim 9, wherein the chroma noise removal technique uses a sigma filter.

11. 10. The method of claim 9, wherein the chroma denoiser is a bilateral filter. Those using filters

12. 10. The method of claim 9, wherein the chroma denoiser is a wavelet denoiser.

13. 10. The method of claim 9, wherein the chroma noise reduction technique is a dictionary-based noise reduction technique. Something.

14. The method of claim 9, wherein the chroma noise reduction technique is a Guided Image Filter. Filters based on a local affine color model, such as the RGB color filter (RGB color filter).

15. The method of claim 9, wherein the chroma denoiser is applied in a multi-scale setting.

16. The method according to claim 15, wherein the image is decomposed into a Laplacian pyramid and a chroma noise reduction technique is used. The art is applied at each level of the pyramid.

17. A method for capturing color images, including: An image sensor having an array of photosensitive photosites of multiple colors, each color corresponding to a spectrum colors that have substantially lower color separation and substantially higher pixel resolution in the photosite array; The colors with higher density are assigned to the actual pixels in the photosite array. Those that are assigned a qualitatively lower density. Digital image signal processor (Digital Image Signal Processor) applicable to the following: Something that is done. When the image sensor receives light, a raw mosaic image is received from the image sensor. believe Reconstructing a full color image from the raw image data.

18. 18. The system of claim 17, wherein at least one of the colors has a substantially higher color separation than the other colors. one color is assigned a substantially higher pixel density than the other colors. thing.

19. 18. The system according to claim 17, further comprising a digital image signal processing device (DIGAS). A 32-bit arithmetic processor (Null Processor) consisting of the following: The raw image data is subjected to demosaic processing to generate demosaic image data. A chroma noise removal function is applied to the demosaic processed image data, and the demosaic processed image data is Converting the data to a specified color space, where the application of the chroma noise reduction function and the The conversion to the specified color space may be performed in any order.

20. 20. The system of claim 17, comprising: The two-color photosites labeled L and M have poor color separation. The third color photosites labeled S are present in lower density than the L and M color photosites. It has high color separation.

21. 21. The system of claim 20, further comprising: a first input for inputting the spectral sensitivities of the colors L, M, and S, the spectral sensitivities of the L, M, and S cones of the human retina; An approximation of the spectral sensitivity of the L, M, and S cones in the human retina. 。

22. 21. The system of claim 20, wherein the L color filter has a higher transmissivity than the Red color filter. Those that have sensitivities.

23. The photosite array that constitutes the image sensor is divided into two fifth arrays. So all the photosites of color L are on one fifth array, and all the photosites of color M are on the other fifth array. The second array is characterized by the fact that the target site is on the fifth array on the other side.

24. 24. The system of claim 23, wherein the photosite has a regular array of photosites of color S. The minimum repeating pattern of the line array is as follows: MSMLS LMLML MLMLM LSLMS MLMLM LMLML In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some things.

25. 24. The system of claim 23, wherein the photosite has an irregular arrangement of photosites of color S. The minimum repeating pattern of the line array is as follows: MSMLS LMLML SLMLM LMLMS MLMLM LMLML In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some things.

26. 24. The system of claim 23, wherein the photosite has a regular array of photosites of color S. The minimum repeating pattern of the line array is as follows: SMLM MLML MSM LML In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some things.

27. 24. The system of claim 23, wherein the photosite has an irregular arrangement of photosites of color S. The minimum repeating pattern of the line array is as follows: SMLMSMLM MLMLMLML MSMLMLS LMLMLML In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some things.

28. 24. The system of claim 23, wherein the photosite array is binnable, have a minimal repeating pattern with a regular arrangement of photosites of color S such that Characterized by: MLM LML MSM LMS In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some things. The minimal repeating pattern is divided into 2x2 photo tiles, and all photos of a color are binned. The raw image data is filtered to produce one value per color for each tile of the raw image data.

29. 24. The system of claim 23, wherein the photosite array is binnable, have a minimal repeating pattern with an irregular arrangement of photosites of color S such that Characterized by: MLMLMLM LMLMLML MSMLMLS LMSMLSL MLMLMLM LMLMLML MLSLMSM LSLMLMS In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some The minimal repeating pattern is divided into 2x2 photo tiles, and all photos of a color are binned. The raw image data is filtered to produce one value per color for each tile of the raw image data.

30. 24. The system of claim 23, wherein the photosite array is binned, such that The device is characterized by having such a minimum repeating pattern. MLMLM LMLML MLMLM LMSMS MLMSM LMSMS In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some The minimal repeating pattern is divided into 3x3 photo tiles, and all photos of a color are binned. The raw image data is filtered to produce one value per color for each tile of the raw image data.

31. 24. The system of claim 23, wherein the photosite array is binned, such that The device is characterized by having such a minimum repeating pattern. MLMLM LMLML MLMLM LMLSL MLSLS LMLSL In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some The minimal repeating pattern is divided into 3x3 photo tiles, and all photos of a color are binned. The raw image data is filtered to produce one value per color for each tile of the raw image data.

32. 24. The system of claim 23, wherein the photosite array is binned, such that The device is characterized by having such a minimum repeating pattern. MLMLMLM LMLMLML MLMLMLM LMLMLML MLMSMSM LMLMSMS MLMSMSM LMLMSMS In addition to this, reflection, rotation, phase, L and M pixel swapping, and combinations of these Including some The minimum repeating pattern is divided into 4x4 photo tiles, and all photos of a color are binned. The raw image data is filtered to produce one value per color for each tile of the raw image data.

33. 20. The system of claim 19, wherein a full color image is reconstructed by: Completely demosaicing the color plane with a higher density of photosites. A particular linear combination of the demosaiced color planes having a higher density of light sites. The image is defined as a guide image. Reconstruct the color plane with the low density of light sites using a guide demosaicing camera do.

34. 20. The system according to claim 19, further comprising: a digital image signal processor for suppressing noise, Something that uses. The full color image is transformed into a luminance-chrominance color space. The chroma noise removal technique is applied to the image in the luminance-chrominance color space. Here, luminance is either a weighted sum of the L, M, and S color planes to achieve a high average SNR, or an average SNR is a highly nonlinear function of the L, M, and S color planes.

Images