System and method for using an image sensor with image and spectral pixels

Abstract

A method includes capturing an image from a scene with an image sensor. The image sensor has a plurality of spectral pixels and a plurality of image pixels. The method also includes collecting gathered data from the spectral imaging array based on received light. The gathered data is separated into spectral pixel data and image pixel data. The spectral pixel data is provided by the spectral pixels and the image pixel data is provided by the image pixels. The method includes both generating spectral information for the image based on the spectral pixel data and generating image information for the image based on the image pixel data.

Description

Technical Field

[0001] Various embodiments generally relate to optical sensor systems, methods, apparatuses, and computer programs, and more specifically, to optical fingerprint and facial sensors. Background Technology

[0002] Image sensors have various pixels used to collect information about received light. These pixels can be image pixels or spectral pixels. Image pixels (or non-spectral pixels) are used to collect information about the detected image. Spectral pixels are pixels that have a response that depends on the spectrum of light and can be used to identify the actual spectrum (or "color") of light. However, when spectral and non-spectral pixels are combined in the pixel array of a CMOS sensor, the pixel response can also depend on its position within the image.

[0003] Integrating spectral pixels directly into the imager array is valuable because it increases the amount of information collected. However, there are problems with doing so. For example, each spectral pixel cannot be used for imaging because the image pattern affects the pixel's response depending on its position in the image plane.

[0004] An image sensor is needed that can benefit from the use of both spectral pixels and image pixels. Summary of the Invention

[0005] The following invention is merely representative and not limiting.

[0006] By using the embodiments, the above-mentioned problems can be overcome and other advantages can be achieved.

[0007] In a first aspect, embodiments provide a method for collecting imager data. The method includes capturing an image from a scene using an image sensor. The image sensor has a plurality of spectral pixels and a plurality of image pixels. The method also includes collecting collected data from a spectral imaging array based on received light. The collected data is separated into spectral pixel data and image pixel data. The spectral pixel data is provided by the spectral pixels, and the image pixel data is provided by the image pixels. The method includes both spectral information for generating an image based on the spectral pixel data and image information for generating an image based on the image pixel data.

[0008] In some embodiments, each of the plurality of spectral pixels has a corresponding spectral response.

[0009] In some embodiments, separating the collected data includes filtering the collected data with a downsampling kernel to remove the spectral pixel data.

[0010] In some embodiments, separating the collected data further includes replacing the spectral pixel data from the spectral pixels with a weighted average of the image pixel data from the image pixels surrounding each of the plurality of spectral pixels.

[0011] In some embodiments, the image includes a finger, and the method further includes determining whether the finger is authorized based at least in part on the spectral information and the image information.

[0012] In some embodiments, determining whether the finger is authorized includes matching the spectral information and the image information with the user's fingerprint data.

[0013] In another embodiment, an image sensor is provided. The image sensor includes a pixel array having a plurality of spectral pixels and a plurality of image pixels, and a processor. The processor is configured to receive collected data from the spectral imaging array and separate the collected data into spectral pixel data and image pixel data. The spectral pixel data is provided by the spectral pixels, and the image pixel data is provided by the image pixels. The processor is also configured to generate spectral information of an image based on the spectral pixel data and image information of an image based on the image pixel data.

[0014] In some embodiments, the processor is further configured to output the spectral information and the image information.

[0015] In some embodiments, the processor is configured to generate the spectral information and the image information simultaneously.

[0016] In some embodiments, each of the plurality of spectral pixels has an associated spectral response.

[0017] In some embodiments, for each of a plurality of nanostructure designs, at least one spectral pixel of the plurality of spectral pixels conforms to the nanostructure design, and each nanostructure design has an associated spectral response.

[0018] In some embodiments, the pixel array includes an imaging region and a spectral collection region, the imaging region including the plurality of image pixels, and the spectral collection region including the plurality of spectral pixels.

[0019] In some embodiments, the imaging region defines a circle centrally located on the pixel array, and the spectral collection region is outside the circle.

[0020] In some embodiments, separating the collected data includes filtering the collected data with a downsampling kernel to remove the spectral pixel data.

[0021] In some embodiments, separating the collected data further includes replacing the spectral pixel data from the spectral pixels with a weighted average of the image pixel data from the image pixels surrounding each of the plurality of spectral pixels.

[0022] In some embodiments, the captured object includes a finger, and the processor is further configured to determine whether the finger is authorized based at least in part on the spectral information and the image information. Attached Figure Description

[0023] The aspects of the described embodiments will become more apparent in the following description when read in conjunction with the accompanying drawings.

[0024] Figure 1 It showcases an array with a unique pixel design.

[0025] Figure 2 The illustration shows the response of spectral pixels to different colors of light.

[0026] Figure 3 An example of an image pixel array and filter kernel is illustrated.

[0027] Figure 4 Another example of an image pixel array and filter kernel with spectral pixels is illustrated.

[0028] Figure 5 An example of an array of spectral pixels is shown, which is used to determine a unique pattern of response in smooth and unboxed images.

[0029] Figure 6 An example of an array of spectral pixels is shown.

[0030] Figure 7 An example of a CCC-type imager is shown.

[0031] Figure 8 An example using a test pattern image is shown.

[0032] Figure 9 Another example using a test pattern image is shown.

[0033] Figure 10 An example of a logic flowchart is shown.

[0034] Figure 11 An example block diagram suitable for implementing various embodiments is shown. Detailed Implementation

[0035] Various embodiments provide image sensors having both image pixels and spectral pixels, as well as methods for using these image sensors. By combining both types of pixels in a single sensor, the sensor can collect more information than if it only had a single type of pixel. This additional information can be used to supplement image data and / or improve image recognition.

[0036] Diffractive nanostructures (e.g., photonic crystals) can be used on CMOS pixels to create spectrum-dependent responses. The responses of these spectral pixels can be used to calculate the spectrum of light. However, using spectral pixels can be problematic if they are not considered. Otherwise, spectral pixels may appear to produce artifacts in the image because their responses relate to the colorimetric spectrum of light, rather than providing image data. In some cases, such as in fingerprint sensors, images are oversampled, and such problems can be corrected.

[0037] In a conventional pixel array, the final image resolution is much lower than the pixel resolution. The final image can be binned into 3x3 pixel sets using an applied smoothing (anti-aliasing) filter. This approach ensures the removal of moiré patterns when imaging through a display.

[0038] When combining image and spectral pixels, the final image can still have sufficient resolution even after removing spectral pixels. This allows the combined array to obtain additional spectral data without sacrificing image quality and improves the ability to authenticate / recognize fingerprints.

[0039] Various embodiments provide image sensor design and operation methods to remove this image dependency, enabling the use of spectral pixels within a pixel array. Another issue with using spectral pixels is that the calibration used to ensure pixel output is correctly correlated with image / spectral information may depend on module-level variations (e.g., for an optical fingerprint sensor located under a display, calibration can be performed on each phone component because pixel response can be affected by numerous manufacturing inhomogeneities). Enabling module-level calibration is both cumbersome and impractical for mass production. In contrast, various embodiments provide a method to remove the effects of these inhomogeneities and enable efficient “one-time” calibration for a large number of devices.

[0040] Photonic nanostructures can be fabricated on top of CMOS sensor pixels to discriminate the spectrum of light incident on the image sensor. In some cases, the nanostructures used are photonic crystals, but other types of designs can also be used. A variety of unique nanostructure designs are available, and each design results in a different spectral response of the pixel to light. By combining the responses of these differently nanostructured pixels, spectral information of the light can be discriminated. For example, if there are enough unique spectral pixels with unique spectral responses, the entire spectrum can be reconstructed.

[0041] Figure 1 An array 100 of unique pixels 110 is shown. Each pixel 110 is shown with a corresponding symbol to demonstrate that each pixel 110 has a corresponding design and associated spectral response Ii, which differs from the spectral response of each other pixel 110. The pixel responses can be transformed using an algorithm to calculate the spectrum of the incident illumination (see below).

[0042] Each of the spectral pixels 110 has a unique design and spectral response. These responses depend on the spectrum of the incident illumination, and each pixel's response can be different. If designed properly, each pixel can have a different response pattern. The spectral information, which is the spectrum of the incident illumination, can then be determined by an algorithm. The resolution of the obtained spectrum depends on the number of unique spectral pixels. For example, without information compression, a spectrum from 430 nm to 570 nm can be achieved with 36 values ​​to achieve a resolution of 4 nm wavelengths. Typically, dozens of unique designs are used to create meaningful spectra.

[0043] CMOS image sensors can be combined in various embodiments to simultaneously determine image and spectral information, specifically the spectrum of incident illumination. For biometric applications such as fingerprint or facial recognition, the spectrum of incident illumination (which could be the spectral content of a finger or face) can be used to create additional information for identity verification or fraud detection. Another application is food inspection, where spectral information can be used to detect spoilage or to inspect the color of manufactured parts.

[0044] Figure 2 The illustration shows the responses 218, 228, and 238 of spectral pixels 212, 222, and 232 to different colors of light (such as red 210, green 220, and yellow 230), as in a CMOS sensor. Integrating spectral pixels directly into a CMOS sensor presents several challenges. Information i from each spectral pixel is represented as the pixel response I. i The data is collected and the pixel response I is executed using an algorithm. i The transformation is used to find the spectrum R of the incident illumination. j For example, a spectrum from 430 nm to 570 nm with a resolution of 4 nm per spectral depth.

[0045] Due to the spectral pixel response (I i It depends on the spectrum of light (R) j Therefore, their response cannot be reliably used for image information. Consequently, spectral pixels do not contribute to image information. For a given CMOS pixel array, this degrades image information and may resemble an array with defective pixels.

[0046] In addition, spectral pixel response (I i The response of a spectral pixel can be influenced by its position within the image. The response of each spectral pixel depends on whether it is located in a darker or brighter area of ​​the image (such as an image of a fingerprint or face). In these cases, the spectral response is incompatible with the image pixel functionality.

[0047] Another challenge relates to the calibration of spectral pixels. Even with uniform illumination, image background can cause non-uniformity across the pixel array, and spectral pixels must be calibrated to address this issue. Depending on the system, this calibration can be cumbersome or impossible. Furthermore, background noise can even be caused by light propagation through the backplane of an OLED display. Background noise is different for each component and can change with each use of the sensor. Calibrating this background response is impractical, and alternative methods are sought.

[0048] In a non-limiting embodiment, the first method relies on the desired image resolution being lower than the pixel array resolution. In other words, the image is oversampled. This is the case with fingerprint CMOS sensors. In fingerprint sensors, the pixel resolution is three times or more than the pixel resolution used in the final image. For this purpose, the image is typically binned to reduce the amount of data.

[0049] Figure 3 The image pixel array 310 and filter kernel 314 are illustrated. During the binning process, the image is filtered with binning kernel 314, which helps to smooth the image and prevent any aliasing artifacts.

[0050] White pixels (image pixels) are used to detect images, such as fingerprints. Image pixels do not use color filters, so the detected image can be a grayscale image. Gray pixels can be used to collect spectral information to check whether the detected fingerprint belongs to a fake or real finger by looking at the spectrum of the incident illumination, where the incident illumination can be light reflected from the finger.

[0051] The value detected at the spectral pixel is not used as the image value because it is overridden by a specific color filter, or it is created using the specific technique mentioned above. The image value at the spectral pixel is calculated using a convolution kernel, and the center of the kernel is placed at the spectral pixel.

[0052] exist Figure 3 In this process, binning is typically done using a 3x3 bin with a Gaussian 5x5 filter, but other combinations can also be used, including 2x2 binning with a 3x3 filter. In these cases, one pixel can be removed from the binning / filtering process so that it contributes nothing to the image. Figure 4 The document provides an explanation regarding the 3x3 compartmentation.

[0053] Figure 4An image pixel array 410 with spectral pixels 412 and a filter kernel 414 is shown. In this case, the filter kernel 414 has been adjusted to remove the center pixel (here, spectral pixel 412) from the bin sum and replace it with a weighted average of the surrounding pixels. Using this new filter kernel, the final binned image is still formed at the same resolution as before, although the total response is reduced.

[0054] Spectral pixel 412 can be assigned to pixels that have been removed from image formation. In fact, two information frames can be read simultaneously: the image frame and the spectral frame. As shown, the kernel center is zero, therefore the original values ​​at the spectral pixels are ignored.

[0055] By performing a convolution at the center of the kernel at the spectral pixels, skipping every two image pixels results in a “downsampling” from 504x504 pixels to 168x168 pixels (504 / 3 = 168). This “downsampling” can also be considered “binning”; however, this type of “binning” is different from “binning” that adds several pixels together to form a single pixel.

[0056] The example shown is based on the operation of a current CMOS sensor with an array of 504x504 image pixels, which can be binned into 168x168 frames in a 3x3 configuration. These figures allow 28k pixels to be used as spectral pixels.

[0057] The large number of spectral pixels across the entire image area offers another advantage: averaging pixels at various locations within the image eliminates the dependence of the spectral pixel response on the image itself. This allows for the distribution of hundreds of spectral pixels of each type across the entire image area. Any response variations caused by image changes can be effectively removed. This means that the response of each type of spectral pixel is independent of the background response pattern or image. The response can be calibrated against the average value of each type of spectral pixel. This averaged response is less dependent on manufacturing variations, allowing calibration to be performed early in the manufacturing process, possibly at the wafer or lot level, and even enabling single-level calibration across all devices.

[0058] Figure 5 An array 510 of spectral pixels is shown for determining the unique pattern 520 of the response in the smoothed image 530 and the unboxed image 540. Figure 6 An array 610 with spectral pixels 612 is shown. The spectral response from the spectral pixels 612 can be taken into account using the techniques described above in order to provide image data from the remaining pixels in the array 610.

[0059] In another non-limiting embodiment, the second method utilizes highly distorted images produced in camera cube chip (CCC) technology, which is used in fingerprint sensors due to its low cost and small footprint.

[0060] Figure 7 An example of a CCC-type imager is shown. Various spectral pixels 710 are located within array 720. Array 720 provides pixel information 730. Useful image areas are marked within circles 734, and the final image is processed from these areas. This circle 734 corresponds to a circle on the object's finger with a diameter of approximately 6.4 mm. However, even outside this circle 734, within the four quadrants of a larger square, there is still a sensor response to light from the object.

[0061] Most of these outer regions exhibit smooth light patterns, with a response starting at approximately 40% of the central image response and decreasing as the radius increases from the central block 732. There is no image information in this region because the image focus is affected by large coma aberrations, and the local downscaling decreases sharply, causing even large image features to be merged and smoothed. However, while the pixels in this location are not useful for image formation, they can be useful for spectral information.

[0062] In this method, spectral pixels are located at non-imaging locations outside the CMOS array. Since imaging does not require these spectral pixels, they can all be used for spectral response purposes. Due to the large image distortion, these peripheral pixels represent a larger radius of the object (finger), and they can assess the origin of light entering these locations from the desired object position.

[0063] The central image region within circle 734 represents a circle with an approximate diameter of 6.4 mm at the finger's location, while the outer pixels represent a circular region with an inner diameter of 7.8 mm and an outer diameter of 10.0 mm at the finger's object plane. If the finger moves to one side or the other, one or two of the outer quadrants shown may not be covered, but information can still be obtained from the remaining quadrants, providing redundancy.

[0064] Based on a specified region, each quadrant contains numerous spectral pixels for spectral detection. With such a large number of available pixels, the response of each type of spectral pixel can be averaged. This averaging means the response is independent of the image background and image-to-image variations. Furthermore, this allows for calibration to be performed early in the manufacturing process.

[0065] For fingerprint devices with a CCC module, the second method offers an additional advantage: the fingerprint sensor can use an infrared cutoff filter (IRCF) to remove the influence of ambient light on image quality. However, this IRCF removes red light, which is an important spectral range for detecting real human fingers.

[0066] Figure 8 An example using a test pattern image is shown. Array 810 includes various pixels for image information. These pixels are located within circle 820. The diagonal intensity profile 830 shows the pattern smoothing and only radial variations are visible.

[0067] Figure 9 Another example using a test pattern image is shown. In this example, spectral pixel 910 is located in array 920, in region 922, which is outside the region used for image information (circle 924). This allows the array to obtain the spectral characteristics of the finger when the fingerprint is imaged, in order to check whether the finger is fake or real.

[0068] The data in graph 934 includes region 932, which corresponds to the pixels in region 922. As can be seen, the information is regular and does not contribute to image information. This means that pixels can be reused for spectral information with minimal impact on the obtained image information.

[0069] In the first approach, the spectral pixels are integrated within the array of imaging pixels, meaning these spectral pixels cannot detect red light blocked by the IRCF. However, when using the second approach, the spectral pixels are separated from the imaging pixels, which allows the IRCF to be removed from the peripheral region rather than the imaging region. This allows the spectral pixels to also discern the red portion of the spectrum, thus providing stronger detection of fake fingers. For example, an IRCF layer can be added to the sensor, but patterned to cover the area inside a smaller inner circle.

[0070] As described above, the first method has the advantage of being easily implemented using binning / filtering methods already used in these CMOS sensors, while the second method has the advantage of removing IRCF from the location of the spectral pixels.

[0071] The first approach has many variations based on different binning and filtering schemes. To improve the sensor's response to the image, fewer spectral pixels can be used and replaced with imaging pixels (e.g., pixels per 6x6 or 9x9). Moreover, depending on the sensor pixel pitch, 2x2 or 4x4 binning can be used, and filters can be designed to accommodate binnings up to 7x7.

[0072] The methods described above use similar techniques to simultaneously collect both spectral and image information. Filters can be used to remove spectral information from the combined sensor data. The remaining data is used as image information. The image information can be further processed to replace the removed data, for example, by replacing the spectral information with a local average of the image data. This can reduce the image response (e.g., by about 11%), but preserve the image resolution. The pixel response I can then be used. i A unique pattern is used to process spectral information, which is the spectral information of the incident illumination R. j .

[0073] As discussed above, the collected pixel response I i By using an algorithm, using matrix H ij Perform a transformation to find the spectrum R of the incident illumination. j For example, from 430nm to 570nm, there is a resolution of 4nm per step.

[0074]

[0075] There exist m spectral pixels, each with m unique spectral responses, I1, I2, ..., I... m The light reflected from an object (e.g., a finger) and detected by these m spectral pixels comprises n spectral units, for example, from 430 nm to 570 nm, with a resolution of 4 nm per unit, and is composed of the spectrum of the incident illumination R1, R2, ... R... n This is represented as follows. For example, the first spectral pixel will detect the spectrum of the incident illumination R1, R2, ... R n Collectively referred to as I1. The second spectral pixel will detect the spectrum of the incident illumination R1, R2, ... R n Collectively referred to as I2, etc.

[0076] matrix This can be determined through calibration or filter design. The collected pixel response is It is generated based on the values ​​of the detected spectral pixels.

[0077] These can then be used by using R j =∑ i H i,j I i To calculate This is the spectrum of the incident illumination. The calculated spectrum of the incident illumination. It is a single, vector-like composite value independent of image pixels. It is not the color of the image pixels. Image pixels can have no color, for example, black and white.

[0078] The spectra of these incident illuminations, R1, R2, ... R nIt is determined by observation and can be compared with the known spectral content of a real finger, thus determining whether the imaged finger is real or fake. The spectrum of the incident illumination. This can be considered as the color or spectral content of a finger or any object captured by an image sensor.

[0079] Figure 10 This is a logical flowchart illustrating a method according to an embodiment, and the result of executing computer program instructions contained in memory. At block 1010, the method includes capturing an image from a scene including an object (e.g., a finger) using an image sensor comprising a plurality of spectral pixels and a plurality of image pixels. At block 1020, collected data is gathered from a spectral imaging array based on received light. At block 1030, the collected data is separated into spectral pixel data and image pixel data, wherein the spectral pixel data is provided by the spectral pixels and the image pixel data is provided by the image pixels. At block 1040, spectral information of the image is generated based on the spectral pixel data, and at block 1050, image information of the image is generated based on the image pixel data. The generated image information is, for example, a processed image from binning and / or filtering, and the generated spectral information is a single composite value representing the spectral content of the object captured by the image sensor. The generated spectral information may be the spectrum of incident illumination as discussed in this disclosure.

[0080] Figure 11 A block diagram of a sensor suitable for use in various embodiments is shown. Figure 11 In system 1100, sensor 1110 includes a controller (such as a data processor (DP) 1112), a computer-readable medium implemented as a memory (MEM) 1114 storing computer instructions (such as a program (PROG) 1115), and a pixel array 1118. Pixel array 1118 includes a combination of image pixels 1120 and spectral pixels 1125. Sensor 1110 may also include a dedicated processor, such as an image processor 1113.

[0081] Program 1115 may include program instructions that, when executed by DP 1112, enable sensor 1110 to operate according to an embodiment, such as performing... Figure 10 The method shown. That is, various embodiments can be performed at least in part by computer software executable by the DP 1112 of sensor 1110, by hardware, or by a combination of software and hardware.

[0082] MEM 1114 can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as magnetic storage devices, semiconductor-based memory devices, flash memory, optical storage devices, fixed memory, and removable memory. As a non-limiting example, DP 1112 can be of any type suitable for the local technical environment and can include general-purpose computers, special-purpose computers, microprocessors, and multi-core processors.

[0083] DP 1112 is configured to receive collected data from pixel array 1118 and separate the collected data into spectral pixel data and image pixel data. Spectral pixel data is provided by spectral pixel 1125, and image pixel data is provided by image pixel 1120. DP 1112 is also configured to generate spectral information of an image based on the spectral pixel data and image information of an image based on the image pixel data.

[0084] In some embodiments, MEM 1114 may include fingerprint data for an authorized user. Image processor 1113 may receive spectral and image information from DP 1112 for comparison with this fingerprint data stored in MEM 1114. If the received information matches the stored information, image processor 1113 may indicate that the user is authorized, for example, through fingerprint matching.

[0085] An embodiment provides a method for collecting imager data. The method includes capturing an image from a scene using an image sensor. The image sensor has multiple spectral pixels and multiple image pixels. The method also includes collecting collected data from a spectral imaging array based on received light. The collected data is separated into spectral pixel data and image pixel data. The spectral pixel data is provided by the spectral pixels, and the image pixel data is provided by the image pixels. The method includes both generating spectral information of an image based on the spectral pixel data, wherein the generated spectral information represents the spectral content of an object, and generating image information of an image based on the image pixel data.

[0086] In a further embodiment of the above method, each of the plurality of spectral pixels has a corresponding spectral response.

[0087] In another embodiment of any of the methods described above, separating the collected data includes filtering the collected data with a downsampling kernel to remove spectral pixel data. Separating the collected data may also include replacing the spectral pixel data from a given spectral pixel with a weighted average of image pixel data from image pixels surrounding each of the plurality of spectral pixels.

[0088] In a further embodiment of any of the methods described above, the method further includes determining whether a finger is authorized, at least in part, based on spectral and image information. Determining that a finger is authorized may include matching the spectral and image information with the user's fingerprint data. Determining that a finger is authorized may also include determining whether the finger is spoofed or genuine.

[0089] Another embodiment provides an image sensor. The image sensor includes a pixel array having a plurality of spectral pixels and a plurality of image pixels, and a processor. The processor is configured to receive collected data from the spectral imaging array and separate the collected data into spectral pixel data and image pixel data. The spectral pixel data is provided by the spectral pixels, and the image pixel data is provided by the image pixels. The processor is also configured to generate spectral information of an image based on the spectral pixel data, wherein the generated spectral information represents the spectral content of an object, and to generate image information of the image based on the image pixel data.

[0090] In a further embodiment of the image sensor described above, the processor is also configured to output spectral information and image information.

[0091] In another embodiment of any of the image sensors described above, the processor is configured to generate spectral information and image information simultaneously.

[0092] In a further embodiment of any of the image sensors described above, each of the plurality of spectral pixels has an associated spectral response. For each of the various nanostructure designs, at least one of the plurality of spectral pixels may conform to the nanostructure design. Therefore, for each nanostructure design used, there exists at least one associated spectral pixel with an associated spectral response.

[0093] In another embodiment of any of the image sensors described above, the spectral imaging array includes an imaging region and a spectral collection region. The imaging region includes a plurality of image pixels, and the spectral collection region includes a plurality of spectral pixels. The imaging region may define a circle centrally positioned on the spectral imaging array, and the spectral collection region may be located outside this circle.

[0094] The various operations described are purely exemplary and do not imply a particular order. Furthermore, these operations can be used in any order as appropriate and can be used in part. In consideration of the above embodiments, it should be understood that additional embodiments may employ various computer-implemented operations involving the transmission or storage of data in a computer system. These operations are those requiring physical manipulation of physical quantities. Typically, but not necessarily, these quantities are in the form of electrical, magnetic, or optical signals that can be stored, transmitted, combined, compared, and otherwise manipulated.

[0095] Any of the operations described that form part of the embodiments currently disclosed can be useful machine operations. Various embodiments also relate to devices or apparatuses for performing these operations. The apparatus may be specifically constructed for the desired purpose, or it may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. In particular, various general-purpose machines employing one or more processors coupled to one or more computer-readable media as described below can be used with computer programs written in accordance with the teachings herein, or more specialized apparatuses can be more readily constructed to perform the desired operations.

[0096] The procedures, processes, and / or modules described herein can be implemented in hardware or software, as a computer-readable medium having program instructions, firmware, or a combination thereof. For example, the functions described herein can be executed by a processor that executes program instructions in memory or other storage devices.

[0097] The foregoing description has been directed to specific embodiments. However, other variations and modifications can be made to the described embodiments to obtain some or all of their advantages. Modifications can be made to the above systems and methods without departing from the concepts disclosed herein. Therefore, the invention should not be considered as limited to the disclosed embodiments. Furthermore, various features of the described embodiments can be used without correspondingly using other features. Thus, this description should be interpreted as merely illustrative of various principles and not as limiting the invention.

Claims

1. A method for collecting imager data, comprising: An image sensor is used to capture an image of an object. The image sensor has a spectral imaging array having: (a) a plurality of spectral pixels, each of which is covered by a color filter; and (b) a plurality of image pixels, each of which is not covered by a color filter. Data is collected from the spectral imaging array, the collected data indicating the received light; The collected data is separated into spectral pixel data provided by the plurality of spectral pixels and image pixel data provided by the plurality of image pixels, at least in part, by replacing the spectral pixel data with a value calculated by a convolution kernel for each of the plurality of spectral pixels, wherein the convolution kernel (a) is centered on the spectral pixel and (b) has a non-zero value at the image pixel surrounding the spectral pixel; The spectral information of the image is generated based on the spectral pixel data, and the generated spectral information represents the spectral content of the object; as well as Image information of the image is generated based on the image pixel data.

2. The method of claim 1, wherein each of the plurality of spectral pixels has a corresponding spectral response.

3. The method of claim 1, wherein the image includes a finger, and the method further includes determining whether the finger is authorized based at least in part on the spectral information and the image information.

4. The method of claim 3, wherein determining whether the finger is authorized includes matching the spectral information and the image information with the user's fingerprint data.

5. An image sensor, comprising: A pixel array having a plurality of spectral pixels and a plurality of image pixels to provide spectral pixel data and image pixel data, respectively; as well as The processor is configured as follows: Receive collected data from the pixel array; The collected data is separated into spectral pixel data and image pixel data, at least in part, by replacing the spectral pixel data with a value calculated using a convolution kernel for each of the plurality of spectral pixels, wherein the convolution kernel (a) is centered on the spectral pixel and (b) has a non-zero value at the image pixel surrounding the spectral pixel; The spectral information of the image is generated based on the spectral pixel data, and the generated spectral information represents the spectral content of the imaged object; as well as Image information of the image is generated based on the image pixel data.

6. The image sensor of claim 5, wherein the processor is further configured to output the spectral information and the image information.

7. The image sensor of claim 5, wherein the processor is configured to simultaneously generate the spectral information and the image information.

8. The image sensor of claim 5, wherein each of the plurality of spectral pixels has an associated spectral response.

9. The image sensor of claim 8, wherein each of the plurality of spectral pixels conforms to a corresponding nanostructure design having an associated spectral response in a plurality of nanostructure designs.

10. The image sensor of claim 5, wherein the pixel array includes an imaging region and a spectral collection region, the imaging region including the plurality of image pixels, and the spectral collection region including the plurality of spectral pixels.

11. The image sensor of claim 10, wherein the imaging region defines a circle centrally disposed on the pixel array, and the spectral collection region is outside the circle.

12. The image sensor of claim 5, wherein the imaged object includes a finger, and the processor is further configured to determine whether the finger is authorized based at least in part on the spectral information and the image information.

Images