Multichannel high-resolution imaging device incorporating metalens for color imaging

Abstract

The apparatus, in some implementations, comprises at least one image sensor, a plurality of metalenses, and readout and processing circuitry. The at least one image sensor includes a pixel array, each of which is associated with a different one of a plurality of optical channels configured to detect an incident light beam of a respective color. The color for each optical channel is different from the color of at least one of the other optical channels. Each of the metalenses is disposed in a different one of the optical channels and configured to respectively focus the incident light beam onto a different one of the pixel arrays. The readout and processing circuitry is operable to read out signals from the pixel array, generate a respective low-resolution image for each of the optical channels, and process the low-resolution images to obtain a high-resolution multi-color image. Methods of operation are also described.

Description

[Technical field]

[0001] Disclosure Fields The present disclosure relates to multi-channel imaging devices. [Background technology]

[0002] background A multi-channel imaging device can capture images using an image sensor. For example, light incident through an aperture at one end of the imaging device is directed to one or more image sensors that include pixels that generate signals in response to sensing the received light. Imaging devices may be incorporated into handheld electronic devices such as smartphones or other portable electronic devices. However, space within such portable devices is often at a premium. Thus, reducing the size or dimensions of imaging devices can be important for such applications. Summary of the Invention [Means for solving the problem]

[0003] overview The present disclosure describes a multi-channel high resolution imaging device incorporating a metalens.

[0004] In one aspect, for example, the present disclosure describes an apparatus comprising at least one image sensor, a plurality of metalenses, and readout and processing circuitry. The at least one image sensor includes a pixel array, each of which is associated with a different one of a plurality of optical channels configured to detect an incident light beam of a respective color. The color for each optical channel is different from the color of at least one of the other optical channels. Each of the metalenses is disposed in a different one of the optical channels and configured to respectively focus the incident light beam onto a different one of the pixel arrays. The readout and processing circuitry is operable to read out signals from the pixel array, generate a respective low-resolution image for each of the optical channels, and process the low-resolution images to obtain a high-resolution multi-color image.

[0005] Some implementations include one or more of the following features. For example, in some cases, each of the metalens is configured to focus incident light of a different respective wavelength, or within a different respective wavelength range, onto a respective one of the pixel array. In some cases, each particular one of the light channels includes a respective optical filter configured to allow incident light of a respective color associated with the particular channel to pass. In some cases, each of the optical filters is disposed between the image sensor and a different respective one of the metalens, while in some cases, each of the optical filters is disposed on a different respective one of the metalens.

[0006] In some implementations, each of the pixel arrays is operable to capture an image of the scene, and an image captured by a first one of the pixel arrays has a sub-pixel shift relative to an image captured by a second one of the pixel arrays. In some implementations, the at least one image sensor includes multiple image sensors, each of which includes a different respective one of the pixel arrays. In some cases, the at least one image sensor is a single image sensor that includes each of the pixel arrays. In some implementations, the device includes at least three optical channels associated with red, green, and blue wavelengths, respectively.

[0007] In some implementations, the readout and processing circuitry is operable to process the low-resolution images to obtain a high-resolution multi-color image using a super-resolution protocol.

[0008] This disclosure also describes a method that includes acquiring a respective low-resolution image of a scene with each of a plurality of pixel arrays associated with a different respective optical channel of an imaging device, each of the low-resolution images being based on light rays passing through a respective metalens in a respective one of the optical channels, each of the optical channels being respectively configured for a different color of light, the method includes reading out a signal representative of the acquired low-resolution image from the pixel array, and acquiring a high-resolution multi-color image of the scene based on the low-resolution images using a super-resolution protocol.

[0009] Some implementations include one or more of the following features: For example, in some cases, a method includes displaying a high-resolution multi-color image on a display screen of a computing device (e.g., a display screen of a smartphone).

[0010] In some implementations, each respective one of the metalens focuses incident light of a different respective wavelength or within a different respective wavelength range onto a respective one of the pixel arrays. In some cases, each of the metalens includes meta-atoms arranged to resonate at a fixed frequency corresponding to the respective wavelength. In some cases, there is a sub-pixel shift in a low-resolution image acquired by a first one of the pixel arrays relative to a low-resolution image acquired by another one of the pixel arrays.

[0011] Some implementations include one or more of the following advantages. For example, using metalenses can be advantageous because they can be relatively flat, ultra-thin, lightweight, and / or compact. Additionally, in at least some implementations, color images can be acquired without using a series of metalenses arranged in a vertical stack to reduce chromatic aberration. Thus, using the metalenses described in this disclosure can help reduce the total track length (TTL) of an imaging device. Additionally, in some implementations, the metalenses can be used in combination with one or more relatively low-cost, low-resolution image sensors in a manner that allows high-resolution multi-color images to be acquired.

[0012] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the following detailed description, the accompanying drawings, and the claims. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 illustrates a first example of a multi-channel imaging device. [Figure 1A] FIG. 2 is a top view of a multi-channel imaging device. [Diagram 2] FIG. 13 is a diagram illustrating a second example of an imaging device. [Diagram 3] FIG. 13 is a diagram illustrating a third example of an imaging device. [Figure 4] FIG. 13 is a diagram illustrating a fourth example of an imaging device. [Diagram 5] 5 is a flow chart of an exemplary method for operation of the imaging devices of FIGS. 1-4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Detailed Description As shown in the example of Figure 1, the multi-channel imaging device 100 is operable to capture images with respective pixel arrays 102A, 102B associated with different channels and that are part of one or more image sensors. Although only two optical channels 106A, 106B are shown in Figure 1, the device 100 may include a greater number of channels. For example, as shown in Figure 1A, there may be four optical channels 106A, 106B, 106C, 106D, each of which has a respective pixel array associated therewith.

[0015] In the illustrated example, a single image sensor 104 is shown, including a pixel array for each of the optical channels. In some implementations, each pixel array (e.g., 102A, 102B) is part of a different respective smaller image sensor rather than a single larger image sensor. In any case, each image sensor can be implemented, for example, as a relatively low-cost, low-resolution CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide-semiconductor) image sensor. That is, more expensive high-resolution image sensors can be employed, but need not be.

[0016] Each optical channel is configured to detect incident light of a respective particular wavelength or a respective particular range of wavelengths. For each channel, a respective metalens is provided to focus the incident light onto a respective one of the pixel arrays 102A, 102B. That is, a first metalens 108A is disposed over a first portion of the image sensor 104 that includes the first pixel array 102A, and a second metalens 108B is disposed over a second portion of the image sensor 104 that includes the second pixel array 102B. If there are more than two channels, a respective metalens can be disposed over each of the pixel arrays.

[0017] Each metalens 108A, 108B has a metasurface, which refers to a surface that has distributed small structures (e.g., meta-atoms) arranged to interact with light in a particular way. For example, a metasurface, which may also be referred to as a metastructure, may be a surface that has a distributed array of nanostructures. The nanostructures are configured to individually or collectively interact with light waves in a way that changes the local amplitude, the local phase, or both of an incident light wave.

[0018] The meta-atoms (e.g., nanostructures) can be arranged to act as metalenses that resonate at a fixed frequency with a relatively sharp bandwidth. That is, the dimensions (e.g., diameter and length), shape, and material of the meta-atoms can be designed to induce a phase delay in an incident wave of a particular wavelength so as to focus the incident wave to a particular spot. Each of the metalenses 108A, 108B can be configured for a different particular wavelength or narrow band of wavelengths. For example, in some implementations, each of the metalenses 108A, 108B is configured for a different respective wavelength or narrow band of wavelengths in the visible portion of the spectrum and configured to focus incident light onto a respective one of the pixel arrays. Thus, for example, a first metalenses of the metalenses 108A in the first channel 106A can be configured for a first portion of the visible spectrum (e.g., red) and can be configured to focus incident light in the first portion of the visible spectrum onto the first pixel array 102A. A second one of the metalens 108B in the second channel 106B can be configured for a different second portion of the visible spectrum (e.g., green) and can be configured to focus incident light in the second portion of the visible spectrum onto the second pixel array 102B. The other channels (e.g., 106C, 106D) can similarly include metalens configured for respective wavelengths or narrow bands of wavelengths in the visible portion of the spectrum and can be configured to focus incident light in a particular portion of the spectrum onto a respective pixel array associated with the particular channel.

[0019] In some implementations, device 100 includes four light channels 106A-106D designed for red, green, blue, and green light, respectively. The channels may be designed for other combinations of colors of light, and in some cases, at least one of the channels can be configured for non-visible light (e.g., infrared). In any case, each of the metalenses can be configured to focus incident light of a respective particular wavelength, or within a respective particular (e.g., narrow) range of wavelengths centered around a particular wavelength, onto a respective one of the pixel arrays.

[0020] The metalenses 108A, 108B can be supported by, for example, a glass or other substrate 110. Although the example of Figure 1 shows the metalenses 108A, 108B on the top surface of the substrate 110, in some cases the metalenses are disposed on the bottom surface of the substrate 110, as shown in Figure 2. In some cases, the metalenses may be disposed on both sides of the substrate 110.

[0021] As described above, each of the light channels is configured to acquire a substantially monochromatic image of a particular color that is different from the color of the image acquired by at least one other channel. In particular, the first pixel array 102A can capture an image based on light rays passing through the first light channel 106A, and the second pixel array 102B can capture an image based on light rays passing through the second light channel 106B. Similarly, if the device 100 includes additional channels (e.g., 106C and 106D), the third pixel array can capture an image based on light rays passing through the third light channel 106C, and the fourth pixel array can capture an image based on light rays passing through the fourth light channel 106D. Furthermore, each pixel array is operable to acquire an image of a scene from a perspective that is slightly different from the perspective of the other pixel arrays. That is, an image of a scene acquired by a particular one of the pixel arrays is slightly different from an image of the same scene acquired by the other pixel arrays. Small differences in viewpoint result in small shifts (or "motion") in the image of the scene captured by the pixel array. The size of the shift may be, for example, sub-pixel.

[0022] On the other hand, introducing metalenses into imaging devices as described herein is counter to common sense because metalenses are known to generally exhibit relatively large chromatic aberrations, and metalenses typically produce relatively small images, which in some cases make it difficult to use the entire active area of ​​a standard image sensor. Nevertheless, by associating each metalens with an optical channel that encompasses only a portion of all pixels of the image sensor, and configuring each of the optical channels in the imaging device for a single wavelength or a relatively narrow band of wavelengths, imaging device 100 can take advantage of the advantages that metalenses can provide. In particular, using metalenses 108A, 108B rather than other types of lenses (e.g., refractive lenses) in imaging device 100 can be advantageous because metalenses can be relatively flat, ultra-thin, lightweight, and compact. Furthermore, as described below, color images can be obtained without using a series of metalenses arranged in a vertical stack to reduce chromatic aberrations. Thus, using metalenses can help to shorten the total track length (TTL) or z-axis height of imaging device 100. Moreover, as described below, the metalens can be used in combination with one or more relatively low-cost, low-resolution image sensors 104 in a manner that enables relatively high-resolution color images to be obtained.

[0023] In some cases, it may be beneficial to include optical filters in each of the optical channels. The filters can help to eliminate or reduce optical noise that may be present. For example, if channel 106A is designed to detect red radiation, a red filter 120A can be included in the channel, as shown in FIG. 3. Similarly, if channel 106B is designed to detect green radiation, a green filter 120B can be included in the channel. Some channels may include color filters designed for another part of the visible portion of the spectrum (e.g., blue) or for a non-visible portion of the spectrum (e.g., infrared). FIG. 3 shows filters 120A, 120B positioned on image sensor 104, i.e., between image sensor 104 and metalenses 108A, 108B. In some implementations, filters 120A, 120B can be positioned on top of metalenses 108A, 108B, as shown in FIG. 4.

[0024] The imaging device 100 may include control circuitry 111 (e.g., logic) operable to control the image sensor 104 to capture an image of a scene 112 including one or more objects. In some implementations, the control circuitry 111 may be responsive to user input (e.g., a user interacts with or otherwise provides input to a user interface of a smartphone or other computing device coupled to the control circuitry).

[0025] The imaging device 100 may also include a readout and processing circuit 114, which may include, for example, a microprocessor and one or more associated memories that store instructions for execution by the microprocessor. The control circuit 111 may be coupled to the readout and processing circuit 114, for example, to provide timing and control signals for reading out pixel signals. Thus, signals from the pixel arrays 102A, 102B in the various channels 106A, 106B of the imaging device 100 may be read out by the readout and processing circuit 114, which may include, for example, one or more integrated circuits in one or more semiconductor chips having appropriate digital logic and / or other hardware components (e.g., readout registers, amplifiers, analog-to-digital converters, clock drivers, timing logic, and / or signal processing circuits).

[0026] Depending on the implementation, the readout circuitry may include, for example, an active MOS readout amplifier for each pixel. In some implementations, the readout circuitry is operable for in-pixel charge transfer together with an in-pixel amplifier to achieve correlated double sampling (CDS). The readout circuitry may include a source follower or charge amplifier with row and column selection. In some cases, the readout circuitry includes a digital readout integrated circuit (DROIC) or a digital pixel readout integrated circuit (DPROIC). In some cases, the pixel is a demodulated pixel. In some implementations, other pixel readout circuits may be used.

[0027] The readout and processing circuitry 114 is operable to process the pixel signals, for example to generate a small, low-resolution image for each channel (e.g., 113A, 113B). Thus, the readout and processing circuitry 114 is operable to read out signals from each of the pixels in the pixel arrays (e.g., 102A, 102B), such that the signal from a pixel in a particular one of the pixel arrays corresponds to a relatively small, low-resolution image of the scene 112.

[0028] The readout and processing circuit 114 is also operable to process the low-resolution images to obtain a high-resolution monochromatic image 118, for example, using a super-resolution protocol 115. Super-resolution restoration generally refers to the process of combining information from multiple low-resolution images to obtain a high-resolution image. In this case, the super-resolution protocol 115 is operable to generate a high-resolution multi-color image from the low-resolution images (e.g., 113A, 113B). Various super-resolution techniques can be used. For example, U.S. Pat. No. 10,147,167, the disclosure of which is incorporated herein by reference, describes a method for generating a super-resolution restored image. The method is suitable in the context of this disclosure for generating a high-resolution multi-color image from a low-resolution image.

[0029] The super-resolution method may include, for example, extracting a high frequency band luma component from a captured monochromatic low resolution image of a scene, generating a high resolution luma image using the high frequency band luma component and motion data for the captured image, and replacing the luma data of an upsampled color image generated from the captured monochromatic image with the high resolution luma data of the high resolution luma image to obtain a super-resolution reconstructed color image.

[0030] In some cases, extracting the high frequency band luma component includes convolving the raw monochrome image data of the monochrome image of the scene with a Gaussian filter to obtain a Gaussian filtered image, and subtracting the raw monochrome image data from the Gaussian filtered image.

[0031] The super-resolution technique may include constructing an array of raw monochromatic images including the raw monochromatic image data based on the acquired image in some cases. Extracting the high frequency band luma component may further include removing a polarity difference after subtracting the raw monochromatic image data from the Gaussian filtered image. Removing the polarity difference may include calculating an absolute value of the difference previously obtained by subtracting the raw monochromatic image data from the Gaussian filtered image. Extracting the high frequency band luma component may further include equalizing the magnitude of the intensity of the high frequency band luma component and / or applying a sigmoid function after removing the polarity difference.

[0032] The super-resolution technique may include performing a motion estimation protocol based on the high frequency band luma component and on a common channel in the raw monochrome image data to obtain additional motion data, and performing a super-resolution protocol to obtain a high-resolution luma image based on the high frequency luma component in a linear transformation and the additional motion data. In some cases, the extracted high frequency band luma component corresponds to at least one of an edge location or a texture feature. The super-resolution technique may include performing a color image fusion based on the raw monochrome image data of the monochrome image to obtain a low-resolution chroma image, and upsampling the low-resolution chroma image to obtain an upsampled color image.

[0033] In some implementations, other super-resolution techniques can be used. The super-resolution reconstructed color image generated by the readout and processing circuitry 114 can be provided to a display 116, which displays the super-resolution reconstructed color image, for example. The display 116 can include, for example, a screen of a computing device (e.g., a smartphone, tablet, personal computer, or other small computing device).

[0034] Imaging device 100 may be used in any of a wide range of applications including, for example, cameras in smartphones and other handheld or portable computing devices, as well as medical imaging, satellite imaging, surveillance, facial recognition, high definition television, etc. In some cases, at least a portion of the readout and processing circuitry 114 for imaging device 100 may be integrated into the smartphone or other computing device's own processing circuitry. In other examples, readout and processing circuitry 114 may be separate from such circuitry within the computing device.

[0035] FIG. 5 illustrates an example of a method of using the imaging device 100 of FIG. 1, 2, 3, or 4. As indicated by 200, two or more pixel arrays associated with different respective optical channels of the imaging device each acquire a respective low-resolution image of a scene including one or more objects. Each low-resolution image is based (at least in part) on light rays passing through a respective metalens in a respective one of the optical channels. The low-resolution images to the optical channels are substantially monochromatic, with each channel acquiring an image based on light of a respective color (e.g., wavelength or narrow range of wavelengths) that is different from at least one of the other channels. In some cases, the low-resolution images are acquired in response to user input (e.g., input provided by a user via an interactive user interface). As indicated by 202, a signal representing the acquired low-resolution image is read out from the pixel array. A super-resolution protocol is then used to acquire a high-resolution color image of the scene based on the low-resolution image, as indicated by 204. In some cases, the high-resolution color image is displayed, for example, on a display screen of a smartphone or other computing device, as indicated by 206.

[0036] Various aspects of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, including the structures disclosed herein and their structural equivalents, or in computer software, firmware, or hardware, or in one or more combinations thereof. Thus, aspects of the subject matter described herein can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or one or more combinations thereof. In addition to hardware, the apparatus can include code that creates an execution environment for the computer program in question, e.g., code that constitutes the processor firmware.

[0037] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program, which can be stored as instructions in one or more memories, can be deployed to run on one computer or multiple interconnected computers.

[0038] The processes and logic flows described herein may be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, or an apparatus may be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or application specific integrated circuit (ASIC).

[0039] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special-purpose logic circuitry.

[0040] Various modifications will become readily apparent from the foregoing detailed description and drawings. Accordingly, other implementations are within the scope of the claims.

Claims

1. At least one image sensor including a plurality of pixel arrays, wherein each of the pixel arrays is respectively associated with a different one of a plurality of optical channels configured to detect incident light rays of respective colors, and the color for each optical channel is different from at least one color of the other optical channels; an image sensor; A plurality of metalenses, each of which is respectively disposed on a different one of the plurality of optical channels and configured to focus incident light rays on a different one of the pixel arrays; Readout and processing circuitry operable to read signals from the plurality of pixel arrays, generate respective low-resolution images for each of the optical channels, and process the low-resolution images to obtain a high-resolution multi-color image. An apparatus comprising the above.

2. The apparatus according to claim 1, wherein each of the plurality of metalenses is configured to focus incident light rays of different respective wavelengths or within different respective wavelength ranges on a respective one of the pixel arrays.

3. The apparatus according to claim 1, wherein each specific one of the plurality of optical channels includes respective optical filters configured to allow incident light rays of respective colors associated with the specific channel to pass through.

4. The apparatus according to claim 3, wherein each of the optical filters is disposed between a different one of the image sensor and the metalenses.

5. The apparatus according to claim 3, wherein each of the optical filters is disposed on a different one of the metalenses.

6. Each of the pixel arrays is operable to acquire an image of a scene, and a sub-pixel shift exists in the image acquired by a first pixel array among the pixel arrays with respect to the image acquired by a second pixel array among the pixel arrays. The apparatus according to any one of claims 1 to 5.

7. The apparatus according to any one of claims 1 to 5, wherein the at least one image sensor includes a plurality of image sensors, each of which includes a different one of the pixel arrays.

8. The apparatus according to any one of claims 1 to 5, wherein the at least one image sensor is a single image sensor including each of the pixel arrays. Claim 9 The apparatus according to any one of claims 1 to 5, comprising at least three optical channels respectively associated with the wavelengths of red light, green light, and blue light. Claim 10 The apparatus according to any one of claims 1 to 5, wherein the readout and processing circuit is operable to process the low-resolution image using a super-resolution protocol to obtain a high-resolution multi-color image. Claim 11 Obtaining, by each of a plurality of pixel arrays respectively associated with different optical channels of an imaging device, a respective low-resolution image of a scene, each of the low-resolution images being based on light rays passing through respective meta-lenses in each of the optical channels, each of the optical channels being configured for a different color of light; Reading out a signal representing the obtained low-resolution image from the pixel array; Obtaining a high-resolution multi-color image of the scene based on the low-resolution image using a super-resolution protocol; A method comprising. Claim 12 The method according to claim 11, comprising displaying the high-resolution multi-color image on a display screen of a computing device. Claim 13 The method according to claim 11, comprising displaying the high-resolution multi-color image on a display screen of a smartphone. Claim 14 The method according to claim 11, wherein each one of the plurality of meta-lenses focuses incident light rays of different respective wavelengths, or within different respective wavelength ranges, onto a respective one of the pixel arrays. Claim 15 The method according to claim 14, wherein each of the meta-lenses comprises meta-atoms arranged to resonate at a fixed frequency corresponding to the respective wavelength. Claim 16 In the low-resolution image obtained by a first pixel array among the pixel arrays, there is a sub-pixel shift with respect to the low-resolution image obtained by another pixel array among the pixel arrays. The method according to any one of claims 11 to 15.

Images