System and method for high quality imaging using a color-splitting meta-optical computational camera

Abstract

In one embodiment, the system includes a first camera including a first filter having a spectral response, a first meta-optical lens, and a first sensor. The system includes a second camera including a second filter having a spectral response different from the spectral response of the first filter, a second meta-optical lens, and a second sensor. The system also includes a processor configured to receive a first image representation of the scene captured by the first camera, receive a second image representation of the scene captured by the second camera, and generate a superimposed image representation of the scene based on the first image representation and the second image representation, where the superimposed image representation has less aberration than the aberration of the first image representation and the aberration of the second image representation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. patent application Ser. No. 63 / 309,911, entitled "SYSTEMS AND METHODS FOR HIGH QUALITY IMAGING USING A COLOR-SPLITTING META-OPTICAL COMPUTATION CAMERA," filed February 14, 2022, which is incorporated by reference in its entirety into this specification.

[0002] Field In one or more embodiments, a meta-optic computational camera can perform color segmentation to generate at least one high quality image. [Background technology]

[0003] background

[0003] When trying to produce smaller cameras (e.g., for ultra-thin phones or laptops), the performance of such cameras can be sacrificed. Meta-optics can be one potential solution to miniaturize optics because the lenses are flat (rather than curved). However, meta-optics cannot always optically correct all aberrations (e.g., in highly constrained systems). Thus, there is a need for improved smaller cameras. Summary of the Invention [Means for solving the problem]

[0004] overview In one or more embodiments, a system includes a first camera including a first filter having a spectral response, a first meta-optical lens, and a first sensor. The system also includes a second camera including a second filter having a spectral response different from the spectral response of the first filter, a second meta-optical lens, and a second sensor. The system also includes a processor configured to receive a first image representation of a scene captured by the first camera, receive a second image representation of the scene captured by the second camera, and generate a superimposed image representation of the scene based on the first image representation and the second image representation, where the superimposed image representation has less aberration than the aberration of the first image representation and the aberration of the second image representation.

[0005] In one or more embodiments, a system includes a plurality of cameras configured to generate a representation of a plurality of images of a scene, each camera from the plurality of cameras including a filter having a different spectral response than a spectral response of a filter of a respective remaining camera, a meta-optic lens, and a sensor. The system also includes a processor, the processor configured to receive the representation of the plurality of images of the scene and generate a superimposed image representation of the scene based on the representation of the plurality of images.

[0006] In one or more embodiments, a method includes receiving a representation of a plurality of images of a scene captured by a plurality of cameras, each camera from the plurality of cameras including a filter, a meta-optical lens, and a sensor. For each camera from the plurality of cameras, a spectral response of the filter of that camera differs from a spectral response of the filter of each remaining camera from the plurality of cameras. Based on the representation of the plurality of images, a superimposed corrected image representation of the scene is generated. [Brief description of the drawings]

[0007] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1]

[0007] According to one embodiment, a system is shown for generating an aberration-free image based on a representation of multiple aberrated images each captured using an individual meta-optical lens. [Diagram 2]

[0008] 1 illustrates a camera system having a baffle and multiple meta-optic lenses, according to one embodiment. [Diagram 3]

[0009] 1 shows a flowchart of a method for generating an image based on a representation of multiple images respectively captured using a meta-optical lens, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Detailed Description

[0010] The techniques discussed herein relate to using multiple meta-optical lenses and circuitry and / or software to reduce the size of a camera system (reducing the track length of the optics) while also maintaining and / or enhancing imaging performance. Similarly, the techniques discussed herein use meta-optical systems that can reduce the size of a camera system, but do not rely on the meta-optical system to directly optically correct all aberrations. Rather, the meta-optical system can work in conjunction with circuitry and / or software to produce high quality (e.g., aberration-free) images.

[0009]

[0011] Multiple meta-optics can be used, with each meta-optic adapted for a subrange (e.g., color) of the full spectral range. Each meta-optic lens can, for example, provide a high-quality blur kernel that is nearly invariant across the field of view in each color channel (i.e., spectral subrange). Thus, the constraints imposed on each meta-optic can be mitigated by using each meta-optic lens dedicated to a distinct narrow-band spectral window.

[0010]

[0012] FIG. 1 illustrates a diagram of a camera system 100 that can generate an image 190 (e.g., a formed image or a representation of an image) based on a representation of images 150A, 150B, 150C captured by cameras 110A, 110B, 110C, according to one embodiment. In some implementations, the thickness and / or track length of the camera system 100 can be reduced by placing the cameras 110A, 110B, 110C adjacent to one another (e.g., side-by-side as shown in FIG. 1). In some implementations, the total track length of the camera system 100 is, for example, less than 1 millimeter. In some implementations, the cameras 110A, 110B, 110C can be slightly angled relative to one another (e.g., in a triangular formation). In some implementations, the camera system 100 can be and / or can be included in a camera, webcam, laptop, tablet, phone, television, smart glasses, smartwatch, vehicle, or any other image capture device (not shown). Each camera 110A, 110B, 110C may include a single meta-optical lens 130A, 130B, 130C.

[0011]

[0013] Filters 120A, 120B, 120C, meta-optic lenses 130A, 130B, 130C, and sensors 140A, 140B, 140C may be included in housing 102. In some implementations, circuitry 160 is included in housing 102. In some implementations, circuitry 160 is not included in housing 102, in which case electronic representations of images 150A, 150B, 150C generated by sensors 140A, 140B, 140C in housing 102 may be transmitted to circuitry 160 via, for example, traces connecting sensor 104A to processor 170 and / or memory 180, traces connecting sensor 140B to processor 170 and / or memory 180, and traces connecting sensor 140C to processor 170 and / or memory 180. Additionally, incident light may pass through holes in housing 102 and contact filters 120A, 120B, 120C.

[0012]

[0014] In some implementations, the housing 102 has a length that is approximately (e.g., within 10%) less than at least one of 6 mm, 5 mm, 4.5 mm, 4 mm, 3.5 mm, or 3 mm. In some implementations, the housing 102 has a width that is approximately (e.g., within 10%) less than at least one of 2 mm, 1.5 mm, 1.4 mm, 1.3 mm, 1.2 mm, 1.1 mm, 1 mm, 0.9 mm, 0.8 mm, 0.7 mm, 0.6 mm, or 0.5 mm. In some implementations, the housing 102 has a height that is approximately (e.g., within 10%) less than at least one of 1.5 mm, 1.1 mm, 1 mm, 0.9 mm, 0.8 mm, 0.7 mm, 0.6 mm, or 0.5 mm. In some implementations, the dimensions (length x width x height) of the housing 102 are approximately (eg, within 10%) 1 mm x 5 mm x 1.4 mm.

[0013]

[0015] Within housing 102, filters 120A, 120B, 120C may each have a substantially similar size relative to one another, may have substantially different sizes relative to one another (e.g., based on different predetermined sampling rates), or a combination thereof (e.g., similar size to one filter but different size to another filter). Within housing 102, meta-optical lenses 130A, 130B, 130C may each have a substantially similar size relative to one another, may have substantially different sizes relative to one another (e.g., based on different predetermined sampling rates), or a combination thereof (e.g., similar size to one filter but different size to another filter). Within the housing 102, the sensors 140A, 140B, 140C may have substantially similar overall sizes, pixel counts and / or pixel sizes relative to one another, may have substantially different overall sizes, pixel counts and / or pixel sizes relative to one another (e.g., based on different predetermined sampling rates), or a combination thereof (e.g., a similar overall size, pixel count and / or pixel size as one filter but a different overall size, pixel count and / or pixel size as another filter).

[0014]

[0016] It should be noted that FIG. 1 shows images 150A, 150B, 150C as formed images. However, it can be understood that images 150A, 150B, 150C are electronic representations (e.g., in a form not visible to humans) provided by sensors 140A, 140B, 140C, respectively, to circuit 160 and do not need to be converted to formed images and / or may be converted to formed images (e.g., in a form visible to humans) at circuit 160. Images 150A, 150B, 150C are shown to provide an appreciation of what representations captured by sensors 140A, 140B, 140C may look like in some implementations. Additionally, image 190 may be generated by circuit 160 as a representation of image 190 and / or a formed version of image 190.

[0015]

[0017] First camera 110A includes filter 120A, meta-optical lens 130A, and sensor 140A (also referred to as a "detector"). Light (e.g., white light) from a scene (e.g., a person and its background) may pass through filter 120A, pass through meta-optical lens 130A, and be incident on sensor 140A to generate a representation of image 150A (e.g., an electronic signal representative of image 150A).

[0016]

[0018] In some implementations, filter 120A can be a color filter. Filter 120A can be any color, such as red, blue, yellow, green, cyan, magenta, or a combination thereof. In some implementations, filter 120A can be one of red, green, or blue. In some implementations, filter 120A can be one of red, green, blue, or yellow. The use of color filters allows meta-optical lens 130A to address a narrower wavelength range than the entire visible spectrum, which allows meta-optical lens 130A to more easily provide a field of view (FoV) invariant blur kernel.

[0017]

[0019] In some implementations, meta-optical lens 130A is substantially flat, transparent at visible and infrared wavelengths (i.e., meta-optical lens 130A has a substantially uniform spectral response at visible and infrared wavelengths), and / or is compatible with a single stage lithography process. Meta-optical lens 130A may include, for example, a patterned dielectric layer having a range of different widths, and in some implementations the dielectric may be gallium nitride, silicon nitride, and / or titanium dioxide. The shape of meta-optical lens 130A is neither substantially convex nor substantially concave. The term "substantially" is used to account for possible variations (e.g., manufacturing variations, environmental variations, etc.).

[0018]

[0020] In some implementations, sensor 140A can convert a light image into an electronic signal that represents an image (e.g., image 150A). In some implementations, sensor 140A is an image sensor (e.g., a charge-coupled device, an active pixel sensor, etc.). Light can contact sensor 140A after passing through filter 120A and meta-optical lens 130A. Sensor 140A then generates a representation of image 150A. In some implementations, sensor 140A is not a Bayer filter sensor.

[0019]

[0021] In some implementations, if filter 120A is a color filter, the representation of image 150A (e.g., when converted to a formed image) has the same hue as filter 120A. For example, if filter 120A is red, image 150A also has a red hue. As another example, if filter 120A is green, image 150A also has a green hue. As another example, if filter 120A is blue, image 150A also has a blue hue. Furthermore, the representation of image 150A (e.g., when converted to a formed image) may include at least one aberration. In some implementations, the representation of image 150A (e.g., when converted to a formed image) is blurry (i.e., the scene captured by first camera 110A is not clear or clearly visible in image 150A).

[0020]

[0022] The second camera 110B and the third camera 110C may each be similar and / or identical to the camera 110A, with at least one difference being that the filters 120A, 120B, and 120C of the first camera 110A, the second camera 110B, and the third camera 110C are different from each other. For example, each filter 120A, 120B, and 120C may have a spectral subrange that is mutually exclusive with the spectral subranges of the other filters 120A, 120B, and 120C. In another example, each filter 120A, 120B, and 120C may have a spectral subrange that is different from, but possibly overlaps with, the spectral subranges of the other filters 120A, 120B, and 120C. In some implementations, each of the filters 120A, 120B, and 120C is associated with a different color. For example, filter 120A may be a red color filter, filter 120B may be a green color filter, and filter 120C may be a blue color filter.

[0021]

[0023] In other cases, such as in the case of the first camera 110A, light from the same scene may pass through filters 120B and 120C, pass through meta-optic lenses 130B and 130C, and be incident on sensors 140B and 140C to produce representations of images 150B and 150C, respectively. As in the case of image 150A, where filters 120B, 120C are color filters, each representation of images 150B, 150C (e.g., when converted to a formed image) may have the same hue as the associated filter 120B, 120C. Each of the representations of images 150B, 150C (e.g., when converted to a formed image) may include and / or be associated with at least one aberration, where, for example, the at least one aberration in each representation of images 150A, 150B, and 150C may be of the same type (e.g., blurry, noisy, etc.) but may differ in degree (e.g., due to spectral response differences). In some implementations, the representation of images 150B, 150C (e.g., when converted to a formed image) is blurry (i.e., the scene captured by the second camera 110B and / or the third camera 110C is not clear or clearly visible in the images 150B, 150C).

[0022]

[0024] The representations of images 150A, 150B, 150C may be transmitted to and received by circuitry 160, which includes a processor 170 and a memory 180 operatively coupled to processor 170 (e.g., via a system bus). Optionally, circuitry 160 may be used to convert the representations of images 150A, 150B, 150C into a formed image. Circuitry 160 may be used to generate (e.g., as a formed image and / or an electronic representation) image 190, which has less aberrations and / or is substantially free of aberrations (e.g., no blur or less blur) than images 150A, 150B, 150C.

[0023]

[0025] Processor 170 may be, for example, a hardware-based integrated circuit (IC) or any other suitable processing device configured to run and / or execute an instruction set or code. For example, processor 170 may be a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array (PLA), a complex programmable logic device (CPLD), and / or a programmable logic controller (PLC), etc. In some implementations, processor 170 may be configured to perform any of the methods and / or portions of the methods discussed herein.

[0024]

[0026] The memory 180 may be, for example, a random access memory (RAM), a memory buffer, a hard drive, a read-only memory (ROM) and / or an erasable programmable read-only memory (EPROM), etc. The memory 180 may be configured to store any data used by the processor 170 to execute the techniques discussed herein. In some cases, the memory 180 may store one or more software programs and / or codes, which may include, for example, instructions that cause the processor 170 to execute one or more processes and / or functions, etc. In some implementations, the memory 180 may include an expandable storage unit that can be incrementally added and used. In some implementations, the memory 180 may be a portable memory (e.g., a flash drive and / or a portable hard disk, etc.) that may be operably coupled to the processor 170. In some cases, the memory 180 may be operably coupled remotely to a computing device (not shown in FIG. 1 ).

[0025]

[0027] Although not shown in FIG. 1, memory 180 may store a software model. Processor 170 may use the software model to convert the representations of images 150A, 150B, 150C generated by sensors 140A, 140B, 140C into formed images 150A, 150B, 150C. In some implementations, the software model performs parallax correction, deconvolution, distortion correction, and / or upsampling on each of images 150A, 150B, 150C (or a transformation generated based on images 150A, 150B, 150C) such that images 150A, 150B, 150C become less aberrated and / or no longer aberrated (e.g., become less blurry or not). In some implementations, the representations of images 150A, 150B, 150C are first deconvolved separately to generate a representation of a deconvolved image. Further details regarding deconvolving representations of images (such as images 150A, 150B, 150C) to generate representations of deconvolved images can be found in “Neural nano-optics for high-quality thin lens imaging” (Tseng, E., Colburn, S., Whitehead, J. et al. Neural nano-optics for high-quality thin lens imaging. Nat Commun 12, 6493 (2021)), the contents of which are incorporated herein by reference in their entirety. Each of these representations of the deconvolved image is then aligned (i.e., parallax corrected) with a single camera in camera system 100 and with multiple (e.g., all) cameras in camera system 100.Further details regarding disparity correction for a single camera can be found in “Disparity Estimation for Image Fusion in a Multi-aperture Camera” (Mustaniemi J., Kannala J., Heikkilae J. (2015) Disparity Estimation for Image Fusion in a Multi-aperture Camera. In: Azzopardi G., Petkov N. (eds) Computer Analysis of Images and Patterns. CAIP 2015. Lecture Notes in Computer Science, vol 9257. Springer, Cham.), the contents of which are incorporated herein by reference in their entirety. Further details regarding parallax correction for multiple cameras can be found in “A unified framework for multi-sensor HDR video reconstruction” (Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Anders Ynnerman, Jonas Unger, A unified framework for multi-sensor HDR video reconstruction, Signal Processing: Image Communication, Volume 29, Issue 2, 2014, Pages 203-215, ISSN 0923-5965, the contents of which are incorporated herein by reference in their entirety.

[0026]

[0028] The software model can then superimpose each representation of the unaberrated version of images 150A, 150B, 150C on top of one another to form image 190. In some implementations, image 190 is an RGB (i.e., red, green, blue) image. In some implementations, after performing the parallax correction and deconvolution discussed above, a demosaicing process can be performed by the software model to generate image 190. Further details regarding the demosaicing that can be performed to generate image 190 can be found in “HIGH-QUALITY LINEAR INTERPOLATION FOR DEMOSAICING OF BAYER-PATTERNED COLOR IMAGES” (H.S. Malvar, Li-wei He and R. Cutler, “High-quality linear interpolation for demosaicing of Bayer-patterned color images,” 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2004, pp. iii-485), the contents of which are incorporated herein by reference in their entirety.

[0027]

[0029] In some implementations, the representation of the image 190 is stored in memory 180. In some implementations, the representation of the image 190 is sent to a computing device (not shown in FIG. 1 ) communicatively coupled to the camera system 100 (e.g., via a wired or wireless connection or network). In some implementations, the representation of the image 190 is converted into a formed image and displayed via a display (not shown in FIG. 1 ). The image 190 may be displayed, for example, to a user associated with the camera system 100 (e.g., using or having access to the camera system 100).

[0028]

[0030] While the camera system 100 discussed above included three cameras 110A, 110B, 110C, in some implementations, two cameras may be used. In other implementations, four or more cameras may be used. Each camera may include a meta-optical lens, a sensor, and a unique color filter. Each camera may also be operatively coupled to circuitry (e.g., circuit 160) configured to generate a representation of an aberrated image (e.g., images 150A, 150B, 150C), a representation of an image (e.g., image 190) substantially free of aberrations from the aberrated images, and / or an image (e.g., image 190) from the aberrated images that has fewer aberrations than the aberrated images.

[0029]

[0031] In some implementations where two cameras are used, the first camera may be associated with a first spectral subrange (e.g., red, orange, and yellow) and the second camera may be associated with a second spectral subrange (e.g., green, blue, indigo, and purple). In some implementations, the camera system 100 is a combined visible / near infrared (VIS / NIR) camera having two channels (i.e., a VIS channel and a NIR channel), in such a situation, the first channel (e.g., camera) may include a VIS filter and the second channel (e.g., camera) may include a NIR filter. In some implementations where four cameras are used, the first camera may include a red color filter, the second camera may include a green color filter, the third camera may include a blue color filter, and the fourth camera may include a yellow camera filter.

[0030]

[0032] In some implementations, the two cameras may be associated with substantially the same (e.g., at least 95% similar, at least 99% similar, etc.) spectral range. For example, the filter 120A of the camera 110A may have substantially the same spectral range as the filter of a camera not shown in FIG. 1 (e.g., both filters are green, both filters are red, both filters are blue and / or both filters are yellow, etc.). Two cameras that have the same spectral response but are slightly offset from each other (e.g., by 1-2 millimeters) (e.g., due to the natural side-by-side positioning of the cameras on the device) may be used to correct for parallax error, since differences between two substantially similar images are not due to spectral differences, but due to the physical offset of the two cameras. Additionally, the two images may be used to improve the signal-to-noise ratio (SNR) when generating the image 190. For example, two nearly identical images (e.g. after disparity correction) can be made to allow noise to be reduced by averaging the two "noisy" images, resulting in less noise in the average (e.g. a noisy signal value at one pixel in one image will be reduced by a less noisy signal value at that pixel in the other image - the noisy and less noisy values ​​will be closer to the true value).

[0031]

[0033] 1 included a single meta-optic lens 130A, 130B, and 130C, respectively, in other implementations, at least one of cameras 110A, 110B, or 110C may include multiple meta-optic lenses. For example, a camera (e.g., camera 110A) may include a first meta-optic lens (e.g., meta-optic lens 130A) and a second meta-optic lens (not shown in FIG. 1), where the second meta-optic lens may be positioned in front of or behind a filter (e.g., filter 120A) and / or in front of or behind the first meta-optic lens.

[0032]

[0034] Although not shown in FIG. 1, each camera 110A, 110B, and 110C may include at least one aperture (e.g., to block stray light and / or substantially off-angle light), where the at least one aperture may be located in front of and / or behind that camera's meta-optic lens to manage collection angle and / or chief ray angle, etc. For example, in some implementations, each camera 110A, 110B, and 110C may include two apertures, one aperture located in front of that camera's meta-optic lens and the other aperture located behind the meta-optic lens. In such implementations, the aperture located in front of a given camera's meta-optic lens may be smaller (i.e., smaller opening) than the aperture located behind that camera's meta-optic lens. The apertures of cameras 110A, 110B, 110C may have the same size, different sizes, or a combination thereof. In some implementations, the apertures of the cameras 110A, 110B, 110C may be arranged in a single common plane, in some implementations, the apertures of the cameras 110A, 110B, 110C may be slightly oblique to one another (e.g., in a triangular formation in implementations where the sensors are in a triangular formation).

[0033]

[0035] 1, each 110A, 110B, and 110C may include at least one refractive element (i.e., a refractive lens) to correct and / or reduce aberrations, in which case the at least one refractive element may be disposed in front of and / or behind that camera's meta-optical lens and / or that camera's filter. For example, incident light may pass through an aperture of a camera (e.g., camera 110A), then pass through a refractive element, then pass through a filter (e.g., filter 120A), then pass through a meta-optical lens (e.g., meta-optical lens 130A), and then enter a sensor (e.g., sensor 140A) to generate a representation of image 150A.

[0034]

[0036] Although not shown in FIG. 1, in some implementations, an infrared (IR) filter may be used. In some implementations, at least one of cameras 110A, 110B, or 110C may include an IR filter. The IR filter may be located in front of the sensor of that camera (e.g., directly above the sensor, directly above a meta-optical lens, directly above a color filter, etc.). In some implementations, a fourth camera (not shown in FIG. 1) may be included in camera system 100. The fourth camera may have a structure that is nearly identical to cameras 110A, 110B, and 110C, but may use an IR filter instead of a color filter. In other cases, light from the same scene as in cameras 110A, 110B, and 110C may pass through an IR filter, pass through a meta-optical lens included in the fourth camera, and be incident on a sensor included in the fourth camera to generate a representation of the image. The representation of the image produced by the fourth camera may be used by circuit 160 in addition to the representations of images 150A, 150B, 150C to produce a representation of image 190.

[0035]

[0037] FIG. 1 shows three sensors 140A, 140B, 140C, but in some implementations, a single sensor (not shown in FIG. 1) may be used. Similarly, in some implementations, camera system 100 may include multiple meta-optic lenses and multiple filters, but may include only a single sensor. In other words, in some implementations, a single sensor may receive light associated with (e.g., passed through) multiple different meta-optic lenses and filters. Light passed through each meta-optic lens from multiple meta-optic lenses may be incident on different sections of the sensor (there may or may not be some overlap between some sections). For example, light passed through meta-optic lens 130A may be incident on a left portion of a single sensor, light passed through meta-optic lens 130B may be incident on a center portion of a single sensor, and light passed through meta-optic lens 130C may be incident on a right portion of a single sensor, with a portion of the center portion overlapping with the left portion and a different portion of the center portion overlapping with the right portion in some implementations.

[0036]

[0038] 1 are in front of (e.g., in contact with) meta-optic lenses 130A, 130B, and 130C, respectively, the locations of filters 120A, 120B, 120C may vary in other implementations. For example, a filter (e.g., filter 120A or an IR filter) may be located in front of (e.g., directly above) a sensor (e.g., sensor 140A). As another example, if an aperture is located between a meta-optic lens (e.g., meta-optic lens 130A) and a sensor (e.g., sensor 140A), a filter (e.g., filter 120A or an IR filter) may be located (1) between the meta-optic lens and the aperture (e.g., directly above the aperture), or (2) between the aperture and the sensor (e.g., directly above the sensor).

[0037]

[0039] 1, in some implementations, anti-reflective (AR) and / or infrared (IR) coatings may be used in camera system 100. In some implementations, the AR and / or IR coatings may be disposed and / or applied to a meta-optical lens (e.g., the entire lens, one side of the lens, etc.). In some implementations, the AR and / or IR coatings may be applied and / or disposed to a filter (e.g., the entire filter, one side of the filter, etc.).

[0038]

[0040] A first object being located "in front of" a second object may refer to the first object being closer to the incident light entering the camera 110A, 110B, 110C than the second object. A first object being located "behind" a second object may refer to the second object being closer to the incident light entering the camera 110A, 110B, 110C than the first object. The first object and / or the second object may be, for example, a meta-optical lens, a color filter, an IR filter, a refractive element, a filter, or a sensor.

[0039]

[0041] In one embodiment, the camera system includes a first camera (e.g., camera 110A) that includes a first filter (e.g., filter 120A) having a spectral response (e.g., one of red, green, or blue), a first meta-optical lens (e.g., meta-optical lens 130A), and a first sensor (e.g., sensor 140A). The camera system also includes a second camera (e.g., camera 110B) that includes a second filter (e.g., filter 120B) having a different spectral response (e.g., one of red, green, or blue) than the spectral response of the first filter, a second meta-optical lens (e.g., meta-optical lens 130B), and a second sensor (e.g., sensor 140B). The camera system also includes a processor (e.g., processor 170) configured to receive a representation of a first image (e.g., image 150A) of the scene captured by the first camera, receive a representation of a second image (e.g., image 150B) of the scene captured by the second camera, and generate a representation of a superimposed image (e.g., image 190) of the scene based on the first image representation and the second image representation, where the superimposed image representation has less aberration than the aberration of the first image representation and the aberration of the second image representation. In some implementations, the superimposed image representation is converted into a formed image and displayed via a display.

[0040]

[0042] In some implementations, the camera system further includes a third camera (e.g., camera 110C) including a third filter having a third spectral response (e.g., one of red, green, or blue) different from the spectral response of the first filter and the spectral response of the second filter, a third meta-optical lens (e.g., meta-optical lens 130C), and a third sensor (e.g., sensor 140C), in such implementations, the processor may be further configured to receive a representation of a third image (e.g., image 150C) of the scene captured by the third camera, in which case generating the representation of the superimposed image of the scene may be further based on the representation of the third image.

[0041]

[0043] In some implementations, the camera system further includes a third camera (e.g., camera 110C) including a third filter having a third spectral response (e.g., one of red, green, or blue) that is substantially similar to the spectral response of the first filter but not substantially similar to the spectral response of the second filter, a third meta-optical lens (e.g., meta-optical lens 130C), and a third sensor (e.g., sensor 140C), in such implementations, the processor may be further configured to receive a representation of a third image (e.g., image 150C) of the scene captured by the third camera, where the representation of the first image and the representation of the third image are used for disparity correction and noise reduction during generation of the representation of the superimposed image.

[0042]

[0044] In some implementations, the spectral response of the first filter includes red, the spectral response of the second filter includes green, and the spectral response of the third filter includes blue. In some implementations, the first camera, the second camera, and the third camera are disposed in a common plane (e.g., side-by-side), and the second camera is disposed between the first camera and the second camera.

[0043]

[0045] In some implementations, an anti-reflective coating is disposed on only the first meta-optic lens, only the second meta-optic lens, or both the first and second meta-optic lenses, hi some implementations, an infrared coating is disposed on only the first meta-optic lens, only the second meta-optic lens, or both the first and second meta-optic lenses.

[0044]

[0046] In some implementations, the camera system further includes at least one aperture stop disposed between (1) the first meta-optic lens and the first filter, and (2) the second meta-optic lens and the second filter. In some implementations, the camera system further includes at least one aperture stop disposed between (1) the first meta-optic lens and the first sensor, and (2) the second meta-optic lens and the second sensor.

[0045]

[0047] In some implementations, the camera system further includes a fourth camera (not shown in FIG. 1 ) including a fourth filter (e.g., a yellow color filter) having a fourth spectral response different from the spectral responses of the first filter, the second filter, and the third filter, a fourth meta-optical lens, and a fourth sensor, in which case the processor may be further configured to receive a fourth image representation of the scene captured by the fourth camera, in which case generating the superimposed image representation of the scene may be further based on the fourth image representation.

[0046]

[0048] In one embodiment, the camera system includes a plurality (e.g., two, three, four, five, etc.) of cameras (e.g., at least two of cameras 110A, 110B, or 110C) configured to generate a representation of a plurality of images (e.g., at least two of images 150A, 150B, or 150C) of a scene, where each camera from the plurality of cameras includes a filter (e.g., filter 120A, 120B, or 120C) having a spectral response different from the spectral response of the filter of the respective remaining camera, a meta-optical lens (e.g., meta-optical lens 130A, 130B, or 130C), and a sensor (e.g., sensor 140A, 140B, or 140C). The camera system further includes a processor (e.g., processor 170) configured to receive the representation of the plurality of images of the scene and generate a representation of a superimposed image (e.g., image 190) of the scene based on the representation of the plurality of images.

[0047]

[0049] In some implementations, generating a representation of the superimposed image may include performing parallax correction, deconvolution, distortion correction and / or upsampling on a representation of the multiple images to generate a representation of the multiple superimposed images, and superimposing the representations of the multiple superimposed images to form the representation of the superimposed image.

[0048]

[0050] In some implementations, the camera system includes only two cameras. In such cases, the multiple images may include at least a first image and a second image (and in some implementations, there are no additional images). In some implementations, the camera system includes only three cameras. In such cases, the multiple images may include at least a first image, a second image, and a third image (and in some implementations, there are no additional images). In some implementations, the camera system includes only four cameras. In such embodiments, the multiple images may include at least a first image, a second image, a third image, and a fourth image (and in some implementations, there are no additional images).

[0049]

[0051] In some implementations, camera system 100 includes a baffle. For example, the baffle may be positioned above meta-optic lenses 130A, 130B, 130C such that light passes through the baffle before passing through meta-optic lenses 130A, 130B, 130C. In some implementations, the baffle is wider at the end farther from the meta-optic lens and narrower closer to the meta-optic lens to eliminate high angle light rays. In some implementations, the baffle is attached to and also acts as an aperture stop. In some implementations, the baffle protrudes vertically along the length of camera system 100. The baffles can eliminate and / or reduce crosstalk by allowing a given camera 110A, 110B, 110C to receive light while blocking light to the other cameras 110A, 110B, 110C (which light can be considered high angle light from the origin of the other cameras 110A, 110B, 110C).

[0050]

[0052] 2 illustrates a camera system with a baffle and multiple meta-optic lenses, according to one embodiment. Filter 206 includes four filters - red filter 206A, green filter 206B, green filter 206C, and blue filter 206D. Meta-optic lens 204 includes four meta-optic lenses - meta-optic lens 204A, meta-optic lens 204B, meta-optic lens 204C, and meta-optic lens 204D. Baffle 202 includes four openings - opening 202A, opening 202B, opening 202C, and opening 202D. Light can pass through opening 202A, through meta-optic lens 204A, and through filter 206A to reach the image sensor. Light can also pass through opening 202B, through meta-optic lens 204B, and through filter 206B to reach the image sensor. Light can pass through aperture 202C, through meta-optic lens 204C, and through filter 206C to reach the image sensor. Light can pass through aperture 202D, through meta-optic lens 204D, and through filter 206D to reach the image sensor.

[0051]

[0053] 2 shows a camera system with a baffle 202 and no separate aperture stop, the baffle 202 may be configured to act as an aperture stop. For example, the perimeter of the top of each of the openings 202A, 202B, 202C, 202D may act as an aperture stop depending on the size of that perimeter relative to the other dimensions of the camera system. Alternatively, the perimeter of the bottom of each of the openings 202A, 202B, 202C, 202D may act as an aperture stop depending on the size of that perimeter relative to the other dimensions of the camera system. Additionally or alternatively, separate aperture stops may be included.

[0052]

[0054] 3 illustrates a flowchart of a method for generating a representation of an image based on a representation of multiple images, each captured using a meta-optical lens and a unique color filter, according to one embodiment. In some implementations, the method discussed with respect to FIG. 3 may be performed by / in processor 170.

[0053]

[0055] At 302, a representation of multiple (e.g., 2, 3, 4, 5) images (e.g., images 150A, 150B, 150C) of a scene captured by a plurality of cameras (e.g., cameras 110A, 110B, 110C), each camera from the plurality of cameras including a filter (e.g., filter 120A, 120B or 120C), a meta-optical lens (e.g., meta-optical lens 130A, 130B or 130C) and a sensor (e.g., sensor 140A, 140B or 140C). For each camera from the plurality of cameras, a spectral response of the camera's filter differs from the spectral response of the filters of each remaining camera from the plurality of cameras. At 304, a representation of a superimposed corrected image (e.g., image 190) of the scene is generated based on the representation of the plurality of images. In some implementations, the representations of the multiple images are converted into a representation of multiple superimposed images (e.g., via parallax correction, deconvolution, distortion correction, and / or upsampling software) and the representations of the multiple superimposed images are superimposed to generate a representation of the superimposed corrected images at 304. In some implementations, the representation of the superimposed corrected images is converted into a formed image at 304 and displayed on a display in response to generating the representation of the superimposed corrected images (e.g., automatically and without further human input).

[0054]

[0056] In some implementations, the multiple cameras include only two cameras. In some implementations, the multiple cameras include only three cameras. In some implementations, the multiple cameras include only four cameras. In some implementations, the multiple cameras include five or more cameras.

[0055]

[0057] In some implementations, when the multiple cameras include four cameras, the spectral response of the filter of a first camera includes red, the spectral response of the filter of a second camera includes green, the spectral response of the filter of a third camera includes blue, and the spectral response of the filter of a fourth camera includes a color different from red, green, and blue (e.g., yellow, cyan, magenta, etc.).

[0056]

[0058] In some implementations, when the multiple cameras include four cameras, the spectral response of the filter of the first camera includes red, the spectral response of the filter of the second camera includes green, the spectral response of the filter of the third camera includes blue, and the filter of the fourth camera is an infrared filter.

[0057]

[0059] In some implementations, the plurality of cameras includes any number of cameras (e.g., at least five cameras, at least ten cameras, at least twenty cameras, etc.). In some implementations, each camera from the plurality of cameras can be associated with a different spectral response. In some implementations, two or more cameras from the plurality of cameras can have the same spectral response. In some implementations, two or more cameras from the plurality of cameras have the same spectral response, and a remaining camera from the plurality of cameras is associated with a different spectral response than every other camera from the remaining cameras, e.g., a first camera is associated with a spectral response that includes green, a second camera is associated with a spectral response that includes green, a third camera is associated with a spectral response that includes red, a fourth camera is associated with a spectral response that includes blue, and a fifth camera is associated with a spectral response that includes yellow.

[0058]

[0060] All combinations of the above concepts and additional concepts discussed herein (where such concepts are not mutually inconsistent) are contemplated as being part of the subject matter disclosed herein. Terms explicitly used herein, which may also appear in any disclosures incorporated by reference, should be accorded the meaning that is most consistent with the particular concepts disclosed herein.

[0059]

[0061] Those skilled in the art will appreciate that the drawings are primarily for illustrative purposes and are not intended to limit the scope of the subject matter described herein. The drawings are not necessarily to scale, and various aspects of the subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference numbers generally refer to like features (e.g., functionally similar and / or structurally similar elements).

[0060]

[0062] In order to address various problems and advance the art, the entire application (including cover page, title, headings, background, summary, brief description of drawings, detailed description, embodiments, abstract, figures, appendices, etc.) shows, by way of example, various embodiments in which the embodiments may be implemented. The advantages and features of the present application are merely representative examples of embodiments and are not exhaustive and / or exclusive. Rather, they are presented to facilitate understanding and teach the embodiments and are not representative of all embodiments. Thus, certain aspects of the present disclosure have not been discussed herein. The fact that alternative embodiments for certain parts of the innovation may not be presented or that further alternative embodiments not described may be available for a part should not be considered as excluding such alternative embodiments from the scope of the present disclosure. It will be understood that many of those undescribed embodiments incorporate the same principles of innovation and that other embodiments are equivalent. Thus, it should be understood that other embodiments may be available and functional, logical, operational, organizational, structural and / or topological modifications may be made without departing from the scope and / or spirit of the present disclosure. Accordingly, all examples and / or embodiments are considered non-limiting throughout this disclosure.

[0061]

[0063] Nor should any inference be drawn with respect to embodiments described herein other than as such to reduce space and repetition with respect to embodiments not described herein. For example, the logical and / or topological structure of any program components (collections of components), other components and / or any combination of any present feature set as illustrated and / or described throughout is not limited to a fixed order of operations and / or placement, but rather, any disclosed order is exemplary and all equivalents, regardless of order, are contemplated by this disclosure.

[0062]

[0064] Various concepts may be implemented as one or more methods, at least one example of which is provided. The acts performed as part of a method may be ordered in any suitable manner. Thus, although illustrated in an exemplary embodiment as sequential operations, embodiments may be constructed in which acts are performed in a different order than illustrated, which may include performing some acts simultaneously. In other words, it should be understood that such features are not necessarily limited to a particular order of execution, but rather any number of threads, processes, services and / or servers, etc., that may be executed sequentially, asynchronously, simultaneously, in parallel, simultaneously and / or synchronously, etc., as consistent with this disclosure. Thus, some of these features may be mutually incompatible in that they cannot exist simultaneously in a single embodiment. Similarly, some features may be applicable to some aspects of the innovation and not to others.

[0063]

[0065] Additionally, the present disclosure may include other innovations not currently described. The applicants reserve all rights in such innovations, including the right to implement such innovations and to file additional, continuation, continuation-in-part and / or divisional applications, etc. As such, it should be understood that the advantages, embodiments, examples, functions, features, logical, operational, organizational, structural, topological and / or other aspects of the present disclosure should not be considered limitations to the present disclosure as defined by the embodiments or limitations to equivalents of the embodiments. Depending on the particular needs and / or characteristics of individual and / or business users, database configurations and / or relational models, data types, data transmission and / or network frameworks and / or syntax structures, etc., various embodiments of the technology disclosed herein may be implemented to allow for great flexibility and customization, as described herein.

[0064]

[0066] All definitions and those used herein are to be understood to take precedence over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0065]

[0067] As used herein, in certain embodiments, the term "about" or "approximately" when preceding a numerical value indicates a range of the value plus or minus 10%. When a range of values ​​is provided, it is understood that each intervening value between the upper and lower limits of that range, to the tenth of the unit of the lower limit, and any other stated or intervening value in that stated range, is encompassed within the disclosure, unless the context clearly dictates otherwise. The upper and lower limits of these smaller ranges may be independently included in the smaller ranges, subject to any specifically excluded limit of the stated range, and are encompassed within the disclosure. Where a stated range includes one or both of the limits, ranges excluding one or both of those included limits are also included in the disclosure.

[0066]

[0068] As used in this specification and the embodiments, unless clearly indicated to the contrary, the indefinite articles "a" and "an" are to be understood to mean "at least one."

[0067]

[0069] As used herein in the specification and embodiments, the term "and / or" should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are sometimes conjunctive and other times disjunctive. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so conjoined. Other elements, other than the elements specifically identified by the "and / or" clause, may optionally be present, whether related or not to those elements specifically identified. Thus, as a non-limiting example, a reference to "A and / or B," when used in conjunction with open-ended language such as "comprising," may refer in one embodiment to only A (optionally including elements other than B), in another embodiment to only B (optionally including elements other than A), and in another embodiment to both A and B (optionally including other elements), etc.

[0068]

[0070] As used herein in the specification and embodiments, "or" shall be understood to have the same meaning as "and / or" defined above. For example, when separating items in a list, "or" or "and / or" shall be interpreted as inclusive, i.e., including at least one of a number or list of elements, but including two or more and optionally additional unlisted items. Only terms clearly indicated to the contrary, such as "only one of" or "exactly one of" or, when used in the embodiments, "consisting of", will refer to the inclusion of exactly one element of a number or list of elements. In general, as used herein, the term "or" shall be interpreted as indicating exclusive alternatives (i.e., "one or the other, but not both") only when preceded by a term of exclusivity, such as "either," "one of," "only one of," or "exactly one of." "Consisting essentially of," when used in the embodiments, shall have its ordinary meaning as used in the field of patent law.

[0069]

[0071] As used herein in the specification and embodiments, the phrase "at least one" in reference to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but does not necessarily include at least one of every element specifically listed in the list of elements, and does not exclude any combination of elements in the list of elements. This definition also allows that there may optionally be elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether related to the specifically identified elements or not. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B" or, equivalently, "at least one of A and / or B") can refer in one embodiment to at least one, optionally more than one, A, and no B (and optionally including elements other than B); in another embodiment to at least one, optionally more than one, B, and no A (and optionally including elements other than A); in yet another embodiment to at least one, optionally more than one, A, and at least one, optionally more than one, B (and optionally including other elements);

[0070]

[0072] In the embodiments and above specification, all transitional phrases such as "comprise," "include," "carry," "have," "contain," "involve," "hold," "comprise," and the like are understood to be open-ended, i.e., meaning "including, but not limited to." Only the transitional phrases "consisting of" and "consisting essentially of" must be closed or semi-closed transitional phrases, respectively, as set forth in Section 2111.03 of the United States Patent Office Manual of Patent Examining Procedures.

[0071]

[0073] Some embodiments and / or methods may be performed by software (executed on hardware), hardware, or a combination thereof. Hardware modules may include, for example, a processor, a field programmable gate array (FPGA), and / or an application specific integrated circuit (ASIC). Software modules (executed on hardware) may include instructions stored in a memory operatively coupled to a processor, and may be expressed in a variety of software languages ​​(e.g., computer code), including C, C++, Java™, Ruby, Visual Basic™, and / or other object-oriented, procedural, or other programming languages ​​and development tools. Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions such as those produced by a compiler, code used to generate web services, and files containing higher level instructions executed by a computer using an interpreter. For example, embodiments may be implemented using imperative programming languages ​​(e.g., C, Fortran, etc.), functional programming languages ​​(Haskell, Erlang, etc.), logic programming languages ​​(e.g., Prolog), object-oriented programming languages ​​(e.g., Java, C++, etc.), or other suitable programming languages ​​and / or development tools. Additional examples of computer code include, but are not limited to, control signals, encryption code, and compression code.

[0072]

[0074] The term "processor" should be interpreted broadly to encompass a general purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some contexts, a "processor" may refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term "processor" may refer to a combination of processing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0073]

[0075] The term "memory" should be interpreted broadly to encompass any electronic component capable of storing electronic information. The term memory may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. Memory is said to be in electronic communication with a processor if the processor can read information from the memory and / or write information to the memory. Memory that is integral to a processor is in electronic communication with the processor.

[0074]

[0076] The terms "instructions" and "code" should be interpreted broadly to include any type of computer-readable statements. For example, the terms "instructions" and "code" may refer to one or more programs, routines, subroutines, functions, procedures, etc. "Instructions" and "code" may include a single computer-readable statement or many computer-readable statements.

[0075]

[0077] While specific embodiments of the present disclosure have been outlined above, many alternatives, modifications, and variations will be apparent to those skilled in the art. Thus, the embodiments, as described herein, are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the present disclosure.

Claims

1. A meta-optical element configured to generate multiple spectrally distinct optical responses corresponding to multiple spectral channels, wherein the multiple spectral channels are defined by an imaging system using the meta-optical element, At least one image sensor, wherein the plurality of spectral channels are incident on the at least one image sensor to generate image data, A processor configured to computationally generate a color image or a multispectral image based on the aforementioned image data. An imaging system including, An imaging system in which the meta-optical element and the computational generation are configured to work together to reduce the optical aberrations of the imaging system.

2. The imaging system according to claim 1, wherein the plurality of spectral channels include three or more spectral channels.

3. The imaging system according to claim 1, wherein the plurality of spectral channels extend over at least one of the visible wavelength range or the visible to infrared wavelength range.

4. The imaging system according to claim 1, wherein the plurality of spectral channels correspond to mutually exclusive spectral subranges of a broadband spectrum.

5. The imaging system according to claim 1, wherein the computational generation includes deconvolution, parallax correction, and demosaicing.

6. The imaging system according to claim 1, wherein the meta-optical element is configured to generate a high-quality blur kernel that enables at least one of chromatic aberration reduction or an extended depth of field through computational generation.

7. The imaging system according to claim 1, further comprising at least one aperture configured to block stray light and / or substantially off-angle light, wherein the at least one aperture is positioned in front of and / or behind the meta-optical element.

8. The imaging system according to claim 1, further comprising one or more refractive optical elements arranged to operate in combination with the meta-optical elements to define the plurality of spectral channels.

9. The imaging system according to claim 1, further comprising one or more spectral filters arranged in combination with the meta-optical elements to define the plurality of spectral channels.

10. The imaging system according to claim 1, wherein the meta-optical element includes a subwavelength structure formed on a substantially flat surface.

11. The imaging system according to claim 10, wherein the subwavelength structure comprises one or more dielectric materials.

12. The imaging system according to claim 1, wherein the plurality of spectral channels are captured by a spatially separated region of a single image sensor or by a plurality of image sensors.

13. A meta-optical element configured to generate multiple spectrally distinct optical responses corresponding to multiple spectral channels from incident broadband light, At least one image sensor configured to generate signals corresponding to the plurality of spectral channels, A controller operably coupled to at least one of the image sensors and An imaging system including, The controller is configured to computationally generate an image based on the signal using collaborative optical computing processing in the imaging system.

14. The imaging system according to claim 13, further comprising a second meta-optical element, wherein the relative positioning of the meta-optical element and the second meta-optical element enables modification of one or more imaging characteristics.

15. Optically generating multiple spectrally distinct images using meta-optical elements, Capturing the plurality of spectrally distinct images on at least one image sensor, The process involves capturing the plurality of spectrally distinct images on at least one image sensor to generate image data, Computationally generating a color image or a multispectral image based on the aforementioned image data. A method of capturing images, including, A method wherein the optical and computational generation of the plurality of spectrally distinct images are configured to work together to reduce optical aberrations.

Images