Metalens 3D Polarized Camera

Abstract

A camera for reconstructing the surface of an object is provided. The camera includes an optical module configured to spatially separate photons reflected from an object in a scene to form at least two focused and distinctly polarized images on a sensor. The sensor is configured to receive the at least two polarized images on pixels of the sensor and generate pixel intensity values. The camera further includes a computation module including a processor and a memory having instructions stored thereon. The processor, in accordance with the instructions, calculates depth values ​​at points on the surface of the object based on ratios of the pixel intensity values ​​for the at least two polarized images and reconstructs the surface of the object from the calculated depth values.

Description

[Technical Field]

[0001] The present disclosure relates generally to metalens 3D cameras, and more particularly to metalens 3D polarization cameras. [Background technology]

[0002] Metalenses are patterned surfaces with nanoscale structures that interact with electromagnetic fields and typically perform some optical functions similar to lenses. Miyata et al. demonstrated how to design metalenses that sort incident light by polarization and focus multiple images with different polarizations within a scene. Zhang and Hancock showed that four different polarization conditions are required to estimate the scene surface normal from a polarized image. Ngo et al. refined this to three illumination conditions × two polarization conditions. It is widely known that 3D surface shapes can be obtained by integrating surface normals, and the results can be extremely sensitive to measurement noise. MetaLenz Inc. has claimed (but has not publicly demonstrated or described) a metalens-based camera system that can do this with four distinctly polarized images. We demonstrate how to directly recover 3D shapes from only two polarized images generated in a single exposure with a polarization-sorting metalens camera and further show that the metalens design can be optimized to better support this task. Therefore, there is a need to develop novel metalens 3D cameras. Summary of the Invention

[0003] The present disclosure provides a metalens 3D camera that reconstructs the three-dimensional (3D) structure of a scene from a single exposure while utilizing all photons that enter the camera's aperture, where more photons per pixel means more information to constrain scene depth estimates.

[0004] Some embodiments of the present invention may provide a method for designing polarization sorting metalens that simultaneously focus multiple distinctly polarized images onto a single image sensor without losing photons to filter, and show how to tune this to minimize spillover that can corrupt depth estimates. We show that 3D scene depth can be inferred from two distinctly polarized images, meaning that twice as much information (pixel depth) can be reconstructed from the same sensor as prior art, or twice as many photons collected per pixel. Importantly, our method directly reconstructs 3D depth without intermediate calculation of surface normals, thereby enabling regularization to make the problem well-posed and avoiding a substantial source of numerical error and noise sensitivity. Our formulation provides robustness against unknown changes in illumination intensity and surface albedo.

[0005] Some embodiments of the present invention are based on the recognition that design and optimization methods are provided for polarization sorting metalenses that focus two or more distinctly polarized scene images onto a sensor. In this case, the scene surface normal and precursors to 3D reconstruction can be calculated from three distinctly polarized images. Some embodiments provide a regularized formulation that yields 3D depth information directly from only two distinctly polarized images produced by the metalens. This allows for the collection of more photons to support accurate estimation.

[0006] Furthermore, some embodiments may provide a method / system for reconstructing the 3D structure of a scene from a single exposure while utilizing all photons that enter the aperture; more photons means more information. Polarization sorting metalens allows multiple distinctly polarized images to be simultaneously focused onto an image sensor without losing photons to filter. We demonstrate how to optimize the creation of two polarization images. State-of-the-art techniques for polarization from 3D use four or more polarization images, which is redundant and therefore wastes sensor pixels. We show that 3D scene depth can be inferred from only two distinctly polarized images, meaning we can reconstruct twice the information (pixel depth) from the same sensor. Importantly, our method directly reconstructs 3D depth without intermediate calculation of surface normals, thereby enabling regularization to make the problem well-posed and avoiding a substantial source of numerical error and noise sensitivity.

[0007] Some embodiments of the present invention provide a metalens 3D polarization camera. The camera includes an optical module configured to spatially separate photons reflected from an object in a scene to form at least two focused and distinctly polarized images on a sensor. The sensor is configured to receive the at least two polarization images on pixels of the sensor and generate pixel intensity values. The camera further includes a computation module including a processor and a memory having instructions stored thereon that cause the processor to calculate depth values ​​at points on a surface of the object based on ratios of the pixel intensity values ​​for the at least two polarization images and to reconstruct the surface of the object from the calculated depth values.

[0008] Furthermore, in accordance with some embodiments of the present invention, a computer-implemented method for reconstructing three-dimensional (3D) depth from the surface of an object in a scene is provided. The method uses a computational module including a processor and a memory having stored thereon instructions for a 3D depth reconstruction algorithm. The instructions include spatially separating photons reflected from an object in the scene to form at least two focused and distinctly polarized images on a sensor. The sensor receives the at least two polarization images on pixels of the sensor. The instructions further include generating pixel intensity values ​​using the sensor, calculating depth values ​​at points on the surface of the object based on ratios of the pixel intensity values ​​for the at least two polarization images, and reconstructing the surface of the object from the calculated depth values.

[0009] Embodiments of the present disclosure will now be further described with reference to the accompanying drawings, in which the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of embodiments of the present disclosure. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an exemplary configuration of a metalens camera in accordance with an embodiment of the present invention. [Figure 2A] FIG. 10 shows a small patch of a metalens in accordance with an embodiment of the present invention. [Figure 2B] FIG. 2B is a top view of the metalens patch of FIG. 2A. [Figure 2C] FIG. 10 shows a larger patch of metalens focusing a wavefront onto a particular pixel in a sensor, in accordance with an embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram illustrating a depth reconstruction algorithm according to an embodiment of the present invention. [Figure 4] FIG. 10 illustrates an exemplary configuration of a computational module for a 3D metalens camera, in accordance with an embodiment of the present invention. [Figure 5]FIG. 10 illustrates the reconstruction of a double-curved surface from a single exposure with a dual-polarizing metalens camera, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] While the above-identified drawing figures illustrate embodiments of the present disclosure, other embodiments are contemplated as set forth in the following description. The present disclosure presents specific embodiments for purposes of illustration and not limitation. Those skilled in the art can devise numerous other modifications and embodiments which fall within the scope and spirit of the principles of the embodiments of the present disclosure.

[0012] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes are contemplated that may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0013] 1 shows an exemplary configuration of a metalens three-dimensional (3D) camera 100 in accordance with one embodiment of the present invention. Metalens camera 100 includes a metalens 120, a sensor 130 having a pixel array 130, and a computer 140 including a processor and memory.

[0014]

number

[0015] Figure 2A shows a small patch of metalens 120 that includes four nanostructures 210, each located on a substrate within a planar grid cell 220 that is smaller than one wavelength of light. Figure 2B shows a top view of this metalens patch 120, illustrating that the nanostructures 210 can be characterized by length, width, and planar orientation angle. Figure 2C shows a larger patch of metalens 120 that focuses a wavefront 230 onto a specific pixel in a sensor 130 according to the direction and polarization of the wavefront. (Design and optimization of polarization sorting metalens)

[0016] Unit cell decomposition (UCD) decomposes metasurfaces into a grid of subwavelength-sized atoms, each of which can be selected independently of its neighbors to impart a localized phase delay in the near field. Observing that the phase delay of silicon nitride nanopillars depends on the polarization direction of the incident wavefront, we can create a table of phase delays for horizontally and vertically polarized light provided by various pillar geometries and use this to piecewise design metalenses that focus 0°, 45°, 90°, and 135° polarized images to spatially separated focal points on the focal plane. We employ a similar approach to design 0° and 90° polarization concentrators, but replace the table with a differentiable function approximator (e.g., bicubic regression) and differentiate using a Rayleigh propagator, allowing us to adjust metasurface design parameters to optimize measured performance at the focal plane. For example, by maximizing the intensity of the polarized light focused to a small region around the correct focal point, the focusing efficiency can be improved by 3% to 5%, as determined from rigorous coupled-wave analysis (RCWA).

[0017] The improvements we can make with respect to the "ideal" phase profile require some explanation. First, we directly optimize the far-field performance rather than piecewise designing the near-field. Second, we can select the closest nanopillar design from a discrete library of simulations to yield the desired phase delay. We have the advantage of a continuous function that fully covers the range of delays and geometries. Third, there are metalens boundary effects that are ignored in piecewise designs but are captured in our differentiated Rayleigh propagator.

[0018] 3 is a block diagram showing a computer-implemented method for performing a depth reconstruction algorithm 300, in which the surface shape is inferred 310 and its polarization-specific reflectance is predicted 320, and then the ratio of these reflectances 330 is compared 340 with the corresponding ratio of measured pixel intensities. If the difference is small enough 350, the depth estimate is output 360; otherwise, the difference is adjusted 370 to reduce the difference, and the process is repeated.

[0019] 4 shows an exemplary configuration of a computing module 400 of a 3D metalens camera 100 according to an embodiment of the present invention. The computing module 400 may include a human machine interface (HMI) 410 connectable to a keyboard 411 and a pointing device / medium 412, one or more processors 420, a storage device 430, memory 440, a network interface controller (NIC) 450 connectable to a network 490, including a local area network, a wireless network, and an internet network, and a sensor interface 460 connected to an optical module / sensor 465. Hereinafter, the one or more processors 420 may be referred to as processor 420 for convenience.

[0020] Memory 440 may be one or more memory units operating in conjunction with storage 430 that stores a computer-executable program (algorithm code) for executing depth reconstruction algorithm 404 in conjunction with processor 420. NIC 450 includes a receiver and a transmitter for connecting to network 490 via wired and wireless networks (not shown). When computation module 400 receives pixel intensity data as input data, computation module 400 executes depth reconstruction algorithm 404 of 3D metalens camera 100 stored in storage 430 by using processor 420 and memory 440. Storage 430 may include image formation algorithm 405 and depth reconstruction data obtained after execution of depth reconstruction algorithm 404. Image formation algorithm 405 can generate a 3D image from a scene captured by optical module / sensor 465 of 3D metalens camera 100. In some cases, image formation algorithm 405 generates a 3D point cloud image using the 3D depth reconstruction data. The memory 440 and the storage device 430 may be referred to as memory for convenience. (depth from polarization)

[0021] When light interacts with a surface, the vertically polarized component becomes more likely to be absorbed or refracted, and the reflected light becomes more horizontally polarized. Some of the vertically polarized component enters the surface, scatters subsurface, and is re-emitted with a Lambertian distribution. The intensity of the reflected light measured by the observer depends on the illumination, illumination direction, surface orientation, surface refractive index, observer direction, and polarizing filter. As explained below, most of these unknowns can be canceled out by taking the ratio of the intensities measured with two different polarizing filters. The remaining unknowns can then be solved by calculating these ratios at several nearby scene points and solving for a smooth surface that matches these derived quantities.

[0022]

number

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027] At this point, we have three unknowns and one observation: a single metalens camera exposure. By calculating the amount of information in advance, we know that four polarization images are needed, or two polarization images under each of three lighting conditions. Without going too far, we assume that the observed surface is generally smooth, and therefore that the majority of imaged pixels do not exhibit depth discontinuities such as occluded edges, and that the refractive index changes very slowly, if at all. This is a good assumption for solid objects, although it is poor for highly porous volumes such as steel wool.

[0028]

number

[0029]

number

[0030] The remaining unknowns for this problem are the refractive indices at the points, which can also be estimated if a reasonable initial guess is provided and deviations from that guess or deviations from a smooth variation are similarly regularized. (metal)

[0031]

number

[0032] FIG. 5 shows an example of the reconstruction of a double-curved surface from a single exposure with a dual-polarization metalens camera, in accordance with an embodiment of the present invention.

[0033] This example assumes a diffuse, unpolarized light source, such as a cloudy sky, and a copper double-curved surface with a refractive index ratio (air to copper) of η = 1.25 + 2.39i at 450 nm. The double-curved surface is positioned 100 μm in front of the metalens, and reflected light is propagated from 9 × 9 = 81 distinct points on the surface through the metasurface to a simulated CCD sensor placed at the focal plane. Intensity measurements are taken at the corresponding 2 × 81 focal points and at sensor wells contaminated with 2% independent and identically distributed (iid) Gaussian noise. The objective function (2) is then minimized by a generalized optimizer, resulting in a good reconstruction of the scene shape. The optimization result remains constant when the regularizer weight λ is varied over five orders of magnitude, indicating that the smoothness constraint makes the problem well-posed without distorting the optimum. We observe that the reconstructed surface is slightly less curved than the true surface, indicating that the sample points are likely closer together and therefore the finite difference between the coordinates of adjacent points provides a better approximation to the local surface gradient.

[0034] In the above description, specific details are provided to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, like reference numbers and names in the various drawings indicate like elements.

[0035] Also, particular embodiments may be described as a process that is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may terminate upon completion of its operations, but may include additional steps not described or included in the diagram. Moreover, not all operations in any specifically described process may be performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to a return of the function to the calling function or the main function.

[0036] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, either manually or automatically. The manual or automatic implementations may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium.

[0037] The above-described embodiments of the present disclosure can be implemented in any of numerous ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. The use of ordinal terms such as "first," "second," etc. to modify claim elements in the claims does not, by itself, imply a priority, precedence, or order of one claim element over another, nor does it imply a chronological order in which method actions are performed; it is merely used as a label to distinguish one claim element having a particular name from another element having the same name (but using ordinal terms) to distinguish between claim elements.

[0038] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications can be made within the spirit and scope of the present disclosure. It is therefore intended by the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the present disclosure.

Claims

1. A camera, an optical module configured to spatially separate photons reflected from objects in a scene to form at least two focused and distinctly polarized images on a sensor, the sensor configured to receive the at least two polarized images on pixels of the sensor and generate intensity values ​​for the pixels, the camera further comprising: a computing module including a processor and a memory having instructions stored thereon, the instructions causing the processor to: calculating a depth value at a point on the surface of the object based on a ratio of the intensity values ​​of the pixel for the at least two polarization images; A camera that reconstructs the surface of the object from the calculated depth values.

2. The camera of claim 1 , wherein the light originates from a diffuse, unpolarized light source of unknown intensity.

3. 3. The camera of claim 2, wherein two or more polarized images are obtained for a single exposure of diffuse unpolarized light.

4. 10. The camera of claim 1, wherein the nanostructures of the optical module are tuned to focus distinct polarization components of incident light from each scene point onto distinct sensor pixels while minimizing spillover to neighboring pixels.

5. The camera of claim 1 , wherein the at least two polarized images have substantially distinct polarization angles.

6. The camera of claim 1 , wherein the at least two polarized images include a horizontally polarized image and a vertically polarized image.

7. 2. The camera of claim 1, wherein the depth value at the point on the surface is calculated by finding depth values ​​for all neighboring surface points in the scene such that the surface orientation indicated for each surface point has polarized reflectances with ratios that match the ratios of polarized illumination intensities measured at sensor pixels representing the scene point.

8. 10. The camera of claim 1, wherein adjacent pixels receive images polarized differently from one another, and the pixels are located in a focal plane.

9. The camera of claim 1 , wherein the optical module is a metalens.

10. The camera of claim 1 , wherein the camera operates in ambient light.

11. 1. A computer-implemented method for reconstructing three-dimensional (3D) depth from surfaces of objects in a scene using a computational module including a processor and a memory having stored thereon instructions for a 3D depth reconstruction algorithm, the method comprising: spatially separating photons reflected from the object in the scene to form at least two focused and distinctly polarized images on a sensor, the sensor receiving the at least two polarized images on pixels of the sensor, the computer-implemented method further comprising: generating intensity values ​​for the pixels using the sensor; calculating a depth value at a point on the surface of the object based on a ratio of the intensity values ​​of the pixel for the at least two polarization images; and reconstructing the surface of the object from the calculated depth values.

12. The computer-implemented method of claim 11 , wherein the light originates from a diffuse, unpolarized light source of unknown intensity.

13. The computer-implemented method of claim 12 , wherein two or more polarization images are obtained for a single exposure of diffuse unpolarized light.

14. 12. The computer-implemented method of claim 11, wherein the nanostructures of the optical module are tuned to focus distinct polarization components of incident light from each scene point onto distinct sensor pixels while minimizing spillover to neighboring pixels.

15. The computer-implemented method of claim 11 , wherein the at least two polarized images have substantially distinct polarization angles.

16. The computer-implemented method of claim 11 , wherein the at least two polarized images include a horizontally polarized image and a vertically polarized image.

17. 12. The computer-implemented method of claim 11, wherein the depth value at the point on the surface is calculated by finding depth values ​​for all neighboring surface points in the scene such that the surface orientation indicated for each surface point has polarized reflectances with ratios that match the ratios of polarized illumination intensities measured at sensor pixels representing the scene point.

18. The computer-implemented method of claim 11 , wherein adjacent pixels receive images polarized differently from one another, and the pixels are located in a focal plane.

19. The computer-implemented method of claim 11 , wherein the optical module is a metalens.

20. The computer-implemented method of claim 11 , wherein the camera operates in ambient light.

Images