Superlens sensor with local deconvolution

Abstract

Systems and techniques for image correction are provided. For example, a process may include obtaining image data of an image from a sensor coupled to a super lens; obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent a light pattern generated by the superlens at a position along an axis of the image; dividing the image into a sub-image set; determining a local PSF of a sub-image in the set of sub-images based on the plurality of PSFs; and performing deconvolution on the sub-image based on the local PSF of the sub-image.

Description

Technical Field

[0001] The present disclosure generally relates to optical systems utilizing metalenses. In some examples, aspects of the present disclosure relate to systems and techniques related to metalenses assemblies. Background Art

[0002] Many devices and systems include optical elements, such as lenses, for focusing light onto an image sensor. For example, a camera or a device including a camera having such an optical element can capture a frame or sequence of frames of a scene (e.g., a video of the scene). To achieve desired optical properties (e.g., including but not limited to clarity, a wide field of view, etc.), the camera or camera device can utilize a refractive lens to focus incident light onto an optical sensor. In some cases, the lens of the camera device can be a composite lens including multiple refractive lens elements stacked together. In some cases, the total thickness of the composite lens stack can add additional dimensions to the device including the composite lens stack.

[0003] Metalenses may provide an alternative to refractive lenses. Metalenses may be formed by fabricating nanometerscale (also referred to herein as nanoscale) geometries on a substrate material. Nanoscale geometries may control the transmission, polarization, and phase of light passing through the nanoscale geometries based on their physical properties (e.g., height, width, length, diameter, etc.). In some cases, metalenses may be fabricated using fabrication techniques such as electron beam (e-beam) lithography. Summary of the Invention

[0004] In some examples, systems and techniques for image correction are described. According to at least one illustrative example, an apparatus is provided. The apparatus includes: at least one memory; and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain image data of an image from a sensor coupled to a metalens; obtain a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at positions along an axis of the image; divide the image into a set of sub-images; determine a local PSF of a sub-image in the set of sub-images based on the plurality of PSFs; and deconvolve the sub-image based on the local PSF of the sub-image.

[0005] As another example, a method for image correction is provided. The method includes obtaining image data of an image from a sensor coupled to a metalens; obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at locations along an axis of the image; dividing the image into a set of sub-images; determining a local PSF of a sub-image in the set of sub-images based on the plurality of PSFs; and deconvolving the sub-image based on the local PSF of the sub-image.

[0006] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided. The instructions, when executed by at least one processor, cause the at least one processor to: obtain image data of an image from a sensor coupled to a metalens; obtain a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at locations along an axis of the image; divide the image into a set of sub-images; determine a local PSF for a sub-image in the set of sub-images based on the plurality of PSFs; and deconvolve the sub-image based on the local PSF for the sub-image.

[0007] As another example, an apparatus for image correction is provided. The apparatus includes: means for obtaining image data of an image from a sensor coupled to a metalens; means for obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at locations along an axis of the image; means for dividing the image into a set of sub-images; means for determining a local PSF of a sub-image in the set of sub-images based on the plurality of PSFs; and means for deconvolving the sub-image based on the local PSF of the sub-image.

[0008] In some aspects, one or more of the devices described herein are, are part of, or include a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device (e.g., a smartwatch, a fitness tracking device, etc.), an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a computing device of a vehicle), or other device. In some aspects, the device includes one or more cameras. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors that can be used to determine the position and / or posture of the device, the state of the device, and / or for other purposes.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. This subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0010] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1A is a perspective view of an example metalens according to some examples;

[0013] Figure 1B is a side view of an example metalens according to some examples;

[0014] Figure 1C is a perspective view of an example metalens unit cell according to some examples;

[0015] Figure 1D is a top view of an example metalens unit cell according to some examples;

[0016] Figure 1E illustrates a graph of pillar diameter versus phase according to some examples;

[0017] Figure 1F illustrates an example metalens having a triangular lattice unit cell according to some examples;

[0018] Figure 2A illustrates a graph of column position versus phase according to some examples;

[0019] Figure 2B illustrates a graph of post position versus diameter according to some examples;

[0020] Figure 2C illustrates example ray diagrams for a hyperbolic metalens according to some examples;

[0021] Figure 2D illustrates example ray diagrams of an optimized metalens according to some examples;

[0022] Figure 2E illustrates an example spot pattern of a hyperbolic metalens at a focal plane according to some examples;

[0023] Figure 2F illustrates an example spot pattern at a focal plane of an optimized metalens according to some examples;

[0024] Figure 3 is a diagram illustrating an example magnified portion of a metalens according to some examples;

[0025] Figure 4 is a diagram illustrating a side view of a composite lens and a corresponding metalens according to some examples;

[0026] Figure 5 The coma aberration of the metalens imaging system according to various aspects of the present disclosure for incident light at different angles is illustrated;

[0027] Figure 6 is an image captured by a metalens imaging system according to aspects of the present disclosure;

[0028] Figure 7 illustrates local deconvolution functions applied to portions of an image according to aspects of the present disclosure;

[0029] Figure 8 Illustrating generating local PSF values ​​for local deconvolution of an image according to aspects of the present disclosure;

[0030] Figure 9 is a flow chart illustrating a process for enhancing an image using local deconvolution according to aspects of the present disclosure;

[0031] Figure 10 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0032] Provide certain aspects and embodiments of the present disclosure below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0033] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0034] Many devices and systems include optical elements that may include lenses for focusing light onto an image sensor. In one example, a camera or a device including a camera with optical elements (e.g., a mobile device, an extended reality (XR) device, etc.) may capture a frame or sequence of frames of a scene (e.g., a video of the scene). To achieve desired optical properties (e.g., clarity, a wide field of view, etc.), the camera or camera device may utilize a refractive lens to focus incident light onto the image sensor. In some cases, the lens of the camera device may include a composite lens that includes multiple refractive lens elements stacked together. In some cases, the total thickness of the composite lens stack may add additional dimensions to a device that includes the camera lens stack as part of a camera system.

[0035] In contrast to refractive lenses, metalenses are lenses made using metasurface technology. Metasurfaces are planar optical components designed at the nanometer (nm) scale with small geometric features on their surfaces. In some cases, the small geometric features can control the transmission, polarization, and phase of light passing through the metalens. In an illustrative example, the small geometric features that make up the metalens can include pillars or columns (sometimes referred to as nanopillars). In some cases, the effect on light passing through the columns can depend on the geometry of the columns, such as the height of the columns, the diameter of the columns, and the pitch of the columns. In some implementations, the columns can have a constant height, and the effect on light passing through the columns can be varied by providing columns of different diameters.

[0036] In some cases, metalenses can be used in devices such as wearable devices, for which thinness is a priority. For example, in some cases, the size constraints in some devices are strict enough that the air gaps between multiple metalenses may make it difficult to integrate a metalens camera into the device. In such cases, it may be difficult to optically correct and / or prevent optical aberrations, such as coma. In some cases, a single-lens metalens may produce aberrations, such as coma, that vary based on the angle of the incident light. In some cases, the metalens may be configured to collimate the incident light so that incident light appearing from increasingly higher angles is focused further away from the center of the image produced by the metalens. Therefore, the aberrations produced by such a metalens may result in the amount of aberration increasing the further away from the center of the resulting image. In some cases, while a deconvolution process based on a point spread function (PSF) can be used to correct certain aberrations, a global deconvolution process based on a single PSF may not be very effective as the amount of aberration changes. Therefore, techniques for enhanced compact metalens sensors with local deconvolution may help correct such aberrations.

[0037] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for an enhanced compact metalens sensor with local deconvolution. Rather than using a single PSF and performing deconvolution of an image, the image can be divided into multiple subportions, each associated with a local PSF. This local PSF can be used to deconvolve the subportions. Additionally, in some cases, such as in the case of a symmetric lens (e.g., where the lens is rotationally symmetric about an axis extending from the center of the lens perpendicular to the lens plane (e.g., the optical axis)), the local PSF values ​​across the image may be circularly symmetric. Thus, a single set of PSF values ​​can be determined for a portion of the axis of an image produced by a lens (such as a metalens). This set of PSFs can be applied to other subportions by selecting a corresponding PSF from the set of PSF values ​​based on how far the subportion is from the center of the image, and rotating the corresponding PSF based on the angle between the subportion and the portion of the axis. The subportion can then be deconvolved based on the determined PSF values, and an image can be output based on the deconvolved subportion.

[0038] Various aspects of the techniques described herein are discussed below with respect to the accompanying figures. Figures 1A to 1C A view of an example metalens is illustrated. Figure 1A In the illustrated example, the metalens 100 includes a substrate 102 (also referred to as a base) having a plurality of pillars 118, including pillars 104, 106, and 108 disposed on a surface of the substrate 102. In some cases, the pillars 118 may be examples of nanoscale geometric structures that form a metasurface. The pillars 104, 106, and 108 may be nanostructures having a height in the nanometer or micrometer range. In some implementations, the height of the nanostructures (e.g., pillars 118) may be on the order of the wavelength of light relevant to a particular application. In one illustrative example, for a metalens in short-wavelength infrared (SWIR) applications (e.g., for wavelengths between 1000 nanometers (nm) and 3000 nm), a pillar height between 1100 nm and 1200 nm may be used. In another illustrative example, for a metalens in visible light applications (e.g., for wavelengths between 350 nm and 800 nm), a pillar height between 300 nm and 400 nm may be used. In some implementations, pillars 104, 106, and 108 can have a common height H. Figure 1A In the illustrated example, posts 104, 106, 108 may have different diameters, with post 104 being shown as having the smallest diameter, post 106 being shown as having a larger diameter than post 104, and post 108 being shown as having a larger diameter than both posts 104 and 106. Figure 1A In the illustration of , additional pillars of different sizes disposed on the substrate 102 are also shown. Figure 1AA light column 110 is illustrated incident on a metalens 100. As will be explained in more detail below, the columns of the metalens 100 (including columns 104, 106, 108) can phase-shift the light rays incident on the light column 110 so that the light rays incident on the light column 110 converge to a focal point 112 having a common phase. In some cases, the light column is collimated. In some cases, the distance between the metalens 100 and the focal point 112 can be referred to as the focal length of the metalens 100. While examples of the present disclosure include example metalenses utilizing columns 118 as geometric features forming a metasurface that forms a metalens, the systems and techniques described herein can be used with metalenses that include features other than columns without departing from the scope of the present disclosure.

[0039] Figure 1B A side view of an example metalens 130 that can be configured to focus light at a focal point 132 is illustrated. In some cases, the metalens 130 may include a plurality of pillars 131 located on one surface of the metalens 130 (the plurality of pillars may correspond to Figure 1A 118 shown in FIG. Figure 1B The column 118 shown in FIG. 1 is for illustration purposes only and is not shown to scale. Figure 1B The number, height, diameter, and / or pitch of the pillars 118 shown in FIG are provided as examples only. Other metalens configurations may be used without departing from the scope of the present disclosure. For example, Figure 1B Each individual one of the pillars 118 shown in FIG may represent a group of pillars in a metalens. Figure 1B In the illustrated example, rods 136A, 136B, and 136C can provide different phase delays to incident light. For example, light passing through rod 136B will experience a greater phase delay than light passing through rod 136A or rod 136C. In some cases, rods 136A, 136B, and 136C can represent a group of rods that provide different phase delays to incident light. Figure 1B In the illustrated example, light rays 134A, 134B, 134C may be incident on metalens 130. Figure 1B In the illustrated example, light ray 134A passes through first rod 136A, light ray 134B passes through second rod 136B, and light ray 134C passes through third rod 136C. Light rays 138A, 138B, 138C represent the paths of light rays 134A, 134B, 134C after passing through respective rods 136A, 136B, 136C. Figure 1B As illustrated, light rays 138A and 138C travel from the edge of metalens 130 and may travel a greater distance than light ray 138B to reach focal point 132. In some implementations, each of the pillars 136A, 136B, 136C may be configured with a phase shift so that each of the light rays 138A, 138B, 138C arrives at focal point 132 at the same phase. As will be described below with respect to Figures 1C to 1E As explained in more detail, the phase shift experienced by light (e.g., 134A, 134B, 134C) passing through rods 136A, 136B, 136C can be controlled based on the geometry of rods 136A, 136B, 136C. In some cases, the amount of phase shift experienced by light passing through rods 118 can depend on the height H, diameter D, wavelength of the light, angle of incidence, and polarization of the light passing through the rods.

[0040] Figure 1C illustrates a perspective view of a unit cell that can be used to design a metalens (such as metalens 100) with desired optical properties, and Figure 1D The top view of the unit cell is shown in FIG. Figure 1C and Figure 1D In the example of FIG. 1 , the unit cell 114 may include a substrate 116, which may be Figure 1A . In some cases, substrate 116 includes a post 118 disposed on substrate 116 and centered at the center of unit cell 114. In some aspects, the unit cell can be a square, triangle, hexagon, or another geometric tile shape with a width of U. As an example, Figure 1F An example metalens 190 is illustrated having a triangular lattice unit cell 192. In some implementations, the width U of the unit cell can be determined based on the wavelength (λ) of light for which the metalens is designed. In some cases, the width U can be less than λ / (-2*NA), where NA is the numerical aperture of the metalens. In some cases, the width U of the unit cell can be between 300 nm and 600 nm. The pillars 118 can have a height H and a diameter D. In some cases, the optical properties of each unit cell 114 can be configured based on a value selected for the value D for each unit cell. In some cases, a metalens can be constructed by arranging an array (also referred to as a lattice) of unit cells having pillars 118 of different diameters to achieve desired optical properties. In the case where each unit cell in the unit cell has the same value of U, the pillars 118 can have a uniform pitch. Although this document discusses Figure 1C and Figure 1D A square unit cell and associated lattice are described, but other unit cell shapes and lattice structures may be utilized without departing from the scope of the present disclosure. In one illustrative example, a hexagonal unit cell may be used to form a hexagonal or triangular lattice.

[0041] Figure 1E A plurality of graphs 150 illustrating the phase shift of light traveling through rods of different diameters D are illustrated. Figure 1E The illustrative example of depicts the relationship between the diameter and phase of transverse electric (TE) polarized light passing through the rod 118. Figure 1EIn the illustrated example, the horizontal axis represents the diameter D (in micrometers (μm)) of the pillars 118 in the unit cell 114, and the vertical axis represents the amount of phase shift experienced by light passing through the pillars 118. The plurality of graphs 150 illustrate the amount of phase shift experienced by light for different incident angles θ. Figure 1E As shown, for a fixed post height H, the phase shift of light passing through the post 118 may increase as the diameter D of the post 118 increases.

[0042] Figure 2A The distance between the metalens and the lens is illustrated for two example positive metalens (e.g., Figure 1A Graph 202 of an exemplary relationship between the distance from the center of the metalens 100 shown in FIG. Figure 2A and Figure 2B In the illustrated example, the metalens may be composed of an array unit cell having a fixed height and width U (e.g., Figure 1C and Figure 1D In one illustrative example, the relationship between the diameters D of the pillars included in the unit cell corresponds to the above Figure 1E The graph 150 shown in FIG. Figure 2A In the illustrated example, the horizontal axis represents the distance from the center of the metalens, and the vertical axis represents the phase shift to be applied at each distance in order to achieve a specific desired metalens optical property. Example graph 206 shows an example of the column size of a metalens designed to have the optical properties of a hyperbolic refractive lens. In an illustrative example, the hyperbolic refractive lens relationship between phase and distance from the lens center illustrated by example graph 206 can represent a hyperbolic lens that provides precise focusing for normally incident light. Example graph 204 illustrates an example of an optimized metalens with a set of desired optical properties. In some cases, optical ray tracing software can be used to determine the optimized properties of the metalens. For example, example graph 204 can represent a lens optimized to minimize the optical path difference (OPD) over a range of incident angles between 0 and 25 degrees. In an illustrative example, the lens represented by example graph 204 can be the optimization result of the following equation (1).

[0043]

[0044] where r is the radial position on the metalens, f is the focal length normalized by the metalens, a0 is the term governing the ratio of the hyperbolic phase law relative to the aspheric term, and a mis adjusted to determine the coefficient of the optimized OPD. In some cases, a0 can also be used to adapt to different materials. For example, a metalens is designed that focuses a normally incident beam into air (s0=1), glass (s0=1.5), and silicon (s0=3.5). The sign of s0 may be changed to negative to simulate a diverging lens. Figures 2C to 2F As illustrated, optimizing the OPD can improve focusing for angles of light that are not perpendicular to the metalens.

[0045] As mentioned above about Figures 1A to 1E As described, the example metalens can be configured so that any incident light rays passing through a column of the metalens (e.g., column 118) arrive at the focal point with the same phase. Figure 2A In the illustration of FIG, the horizontal axis of graph 202 represents the distance from the center of the metalens in millimeters (mm), and the vertical axis of graph 202 represents the amount of phase shift in radians required to achieve the desired optical properties of the example metalens.

[0046] Figure 2B Graph 212 of metalens rod diameter plotted against distance from the center of the metalens is illustrated. Figure 2B In the illustrated example, the horizontal axis represents the distance from the center of the metalens, and the vertical axis represents the diameter of the rod that achieves the specific metalens optical properties. Figure 2B The example column diameters shown correspond to the above description of Figure 2A Graph 202 of an optimized metalens is depicted. Because the propagation of light can be described as a sinusoid, the phase of the light can repeat every cycle of the wavelength of the light (e.g., every 360 degrees or every 2×pi (π) radians). Thus, for example, the same rod diameter can be used when the desired phase shift is 180 degrees and when the desired phase shift is 540 degrees. Thus, graph 212 illustrates a range of rod diameters that can provide a phase shift corresponding to example graph 204 of an optimized metalens. In the illustrated example, diameter D can have a maximum value at the center 214 of the metalens. In some cases, as the distance from the center 214 of the metalens increases, the diameter D of the rods in the unit cell can decrease until a minimum diameter 216 is reached. At a distance from the center 214 of the metalens corresponding to the minimum diameter 216, the desired phase shift of the rods can be separated by 2π radians from the desired phase shift of the rods at the center 214 of the metalens. In some cases, the diameter D of the pillar can be reset to a maximum size at each point where the desired phase shift is a multiple of 2π radians apart from the phase shift of the pillar at the center 214 of the metalens. In some cases, the location where the pillar diameter D is reset to a maximum value can be referred to as a phase reset point 218.

[0047] Figure 2CAn example ray diagram 220 of a hyperbolic metalens is illustrated. In one illustrative example, the hyperbolic lens shown in ray diagram 220 may correspond to the above Figure 2A An example hyperbolic lens phase characteristic is shown in the example graph 206 . Figure 2C An aperture 222, a metalens 224, and light rays 226, 228, 230, 232, 234, 236 are illustrated. Figure 2C In the illustrated example, ray 226 has an incident angle of 0 degrees, ray 228 has an incident angle of 5 degrees, ray 230 has an incident angle of 10 degrees, ray 232 has an incident angle of 15 degrees, ray 234 has an incident angle of 20 degrees, and ray 236 has an incident angle of 25 degrees. Figure 2C As shown, light rays 226, 228, 230, 232, 234, 236 show increasing amounts of etendue at focal plane 238 as the angle of incidence increases.

[0048] Figure 2D An example ray diagram 240 of an optimized metalens configuration is illustrated. In the illustrated example, the lens configuration may be optimized for wide-angle performance. In one illustrative example, the optimized metalens 244 shown in ray diagram 240 may correspond to the above Figure 2A The example optimized lens phase characteristics shown in the example graph 204 shown in FIG. Figure 2D An aperture 242, a metalens 244, and light rays 246, 248, 250, 252, 254, 256 are illustrated. Figure 2D In the illustrated example, ray 246 may have an angle of incidence of 0 degrees, ray 248 may have an angle of incidence of 5 degrees, ray 250 may have an angle of incidence of 10 degrees, ray 252 may have an angle of incidence of 15 degrees, ray 254 may have an angle of incidence of 20 degrees, and ray 256 may have an angle of incidence of 25 degrees. Figure 2D As shown, when Figure 2C Light rays 246 , 248 , 250 , 252 , 254 , 256 show a relatively reduced amount of spread at focal plane 258 when compared to light rays 226 , 228 , 230 , 232 , 234 , 236 shown in FIG.

[0049] In one illustrative example, Figure 2C and Figure 2D The example metalenses 224 and 244 shown in Figures 1 and 2, respectively, may represent metalenses configured as follows: metalenses 224 and 244 may be designed for a wavelength of 1380 nm or 1550 nm; apertures 222 and 242 may have a diameter of 1 mm; the spacing between apertures 222, 242 and the corresponding metalenses 224, 244 may be 1.5 mm; metalenses 224, 244 may be fabricated on a 0.5 mm thick crystalline silicon wafer substrate; and metalenses 224, 244 may have a focal length of 2 mm.

[0050] Figure 2E Illustrated through the Figure 2C 26 , 268 , 270 , 272 , 274 , 276 , the grids drawn represent areas of size 200 μm x 200 μm, and the center of each grid may correspond to the focal plane (e.g., Figure 2C The ideal focus is at the focal plane 238 shown in FIG. Figure 2E In the illustration of FIG, each of the spot diagrams 266, 268, 270, 272, 274, 276 includes a circle representing the diffraction limit for focusing incident light at the focal plane. For example, circle 278 (which may appear as a dot) illustrates the diffraction limit illustrated on the spot diagram 266. Each of the remaining spot diagrams 268, 270, 272, 274, 276 includes a similar circle (not labeled). Figure 2E As shown, spot diagram 266 may correspond to light ray 226 having an incident angle of 0 degrees, spot diagram 268 may correspond to light ray 228 having an incident angle of 5 degrees, spot diagram 270 may correspond to light ray 230 having an incident angle of 10 degrees, spot diagram 272 may correspond to light ray 232 having an incident angle of 15 degrees, spot diagram 274 may correspond to light ray 234 having an incident angle of 20 degrees, and spot diagram 276 may correspond to light ray 236 having an incident angle of 25 degrees. In the illustrated spot diagrams 266, 268, 270, 272, 274, 276, the dots represent the light rays passing through different portions of the metalens 224 at the focal plane (e.g., Figure 2C 238). As shown in spot diagram 266, hyperbolic metalens 224 provides ideal focus at an angle of incidence of 0. However, as shown in spot diagrams 268, 270, 272, 274, and 276, as the angle of incidence increases, the amount of etendue also increases. As shown in spot diagram 276, some light rays with an incident angle of 25 degrees can reach the focal plane more than 100 μm away from the ideal focus on the focal plane.

[0051] Figure 2F Illustrated through the Figure 2D The spot diagrams of light rays 246, 248, 250, 252, 254, 256 at the focal plane of the metalens 244 are shown. In the spot diagrams 286, 288, 290, 292, 294, 296, the drawn grids represent an area with a size of 20 μm x 20 μm, and the center of each grid may correspond to the focal plane (e.g., Figure 2D The center of the focal plane 258 shown in FIG. Figure 2FIn the illustration of FIG, each of the spot diagrams 286, 288, 290, 292, 294, 296 includes a circle representing the diffraction limit for focusing the incident light. For example, circle 298 shows the diffraction limit illustrated on the spot diagram 286. Each of the remaining spot diagrams 288, 290, 292, 294, 296 includes a similar circle (not labeled). Figure 2F As shown, spot diagram 286 may correspond to light ray 246 having an incident angle of 0 degrees, spot diagram 288 may correspond to light ray 248 having an incident angle of 5 degrees, spot diagram 290 may correspond to light ray 250 having an incident angle of 10 degrees, spot diagram 292 may correspond to light ray 252 having an incident angle of 15 degrees, spot diagram 294 may correspond to light ray 254 having an incident angle of 20 degrees, and spot diagram 296 may correspond to light ray 256 having an incident angle of 25 degrees. In the illustrated spot diagrams 286, 288, 290, 292, 294, 296, the dots represent the light rays passing through different portions of the metalens 244 at the focal plane (e.g., Figure 2D As shown in the spot diagram 286, the optimized metalens 244 can provide ideal focusing at zero angle of incidence. As shown in the spot diagrams 288, 290, 292, 294, and 296, as the angle of incidence increases, the Figure 2E The amount of expansion is increased only to a lesser extent compared to the hyperbolic lens spot patterns 266, 268, 270, 272, 274, and 276 illustrated in FIG. As shown in the spot patterns 288, 290, 292, 294, and 296, light passing through the optimized metalens 244 can be focused within a radius of 10 μm in any direction from the ideal focus on the focal plane.

[0052] Figure 3 An example magnified portion of a metalens 300 is illustrated, illustrating a pattern of unit cells having varying pillar diameters. Figure 3 , metalens 300 is disposed on lens plane 332, and the optical axis extends from center 306 of metalens 300 perpendicular to lens plane 332 (e.g., extends outward from lens plane 332 and back through the lens plane). In one illustrative example, the column size of metalens 300 may correspond to Figure 2B The example metalens shown in the graph 212 shown in FIG. Figure 3 As illustrated, a low magnification horizontal view 302 of the metalens 300 shows a column 318 of the metalens 300 (which may correspond to the Figure 1C and Figure 1D The pattern of pillars 118 shown in FIG3 may have a radially symmetric 330 pattern around the optical axis from the center 306 of the metalens 300 to the periphery of the metalens 300. Figure 3 In the illustration of FIG, a line segment 304 is drawn extending radially from the center 306 of the metalens 300. Figure 2B As shown in graph 212 of FIGURE 212, the diameter of pillars 318 may have a maximum value near the center 306 of metalens 300. In one illustrative example, the diameter of pillars 318 at the center of metalens 300 may be approximately equal to or slightly smaller than the diameter of the unit cell (e.g., as shown in FIGURE 212). Figure 1C and Figure 1D Moving away from the center 306 of the metalens 300, the pillar size may decrease (providing a correspondingly smaller phase shift) relative to the pillars at the center 306 of the metalens until a phase reset point 308 (e.g., Figure 2B ). At phase reset point 308, the size of pillars 318 may be reset to their maximum diameter. In some cases, the different diameters of pillars 318 may create a ring-like appearance. Medium magnification level 310 and high magnification level 312 also illustrate the appearance of pillars within a unit cell. As illustrated, pillars 318 may be centered on a common pitch, and the diameter of large pillars 320 may be slightly smaller than the width U of unit cell 322 (depicted as white squares).

[0053] Figure 4 A side view of a composite lens 400 and a corresponding metalens 410 are illustrated, which may have similar optical properties. Figure 4 In the illustrated embodiment, composite lens 400 includes lens elements 402A, 402B, 402C, 402D, 402E, and sensor cover glass 402F, which, when stacked together, can provide desired optical properties for a particular application. For example, composite lens 400 can be designed to have a specific target focus range, a wide-angle field of view, and desired limits on spherical and chromatic aberration, among other properties. In composite lens 400, the various optical elements 402A, 402B, 402C, 402D, 402E, 402F can each refract incident light 406A, 406B, 406C, 406D in different ways, such that the overall effect of optical elements 402A, 402B, 402C, 402D, 402E, 402F, when stacked together, provides the desired optical performance. In the illustrated example, the composite lens 400 is operable to focus incident light rays 406A, 406B, 406D, 406D at a focal plane 404. In some examples, an optical sensor (also referred to herein as an image sensor, image detector, or light-sensitive device) can be positioned at the focal plane 404 to detect the incident light. Because multiple elements may be required to achieve the desired characteristics of the composite lens 400, the composite lens may significantly increase the height, weight, and / or cost of a device (e.g., a mobile device) that uses the composite lens 400. In some cases, a device may have more than one camera and other optical sensors, each of which may require multiple separate composite lenses.

[0054] In some cases, the metalens 410 can be configured to perform similar optical properties as the composite lens 400. In some implementations, a single-layer metalens 410 can provide desired optical properties for an imaging system (e.g., a camera, a range imager, etc.). In such cases, the metalens 410 can provide significant weight and thickness savings relative to the composite lens 400. The metalens 410 can include a substrate 412 and a pillar 414 (e.g., Figure 1A 、 Figure 1C and Figure 1D 418). In some cases, light rays 416A, 416B, and 416C may arrive at metalens 410 from different angles after passing through aperture 406. Light rays 416A, 416B, and 416C may propagate through the lens along paths about optical axis 420. Optical axis 420 may be at the center of metalens 410. In some cases, varying dimensions (e.g., diameter, height, pitch, etc.) of pillars 414 may be used to control the transmission, polarization, phase / retardation of incoming light by varying the effective refractive index. In some cases, metalens 410 may include pillars 414 configured to focus collimated incident light into a spot on focal plane 418. In some cases, pillars 414 at positions (x, y) may be configured to impart a focal length defined by φ. t (x, y) is given a phase (e.g., delay) to compensate for the path differences between the light rays. As a result, all light rays travel to point O on the focal plane 418 at the same time. Figure 4 As illustrated, metalens 410 can focus light at focal plane 418. In some examples, an optical sensor can be positioned at focal plane 418 to detect incident light.

[0055] As discussed above, in some cases, the metalens 410 may be made of a silicon material (such as a silicon wafer). In some cases, the silicon-based metalens 410 may have a relatively high optical index in the SWIR wavelengths compared to, for example, air or glass. This high optical index allows light to be focused into a very sharp (e.g., small) focal spot, thereby producing a relatively small image. However, replacing multiple lenses (such as optical elements 402A, 402B, 402C, 402D, 402E, 402F) with a metalens may sometimes introduce aberrations. In some cases, these aberrations can be optically corrected using multiple layers of the metalens (e.g., layers of substrate 412 and pillars 414) and / or spacers between the metalens layers or between the metalens and the focal plane. In some cases, size limitations of an imaging system with a metalens may prevent the use of multiple layers of metalens and / or sufficient spacers to avoid optical aberrations, such as coma.

[0056] Figure 5Coma for metalens imaging systems according to aspects of the present disclosure for incident light at different angles is illustrated. In some cases, the coma or coma aberration is a cone-like, asymmetric shape that is formed, rather than a point of light. It is worth noting that although discussed with reference to coma aberration, the techniques discussed in the present disclosure can be applied to other aberrations, such as defocus, spherical aberration, chromatic aberration (for color images at visible, NIR or SWIR wavelengths), etc. As an example, a collimated laser beam focused onto a light sensor through a metalens can be used to measure a point spread function (PSF) based on the resulting light pattern detected by the light sensor. The PSF can be a function that describes how a point source (e.g., a point object) appears in an optical imaging system. In some cases, the PSF can be measured at different angles of incidence. For example, 0 degrees (e.g., directly perpendicular to the metalens and sensor, such as Figure 4 First light pattern 502 is generated using a collimated laser beam (e.g., similar to light 416B but offset only 3 degrees from light 416C). Second light pattern 504 can be generated using a 3-degree laser beam (e.g., similar to light 416B but offset only 3 degrees from light 416C), third light pattern 506 can be generated using a 6-degree laser beam, and fourth light pattern 508 can be generated using a 13-degree laser beam. In this example, first light pattern 502 is circular and slightly blurred rather than a single dot. Second light pattern 504 appears slightly triangular due to cone coma 510. In some cases, the size of the coma can increase as the angle of the incident light increases. For example, in third light pattern 506, the size and amount of cone coma 512 has increased. Similarly, in fourth light pattern 508, the size and amount of cone coma 514 has also increased.

[0057] As indicated by the increase in the size of the coma aberration as the angle of the incident light increases, the PSF may also change based on the angle of the incident light. For example, an image captured using a metalens imaging system having such coma aberration (such as Figure 6 As shown in image 600, the center of the image may be relatively sharp, while the amount of coma increases as the image moves away from the center. Furthermore, the direction of the coma may vary depending on the direction away from the center. For example, in the left portion of image 600, cone coma may appear to be shifted to the left, while in the right portion of image 600, cone coma may appear to be shifted to the right, and vice versa.

[0058] In some cases, if the PSF is consistent across an image (e.g., for incident light at multiple angles), a deconvolution algorithm based on a single PSF (such as the PSF of the first light pattern 502) can be applied to the entire image (e.g., global deconvolution) to sharpen the image and improve image quality. However, as the PSF varies, global deconvolution may only significantly improve image quality in a portion of the image. In some cases, local deconvolution can be applied to help account for the varying PSF across the image.

[0059] Figure 7 700 , and a local deconvolution function is illustrated as being applied to portions of an image 700 in accordance with aspects of the present disclosure. In some cases, local deconvolution may be applied, where different portions of the image 700 may be deconvolved based on different PSFs. For example, the image 700 may be divided into nine different sub-images 702A-702H using a 3×3 grid, and different PSFs may be applied to the different sub-images. For example, the image 700 may be divided into nine different sub-images 702A-702H based on a predetermined first PSF (such as a PSF based on a 3×3 grid). Figure 5 The pixel values ​​in the sub-image 702E are deconvolved based on the 0 degree PSF 704 determined by the first light pattern 502. Figure 5 The pixel values ​​in sub-images 702A-702D and 702F-702H are deconvolved using a 13-degree PSF 706 determined by the fourth light pattern 508 of FIG. 1 . In some cases, the PSF used for local deconvolution with a metalens may be predetermined, such as during research and development and / or production of the metalens. In some cases, a PSF may be determined once for an imaging device having a metalens. In some cases, deconvolution may be performed using any deconvolution algorithm, examples of which include the Richardson-Lucy algorithm, maximum likelihood deconvolution, a Wiener filter, and the like. It is noteworthy that although Figure 7 Local deconvolution using two predetermined PSFs is illustrated, but it should be understood that any number of predetermined PSFs may be used.

[0060] In some cases, blind deconvolution can be performed for each sub-image. In blind deconvolution, one or more PSFs can be predetermined and used as hints for the blind deconvolution algorithm. For example, based on Figure 5 The 0 degree PSF determined from the first light pattern 502 may be used as a hint for the blind deconvolution algorithm applied in sub-image 702E. The same PSF or another PSF (such as one based on Figure 5 The 13 degree PSF determined by the fourth light pattern 508 may be used as a cue in the other sub-images 702A-702D and 702F-702H.

[0061] In some cases, a single set of PSF values ​​may be determined along a portion of the axis and rotated for use across the image. For example, in the case of a metalens symmetry, the local PSF values ​​across the image may be circularly symmetric and may be determined based on predetermined PSF values ​​for points along a portion of the axis. In some cases, a local PSF may be determined for a sub-portion of the image for local deconvolution of the sub-portion. In some cases, the local PSF may be determined for the center point of the sub-portion.

[0062] Figure 8 800 for local deconvolution. In some cases, a portion of axis 810 may extend from the center of image 800 to an edge of the image. Figure 8 In , the portion of axis 810 may be from the center of image 800 (e.g., at point 802) to any edge of image 800 on the horizontal axis (in this example, the right edge). Figure 8 800 ). In FIG. 800 , an image of a collimated light spot can be used to obtain predetermined PSFs at points 802, 804, 806, and 808 along a portion of an axis 810 of an image plane 830 (associated with image 800) on which PSF values ​​can be measured. As indicated above, in some cases, a metalens can be rotationally symmetric about the optical axis. For a symmetric metalens, the PSF can also be rotationally symmetric about the center of the image (e.g., at point 802, corresponding to the intersection between image plane 830 and the optical axis). Thus, the predetermined PSFs at points 802, 804, 806, and 808 can be used to describe the local PSFs at other points in image 800 based on their distance from the center of the image (e.g., the location of the zero-degree beam focus) and the angle of the point with respect to that portion of axis 810. For example, point 812 can be offset by 90 degrees from that portion of axis 810, and the distance from point 812 to the center is the same as that of point 804. Thus, the local PSF at point 812 can be described based on the predetermined PSF at point 804 by rotating the predetermined PSF at point 804 by 90 degrees. Similarly, point 814 can be located at the same distance from the center of the image as point 806 and offset by an angle of 180 degrees from the portion of axis 810. Thus, the local PSF at point 814 can be described based on the predetermined PSF at point 806 rotated by 180 degrees (e.g., flipped). After determining the local PSFs for points across the image, the local PSFs can be used for local convolution of sub-images of the image to generate an image for output.

[0063] As a more graphical example, refer back to Figure 7, a 13-degree PSF 706 may be obtained along a portion of the axis on which the PSF values ​​are measured, with a 0-degree PSF 704 obtained at the center of image 700. In some cases, 13-degree PSF 706 may be used for local convolution in sub-image 702D. Sub-image 702A may be offset 45 degrees from the portion of the axis (e.g., between sub-portion 702E and sub-portion 702D), and thus 13-degree PSF 706 may be rotated 45 degrees clockwise to obtain PSF 708 for sub-image 702A. Similarly, PSF 710 for sub-image 702B may correspond to a 90-degree rotation of 13-degree PSF 706, PSF 712 for sub-image 702C may correspond to a 135-degree rotation of 13-degree PSF 706, and so on.

[0064] In some cases, one or more machine learning (ML) techniques can be used to determine a local PSF (e.g., the PSF for various portions of an image) based on a set of predetermined PSFs for a portion of an axis. For example, an ML algorithm can be trained to predict PSFs for different points about an image, and the ML algorithm inherently divides the image into sub-images based on the number of PSFs being predicted to determine the PSF.

[0065] Figure 9 is a flow chart illustrating an example of a process 900 for image correction according to aspects of the present disclosure. The process 900 may be performed by a computing device (or apparatus) or a component (e.g., a chipset, a codec, etc.) of a computing device (e.g., a processor ... Figure 10 The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other type of computing device. The operations of process 900 may be implemented as a processor (e.g., Figure 10 software components executed and run on the processor 1010 and / or other processors).

[0066] At block 902, a computing device (or a component thereof) may obtain image data for an image from a sensor coupled to a metalens. The computing device (or a component thereof) may include a sensor coupled to the metalens. In some cases, the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis. In some cases, the image includes coma.

[0067] At block 904, the computing device (or a component thereof) may obtain a plurality of point spread functions (PSFs) (e.g., Figure 7 PSF704, 706, Figure 8In some cases, the multiple PSFs represent the PSFs generated by the metalens along an axis associated with the image plane (e.g., Figure 8 800) is associated with an image plane 830. In some cases, the multiple PSFs represent coma at different angles of incident light. In some cases, the multiple PSFs represent coma on the axis of the image plane.

[0068] At block 906, the computing device (or a component thereof) may divide the image into a set of sub-images (eg, Figure 7 sub-images 702A-702H).

[0069] At block 908, the computing device (or a component thereof) may determine a local PSF (eg, Figure 7 The computing device (or a component thereof) may rotate the determined local PSF based on an angle between the sub-image and an axis of the image plane. The computing device (or a component thereof) may determine that the PSF is the local PSF of the sub-image based on a comparison between a first distance of the sub-image from the center of the image and a second distance of a PSF from the center of the image in the plurality of PSFs.

[0070] At block 910, the computing device (or a component thereof) may deconvolve the sub-image based on the local PSF of the sub-image.The computing device (or a component thereof) may output a deconvolved image based on the deconvolved sub-image.

[0071] In some examples, the processes described herein (e.g., process 900 and / or other processes described herein) can be performed by a computing device or apparatus. For example, Figure 10 The computing system 1000 shown in FIG. 1 may implement Figure 9 In some examples, Figure 10 The computing system 1000 shown in FIG. 1 may include a metalens camera module including a metalens described herein (e.g., Figure 1A ).

[0072] The computing device may include any suitable device, such as a vehicle or a computing device of a vehicle (e.g., a driver monitoring system (DMS) of a vehicle), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a web-connected watch or smartwatch, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device with the resource capacity to perform the processes described herein (including process 900 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0073] A component of a computing device can be implemented in circuitry. For example, a component may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0074] Process 900 is illustrated as a logical flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0075] Additionally, process 900 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that may be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0076] Figure 10 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 10 An example of a computing system 1000 is illustrated, which can be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 1005. Connection 1005 can be a physical connection using a bus, or a direct connection to processor 1010, such as in a chipset architecture. Connection 1005 can also be a virtual connection, a networked connection, or a logical connection.

[0077] In some embodiments, computing system 1000 is a distributed system, in which the functionality described in this disclosure can be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component can be a physical device or a virtual device.

[0078] Example system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that couple various system components including system memory 1015, such as read-only memory (ROM) 1020 and random access memory (RAM) 1025, to processor 1010. Computing system 1000 may include a cache 1012 of high-speed memory directly connected to, in close proximity to, or integrated as part of processor 1010.

[0079] Processor 1010 may include any general-purpose processor and hardware or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, configured to control processor 1010 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1010 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0080] To enable user interaction, the computing system 1000 includes an input device 1045 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1000 may also include an output device 1035 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1000. The computing system 1000 may include a communication interface 1040 that generally governs and manages user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, The communication interface 1040 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1000 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and thus the base features herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0081] The storage device 1030 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, a digital video disc (DVD) optical disc, a Blu-ray disc (BDD) optical disc, a holographic optical disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0082] Storage device 1030 may include software services, servers, services, etc. that, when code defining such software is executed by processor 1010, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1010, connection 1005, output device 1035, etc.

[0083] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data can be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0084] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0085] Specific details are provided in the description above to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments can be put into practice without these specific details. For clarity of explanation, in some cases, the present technology can be presented as comprising separate functional blocks, including functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the accompanying drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid these embodiments becoming difficult to understand in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques can be shown in order to avoid making each embodiment difficult to understand without necessary details.

[0086] Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed, but a process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, termination of the process may correspond to the function returning to the calling function or main function.

[0087] The processes and methods according to the examples described above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0088] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0089] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0090] In the foregoing description, various aspects of the present application have been described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. The various features and aspects of the application described above can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0091] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present description.

[0092] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0093] The phrase “coupled to” refers to any component being physically connected directly or indirectly to another component, and / or any component being in communication directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0094] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0095] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Technicians can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0096] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium comprising program code, the program code comprising instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0097] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0098] Illustrative aspects of the present disclosure include:

[0099] Aspect 1. A device comprising: at least one memory; and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: obtain image data of an image from a sensor coupled to a metalens; obtain a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at positions along an axis of the image; divide the image into a set of sub-images; determine local PSFs of sub-images in the set of sub-images based on the plurality of PSFs; and deconvolve the sub-images based on the local PSFs of the sub-images.

[0100] Aspect 2. The apparatus of aspect 1, wherein the apparatus further comprises the sensor coupled to the metalens.

[0101] Aspect 3. The device of any one of Aspects 1 to 2, wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis.

[0102] Aspect 4. The apparatus of any one of aspects 1 to 3, wherein the image includes coma.

[0103] Aspect 5. The apparatus according to aspect 4, wherein the plurality of PSFs represent the coma aberration at different angles of incident light.

[0104] Clause 6. The apparatus of clause 4, wherein the plurality of PSFs represent the coma on-axis of an image plane associated with the image.

[0105] Clause 7. The apparatus of clause 6, wherein the at least one processor is further configured to rotate the determined local PSF based on an angle of the sub-image with the axis of the image plane.

[0106] Aspect 8. An apparatus according to any one of Aspect 107, wherein the at least one processor is further configured to determine that the PSF is the local PSF of the sub-image based on a comparison between a first distance of the sub-image from the center of the image and a second distance of a PSF among the multiple PSFs from the center of the image.

[0107] Aspect 9. The apparatus according to any one of aspects 1 to 8, wherein the at least one processor is further configured to output a deconvolved image based on the deconvolved sub-image.

[0108] Aspect 10. A method for image correction, the method comprising: obtaining image data of an image from a sensor coupled to a metalens; obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at positions along an axis of the image; dividing the image into a set of sub-images; determining local PSFs of sub-images in the set of sub-images based on the plurality of PSFs; and deconvolving the sub-images based on the local PSFs of the sub-images.

[0109] Aspect 11. The method of aspect 10, wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis.

[0110] Clause 12. The method according to any one of clauses 10 to 11, wherein the image includes coma.

[0111] Clause 13. The method according to clause 12, wherein the plurality of PSFs represent the coma aberration at different angles of incident light.

[0112] Clause 14. The method of clause 12, wherein the plurality of PSFs represent the coma on an axis of an image plane associated with the image.

[0113] Clause 15. The method according to clause 14, further comprising rotating the determined local PSF based on an angle of the sub-image with the axis of the image.

[0114] Aspect 16. A method according to any one of Aspects 10 to 15, the method further comprising determining that the PSF is the local PSF of the sub-image based on a comparison between a first distance of the sub-image from the center of the image and a second distance of a PSF among the multiple PSFs from the center of the image.

[0115] Aspect 17. The method according to any one of aspects 10 to 16, further comprising outputting a deconvolved image based on the deconvolved sub-image.

[0116] Aspect 18. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by at least one processor, causing the at least one processor to: obtain image data of an image from a sensor coupled to a metalens; obtain a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at positions along an axis of the image; divide the image into a set of sub-images; determine local PSFs of sub-images in the set of sub-images based on the plurality of PSFs; and deconvolve the sub-images based on the local PSFs of the sub-images.

[0117] Aspect 19. The non-transitory computer-readable medium of aspect 18, wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis.

[0118] Clause 20. The non-transitory computer-readable medium of any one of clauses 18 to 19, wherein the image includes coma.

[0119] Clause 21. The non-transitory computer-readable medium of clause 20, wherein the plurality of PSFs represent the coma at different angles of incident light.

[0120] Clause 22. The non-transitory computer-readable medium of clause 20, wherein the plurality of PSFs represent the coma on an axis of an image plane associated with the image.

[0121] Clause 23. The non-transitory computer-readable medium of clause 22, wherein the instructions cause the at least one processor to rotate the determined local PSF based on an angle of the sub-image to the axis of the image.

[0122] Aspect 24. A non-transitory computer-readable medium according to any one of Aspects 18 to 23, wherein the instructions cause the at least one processor to determine that the PSF is the local PSF of the sub-image based on a comparison between a first distance of the sub-image from the center of the image and a second distance of a PSF among the multiple PSFs from the center of the image.

[0123] Clause 25. The non-transitory computer-readable medium of any one of clauses 18 to 24, wherein the instructions cause the at least one processor to output a deconvolved image based on the deconvolved sub-image.

[0124] Aspect 26. An apparatus for image correction, the apparatus comprising: means for obtaining image data of an image from a sensor coupled to a metalens; means for obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at positions along an axis of the image; means for dividing the image into a set of sub-images; means for determining local PSFs of sub-images in the set of sub-images based on the plurality of PSFs; and means for deconvolving the sub-images based on the local PSFs of the sub-images.

[0125] Aspect 27. The device of aspect 26, wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis.

[0126] Clause 28. The apparatus of any one of clauses 26 to 27, wherein the image includes coma.

[0127] Clause 29. The apparatus of clause 28, wherein the plurality of PSFs represent the coma at different angles of incident light.

[0128] Clause 30. The apparatus of clause 28, wherein the plurality of PSFs represent the coma on-axis of an image plane associated with the image.

[0129] Aspect 31: An apparatus comprising means for performing any of the operations according to aspects 1 to 25.

Claims

1. A device, comprising: at least one memory; and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: obtaining image data of an image from a sensor coupled to the metalens; obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at locations along an axis of the image; dividing the image into a set of sub-images; determining a local PSF for a sub-image in the set of sub-images based on the plurality of PSFs; as well as The sub-image is deconvolved based on the local PSF of the sub-image.

2. The apparatus of claim 1 , further comprising the sensor coupled to the metalens.

3. The apparatus of claim 2 , wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis. The apparatus of claim 1 , wherein the image includes coma. The apparatus of claim 4 , wherein the plurality of PSFs represent the coma aberration at different angles of incident light.

6. The apparatus of claim 4, wherein the plurality of PSFs represent the coma on an axis of an image plane associated with the image. 7 . The apparatus of claim 6 , wherein the at least one processor is further configured to rotate the determined local PSF based on an angle of the sub-image with the axis of the image plane.

8. The apparatus of claim 1 , wherein the at least one processor is further configured to determine that the PSF is the local PSF of the sub-image based on a comparison between a first distance of the sub-image from a center of the image and a second distance of a PSF among the plurality of PSFs from the center of the image.

9. The apparatus of claim 1, wherein the at least one processor is further configured to output a deconvolved image based on the deconvolved sub-image.

10. A method for image correction, the method comprising: obtaining image data of an image from a sensor coupled to the metalens; obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at locations along an axis of the image; dividing the image into a set of sub-images; determining a local PSF for a sub-image in the set of sub-images based on the plurality of PSFs; as well as The sub-image is deconvolved based on the local PSF of the sub-image.

11. The method of claim 10, wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis. The method of claim 10 , wherein the image includes coma. The method of claim 12 , wherein the plurality of PSFs represent the coma at different angles of incident light.

14. The method of claim 12, wherein the plurality of PSFs represent the coma on an axis of an image plane associated with the image.

15. The method of claim 14, further comprising rotating the determined local PSF based on an angle of the sub-image to the axis of the image plane.

16. The method of claim 10, further comprising determining that the PSF is the local PSF of the sub-image based on a comparison between a first distance of the sub-image from a center of the image and a second distance of a PSF among the plurality of PSFs from the center of the image.

17. The method of claim 10, further comprising outputting a deconvolved image based on the deconvolved sub-image.

18. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: obtaining image data of an image from a sensor coupled to the metalens; obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at locations along an axis of the image; dividing the image into a set of sub-images; determining a local PSF for a sub-image in the set of sub-images based on the plurality of PSFs; as well as The sub-image is deconvolved based on the local PSF of the sub-image.

19. The non-transitory computer-readable medium of claim 18, wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis.

20. The non-transitory computer readable medium of claim 18, wherein the image includes coma.

21. The non-transitory computer readable medium of claim 20, wherein the plurality of PSFs represent the coma at different angles of incident light.

22. The non-transitory computer readable medium of claim 20, wherein the plurality of PSFs represent the coma on an axis of an image plane associated with the image.

23. The non-transitory computer-readable medium of claim 22, wherein the instructions cause the at least one processor to rotate the determined local PSF based on an angle of the sub-image to the axis of the image.

24. The non-transitory computer-readable medium of claim 18, wherein the instructions cause the at least one processor to determine that the PSF is the local PSF of the sub-image based on a comparison between a first distance of the sub-image from the center of the image and a second distance of a PSF among the plurality of PSFs from the center of the image.

25. The non-transitory computer-readable medium of claim 18, wherein the instructions cause the at least one processor to output a deconvolved image based on the deconvolved sub-image.

26. A device for image correction, the device comprising: means for obtaining image data of an image from a sensor coupled to the metalens; means for obtaining a plurality of point spread functions (PSFs), wherein the plurality of PSFs represent light patterns generated by the metalens at positions along an axis of the image; means for dividing the image into a set of sub-images; means for determining local PSFs of sub-images in the set of sub-images based on the plurality of PSFs; and Means for deconvolving the sub-image based on the local PSF of the sub-image.

27. The device of claim 26, wherein the metalens is rotationally symmetric about an optical axis of the metalens, and wherein the plurality of PSFs are rotationally symmetric about the optical axis.

28. The apparatus of claim 26, wherein the image includes coma.

29. The apparatus of claim 28, wherein the plurality of PSFs represent the coma at different angles of incident light.

30. The apparatus of claim 28, wherein the plurality of PSFs represent the coma on-axis of an image plane associated with the image.

Images