Systems and methods for sensing using flat optics

Abstract

An optical system includes a metasurface formed of an array of meta-pixels configured to manipulate the phase, amplitude, polarization, etc. of an incident light. Each meta-pixel integrates multiple functions such as lensing, beam steering, shaping, splitting, filtering, and / or converting light based on properties such as polarization, wavelength, etc. The meta-pixel modulates and steers the incident light based on its properties into multiple designated directions onto multiple designated photodetectors of a photodetector array. A design method incorporating end-to-end optimization approaches facilitates precise control over the metasurface's response to different light properties. The optical system effectively separates polarization states or wavelengths into distinct optical channels, for reconstructing images (e.g., using an artificial neural network) and / or enhancing detection capabilities of the photodetectors. The system can achieve higher broadband performance and chromatic aberration tolerance, and is suitable for advanced sensing applications in diverse fields.

Description

SYSTEMS AND METHODS FOR SENSING USING FLAT OPTICSBACKGROUND OF THE INVENTION

[0001] This invention relates to systems and methods for imaging and sensing using flat optics.SUMMARY OF THE INVENTION

[0002] The present invention is directed to a metasurface-based optical sensing system and related method that substantially obviates one or more of the problems due to limitations and disadvantages of the related art.

[0003] Additional features and advantages of the invention will be set forth in the descriptions that follow and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.

[0004] In one aspect, the present invention provides a metasurface structure, which includes: an array of meta-pixels formed on a substrate, each meta-pixel including an array of meta-atoms formed on the substrate, configured to receive an incident light, wherein each meta-pixel is configured to form a plurality of output light distributions corresponding to a plurality of states of the incident light.

[0005] In another aspect, the present invention provides an optical sensing system, which includes: a metasurface including an array of meta-pixels, each meta-pixel including an array of meta-atoms, configured to receive an incident light representing an image; and a light intensity detector array including an array of photodetectors, disposed to receive light from the metasurface to generate detection data, wherein the light intensity detector array includes a plurality of photodetector groups, each photodetector group corresponding to a meta-pixel and including one or a plurality of photodetectors; wherein each meta-pixel is configured to form a plurality of output light distributions based on a plurality of states of the incident light, and each of the plurality of photodetectors of the corresponding photodetector group is configured to receive one or a collection of the plurality of output light distributions formed by the meta-pixel.

[0006] In some embodiment, each of the plurality of states of the incident light includes one or more of a phase, amplitude, incident angle, wavelength, and polarization of the incident light.

[0007] In one example, each meta-pixel is configured to select four polarization states of the incident light including x-polarized, y-polarized, left-hand-circular-polarized (LCP), and right-hand-circular-polarized (RCP), and to form four corresponding output light spots on the four photodetectors. In another example, each meta-pixel is configured to select a red, a green and a blue wavelength range of the incident light, and to form four output light distributions on the four photodetectors. Exemplary designs can operate within a wavelength range of the incident light having a width that is up to 55% of a center wavelength of the range, e.g., between approximately 0.4 pm and 0.7 pm. For example, an exemplary design can generate the four output light distributions forming a Bayer pattern.

[0008] In some embodiments, all meta-pixels of the metasurface are identical to each other. In some other embodiments, the metasurface includes a plurality of identical supercells, each supercell including a plurality different meta-pixels. In some other embodiments, the metasurface includes a plurality of different supercells, each supercell including a plurality different or identical meta-pixels.

[0009] In some embodiments, each meta-pixel is configured to form the plurality of output light distributions based on a plurality of polarization states of the incident light, and wherein the metasurface generates similar light distribution patterns within a wavelength range of the incident light having a width that is up to 35% of a center wavelength of the range, e.g., between 0.7 pm and 1 pm.

[0010] In some embodiments, the optical sensing system further includes optical components located either between the metasurface and the light intensity detector array or on a side of the metasurface opposite of the light intensity detector array, wherein the optical components includes one or more of: a metalens, a refractive lens, a reflective mirror, a spectral filter, a polarization filter, a deflector, an aperture, and diffractive optical elements.

[0011] In some embodiments, the optical sensing system further includes a data processing system coupled to the light intensity detector array and configured to extractmulti-dimensional information from the detection data, the data processing system storing a trained artificial neural network, wherein the data processing system is configured to: inputting the detection data into the trained artificial neural network; and obtaining a reconstructed image from the artificial neural network in response to the inputted detection data.

[0012] In some embodiments, the trained artificial neural network is a convolutional neural network and has been trained by: generating a data library based on full-wave simulated results obtained for the metasurface, which includes input polarization states covering an entire Poincare sphere; for each input polarization state, using a corresponding Stokes parameter as a ground truth label; and training the artificial neural network by comparing output values of the artificial neural network and the ground truth label to calculate a mean square error (MSE) loss function and updating parameters of the artificial neural network based on the loss function.

[0013] In another aspect, the present invention provides an imaging apparatus which includes: an illumination source configured to generate light having a plurality of states; the optical sensing system described above disposed to receive light from the illumination source after reflection or transmission through a scene and to generate detection data by the light intensity detector array; and a processor configured to generate a reconstructed multi-dimensional image from the detection data.

[0014] In another aspect, the present invention provides a method of imaging, which includes: generating, by an illumination source, light having a plurality of states including at least one of polarization, wavelength, amplitude, or phase; directing the light onto a scene and receiving light reflected from or transmitted through the scene; coupling the received light into a metasurface of an optical sensing system, the metasurface comprising an array of meta-pixels each configured to form a plurality of output light distributions corresponding to the plurality of states of the incident light; detecting the plurality of output light distributions by a light intensity detector array comprising a plurality of photodetector groups, each photodetector group corresponding to a metapixel and including one or a plurality of photodetectors; and processing the detection data using a processor configured to reconstruct a multi-dimensional image of the scene basedon the plurality of output light distributions. In some embodiments, the processing step employs a trained neural network.

[0015] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 schematically illustrates a meta- atom used for metasurface.

[0017] Figures 2a- 2d schematically illustrate the metasurface functionality according to embodiments of the present invention. Each meta-pixel is formed of an array of metaatoms that are configured to select and modulate the light according to the input light properties (e.g., (Fig. 2a) polarization states, (Fig. 2b) wavelengths, and (Fig. 2c) a combination of different light properties) and focus the light onto the designated photodetectors.

[0018] Figure 3 schematically illustrates a design method based on the angular spectrum method according to an embodiment of the present invention.

[0019] Figure 4 shows an exemplary layout of a meta-pixel that contains an array of meta- atoms.

[0020] Figure 1 shows simulated intensity distribution of the output field after the single pitch metasurface area for the example of Fig. 4. The top label indicates the polarization states of the input field.

[0021] Figure 2 schematically illustrates the workflow for Stokes parameters reconstruction based on artificial neural networks according to an embodiment of the present invention. The backpropagation process is used to update the neural network parameters.

[0022] Figure 3 schematically illustrates a detailed structure of an exemplary convolutional neural network that may be used to reconstruct Stokes parameters.

[0023] Figure 8 schematically illustrates an exemplary data library generation used to train the neural network.

[0024] Figure 4 shows zoom-in images of the intensity distribution after the input image is shed on the metasurface.

[0025] Figure 10 illustrates a simulation of the signal captured by the camera.

[0026] Figure 11 illustrates a comparison of the ground truth label and reconstructed Stokes parameters by the pre-trained neural networks.

[0027] Figure 12 schematically illustrates an integration structure of the metasurface over the photodetector chip. The figure on the left shows the intensity distribution after the metasurface area. The label above each intensity distribution indicates the corresponding polarization states of the input light. The figure on the right shows the alignment of metasurface upon the photodetectors.

[0028] Figure 13 shows simulated intensity distribution after the basic metasurface area for a number of wavelengths within a working wavelength range.

[0029] Figure 14 shows measured intensity distribution after a fabricated metasurface area. The labels above the images indicate the corresponding polarization state of the input light.

[0030] Figure 15 shows exemplary target phase responses of the meta-pixel for different wavelengths in the embodiment of Fig. 2b.

[0031] Figures 16a-c show three exemplary implementations of the meta-pixel in the embodiment of Fig. 2b and Fig. 15.

[0032] Figure 17 shows full wave simulation results for a device employing the meta- pixel of Fig. 16c.

[0033] Figure 18 shows the efficiency as a function of wavelength of the device employing the meta-pixel of Fig. 16c.

[0034] Figure 19 shows an exemplary free form meta- atom design and the method to quantify the free from shape by assigning a set of geometrical parameters.

[0035] Figure 20 shows an example of modeling of the free form meta-atom fabrication quality by overlapping the edge of the structure from SEM image to the edge of the original layout, and modeling of the optical properties of the fabricated free form metaatoms by simulating phase and transmittance in the desired wavelength range.

[0036] Figure 21 schematically illustrates a metasurface optical coupler. Metasurfaces are used to couple the input channel to multiple output channels depending on the wavelength and / or the polarization of light.DETAILED DESCRIPTION OF THE INVENTION

[0037] An optical system according to embodiments of the present invention includes one or more metasurfaces configured for manipulating the phase, amplitude, and / or polarization of incident light. The metasurface integrates multiple functions, e.g., lensing, beam steering, filtering, and / or converting light based on its properties (e.g., polarization, wavelength, etc.) and can operate over a broad spectral range. In one example, the metasurface focuses and steers the light based on its polarization state or wavelength into designated directions or detectors.

[0038] This optical system further utilizes a design method that incorporates end-to-end optimization approaches to facilitate precise control over the metasurface’s response to different properties of light. The design effectively separates various polarization states or wavelengths into distinct optical channels (e.g., in the form of focal spots or any other optical intensity distribution patterns), which can be used to reconstruct an image and / or enhance the detection capabilities of integrated photodetectors.

[0039] In addition, embodiments of the present invention include a method for image reconstruction to extract detailed information of a scene (e.g., spatial, polarization, and / or spectral information). The system can achieve higher broadband performance and chromatic aberration tolerance, and is suitable for advanced sensing applications in diverse fields.

[0040] This patent application describes the components, systems, design methodology, and practical implementations of the flat optics system, providing a detailed framework for applications in optical imaging and sensing.

[0041] Metasurface

[0042] Optical metasurfaces, also alternatively termed sub-wavelength diffractive optics, are artificial media comprising 2-D arrays of sub-wavelength optical structures (commonly called meta-atoms), typically positioned on a substrate. The optical system according to embodiments of the present invention includes at least one metasurface optimized for manipulating the phase, amplitude, and / or polarization of incident light. The metasurface integrates multiple functions, e.g., lensing, beam steering, filtering, and / or converting light based on its properties (e.g., polarization, wavelength, etc.). Inone example, the metasurface focuses and steers the light based on its polarization states into designated directions or detectors.

[0043] Figure 1 schematically illustrates an exemplary meta-atom used for metasurfaces. In this example, silicon nanopillar is utilized for the basic meta-atom structure, and fused silica is used as the substrate.

[0044] In one embodiment, the meta-atom comprises silicon pillar structures sitting on the fused silica wafer. The meta-atoms can be immersed in different mediums, e.g., air, or other optical materials, such as a transparent polymer or dielectric layer. The pitch of meta-atoms is chosen around O.34Xo, while the height is set around O.96 o. Here, Xo is the device operating wavelength in free space. Rectangular or elliptical shapes can be used in the fin-like structure as the basic meta-atom (see Fig. 1). The shape, width, length, and rotation angle of each meta-atom is designed to control the polarization-sensitive response to the incident light, e.g., via the following Jones matrix:

[0045]

[0046] where Ex\y inis the xly polarized input electric field, Ex y outis the corresponding output electric field, 9 is the rotation angle of the meta-atom, and (px\(pyis the phase delay of the meta-atom (9 = 0) under xly polarized incident light illumination, respectively, which can be controlled by the meta-atom’ s length and width. The meta- atoms can be designed to operate at other wavelengths or a range of wavelengths. Additionally, appropriate meta-atoms may also be used to provide wavelength- sensitive responses to the incident light.

[0047] The meta-atoms may have the same or different geometries, dimensions, and orientations. Exemplary geometries may include rectangular, cylindrical, elliptical, free form, or any other suitable shapes or combinations of different shapes, etc. The lattice of the meta-atoms may have any suitable shape and period (e.g., square, rectangular, or hexagonal). The lattice may also be aperiodic, with varying or random distances between adjacent meta-atoms. In some examples, the gap between adjacent meta-atoms may be designed to have a constant gap distance. The meta-atoms may be immersed in air or other optical mediums (e.g., epoxy, glass, etc.). The meta-atoms and the substrate may be made of the same or different optical materials. In alternative embodiments, the meta-atoms (e.g., pillar or other structures) may be formed directly on the photodetectors that will be described later.

[0048] Metasurface optical functionality

[0049] In embodiments of this invention, an imaging optical system may include imaging optics, a meta-pixel array (each meta-pixel formed by a plurality of meta-atoms), and a photodetector array. The imaging optics first couples light onto the meta-pixel array (e.g., by forming an image), which further modulates and couples the light onto the photodetector array. In the photodetector array, multiple adjacent photodetectors constitute a group, such that each photodetector group spatially corresponds to one meta- pixel and receives the modulated light from that meta-pixel. Multi-dimensional information (e.g., spatial, spectral, angular, depth information, etc.) can be extracted by such an optical system and subsequently reconstructed via signal postprocessing. In some embodiments, all meta-pixels are identical to each other. In some other embodiments, multiple different meta-pixels disposed adjacent to each other may constitute a supercell, and the meta-pixel array is formed of an array of identical supercells. One example of a supercell array is shown in the embodiment of Fig. 12 described later. The metasurface may include a plurality of identical supercells, each supercell including a plurality different meta-pixels; or it may include a plurality of different supercells, each supercell including a plurality different or identical meta-pixels.

[0050] The meta-pixels may provide different functionalities depending on the properties of the incident light, e.g., polarization, wavelength, incident angles, etc. In one example, each meta-pixel multiplexes two different functions into a single device layer with polarization sensitivity. The first function is lensing, which can focus the input light within the aperture into a spot. Another function is light deflection based on the polarization states, which can deflect the light with a specific polarization state(s) into the designed direction. One example design is shown in Fig. 2a, where a meta-pixel (e.g., with a 4.4 pm pitch) is formed of an array of meta-atoms that are configured to select four different polarization states of the incident light, i.e. x-polarized, y-polarized, left- hand-circular-polarized (LCP), and right-hand-circular-polarized (RCP), then direct the light onto four spatially separated focal spots (or light distributions) respectively. A group of four adjacent photodetectors are disposed to respectively receive the four lightdistributions. The output field indicates distinguished response to the different input polarization states. A full wave simulation results are shown in Fig. 5, which can match well with the designed function (a more detailed description of the simulation will be provided later). Other polarization states (e.g., 45° and 135°) may also be included.

[0051] In another embodiment, the metasurface multiplexes two different functions into a single device layer with wavelength selectivity. The first function is lensing, which can focus the input light within the aperture into a spot. Another function is light deflection or splitting based on its wavelength, which can direct the light with a specific wavelength(s) into the designed direction or photodetector. The device is schematically shown in Fig. 2b, where a meta-pixel (e.g., with a 4.4 pin pitch) is formed of an array of meta-atoms that are configured to direct different wavelengths into spatially separated focal spots (or light distributions) on the photodetector array. To achieve this functionality, the meta- pixel must provide a wavelength- specific phase shift to the incident light.

[0052] As an example, a meta-pixel may be designed to route red, green, and blue wavelengths onto a photodetector array according to certain patterns, such as the Bayer pattern, to provide both color routing and micro-lensing functions in RGB cameras. An exemplary target phase response of the meta-pixel for each different wavelength is shown in Fig. 15. At each location on the device, the meta- atom that provides the best phase response across all three wavelengths is chosen from a data library. Three different exemplary meta-pixel implementations of this device are presented in Figs. 16a-c. The first version, shown in Fig. 16a, consists of only circular meta-atoms. The design and fabrication of this version is simple, but the performance is limited since the circular meta-atoms have limited phase variety over the different wavelengths. The second version, shown in Fig 16b, uses uniquely shaped “free form” meta-atoms. These shapes have been designed to provide a more varied phase response for each different wavelength, commonly referred to as dispersion engineering. This version can achieve better performance, but the complexity of the shapes makes fabrication prohibitive.

[0053] A third version, shown in Fig. 16c, uses free form meta-atoms with fabricationcompatible shapes. This version can achieve improved performance with reduced shape complexity, making fabrication feasible. Full wave simulation results of the fabricationcompatible free form device of Fig. 16c are shown in Fig 17, demonstrating thesuccessful functionality of this device. Fig. 18 shows the efficiency of this device as a function of wavelength. Here, efficiency is defined as the percentage of incident power routed to the corresponding photodetector (note that the values for the two photodetectors for green are summed). The meta-pixel device clearly shows better efficiency than the conventional color filter approach, which is typically limited to 25% efficiency. This specific design used in the simulation is formed of niobium oxide structures in air cladding with a thickness of 650 nm. In general, different materials, different thickness, different number of wavelengths, or different target functionality may also be used.

[0054] In yet another embodiment, the polarization and wavelength selectivity functions may be combined in one metasurface, as schematically illustrated in Fig. 2c. In this example, the four states are four different combinations of two polarizations and two wavelengths.

[0055] In another embodiment (Fig. 2d), one or more additional optical components may be located on the top or bottom of the meta-pixel. The additional optical component may function as a lens or lens array (e.g., a metalens, refractive lens, reflective mirror, etc.), filter (e.g., a spectral or polarization filter), deflector(s), aperture(s), diffractive optical elements, a combination of such functions, etc. For example, an imaging lens may be included to form an image on the meta-pixel array. The additional optical components may be a hybrid metalens-metasurface arrangement configured to achieve extended depth of focus or wide-angle field of view for the sensing system. There may or may not be a gap between the additional optical component and the meta-pixel layer.

[0056] In another embodiment (Fig. 21), metasurfaces function as a coupler, which receive an input light from one optical channel and spit the input light into different output channels depending on the wavelength, incident angle and / or the polarization states of the light. Such metasurface couplers may be applied to optical interconnects between fibers, waveguides, grating couplers, lasers, photodetectors, etc.

[0057] Stated more generally, the meta-pixels may be configured to effectively couple light or light sources with different properties (e.g., polarization, wavelength, angular, depths, etc.) into different optical channels (e.g., in the form of focal spots or any other optical intensity and / or phase distribution patterns), which can be used to reconstruct the image with detailed spatial and / or polarization and / or spectral information, and / or toenhance the detection capabilities of integrated photodetectors. In addition to the polarization and wavelength dependent designs described, 3D depth information may also be extracted using this approach. For example, the meta-pixels may be configured to be depth and / or angular sensitive and thereby extract depth-dependent information. Grating or metasurface- grating structures may be utilized to enhance such functions.

[0058] A spacer or materials may be positioned or filled between the meta-pixels and the photodetectors, which may be made of air, glass, polymer, dielectrics, semiconductors, or other materials. In alternative embodiments, the meta-atoms (e.g., pillar or other structures) of the metasurface may be formed directly on the photodetectors rather than on another substrate.

[0059] Design method

[0060] End-to-end optimization method can be utilized to design the metasurface devices as shown in Fig. 3, which is based on a physical model of light propagation within the meta-system. For example, stochastic gradient decent (SGD) solver can be employed to optimize the output intensity at a position in space after the metasurface. In the forward analytical model, light propagating in free space is described by the angular spectrum propagation operator,

[0062] where J7is a Fourier transform operator, d is the propagating distance, and 5f(d) = exp[i ■is the transfer function of light in fc-space. Here, A is the effective wavelength in the medium and kxand kyrepresent the lateral wavenumbers. Light propagating through the meta-optic is calculated by cascading the various elements and free-space regions and is given by,

[0064] where p is the polarization state. Multiple output fields in response to different polarization state input may be optimized simultaneously. Mean square error (MSE) may be calculated by comparing the target and simulated field intensity as the loss function. The target field profiles used to design the meta-optics are presented in Fig. 3, which corresponds to a specified polarization input as labelled on the right.

[0065] The free form shape of free from meta-atoms may be designed and quantified by assigning a set of geometrical parameters. In one example, free form meta-atoms aregenerated by defining the outer boundary and inner boundary. The outer boundary is defined in polar coordinate, with center to edge distance r and angle 0: r = a0+ a • cos(40) + a2• cos (80) where a0, a , and a2are random numbers to give enough diversity to generate a large meta- atom library. Inner boundary may be circular holes with different radius, or other free form shape holes defined by the same method above. One example of such a free form shape is shown in Fig. 19. More generally, the free form is characterized by variable edge profiles or embedded apertures, and designed to enhance dispersion control, multiwavelength, and / or polarization-dependent performance.

[0066] Fig. 20 shows an example of modeling of the free form meta-atom fabrication quality by overlapping the edge of the meta-atom structure from SEM image to the edge of the original designed layout (plot a), and modeling of the optical properties of the fabricated free form meta-atoms by simulating phase and transmittance in the desired wavelength range (plot b).

[0067] Layout and full-wave simulation of metasurface

[0068] An exemplary layout of a meta-pixel obtained using the above-described design method is shown in Fig. 4. The length and width of the meta-pixel is fixed at 4.4 pm, corresponding to a group of 2 by 2 photodetectors on the CMOS camera sensor with a 2.2 pm photodetector pitch. Different numbers of photodetectors may be used under a single meta-pixel. The meta-atoms in this example are in elliptical shapes with different orientations, while other geometries of meta-atoms can also be utilized. Here, a square lattice architecture is utilized to fit all meta-atoms with a period of 0.34 pm in the metasurface, while other lattice structure can be utilized. The height of the meta-atoms is 0.9 pm, although other heights may also be used. The substrate is a fused silica material and the meta-atoms are silicon nanopillars, although other materials may also be used. The refractive index of meta-atom was set at 3.55 during design with the working wavelength of 0.94 pm. The meta-atom is embraced by a protection layer, which may be polymer or dielectrics, with an index of 1.5 for index matching. The designed geometry parameters may be scaled to the other working wavelength, s ■ 2. where s is the scaling factor and 2 = 0.94 is the current working wavelength. To verify the functionality of the designed metasurface, a full-wave simulation was performed by FDTD (Finite-DifferenceTime-Domain) method. In this simulation, periodic boundary condition was used with a specified polarized plane wave as the input.

[0069] The full-wave simulated field results for the above exemplary meta-pixel structure are shown in Fig. 5, where the top label on each field indicates the polarization state of the input plane wave. Each field was obtained by propagating the output light from the metasurface with a certain distance (2.2 pm in this case). With different polarized input, the output intensity distribution demonstrates a distinguished pattern, which can be used to reconstruct the input polarization state, i.e. the Stokes parameters.

[0070] Stokes parameters reconstruction

[0071] After an image is captured by the polarization- sensitive metasurface, the spatial and / or polarization and / or spectral information of the object or scene may be reconstructed using post-processing algorithms. For example, artificial neural networks can be utilized to assist the signal reconstruction process as shown in Fig. 6. In this example, the metasurface is configured in a periodic architecture with the meta-pixel period of 4.4 pm, corresponding to 2 by 2 photodetectors on the CMOS camera sensor with a 2.2 pm photodetector pitch. Different numbers of photodetectors may be used under a single meta-pixel. Under this design, each meta-pixel generates 4 signals captured by the photodetectors, which will be used as the input of the neural network. During the imaging process, each meta-pixel is independent from others, corresponding to a single image pixel of the image. Hence, the above trained neural network can be performed in parallel for the single reconstruction in the imaging application.

[0072] In order to train the neural network model, a data library may be manually generated based on the previous full-wave simulated results, which includes all possible input polarization states covering the entire Poincare sphere. For each polarization state input, the corresponding Stokes parameter may be used as the label during model training. By comparing output values of the neural network and the ground truth label by the mean square error (MSE) function, the loss can be calculated to update the neural network parameters during the training process.

[0073] Neural network model

[0074] A convolutional neural network may be trained to reconstruct the full Stokes parameters, as shown in an example in Fig. 7. Training data may be first generated byoptical simulation of the imaging process, which can enforce a large number of, e.g., 2001 in some examples, kinds of polarization states as the data library. During the forward propagation in the neural network, 3 layers of convolution as well as 3 fully connected layers combined with ReLu (Rectified Linear Unit) activation function may be used for prediction. In training of the neural network an Adam optimizer may be utilized with the learning rate set as, for example, 0.001. In one experiment, training occurred over 500 epochs.

[0075] Full Stokes parameters reconstruction example

[0076] In order to verify the functionality for Stokes parameters reconstruction with the metasurface described above, an example to retrieve the artificial polarization states over a human face was demonstrated with optical simulation as shown in Fig. 8. Here, a variation of polarization states was manually created by setting the complex-valued optical field along x and y directions. Such polarization field was converted to the Stokes parameters by the following equations:2

[0077] So= |EX|2+ |Fy|

[0081] The calculated Stokes parameters were used as the ground truth label during the training of neural network. The imaging process by the metasurface device was based on angular spectrum method, where each pixel of the input image corresponded to a single basic metasurface area. During the imaging process, assuming the human face information is relayed by a front lens then overlapped with the metasurface layer, the optical field propagating by a certain distance was simulated and the result is shown in Fig. 9. Depending on the input polarization states, each pixel in the images exhibited various patterns after passing through the metasurface, which was used to reconstruct the original information.

[0082] Since each image pixel corresponded to a single meta-pixel, which overlapped with 2 by 2 photodetectors on the camera chip, the signal captured by camera was simulated by summing the optical intensity within each photodetector area. A zoom-inexample of the captured image (simulated) is shown in Fig. 10, where the information of each image pixel was divided into 4 separated channels by the metasurface. These four different values were used as the input of the trained neural network to retrieve the Stokes parameters they stand for.

[0083] The reconstructed Stokes parameters of the original image by the trained neural network is shown in Fig. 11, where the top row indicates the ground truth label, and the bottom row shows the reconstructed results. The results from pre-trained neural network can match well with the ground truth values, which can verify the functionality of the device.

[0084] The reconstruction processes described above, including the neural network model, may be implemented by a data processing system such as processors which execute computer executable program code stored in computer readable non-volatile memories.

[0085] Integration with photodetectors

[0086] An exemplary integration structure of the metasurface over the photodetector chip is shown in Fig. 12. In this example, both the photodetector array and the meta-pixel array are square lattices and are rotated 45 degrees relative to each other, and the four sides of each meta-pixel correspond to four diagonals of 2 by 2 photodetectors. The metasurface, which can deflect the input light onto the photodetectors, are aligned according to the polarization states. In this example, one type of meta-pixel (labeled “Split x, y-pol”) is configured to deflect x and y polarized light along the -45 deg and +135 deg directions (relative to the photodetector array lattice) onto two photodetectors located diagonally from each other, and another type of meta-pixel (labeled “Split LCP and RCP”) is configured to deflect LCP and RCP light along the +45 deg and -135 deg direction onto two other photodetectors located diagonally from each other, as shown in Fig. 12. The two types of meta-pixels are arrayed in a checkerboard pattern, where each pair of the two types of meta-pixels constitutes a supercell. The simulation result is shown in the left-hand side of Fig. 12. This integration structure can ensure the symmetrical aperture of basic metasurface area thus guaranteeing the focusing efficiency and quality of metasurface devices. The two types of meta-pixels may also be configuredto distinguish other states of the input light, such as wavelength, wavelength and polarization combination, etc.

[0087] Broadband operation

[0088] The structure and method described above can be used for broadband imaging and sensing. In order to demonstrate the broadband performance of the designed metasurface device, a full-wave simulation was performed by FDTD technique under different wavelength illumination as shown in Fig. 13. During the simulation, periodic boundary condition and LCP plane wave were used as an example. The simulated spectral range was chosen between 0.7 pm to 1 pm of wavelength. The output field exhibited similar intensity distribution (e.g., near diffraction limit or similar spot sizes) within the simulated spectral range, indicating minimal dispersion effects due to chromatic aberrations. In some examples, the metasurface achieves similar focal spot sizes (e.g., with spot size variation or light distribution variation less than 20%) or near diffraction limit within a broad spectral range such as 300-400 nm width in the visible and nearinfrared range. Such broadband performance can also be extended to other spectral bands such as mid-infrared, long wave infrared, etc. For example, the meta-atoms / meta-pixels may be scaled down or up or re-designed and optimized for the desired spectral band. Therefore, for example, the width of the spectral band within which a metasurface provides acceptable light intensity distribution quality, dispersion and chromatic aberration may be about 35% of the center wavelength.

[0089] Experimental verification

[0090] In order to verify functionality of the above-described device, a metasurface structure was experimentally fabricated using the lithography process. More specifically, plasma-enhanced chemical vapor deposition (PECVD) was utilized to deposit the amorphous silicon device layer on a fused silica substrate. MaN photoresist (Micro resist technology, Germany) was then spin-coated on the silicon layer, followed by baking at a temperature of 90 °C for 2 min and coating a conductive layer of e-spacer (Resonac, Japan). The resist was then exposed in the electron beam lithography (EBL) system and developed in AZ 726 (Microchemicals, Germany). The silicon film was then patterned using reactive ion etching (RIE), and a 1 pm-thick layer of SU8 (Kayaku advanced materials, Japan) was spin-coated to encase the nanopillar structures as a protective layer.More generally, the metasurfaces can be fabricated using any suitable microfabrication approaches, such as lithography, nanoimprint, laser direct writing, molding, casting, etc. The optical characterization results are shown in Fig. 14. The labels above the images indicate the corresponding input polarization states during the characterization process, while the square box in the measured image for each polarization state represents one meta-pixel, which can deflect and focus the light into different directions depending on the polarization states. As a comparison, a full- wave simulated intensity distribution from a single basic metasurface area corresponding to the square box is shown to the right of each measured result. The intensity distribution matches well between the simulated and measured results. The background noise in the measurement can be reduced by further improving the fabrication process.

[0091] Additional information and variation embodiments

[0092] The meta-atoms may have the same or different geometries, dimensions, and orientations. Exemplary geometries may include elliptical, rectangular, cylindrical, free form, or any other suitable shapes or combinations of different shapes, etc. The lattice of the meta-atoms may have any suitable shape and period (e.g., square, rectangular, or hexagonal). The lattice may also be aperiodic, with varying or random distances between adjacent meta-atoms. In some examples, the gap between adjacent meta-atoms may be designed to have a constant gap distance. Exemplary metasurface materials include, but are not limited to, dielectric materials (e.g., silicon, silicon nitride, titanium dioxide, niobium oxide, gallium nitride, chalcogenide glasses, etc.), polymers and organic materials (e.g., polymethyl methacrylate, polydimethylsiloxane, etc.), metals (e.g., gold, silver, aluminum, copper, etc.), semiconductors (e.g., silicon, silicon carbide, gallium arsenide, indium phosphide, etc.), transparent conducting oxides (e.g., indium tin oxide, fluorine-dope tin oxide, etc.), phase change materials (e.g., vanadium oxide, germanium antimony tellurium related alloys, etc.), two dimensional materials (e.g., graphene, molybdenum disulfide, etc.), ceramics (e.g., barium titanate, zirconia, etc.), metamaterials, etc. Exemplary substrate materials include, but are not limited to, silicon, glass, sapphire, polymers, quartz, alumina, polyimide, polyethylene terephthalate, ceramics, semiconductors, etc. The metasurface and substrate may be made of the same or different materials.

[0093] The metasurface or meta-atoms described in the invention may be configured to operate at a wide range of wavelengths (e.g., from the visible to the infrared (IR)), depending on the design of the meta-surface and the substrate and meta-surface materials. In some examples, a metasurface or meta-atom can be designed to operate at any wavelength from the microwave to ultraviolet (UV) regions of the electromagnetic spectrum, with a bandwidth that spans up to an octave.

[0094] The metasurface may be flat, curved or conformally integrated with the substrate. One or both sides of the substrate may be flat or curved. Both the metasurface and the substrate may be rigid, flexible, or stretchable. The geometry, dimensions, and layout of the meta-atoms and substrate are designed to provide the target optical functions. The metasurfaces may be designed to operate at a single wavelength, multiple wavelengths, or over a continuous spectral range.

[0095] The sensing system may further include an illumination system to illuminate the scene being imaged (see Fig. 2d). The illumination system may emit light with certain spatial and / or polarization and / or spectral properties according to the sensor design to facilitate the imaging and sensing processes. Metasurfaces may also be incorporated in the illumination system to provide the desired functions, e.g., to match the polarizations, wavelength and / or angle of incidences that can be accepted by the sensor.

[0096] It will be apparent to those skilled in the art that various modification and variations can be made in the metasurface structure, the sensing device and related design method of the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention cover modifications and variations that come within the scope of the appended claims and their equivalents.

Claims

CLAIMS1. An optical sensing system, comprising: a metasurface including an array of meta-pixels, each meta-pixel including an array of meta- atoms, configured to receive an incident light; and a light intensity detector array including an array of photodetectors, disposed to receive light from the metasurface to generate detection data, wherein the light intensity detector array includes a plurality of photodetector groups, each photodetector group corresponding to a meta- pixel and including one or a plurality of photodetectors; wherein each meta-pixel is configured to form a plurality of output light distributions based on a plurality of states of the incident light, and each of the plurality of photodetectors of the corresponding photodetector group is configured to receive one or a collection of the plurality of output light distributions formed by the meta-pixel.

2. The optical sensing system of claim 1, wherein the meta- atoms are nano-structures formed on a substrate or the array of photodetectors, wherein the meta-atoms comprise one or a combination of materials selected from a group consisting of dielectric materials, polymers and organic materials, metals, semiconductors, transparent conducting oxides, phase change materials, and two dimensional materials, wherein the substrate materials include one or a combination of materials selected from a group consisting of silicon, silicon dioxide, glass, sapphire, polymers, quartz, alumina, polyimide, polyethylene terephthalate, ceramics, and semiconductors, and wherein the meta-atoms are immersed in air, a transparent polymer, or a dielectric layer.

3. The optical sensing system of claim 1, wherein at least some of the meta-atoms have free form geometries characterized by variable edge profiles or embedded apertures, configured to enhance dispersion control, multi-wavelength, and / or polarization-dependent performance.

4. The optical sensing system of claim 1, wherein each of the plurality of states of the incident light includes one or more of a phase, amplitude, incident angle, wavelength, and polarization of the incident light.

5. The optical sensing system of claim 1, wherein each photodetector group includes four photodetectors, and wherein either: each meta-pixel is configured to select multiple polarization states of the incident light, and to form four corresponding output light distributions on the four photodetectors; or: each meta-pixel is configured to select multiple wavelengths of the incident light, and to form four corresponding output light distributions on the four photodetectors; or: each meta-pixel is configured to select multiple combinations of multiple polarizations and multiple wavelengths of the incident light, and to form four corresponding output light distributions on the four photodetectors.

6. The optical sensing system of claim 1, wherein each photodetector group includes four photodetectors, and wherein each meta-pixel is configured to select a red wavelength range, a green wavelength range and a blue wavelength range of the incident light, and to form four output light distributions on the four photodetectors, the four output light distributions forming a color pattern of red, green, and blue light distributions.

7. The optical sensing system of claim 6, wherein the color pattern is a Bayer pattern including one red light distribution, two green light distributions, and one blue light distribution.

8. The optical sensing system of claim 1, wherein the photodetector array and the meta- pixel array are respective square lattices rotated by 45 degrees relative to each other, wherein four sides of each meta-pixel correspond to four diagonals of 2 by 2 photodetectors, wherein the meta-pixel array includes a first type of meta-pixels each configured to modulate and redirect light of two states respectively into two photodetectors located diagonally from each other and a second type of meta-pixels each configured to deflect light of two other states respectively into two other photodetectors located diagonally from each other, and wherein the first and second types of meta-pixels form a checkerboard pattern in the meta-pixel array.

9. The optical sensing system of claim 1, wherein all meta-pixels of the metasurface are identical to each other.

10. The optical sensing system of claim 1, wherein either the metasurface includes a plurality of identical supercells, each supercell including a plurality different meta-pixels, or the metasurface includes a plurality of different supercells, each supercell including a plurality different or identical meta-pixels.

11. The optical sensing system of claim 1, wherein each meta-pixel is configured to form the plurality of output light distributions based on a plurality of polarization or spectral states of the incident light, and wherein the metasurface generates similar light distribution patterns within a wavelength range of the incident light having a width that is up to 55% of a center wavelength of the range.

12. The optical sensing system of claim 1, further comprising optical components located either between the metasurface and the light intensity detector array or on a side of the metasurface opposite of the light intensity detector array, wherein the optical components includes one or more of: a metalens, a refractive lens, a reflective mirror, a spectral filter, a polarization filter, a deflector, an aperture, and diffractive optical elements.

13. The optical sensing system of claim 12, wherein the optical components include a hybrid metalens-metasurface arrangement configured to achieve extended depth of focus or wide-angle field of view.

14. The optical sensing system of claim 1, wherein each meta-pixel is further configured to separate incident light according to its angle of incidence to extract depth information from the incident light.

15. The optical sensing system of claim 1, further comprising a data processing system coupled to the light intensity detector array and configured to extract multi-dimensional information from the detection data, the data processing system storing a trained neural network, wherein the data processing system is configured to: inputting the detection data into the trained neural network; andobtaining a reconstructed image from the neural network in response to the inputted detection data.

16. The optical sensing system of claim 15, wherein the trained neural network is a convolutional neural network and has been trained by: generating a data library based on full-wave simulated results obtained for the metasurface, which includes input polarization states covering an entire Poincare sphere; for each input polarization state, using a corresponding Stokes parameter as a ground truth label; and training the neural network by comparing output values of the neural network and the ground truth label to calculate a mean square error (MSE) loss function and updating parameters of the neural network based on the loss function.

17. A sensing apparatus comprising: an illumination source configured to generate light having a plurality of states; the optical sensing system of claim 1 disposed to receive light from the illumination source after reflection or transmission through a scene and to generate detection data by the light intensity detector array; and a processor configured to generate a reconstructed multi-dimensional image from the detection data.

18. A metasurface structure, comprising: an array of meta-pixels formed on a substrate or one or a plurality of photodetectors, each meta-pixel including an array of meta-atoms formed on the substrate or one or a plurality of photodetectors, configured to receive an incident light, wherein each meta-pixel is configured to form a plurality of output light distributions corresponding to a plurality of states of the incident light.

19. A method of sensing, comprising: generating, by an illumination source, light having a plurality of states of at least one of polarization, wavelength, amplitude, or phase;directing the light onto a scene and receiving light reflected from or transmitted through the scene; coupling the received light into a metasurface of an optical sensing system, the metasurface comprising an array of meta-pixels each configured to form a plurality of output light distributions corresponding to the plurality of states of the incident light; detecting the plurality of output light distributions by a light intensity detector array comprising a plurality of photodetector groups, each photodetector group corresponding to a meta-pixel and including one or a plurality of photodetectors; and processing the detection data using a processor configured to reconstruct a multidimensional image of the scene based on the plurality of output light distributions.

20. The method of claim 19, wherein the processing step includes: inputting the detection data into a trained neural network; and obtaining the reconstructed image from the neural network in response to the inputted detection data, wherein the trained neural network is a convolutional neural network and has been trained by: generating a data library based on full-wave simulated results obtained for the metasurface, which includes input polarization states covering an entire Poincare sphere; for each input polarization state, using a corresponding Stokes parameter as a ground truth label; and training the neural network by comparing output values of the neural network and the ground truth label to calculate a mean square error (MSE) loss function and updating parameters of the neural network based on the loss function.

Images