Real-time machine learning-enhanced hyperspectro-polarimetric imaging via an encoding metasurface

Abstract

Embodiments can relate to systems and methods for generating hyperspectro-polarimetric images. The system can include a metasurface having a plurality of super pixels configured to encode spectral and polarization image data into spatial intensity distributions. The system can include an image capture device having at least one image sensor and a plurality of subsets of pixels configured to capture the spatial intensity distributions encoded by the metasurface. The system can include a processor in communicative connection with the image capture device. The processor can be configured to receive pixel data from the image capture device, generate captured image input data based on the pixel data, and access at least one computational reconstruction model configured to decode the captured image input data, and generate image output data based on the decoded captured image input data.

Description

Real-Time Machine Learning-Enhanced Hyperspectro-Polarimetric Imaging Via an Encoding MetasurfaceCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application is related to and claims the benefit of priority of U.S. provisional patent application no. 63 / 682,506, filed on August 13, 2024, the entire contents of which are incorporated herein by reference.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

[0002] This invention was made with government support under Grant Nos. ECCS2047446, ECCS2114266, and CHE2305139 awarded by the National Science Foundation and under Grant No. EY031853 awarded by the National Institutes of Health. The Government has certain rights in the invention.FIELD OF THE INVENTION

[0003] Embodiments can relate to encoding metasurfaces and computational reconstruction models (e.g., machine learning models) for generating hyperspectral-polarimetric images in a single shot and hyperspectral-polarimetric video in real-time with a standard camera. The encoding metasurfaces can include nanostructures capable of tailoring various optical properties, such as polarization, amplitude, phase, and spectral response. The nanostructures form superpixels that can be used to encode spectral and polarimetric information into spatial intensity distributions that can be captured by a standard camera image sensor. The computational reconstruction models (e.g., machine learning models) can include a neural network capable of decoding the information (e.g., the spatial intensity distributions) captured by the standard camera and / or image sensor and generating hyperspectral-polarimetric single-shot images and / or real-time video.BACKGROUND OF THE INVENTION

[0004] The ability to observe and utilize multidimensional information of light has been a long- standing human pursuit, offering a more comprehensive understanding of light-matter interaction and promoting more accurate characterizations across many fields. For instance, the spectral information can significantly improve the precision of medical diagnosis and agricultural monitoring, while the polarization information can aid in material classification and stress analysis. Obtaining multidimensional information within a single system presents substantial advantages.

[0005] However, current hyperspectral and polarimetric imaging techniques often capture these two distinct types of data separately, making it difficult to rapidly obtain comprehensive information. Furthermore, conventional spectral and polarization imaging systems, which predominantly rely on diffractive optics or optical filters, tend to be bulky and heavy. These optical elements are typically sensitive to either the spectrum or the polarization of incoming light. A common solution involves the incorporation of polarization elements into spectral systems. For example, stacking layers of organic filters can create integrated hyperspectral- polarimetric (HSP) cameras. Another method entails integrating linear micro-polarizer arrays onto traditional RGB Bayer sensor. However, both of those approaches have limited number of wavelength channels, and are only capable of analyzing linear polarization states. These limitations not only confine the imaging systems’ capacity to incorporate multiple functionalities into a single platform but also obstruct their potential for further miniaturization and integration.

[0006] Recently, a few HSP cameras have been developed leveraging optical metasurfaces. A metasurface is a thin layer of nanostructures capable of tailoring various light properties, including polarization, amplitude, and phase, presenting a paradigm shift in miniaturizing optical devices. A metasurface can enable various applications such as beam bending, beam shaping, holography, and even invisibility cloaks. One type of spectral -polarimetric camera relies on a spatial-multiplexing metasurface lens that directs light of different polarizations and wavelengths onto separate focal points. However, the method suffers from crosstalk among different polarization and spectral channels, and it has limited spectral resolution due to the inherent coupling of the spectral and spatial dimensions. Another approach employs multiple rotating metasurfaces but is limited to linear polarization analysis. Moreover, it involves mechanicallymoving parts, constraining the system’s compactness and making real-time imaging considerably more challenging.SUMMARY OF THE INVENTION

[0007] A metasurface-enhanced camera system, and method of use thereof, capable of capturing HSP images in a single shot and videos in real-time are presented. The metasurface-enhanced camera system / method can obtain comprehensive information about the broadband spectrum and all four Stokes parameters simultaneously. The uniquely designed metasurface encoder can encode spectral and polarization information into a spatial intensity distribution that can be captured by a standard camera. The system can include a custom machine learning (ML) neural network decoder to enable real-time image recovery. The metasurface is designed such that each super pixel possesses both distinct spectral and polarization responses. Specifically, the metasurface incorporates chiral meta-atoms exhibiting strong chirality, which can enable different polarization responses for all polarization states, including both left- and right-circular polarizations. Coupling the metasurface encoder with the ML neural network decoder can achieve real-time acquisition of HSP images at a rate of 28 frames per second (FPS), mainly limited by the camera’s maximal readout speed.

[0008] The system can operate within the wavelength range from 700 to 1150 nm, amongst the highest bandwidths reported for integrated spectrum polarization devices. The system can have a spectral step size as small as 0.23 nanometers (nm), and can extract complete Stokes polarization information in addition to wavelength data. The system demonstrates the ability to distinguish test images with arbitrary wavelength and polarization distribution, as well as natural objects like beetles, due to its capability to acquire spectral and complete polarization data simultaneously. It is envisioned that the metasurface-enhanced camera system can greatly enhance imaging systems functioning in dynamic, high-speed environments such as surveillance systems, autonomous vehicles, or biomedical systems. This advancement can be leveraged in diverse fields, including biodiversity research, behavioral studies, and environmental monitoring.

[0009] Embodiments can relate to systems and methods for generating hyperspectro-polarimetric (HSP) images with an encoding metasurface and a computational reconstruction model (e.g., machine learning model, such as a trained neural network). The metasurface and accompanying computational reconstruction model can be used in conjunction with a standard image capturedevice (e.g., a camera) to facilitate the generation of HSP images. As will be explained herein, the metasurface and machine learning model can resolve spectral information and all four Stokes polarization parameters across a broad spectral range.

[0010] An exemplary embodiment can relate to a hyperspectro-polarimetric image processing system. The system can include a metasurface configured to encode spectral and polarization image data. The metasurface can include a plurality of super pixels disposed on the metasurface. Each super pixel can include a plurality of arrays of nanostructures, and each array can include a plurality of nanostructures and can be configured such that each array can have different spectral and polarization responses. Each super pixel can be configured to encode spectral and polarization image data into a plurality of distinct spatial intensity distributions, and each distinct spatial intensity distribution can correspond to a predetermined wavelength and / or polarization state. The system can include an image capture device configured to capture each distinct spatial intensity distribution. The image capture device can include at least one image sensor having a plurality of subsets of pixels. Each subset of pixels can correspond to one super pixel such that each subset of pixels can be configured to capture one distinct spatial intensity distribution of the plurality of distinct spatial intensity distributions and to generate pixel data corresponding to each distinct spatial intensity distribution captured by the subset of pixels. The system can include at least one processor in communicative connection with the image capture device. The at least one processor can be configured to receive the pixel data for each subset of pixels of the plurality of subsets of pixels from the at least one image sensor and to generate captured image input data based on the pixel data. The at least one processor can be configured to access at least one computation reconstruction model from a non-transitory memory, to input the captured image input data to the at least one computational reconstruction model, to decode the captured image input data via the at least one computational reconstruction model, and to generate image output data based on the decoded captured image input data via the at least one computational reconstruction model.

[0011] In some embodiments, the at least one computational reconstruction model can be a machine learning model. In some embodiments, the machine learning model can be a trained neural network.

[0012] In some embodiments, the nanostructures can be chiral meta-atoms. In some embodiments, each of the chiral meta-atoms can be at least one of a split-ring-shaped meta-atom or a split-door-shaped meta-atom. In some embodiments, the metasurface can include a predetermined number of chiral meta-atoms designs based on a desired condition number.

[0013] In some embodiments, the at least one computational reconstruction model can be configured to compensate, during the decoding, for aberrations and / or distortion caused by one or more super pixels based on a known location of the one or more super pixels.

[0014] In some embodiments, the metasurface can be disposed on a surface of the at least one image sensor.

[0015] In some embodiments, the system can be further configured to, via the at least one processor, generate one or more hyperspectro-polarimetric images based on the image output data.

[0016] In some embodiments, the at least one computational reconstruction model can be configured to generate image output data for each of one or more linear polarization states, one or more circular polarization states, and one or more elliptical polarization states.

[0017] Another exemplary embodiment can relate to a method for generating one or more hyperspectro-polarimetric images. The method can include encoding, via a metasurface, spectral and polarization image data. The metasurface can include a plurality of super pixels disposed on the metasurface. Each super pixel can include a plurality of arrays of nanostructures, and each array can include a plurality of nanostructures and can be configured such that each array can have different spectral and polarization responses. Each super pixel can be configured to encode spectral and polarization image data into a plurality of distinct spatial intensity distributions, and each distinct spatial intensity distribution can correspond to a predetermined wavelength and / or polarization state. The method can include capturing, via an image capture device, each distinct spatial intensity distribution. The image capture device can include at least one image sensor having a plurality of subsets of pixels. Each subset of pixels can correspond to one super pixel such that each subset of pixels can be configured to capture one distinct spatial intensity distribution of the plurality of distinct spatial intensity distributions and to generate pixel data corresponding to each distinct spatial intensity distribution captured by the subset of pixels. The method can include receiving, by at least one processor in communicative connection with theimage capture device, the pixel data for each subset of pixels of the plurality of subsets of pixels from the at least one image sensor. The method can include generating, by the at least one processor, captured image input data based on the pixel data. The method can include accessing, by the at least one processor, at least one computational reconstruction model stored in a non- transitory memory. The method can include inputting, by the at least one processor to the at least one computational reconstruction model, the captured image input data. The method can include decoding, by the at least one processor via the at least one computational reconstruction model, captured image input data. The method can include generating, by the at least one processor via the at least one computational reconstruction model, image output data based on the decoded captured image input data. The method can include generating, by the at least one processor, the one or more hyperspectro-polarimetric images based on the image output data.

[0018] In some embodiments, the at least one computational reconstruction model can be a machine learning model. In some embodiments, the machine learning model can be a trained neural network.

[0019] In some embodiments, the nanostructures can be chiral meta-atoms. In some embodiments, each of the chiral meta-atoms can be at least one of a split-ring-shaped meta-atom or a split-door-shaped meta-atom. In some embodiments, the metasurface can include a predetermined number of chiral meta-atoms designs based on a desired condition number.

[0020] In some embodiments, the at least one computational reconstruction model can be configured to compensate, during the decoding, for aberrations and / or distortion caused by one or more super pixels based on a known location of the one or more super pixels.

[0021] In some embodiments, the metasurface can be disposed on a surface of the at least one image sensor.

[0022] In some embodiments, the at least one computational reconstruction model can be configured to generate image output data for each of one or more linear polarization states, one or more circular polarization states, and one or more elliptical polarization states.

[0023] In some embodiments, the metasurface can be configured to encode spectral and polarization image data within a spectral range of 700 nanometers to 1150 nanometers.

[0024] Further features, aspects, objects, advantages, and possible applications of the present invention will become apparent from a study of the exemplary embodiments and examples described below, in combination with the Figures, and the appended claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, aspects, features, advantages and possible applications of the present innovation will be more apparent from the following more particular description thereof, presented in conjunction with the following drawings. Like reference numbers used in the drawings may identify like components.

[0026] FIG. l is a schematic block diagram of an exemplary hyperspectro-polarimetric image processing system.

[0027] FIG. 2 is a schematic block diagram of an exemplary hyperspectro-polarimetric image processing system.

[0028] FIG. 3 is a flowchart illustrating an exemplary method for generating one or more hyperspectro-polarimetric images.

[0029] FIG. 4 is an illustration of an exemplary hyperspectro-polarimetric image processing system having an image capture device, an encoding metasurface, and a computational reconstruction model for generating HSP images of a target object.

[0030] FIG. 5 is an illustration of exemplary full-Stokes parameters generated by an exemplary hyperspectro-polarimetric image processing system for a target object across a wavelength range.

[0031] FIG. 6 depicts an illustration and SEM images of an exemplary metasurface configuration.

[0032] FIG. 7 is a graphical representation of the spectral responses of a split-ring-shaped meta- atom for X Y, 45° 135°, and LCP and RCP polarization states.

[0033] FIG. 8 is a graphical representation of transmission matrix values for 100 selected chiral meta-atoms, the transmission matrix having a condition number of 37.15.

[0034] FIG. 9 is an illustration of distinct spatial intensity distributions encoded by a single super pixel for a wavelength range from 745nm to 950 nm for X Y, 45° 135°, and LCP and RCP polarization states.

[0035] FIG. 10 is a graphical representation of a comparison of recovered Stokes spectra results as compared to input states for a wavelength range of 700nm to 1150 nm.

[0036] FIG. 11 is a graphical representation of a comparison of recovered Stokes spectra results as compared to input states for a wavelength range of 760nm to 776nm with a spectral step size of 0.23nm.

[0037] FIG. 12 depicts raw images of a target object captured by a metasurface and recovered HSP images of the target object generated by an exemplary hyperspectro-polarimetric image processing system.

[0038] FIG. 13 is an illustration of an exemplary hyperspectro-polarimetric image processing system having an image capture device, an encoding metasurface, a computational reconstruction model, and a machine learning model for generating HSP images of a target object.

[0039] FIG. 14 depicts raw images of a target object captured by a metasurface and recovered HSP images of the target object generated by an exemplary hyperspectro-polarimetric image processing system.

[0040] FIG. 15 is a graphical representation of camera capture time, data preprocessing time, and machine-learning interface time for 100 tests conducted with an exemplary hyperspectro- polarimetric image processing system.

[0041] FIG. 16 depicts HSP images of a Chrysina Gloriosa green gold scarab beetle across Stokes parameters S0-S3 generated by an exemplary hyperspectro-polarimetric image processing system under laser illumination at 750nm.

[0042] FIG. 17 depicts ground truth photos of a Chrysina Gloriosa green gold scarab beetle taken with a LCP and RCP illuminations as compared to HSP images generated by an exemplary hyperspectro-polarimetric image processing system.

[0043] FIG. 18 depicts LCP and RCP HSP images generated by an exemplary hyperspectro- polarimetric image processing system at 750nm, 780nm, 800nm, 830nm, 850nm, and 890nm.

[0044] FIG. 19A is an illustration of the working principle of a narrowband filter-based approach to distinguish spectral information.

[0045] FIG. 19B is an illustration of the working principle of a broadband filter-based approach to distinguish spectral information.

[0046] FIG. 19C is an illustration of the working principle of metasurface-based multichannel encoders for distinguishing both spectral and polarimetric information.

[0047] FIG. 20 shows an illustration of a library of 100 meta-atom designs and illustrations of tunable geometric parameters for split-ring-shaped and split-door-shaped meta-atoms.

[0048] FIG. 21 is an illustration of an experimental setup for the characterization of an exemplary metasurface.

[0049] FIG. 22A is an illustration of a Poincare Sphere for the graphical representation of a comparison of recovered Stokes spectra results as compared to input states for a wavelength range of 700nm to 1150 nm in FIG. 10.

[0050] FIG. 22B is an illustration of a Poincare Sphere for the graphical representation of a comparison of recovered Stokes spectra results as compared to input states for a wavelength range of 760nm to 776nm with a spectral step size of 0.23nm. in FIG. 11.

[0051] FIG. 23 is an illustration of an experimental setup generating HSP images of a target object with an exemplary hyperspectro-polarimetric image processing system.

[0052] FIG. 24 depicts ground truth images of the target object shown in FIG. 12 for Stokes parameters S0-S3 at 750nm, 800nm, 850nm, 900nm, and 950nm.

[0053] FIG. 25 is a graphical representation of the calculated structural similarity index measures (SSIMS) between the ground truth images of FIG. 24 and the HSP images of FIG. 12.

[0054] FIG. 26A depicts a raw image captured by a CMOS camera.

[0055] FIG. 26B depicts a down-sampled version of the raw image of FIG. 26A.

[0056] FIG. 27A depicts exemplary data preprocessing of segmenting the down-sampled image of FIG. 26B into super pixels.

[0057] FIG. 27B depicts resulting super pixels of the down-sampled image of FIG. 26B generated by the data preprocessing.

[0058] FIG. 28 is an illustration of a Poincare sphere showing 14 polarization states used in the calibration of an exemplary hyperspectro-polarimetric image processing system.

[0059] FIG. 29 is a visualization of elements 12 elements from S0-S3 describing the polarization and spectrum of 6 ground truth image samples.

[0060] FIG. 30 is a visualization of elements 12 elements from S0-S3 describing the polarization and spectrum of 6 HSP image samples generated by an exemplary hyperspectro-polarimetric image processing system.

[0061] FIG. 31 is a visualization of a comparison of the visualization or FIG. 29 and the visualization of FIG. 30.

[0062] FIG. 32 depicts ground truth images of the target object shown in FIG. 14 for Stokes parameters S0-S3 at 750nm, 800nm, 850nm, 900nm, and 950nm.

[0063] FIG. 33 an illustration of an experimental setup for generating HSP images of the Chrysina Gloriosa green gold scarab beetle depicted in FIG. 16 with an exemplary hyperspectro- polarimetric image processing system.

[0064] FIG. 34 depicts HSP images of a side-by-side glass substrate and acrylic plate sample generated by an exemplary hyperspectro-polarimetric image processing system for Stokes parameters S0-S3 at 750nm, 800nm, 850nm, 900nm, 950nm, and lOOOnm.

[0065] FIG. 35 depicts HSP images of a side-by-side glass substrate and acrylic plate sample generated by an exemplary hyperspectro-polarimetric image processing system for Stokes parameters S0-S3 at 700nm, 705nm, 710nm, 715nm, 720nm, 725nm, 730nm, 735nm, 740nm, and 745nm.DETAILED DESCRIPTION OF THE INVENTION

[0066] The following description is of exemplary embodiments that are presently contemplated for carrying out the present invention. This description is not to be taken in a limiting sense, but is made merely for the purpose of describing the general principles and features of the present invention. The scope of the present invention is not limited by this description.

[0067] Referring to FIGS. 1-2, an exemplary embodiment can relate to a hyperspectro- polarimetric image processing system 100. The system 100 can include an image capture device 102. In some embodiments, the image capture device 102 can be a standard camera, e.g., a monochromatic camera. In some embodiments, the image capture device 102 can be a user device having a camera (e.g., cell phone, smartphone, tablet, computer, etc.). It should be understood that the above image capture devices are merely exemplary and that the system 100 can include any suitable type of image capture device. The image capture device 102 can have at least one lens 106 and at least one image sensor(s) 104. The lens(es) 106 can be any type of lens suitable for capturing images. The image sensor(s) 104 can be any suitable type of image sensor. For example, in some embodiments, the image sensor(s) 104 can be a CMOS image sensor. In some embodiments, the image sensor(s) 104 can include a plurality of pixels 104a. In someembodiments, the plurality of pixels 104a can include a plurality of subsets of pixels 104b each configured to generate pixel data based on encoded spectral and polarization data captured by the plurality of subsets of pixels 104b. For example, each subset 104b of the plurality of subsets of pixels 104b can be configured to generate pixel data (e.g., an electric charge / signal) based on light information (e.g., light reflecting off of a target imaging object 101 (FIG. 4)) captured by the subset of pixels 104b.

[0068] The system 100 can include an encoding metasurface 108 configured to encode spectral and polarization image data. In some embodiments, the system 100 can include more than one encoding metasurface 108. In some embodiments, the encoding metasurface 108 can be disposed intermediate the lens(es) 106 and the image sensor(s) 104. In some embodiments, the metasurface 108 can be disposed directly on the image sensor(s) 104. In some embodiments, the metasurface 108 can be disposed directly on the lens(es) 106. It should be understood that the above configurations are merely exemplary and that the metasurface 108 can be physically configured within the system 100 in any configuration suitable for encoding spectral and polarization image data.

[0069] Referring now to FIGS. 1-2 and 4-6, in an exemplary embodiment, the metasurface 108 can include a plurality of super pixels 108a. In some embodiments, the metasurface 108 be a 2- dimensional (2D) grid including 20 x 20 super pixels 108a. Each super pixel 108a can include a plurality of arrays 108b of nanostructures, such as meta-atoms 108c. For example, in some embodiments, each super pixel 108a can include a 2D 10 x 10 grid of arrays 108b, that is, a total of 100 arrays 108b arranged in a 10 x 10 grid. Each array 108b can include a plurality of meta- atoms 108c. In some embodiments, each array 108b can include a 10 x 10 grid of meta-atoms (e.g., 100 total meta-atoms 108c) arranged in a periodic fashion. For example, each of the meta- atoms 108c can be spaced apart from one another in a periodic fashion with a predetermined separation distance, such as 80 nanometers (nm), to mitigate or eliminate strong near-field coupling between meta-atoms. In some embodiments, the metasurface 108 can include meta- atoms 108c disposed on an ITO glass substrate, for example, the meta-atoms 108c can be silicon meta-atoms and can be disposed on (e.g., etched into a layer of silicon disposed on the ITO glass) the metasurface, as further described below.

[0070] It should be understood that such physical configuration(s) and / or material composition(s) of the metasurface 108 and / or meta-atoms 108c is merely exemplary, and that the system 100 can include any suitable metasurface 108 and / or meta-atom 108c configuration(s) and / or composition(s). For example, in some embodiments, the metasurface 108 can include any suitable transparent material within a desired spectral band, such as silicon carbide (SiC), sapphire glass, silicon (Si), zinc selenide (ZnSe), etc. In some embodiments, the meta-atoms 108c can be any meta-atoms material suitable for encoding spectral and polarization image data, such as gold (Au), silver (Ag), aluminum (Al), dielectric materials (e.g., titanium dioxide (TiO2), Silicon Nitride (Si3N4), Gallium Nitride (GaN), etc.), etc.

[0071] In an exemplary embodiment, each array 108b can be configured to act as a unique polarization and / or wavelength encoder. For example, each array 108b can include a unique configuration of meta-atoms 108c relative to the other arrays 108b, such that each array 108b exhibits a unique response to different wavelengths and polarizations. The configuration of meta-atoms 108c for each array 108b can be uniquely configured, relative to each other array 108b, based on the physical arrangement / orientation and / or geometric parameters of the plurality of meta-atoms 108c included in each array 108b. For example, each array 108b can include a plurality of meta-atoms 108c each having the same / similar design (e.g., a plurality of met-atoms 108c each having the same physical arrangement / orientation and geometric parameters) that are arranged periodically (e.g., repeated and spaced apart), such as in a 10 x 10 grid orientation.

[0072] In other words, each array 108b can include a plurality (e.g., 100) of meta-atoms 108c, each meta-atom 108c having the same / similar physical arrangement / orientation and geometric parameters, and each array 108b can include a different meta-atom 103c design (e.g., physical arrangement / orientation and / or geometric parameters) relative to each other array 108b, such that each array 108b can act as a unique polarization and / or wavelength encoder. In some embodiments, each super pixel 108a can include the same / similar plurality of arrays 108b arranged in the same / similar configuration. That is, each super pixel 108a can include the same / similar configuration of the plurality of arrays 108b.

[0073] In some embodiments, as depicted in FIG. 20, each meta-atom 108c design can be a chiral shape (e.g., split-ring shape 103a, split-door shape 103b, Z shape, S shape, etc.) having varied structural parameters (e.g., length, width, gap distance, diameter, radius, etc.) and / orphysical arrangement / orientation (e.g., rotated, mirrored, etc.), such that the metasurface 108 can exhibit strong chirality and anisotropy, as further described below. For example, each array 108b can include a met-atom 108c design having different structural / geometric parameters and / or physical orientation / arrangement (e.g., rotated, mirrored, etc.) relative to each other array 108b, such that each array 108b exhibits distinct responses to different wavelengths and polarizations, which can determine the resolving power of reconstructive spectrometer polarimetry. It should be understood that the above configurations are merely exemplary and that the metasurface 108 can include any suitable configuration / design / number of super pixels 108a, arrays 108b, and / or meta-atoms 108c.

[0074] In some embodiments, the structural and geometric parameters, quantity, and / or physical orientation of meta-atoms 108c selected for each array 108b can be determined based on a target condition number of a response matrix. In this response matrix, each four columns comprise the spectral response and polarization response as quantified by the four Stokes parameters (S0-S3) of a corresponding meta-atom 108c design, and the condition number provides a measure of the degree of distinguishability among the response of the meta-atoms 108c, as further described below. For example, in some embodiments, the selected meta-atom 108c designs (e.g., shapes, physical arrangements / orientations, geometric parameters, etc.) can be chosen to achieve a target condi ti on numb er of 37.15.

[0075] Each super pixel 108a can be configured, as described above, such that each super pixel 108a can encode spectral and polarization data (e.g., light signals in a captured raw image) into a plurality of distinct spatial intensity distributions 105 (FIG. 9), and each distinct spatial intensity distribution 105 can correspond to a different wavelength and / or polarization. In some embodiments, each super pixel 108a of the metasurface 108 can be configured to encode spectral and polarization data for a segment of the captured raw image corresponding to the location of each super pixel 108a relative to the metasurface 108. That is, a raw image can be segmented into a number of segments equal to the number of super pixels 108a, such that each super pixel 108a can encode the plurality of distinct spatial intensity distributions 105 for its corresponding segment of the captured raw image. For example, as depicted in FIG. 9, each distinct spatial intensity distribution 105 (depicted as individual tiles) can be encoded by one super pixel 108a of the metasurface 108, with each of the distinct spatial intensity distributions 105 depicting thespatial intensity distribution for the segment of the captured raw image corresponding to the one super pixel 108a at various spectral wavelengths and polarizations.

[0076] Each distinct spatial intensity distribution 105 includes the spectral and polarization data encoded by its corresponding super pixel 108a and can be captured by the image sensor(s) 104 of the image capture device 102. For example, in some embodiments, each subset of pixels 104b of the plurality of subsets of pixels 104b can correspond to, or be otherwise associated with, one super pixel 108a of the metasurface 108, such that each subset of pixels 104b can capture the distinct spatial intensity distributions 105 encoded by the super pixel 108a to which the subset of pixels 104b corresponds. In other words, each subset of pixels 104b can be configured to generate pixel data (e.g., electric charges / signals) corresponding to, and / or based on, the distinct spatial intensity distributions 105 captured by the subset of pixels 104b.

[0077] The system 100 can include at least one processor 110. The processor(s) 110 can include any type of processor suitable for processing light information and / or pixel data, such as classical processors (e.g., CPUs, GPUs, DSPs, etc.), quantum processors, etc. The processor(s) 110 can be configured to receive the pixel data from each subset of pixels 104b of the plurality of subsets of pixels 104b. For example, in some embodiments, the processor(s) 110 can be communicatively connected (e.g., via USB, Bluetooth, WiFi, Cellular Network, etc.) to the image capture device 102 and configured to receive, from the image capture device 102 (e.g., from the image sensor(s) 104) pixel data for each subset of pixels 104b of the plurality of subsets of pixels 104b. In some embodiments, the processor(s) 110 can be a part of the image capture device 102 and can be in communicative connection with one or more additional processors 114 (FIG. 2).

[0078] In an exemplary embodiment, the processor(s) 110 can be configured to receive the pixel data for each subset of pixels 104b of the plurality of subsets of pixels 104b. For example, the processor(s) 110 can be a part of, or in communicative connection with, the image capture device 102 such that the processor(s) 110 can receive the pixel data from the image sensor(s) 104. The processor(s) 110 can be configured to generate captured image input data (Fig. 13) based on the received pixel data. In some embodiments, the captured image input data can include the location (e.g., position encoding) of each super pixel 108a (e.g., relative to the metasurface), and the pixel data for each distinct spatial intensity distribution 105 captured by the subset of pixels 104b corresponding, or otherwise associated with, each super pixel 108a.

[0079] For example, the processor(s) 110 can receive the pixel data corresponding to, and / or based on, each distinct spatial intensity distribution 105 and can generate a raw image (FIG. 13) based on the received pixel data. The processor(s) 110 can then preprocess (e.g., via the computational reconstruction model(s) 112) the raw image to generate the captured image input data. For example, the processor(s) 110 can be configured to (e.g., via the execution of one or more algorithms stored in a memory 116) partition the raw image captured by the image sensor(s) 104 into a plurality of tiles having an encoded location (e.g., location relative to the raw image), as depicted in FIGS. 13 and 26A-27B. For example, each tile can correspond to one super pixel 108a of the metasurface 108, such that the raw image is divided into a number of segments equal to the number of super pixels 108a of the metasurface 108, and the encoded location for each tile can correspond to the location of each super pixel 108a of the metasurface 108. In some embodiments, the raw image may undergo additional pre-processing prior to being partitioned, such as being down-sampled, resized, etc., to increase efficiency / reduce computational requirements of the system 100, as shown in FIG. 26B and further described below.

[0080] In an exemplary embodiment, the location / spatial position of each tile of the plurality of tiles can be encoded using the one-hot method, as further described below. Each tile of the plurality of tiles can include the distinct spatial intensity distribution(s) 105 encoded by one super pixel 108a of the metasurface 108. In an exemplary embodiment, the captured image input data can include each tile of the plurality of tiles (e.g., the distinct spatial intensity distribution(s) 105) and its corresponding location / position within the raw image (e.g., the location of its corresponding super pixel 108a relative to the metasurface 108).

[0081] Referring now to FIGS. 1-2, 4, 13, and 19C, the processor(s) 110 can be configured to access at least one computational reconstruction model 112 stored in a memory in communicative and / or wired connection with the processor(s) 110, such as memory 116. In some embodiments, the computational reconstruction model(s) 112 can include at least one machine learning model 112a. The processor(s) 110 can be configured to input the captured image input data to the machine learning model(s) 112a. The processor(s) 110 can be configured to, via the machine learning model(s) 112a, decode the captured image input data and / or generate image output data based on the decoded captured image input data. In some embodiments, themachine learning model 112 can be a neural network, such as trained neural network 112a. In some embodiments, the processor(s) 110 can be configured to generate, and / or store in the memory 116, the at least one computational reconstruction model 112.

[0082] In an exemplary embodiment, the trained neural network 112a can be a multi-layered neural network having one or more hidden layers. For example, the trained neural network 112a can have a first hidden layer that acts as a position-dependent system calibration layer, which can assign different weights to super pixels based on their positions, and can have two subsequent hidden layers, which can serve as hyperspectro-polarimetric image recovery layers that can decode the spectra and polarization information from each tiles of the plurality of tiles, as described in further detail below. It should be understood that the above neural network configuration is merely exemplary, and that the system 100 can include any suitable neural network configuration / architecture for decoding spectra and polarization data and generating image output data. In some embodiments, the trained neural network 112a can be configured to identify and correct local aberrations and / or distortions (e.g., compensate for during decoding) based on the known location / position of each superpixel 108a. That is, the trained neural network can be trained using previous captured image data encoded by the metasurface 108 and can learn, e.g., based on repetitive training, of the existence of local aberrations and / or distortion that are caused by a given super pixel 108a, for example, due to fabrication errors in the construction of the given super pixel 108a, and can compensate for or correct said aberrations and / or distortion in future image input data decoding processes.

[0083] In some embodiments, the trained neural network 112a can be generated by the processor(s) 114 and can be in communicative connection (e.g., wired and / or wireless) with the image capture device 102 (e.g., processor(s) 110) such that the trained neural network 112a is configured to receive the captured image input data from the image capture device 102 for decoding.

[0084] In an exemplary embodiment, once the trained neural network 112a has decoded the encoded spectra and polarization information for the received image input data (e.g., via the processor(s) 110), the trained neural network 112a can be configured to generate image output data (e.g., via the processor(s) 110), that can be recovered / assembled (e.g., by the processor(s) 110, the computational reconstruction model(s) 112, etc.) into one or more HSP images. Forexample, the image output data can include the full Stokes parameters (e.g., So, Si, S2, S3,) within a specified wavelength range for each super pixel 108a (e.g., the hyperspectro-polarimetric response of each superpixel 108a), and the spatial location / coordinates (e.g., location relative to the entirety of the captured raw image). The image output data can then be assembled (e.g., by the processor(s) 110, the computational reconstruction model(s) 112, etc.) into one or more HSP images based on desired wavelength and / or Stokes parameters. In other words, Full-Stokes parameters across a range of wavelengths can be generated for each segment or portion of a raw captured image, as captured by the superpixel 108a corresponding to, or otherwise associated with, each segment or portion of the captured raw image, and assembled (based on the known position / location of each superpixel 108a) into an HSP image at a desired wavelength and Stokes parameter.

[0085] In some embodiments, the system 100 can include one or more displays 118 (FIG. 2) in communicative connection (e.g., wired and / or wireless) with the processor(s) 110, such that the processor(s) 110 can be configured to display the assembled HSP images on the display(s) 118. In some embodiments, the image input data, image output data, and / or assembled HSP images can be stored in a memory of the system 100, such as memory 116.

[0086] Referring now to FIG. 3, an exemplary embodiment can relate to a method 200 for generating one or more hyperspectro-polarimetric images. The method 200 can utilize any of the above-described embodiments and / or elements of the hyperspectro-polarimetric image processing system 100 for generating one or more hyperspectro-polarimetric images, the details of which are not repeated here in the interest of brevity. The method 200 can include first step 202, encoding, via a metasurface, spectral and polarization image data. In some embodiments, the metasurface can include a plurality of super pixels, and each super pixel can be configured to encode the spectral and polarization image data into a plurality of distinct spatial intensity distributions.

[0087] The method 200 can include second step 204, capturing, via an image capture device, each spatial intensity distribution. In some embodiments, the image capture device can include at least one image sensor having a plurality of subsets of pixels. In some embodiments, each subset of pixels of the image sensor can correspond to one super pixel such that each subset of pixels can be configured to capture the distinct spatial intensity distribution(s) encoded by onesuperpixel and to generate pixel data corresponding to the captured distinct spatial intensity distribution(s).

[0088] The method 200 can include third step 206, receiving, by at least one processor in communicative connection with the image capture device, the pixel data for each subset of pixels of the plurality of subsets of pixels of the at least one image sensor. The method 200 can include fourth step 208, generating, by the at least one processor, captured image input data. The method 200 can include fifth step 210, accessing, by the at least one processor, at least one computational reconstruction model stored in a memory. The method 200 can include sixth step 212, inputting, by the at least one processor to the at least one computational reconstruction model, the captured image input data. The method 200 can include seventh step, 214, decoding, by the at least processor via the at least one computational reconstruction model, the captured image input data.

[0089] The method 200 can include eighth step 216, generating, by the at least one processor via the at least one computational reconstruction model, image output data based on the decoded captured image input data. The method 200 can include ninth step 218, generating, by the at least one processor, the one or more hyperspectro-polarimetric images based on the image output data. In some embodiments, the method 200 can include an additional step (not shown), displaying the one or more hyperspectro-polarimetric images on a display. In some embodiments, the method 200 can include an additional step (not shown), storing the one or more hyperspectro-polarimetric images in a memory.

[0090] The processor(s) 110 / 114 can be any of the processors disclosed herein. The processor(s) 110 / 114 can be part of or in communication with a machine (logic, one or more components, circuits (e.g., modules), or mechanisms). The processor(s) 110 / 114 can be hardware (e.g., processor, integrated circuit, central processing unit, microprocessor, core processor, computer device, etc.), firmware, software, etc. configured to perform operations by execution of instructions embodied in algorithms, data processing program logic, artificial intelligence programming, automated reasoning programming, etc. Use of processors 110 / 114 herein can include any one or combination of a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a Digital Signal Processor (DSP), etc. The processor(s) 110 / 114 can include one or more sub-processors or processing modules. Asub-processor or processing module can be a software or firmware operating module configured to implement any of the method steps disclosed herein. The sub-processor or processing module can be embodied as software and stored in memory 116, the memory 116 being operatively associated with the processor(s) 110 / 114. A sub-processor or processing module can be embodied as a web application, a desktop application, a mobile (smartphone) application, a console application, etc.

[0091] The processor(s) 110 / 114 can include or be associated with a computer or machine readable medium. The computer or machine readable medium can include memory 116. The computer or machine readable medium can be configured to store one or more instructions thereon. The instructions can be in the form of algorithms, program logic, a model, etc. that cause the processor(s) 110 / 114 to perform any of the functions described herein.

[0092] Any of the memory 116 discussed herein can be computer readable memory configured to store data. The memory 116 can include a volatile or non-volatile, transitory or non-transitory memory, and be embodied as an in-memory, an active memory, a cloud memory, etc. Embodiments of the memory 116 can include a sub-processor or processor module and other circuitry to allow for the transfer of data to and from the memory 116, which can include to and from other components of a communication system. This transfer can be via hardwire or wireless transmission. The communication system can include transceivers, which can be used in combination with switches, receivers, transmitters, routers, gateways, wave-guides, etc. to facilitate communications via a communication approach or protocol for controlled and coordinated signal transmission and processing to any other component or combination of components of the communication system. The transmission can be via a communication link. The communication link can be electronic-based, optical-based, opto-electronic-based, quantum- based, etc.

[0093] The processor(s) 110 / 114 can be in communication with other processors of other devices (e g., a computer device, a desktop computer, a laptop computer, a computer system, etc.). Any of those other devices can include any of the exemplary processors disclosed herein. Any of the processors 114 can have transceivers or other communication devices / circuitry to facilitate transmission and reception of wireless signals. Any of the processors 110 / 114 can include an Application Programming Interface (API) as a software intermediary that allows twoapplications to talk to each other. For example, use of an API can allow software of the processor 110 of the image capture device 102 to communicate with software of the processor 114 of the other device(s).

[0094] Any data transmission between the processor(s) 110 / 114 and memory 116, between the processor(s) 110 / 114 and a database, and between the processor(s) 110 and processor(s) 114 of other devices, etc. can be via a pull operation (e.g., the processor(s) 110 / 114 can pull the data) or a push operation (e.g., the data can be pushed to the processor(s) 110 / 114). The processor(s) 110 / 114 can receive the data in streaming format, or store it in memory 116 before being processed. In addition, embodiments of the algorithm, model, etc. disclosed herein can be developed as an application software (an “App”) to be implemented on a processor(s) 110 / 114 of a device. The App can be sent via a streaming format, or the App can be sent and stored on a memory associated with or accessed by the device.

[0095] As noted herein, the processor(s) 110 / 114 can be configured to be a component of, used in combination with, or in communication with another device / system - e g., this can include the processor(s) 110 / 114 being part of the device / system, the device / system being part of the processor(s) 110 / 114, the processor(s) 110 / 114 in communication with the device / system, etc. “Being part of’ can include being on the same substrate or integrated circuit. For instance, the processor(s) 110 / 114 can be a component of, used in combination with, or in communication with a predictive modeling system, a decision support system, an automated control system, etc. processor(s) 110 / 114 can use a model (e.g., machine learning model) or algorithm disclosed herein or provide the model or algorithm to the device / system to assist with or augment the performance of these devices / sy stems.

[0096] EXAMPLES

[0097] The following are exemplary compositions, devices, methods, and implementations of the embodiments disclosed herein. While the examples may focus on one implementation, it is understood that this is exemplary and the embodiments disclosed herein are not limited thereto.

[0098] EXAMPLE 1

[0099] The metasurface features thousands of spectro-polarimetric super pixels, each comprising arrays of judiciously designed chiral meta-atoms. The metasurface was placed in conjunction of the detector chip of a standard monochromatic camera, enabling it to imageobjects with both wavelength and polarization data. In contrast to traditional narrowband filter- based spectrometers or single-channel polarizers – where each wavelength or polarization state is assigned to a single pixel in the spatial domain – the metasurface can multiplex all spectro- polarimetric channels through a single super pixel (See Equation S1 below). Each array of meta- atoms acts as a unique polarization and wavelength encoder, with h(λ, S) denoting its transmission function. By combining multiple arrays with distinct responses, the spectro- polarimetry that functions as an image pixel for the HSP camera system can be established. The intensity information I(x,y) captured by the camera is subsequently decoded by a ML backend, producing four-dimensional (4D) images f’ (x, y, λ, S) of the input. FIG.5 shows an artistic depiction of such a 4D image, with distinct wavelength and polarization views, each emphasizing different features of the objects.

[0100] The image of an object (e.g., a fish tank), which has spectral and polarization information denoted by f(x, y, λ, S) in FIG.4, is transformed into a 2D intensity mapping, I(x, y), by the hyperspectro-polarimetric (HSP) camera. This HSP camera system comprises two main components: encoding metasurfaces at the frontend to encode the information, and a ML algorithm at the backend to decode the information. Through the HSP camera, the encoded 2D intensity information of the tank image is recovered into 4D information f’ (x, y, λ, S), which ideally should equal to the real spectral and polarization information f(x, y, λ, S) of the object. Certain parts of this image (like the fish and the rock) in S2 become more pronounced because they reflect more 45° or 135° linear polarizations. The encoding metasurfaces encompass multiple wavelength and polarization encoders h(λ, S), which can be flexibly combined to construct an HSP super pixel T(x, y, λ, S) according to the required resolution.

[0101] The encoding process of spectro-polarimetry by the metasurface can be elucidated using a mathematical model. The wavelength and polarization information are encoded such that the intensity received by each camera pixel is the integral of the entire spectrum, weighted by the spectral and polarization response of the corresponding meta-atoms directly above it. The polarization state is described with the Stokes parameters the transmitted intensity received by camera pixels behind the i-th spectro-polarization encoder can be expressed as:where n represents the number of detectors and m denotes the number of wavelength sampling points. Although mathematically the spectral and polarization information can be recovered by directly pseudo-inverting the transmission matrix T, the recovery may be compromised, leading to inaccurate results, due to various unavoidable errors and uncertainties, such as calibration noise, measurement noise, and nonuniform responses of the camera pixels, etc. To address this, employed linear regression was first employed with the least-square optimization method to retrieve spectral and polarization information, which can be formulated asSubject toThe objective is to minimize the discrepancy between the measured intensity imeasand the intensity Tfscalculated from the transmission matrix. It subjects to both linear and nonlinear constraints because the Stokes parameters must satisfy the inequities as shown in Eq. (3). These constraints introduce nonconvexity to the problem, making it challenging to solve. It can be addressed using local optimization methods like the interior-point method, or using globaloptimization strategies such as particle swarm algorithms. While the latter might produce better images, it comes at the expense of increased computational time. To further enhance image quality, advanced algorithms incorporating sparsity in both time and space dimensions can be utilized. However, this approach is considerably more time intensive. Therefore, a method based on the ML neural network was developed to perform fast decoding of spectral and polarization information.

[0102] Machine Learning-Based Fast Decoding

[0103] Machine Learning (ML) can approximate complex functions given sufficient training data and it does not necessitate an explicit physical model. Here a customized ML network can be used to reconstruct HSP images accurately and quickly. Given training data, the model can learn the mapping relationship between an encoded input image and its corresponding wavelength and polarization information. In some embodiments, each super pixel in the captured image corresponds to a unique spectrum and polarization information, independent of the other pixels.

[0104] For traditional optimization methods, a unique model is required for each super pixel due to fabrication-induced non-uniformity across the super pixels and optical aberrations of the system. As a result, thousands of models are required, leading to high computational costs. In contrast, with the proposed ML approach, a unified network can be established which takes in both the position encodings of the super pixels and the encoded intensity distributions for polarization and spectrum reconstruction. This way, all super pixels can be processed using a single network. Since the spatial locations of the super pixels are also part of the input, the network can learn the effects of the non-uniformity and aberrations and compensate for it. Once the ML model is trained, the decoding process only requires simple calculations, making real- time recovery of HSP images possible.

[0105] Design and Characterization of the Metasurface

[0106] As depicted in FIGS. 5-6, the metasurface 108 for the HSP camera system 100 can be composed of thousands of spectro-polarimetric encoders (super pixels, e.g., 108a), each consisting of 10x 10 arrays (e.g., 108b) of meta-atoms (e.g., 108c). For discerning the multichannel information, each meta-atom array exhibits a unique response to different wavelengths and polarizations. The meta-atoms are silicon-based and have shapes of split ringsor split doors. Such designs manifest not only anisotropy but also strong chirality. For example, the split ring design features two unequal splits and arm lengths. The coupling between the bright mode (the electric dipole resonance supported by the arms) and the dark mode (the magnetic dipole resonance supported by the entire ring) leads to a sharp Fano resonance, essential for achieving high spectral resolution. In addition, the split ring also exhibits strong chirality, which can be vital for distinguishing left- and right-handed elliptical / circular polarizations. A large library of meta-atoms was established by varying the structural parameters of the split ring and split door designs while keeping their height and period constant (see Note S2 below). Numerical simulations were performed using a commercial finite element method solver package, COMSOL Multiphysics, to calculate the spectral response with the x and y linear polarization inputs, respectively.

[0107] Each meta-atom is modeled in a periodic array where the period is chosen to give at least 80 nm distance between neighboring meta-atoms to avoid strong near-field coupling. The transmission of the meta-atoms for different polarization and different wavelength were used to obtain the Jones matrix, which was then converted to the Muller matrix. FIG. 7 shows a typical response from a meta-atom, showcasing rich spectral features for different polarization inputs. This highlights its capability to distinguish full-Stokes polarization and wavelength information. For added diversity to the library, structures with identical geometric parameters but modified by either a 90-degree rotation, a mirrored orientation, or both were incorporated. In total, the meta- atom library comprises 1,936 split-door shapes and 1,294 split-ring shapes.

[0108] FIG. 6 shows an artistic illustration of the metasurface 108 and SEM images of the fabricated metasurface. Scale bars from top right to bottom right: 1 pm, 1 pm, and 200 pm. FIG.7 shows spectral response for different polarization inputs for a typical split-ring-shaped meta- atom. FIG. 8 shows the transmission matrix for the selected 100 chiral meta-atoms, with the condition number (CN) minimized to 37.15, indicating good distinguishability of different meta- atom’s spectral and polarization responses. FIG. 9 depicts exemplary intensity responses of super pixels. Each small box represents the intensity response of a single super pixel, which consists of a 10x 10 array of meta-atoms, as depicted in FIG. 6. The intensity responses under different wavelength and polarization illuminations captured using a regular camera. For demonstration purposes, the intensity response of one typical super pixel is shown in FIG. 9.

[0109] It can be essential to select meta-atom designs that exhibit distinct responses to different wavelengths and polarizations, as the dissimilarity between the responses of the structures determines the resolving power of the reconstructive spectrometer polarimetry. If two designs produce similar responses, it becomes challenging to discern the wavelength and polarization. Mathematically, the response of each meta-atom design is represented by a column in the transmission matrix, T, described in Eq. (2). Consequently, the condition number of the transmission matrix indicates the similarity of the responses. 100 designs from the meta-atom library were chosen to construct the metasurface so that the resulting transmission matrix has a minimum condition number (as shown in FIG. 8). As depicted in FIG. 6, each meta-atom (e.g., 108c) design can be arranged in a 9 pm x 9 pm 2D array (e.g., 108b). In total, there are 100 different arrays.

[0110] These arrays were then organized into a two-dimensional grid to construct a 90 pm x 90 pm super pixel (e.g., 108a). Each metasurface can consist of 20 x 20 super pixels, which can also be arranged in a two-dimensional grid. When aiming for different spatial, spectral and polarization resolution, the super pixel size and number can be adjusted accordingly. After fabricating the metasurface (see the detailed fabrication process in Methods), the transmission of the pixels was calibrated under a variety of polarization and wavelength combinations (see the detailed setup in FIG. 21). FIG. 9 shows the results of 30 different wavelengths and 6 incident polarizations. It was clearly observed that different wavelengths and polarizations exhibit distinct intensity distribution patterns, as captured by the camera (DMK 33GX265). These patterns were then used to determine the transmission matrix or for training the ML neural network.

[0111] Optimization-Based Recovery

[0112] The capability of the HSP camera system was first demonstrated using optimization- based recovery. By knowing the captured intensity distribution along with its corresponding input, the transmission matrix, T, of each super pixel can be calculated based on Eq. (2). Utilizing the transmission matrix, it was demonstrated that the spectral polarimetry of unknown incident beams could be retrieved through the optimization-based recovery method described in Eq. (3).

[0113] An input beam covering a broad wavelength range was prepared by combining two laser beams: one with left circular polarization (LCP) at 800 nm and the other with x-polarization at1100 nm. After passing through the metasurface, the encoded patterns were captured by the camera. FIG. 10 shows the recovered Stokes spectra ranging from 700 nm to 1150 nm, aligning well with those measured directly by a spectrometer with Stokes analysis (FIGS. 22A-B). It was demonstrated that the device operates over a bandwidth of 450 nm, which is limited by the range of the tunable laser source used in the experiment.

[0114] Furthermore, to achieve high spectral resolution, the metasurface was calibrated with a spectral step size of 0.23 nm, limited by the spectrometer used. An input beam was introduced with wavelengths ranging from 760 nm to 776 nm, with x-polarization at 761.8 nm and right circular polarization (RCP) at 770.2 nm. The recovered Stokes spectra again showed that the recovery results matched the input states with a spectral sensitivity of 0.23 nm (FIG. 11). The recovered spectra resolved minor oscillating peaks, demonstrating the accuracy and high spectral resolving power of the system.

[0115] Spectro-Polarimetry and Hyperspectro-Polarimetric Imaging Using Optimization Methods

[0116] FIG. 10 depicts recovery results for a broadband wavelength range (700 nm to 1150 nm). FIG. 11 depicts recovery results from a wavelength range of 760 nm to 776 nm with a step size of 0.23 nm. FIG 12 depicts three raw images captured by the metasurfaces and HSP imaging recovery results of the three letters ‘PSU’ contained in the raw images that are recovered with the optimization methods. Scale bar: 600 pm. The size of the raw image is 1536x2048 pixels, while the size of each retrieved HSP image is 42* 17 pixels.

[0117] To validate the imaging performance, HSP imaging was conducted with a test target. The test target consists of the letters ‘PSU’ (see the fabrication process in Methods) with each letter being illuminated by laser beams of different polarizations and wavelengths. An inverse telescope system, comprising two lenses (fl = 50 mm and f2 = 150 mm), projected the target onto the metasurface and camera (FIG. 23). The captured raw images and the recovered HSP images are shown in FIG. 11. Clearly, the three ‘PSU’ letters exhibit wavelengths of 950 nm, 850 nm, and 750 nm, respectively. Additionally, the ‘P’ letter is in x-polarization, ‘U’ in -45- degree polarization, and ‘S’ in LCP. These results are sharp, with low background noise, and closely match the ground truth. For single image recovery, both local and global optimization methods were tested, which required three minutes and around two hours, respectively. Toquantitatively evaluate the quality of the recovered HSP images, the structural similarity index measure (SSIM) was calculated between the results shown in FIG. 12 and their corresponding ground truth. It was observed that the SSIM values (FIG. 25) are almost all greater than 0.6 for all Stokes’ parameters at all wavelengths, with an average of about 0.9. Notably, at some wavelengths, the SSIM values are close to unity because the images are almost blank.

[0118] The high SSIM values indicate the recovered HSP images capture most of the features from the testing object. However, the results in FIG. 12 still exhibit noise and artifacts. These issues likely stem from several possible sources. First, misalignment of optical components introduced phase errors and misfocus, and power fluctuations of the light source resulted in varying illumination intensity and its distribution during the calibration and imaging experiments.

[0119] FIGS. 10-12 show the hyperspectro-polarimetric responses and imaging of the three letters£PSU,’ and depict the results retrieved using the linear regression with the least squares optimization method. This method, while effective in certain scenarios, can be slow and highly susceptible to noise and errors, and can lead to reduced accuracy and sometimes severe artifacts. In contrast, the neural network (NN)-based approach, used for the retrieval of the rest of the images and for real-time retrieval, offers significant advantages. The NN can learn complex patterns and relationships within the data, making it more robust against noise and errors, resulting in higher-quality image reconstruction. Additionally, the NN can perform inference much faster than the optimization method, enabling real-time data HSP image recovery.

[0120] Neural Network Architecture, Data Synthesis, and Training

[0121] To accelerate the recovery process, a ML neural network was designed to decode the image information. The workflow of the decoding process is depicted in FIG. 13. Once the camera captures an image, it is divided into small tiles, with each tile containing the light intensity distribution after passing through one super pixel. The spatial positions of the tiles are encoded using the one-hot method. Both the tiles and their encoded positions serve as inputs for the neural network. The neural network architecture comprises three hidden layers. Both the number of layers and the number of neurons in each layer were minimized to achieve a compact design, thus optimizing the processing speed. The first hidden layer acts as a position-dependent system calibration layer, assigning different weights to super pixels based on their positions. Thesubsequent two hidden layers, serving as HSP image recovery layers, decode the spectra and polarization from each tile. The outputs are then assembled into HSP images.

[0122] To prepare the training dataset, the first step is to collect experimental calibration data. This was done by uniformly illuminating the metasurface with light at different wavelengths and polarization states and then capturing the corresponding output images. A total of 5 wavelengths spanning from 750 nm to 950 nm with an interval of 50 nm were used for FIG. 14 (for FIG. 18, seven wavelengths were used), and at each wavelength 14 polarization states were prepared as the input, producing a total of 70 (5>< 14) calibration images. The 14 polarization states were selected to represent a uniform sampling on the surface of the polarization Poincare sphere (see

[0123] To train a neural network, a large quantity of training data is required; however experimentally generating such large training datasets is time-consuming, costly, and often not practical. The next step is to augment the calibration data into training and validation datasets. A training image can be synthesized as follows:where cks are random real numbers uniformly distributed within [0,1.2] and rk's are random integers selected from 1 to 14 with equal probability. In other words, through linear combinations of the calibration data augmented training datasets can be generated with new wavelength / polarization combinations to train the neural network. The positions of these super pixels (z,j) can also be embedded within the images input into the training set in the form of one- hot encoding. This positional information can reduce errors caused by optical aberrations, metasurface fabrication defects, and other imperfections. The final size of the training set is 1.8 million. The validation dataset was constructed similarly, with the only difference being that the random weights ck's are uniformly distributed within (1.2, 1.5). This ensures that the validation data is different from the training data. The final size of the validation dataset is 180k.

[0124] The training and validation datasets were then used to develop the ML model. The mean square error (MSE) was reduced to 10-4 upon convergence. The training time was around 12hours using a computer equipped with an Intel i9-9900K CPU, an NVIDIA GeForce RTX 2080 Ti (11 GB) GPU, and 32 GB of RAM. The ML model was validated with the synthesized validation data (FIGS. 29-31), and the recovery results matched well with the ground truth. The recovery process can be significantly faster compared to the optimization-based methods, enabling real-time video recovery. During inference, processing each raw image to obtain the HSP image took less than 0.01 seconds, corresponding to a processing rate of over 100 frames per second (FPS).

[0125] The testing dataset was prepared experimentally. The letters (‘PSU’), a cartoon image, and beetle specimens were used as testing samples. These samples were inserted into the illumination beam and placed in front of the metasurface; the response of the metasurface was then captured. For each sample, experimental data for images at different wavelengths and polarizations was collected. The testing data was directly used to evaluate the trained neural network model, mimicking real-world applications, data augmentation was not performed on these experimental test images. The testing procedures are shown in FIGS 13-15.

[0126] FIG. 13 shows the architecture of the computational backend of the full-stokes HSP camera system. The images captured from the HSP camera are cropped to a resolution of 1505 x 1764 pixels to align with the metasurface areas. Subsequently, these images are divided into segments of 42 x 50 super pixels. These segmented pixels, along with position encoding, are used as input for the ML model. The model, which includes three hidden layers, is responsible for position-dependent system calibration and HSP image processing. The output, consisting of 12 x 2100 values, is then assembled to form HSP images. FIG. 14 shows image reconstruction results of a cartoon figure. Horizontally polarized illumination light at 750 nm is used. The left subset shows the raw image for the cartoon figure. Scale bar: 600 pm. FIG. 15 illustrates the camera capture time, data preprocessing, and machine-learning interface time for 100 tests. FIG. 32 depicts the ground truth for FIG. 14.

[0127] To demonstrate the ML-assisted real-time recovery, a cartoon figure and the ‘PSU’ letters were used as test targets for HSP imaging (See experimental setup in FIG. 23). The cartoon figure was illuminated with linearly polarized light at 750 nm. Its HSP image, recovered by the ML-assisted method from a single snapshot, clearly shows the cartoon figure along with the wavelength and polarization information (FIG. 14). This method enables the acquisition ofcomplete Stokes parameters, facilitating the accurate analysis of all polarization states, including circular or elliptical ones. This contrasts with other reported methods that can only ascertain linear polarization states. The results in FIG. 14 face similar quality and artifacts like FIG. 12.

[0128] Owing to the high speed of ML-assisted recovery, real-time HSP video acquisition was demonstrated. For this test, the camera’s output was directly fed to the trained network and displayed the recovered HSP video in real time. During the experiment, the wavelengths and polarizations of the incident light were varied while recording. The video stream was processed by the neural network and the HSP video was recovered in real-time. From the recorded HSP video, it is evident that the HSP camera system accurately captured variations in polarizations and wavelengths. FIG. 15 breaks down the contributions from the camera readout, data preprocessing, and ML-assisted recovery to the processing time for each frame, demonstrating that the HSP camera system can operate at approximately 28 FPS when frames are processed synchronously. In asynchronous processing, the speed can increase to 36 FPS, which corresponds to, and is limited by, the maximal camera readout frame rate, with a latency of 0.02 seconds.

[0129] To further evaluate the capability and robustness of the HSP camera system, it was tested on a natural object with a more complex surface than the prior test targets. A beetle specimen, Chrysina gloriosa, the green gold scarab beetle Arizona, known for its reflection of a high degree of left-handed circular polarization, was chosen as the test target. The light reflected by the beetle was captured with the HSP camera system (see experimental setup in FIG. 33). The recovered multidimensional images (FIG. 16) distinctly show stripes in S3image, indicating selective reflection of left-handed circular or elliptical polarization in those areas. From the recovered Stokes images, images of the beetle under RCP or LCP illuminations were recovered (FIG. 17). These results closely match the experimental images captured under the same conditions. The LCP and RCP views for seven wavelengths are shown in FIG. 18. For improving the density of the information in the hyperspectro-polarimetric images, new experiments were conducted with a glass / acrylic sample, and hyperspectro-polarimetric images were obtained with both 50-nm wavelength intervals and 5-nm wavelength intervals, as shown in FIGS. 34 and 35, respectively.

[0130] FIG. 16 depicts imaging reconstruction results of a Chrysina gloriosa green gold scarab beetle from the ML model with laser illumination (750 nm). The training dataset includes seven wavelengths. FIG. 17 depicts LCP and RCP views of the beetle at 750 nm: the left-side views are captured by a camera with LCP or RCP illuminations, while the right-side ones are calculated using retrieved Stokes’ parameters. The RCP and LCP views are defined as the reflected image of the beetle with RCP and LCP illuminations, respectively. FIG. 18 depicts LCP and RCP views for seven wavelengths. In the LCP views, from 750 to 850 nm, the vertical stripes on the back of the beetle can be clearly observed. At 750 nm, the stripes are consistent with the ground truth. As the wavelength increases, the stripes on the back of the beetle gradually become less distinguishable. By 890 nm, the stripes become vague, and the LCP view is no longer significantly different from the RCP view. This is because the response of the beetle’s back to polarized light changes as the wavelength increases. In the RCP view, there are no clear bright stripes on the back of the beetle; instead, there are widely distributed spots at the 750 nm, consistent with ground truth.

[0131] These tests have demonstrated the utility of the system / device across a range of test targets, including real-world objects, highlighting its robustness, accuracy, and the effectiveness of the machine learning approach for real-time data recovery. The system exhibits remarkable versatility. For instance, through different calibrations, the same system can cater to diverse requirements - ranging from broad bandwidth applications with large wavelength steps (from 700 nm to 1150 nm) to high spectral resolution (0.23 nm) within a more limited bandwidth. Furthermore, flexibility is provided to adjust the emphasis on spatial, spectral, and polarimetric channels by allocating a varying number of metasurface encoder pixels to resolve the information within those channels. For example, spectral and polarization resolutions can be enhanced at the expense of spatial resolution by incorporating more metasurface encoder pixels within each super pixel. Alternatively, it may be chosen to sacrifice polarization data to boost spectral performance, dedicating all pixels to constructing spectral responses within the transmission matrix. Conversely, polarization accuracy can be improved by narrowing the range of wavelengths resolved. Thus, depending on specific application scenarios, the system’s capabilities can be tailored to provide on-demand multidimensional imaging.

[0132] Discussion

[0133] In this work, a conventional camera was transformed into a compact HSP camera by integrating a judiciously designed metasurface. This metasurface, consisting of meta-atoms possessing rich polarization-dependent resonances, can encode spectral and polarimetric information into distinct intensity distributions. These distributions can be decoded using either an optimization-based or a ML-based method. Experimental results showed that this metasurface-enhanced camera system can resolve full-Stokes polarization across a broad spectral range (700 nm to 1150 nm) from a single snapshot, achieving a spectral sensitivity as high as 0.23 nm. The HSP camera, backed with the ML-based recovery, achieves real-time recovery on a standard laptop. The system can record full-Stokes HSP videos in real-time at 28 frames per second, limited only by the camera’s readout rate. The metasurface-based camera system offers a compact, speedy, and cost-effective solution for a multi-dimensional imaging system that harness spatial, spectral, and polarization information within light fields. In addition, the system’s compactness allows for potential integration into other platforms, such as microfluidic systems or fiber-optic probes, which could pave the way for new applications in real-time monitoring or in situ analysis.

[0134] Materials and Methods

[0135] Fabrication of The Metasurfaces and Imaging Targets

[0136] The encoding metasurfaces were fabricated on ITO glass, which was thoroughly cleaned through sonication in acetone and IP A, each for a duration of 3 minutes. Following this, a 600 nm layer of amorphous silicon was grown using plasma enhanced chemical vapor deposition (PECVD). A 1 : 1 diluted ZEP 520A e-beam resist was then spun at 3000 r.p.m. for 45 seconds and prebaked at 180 °C for 3 minutes. The meta-atoms were inscribed using a Vistec 5200 lOOkV, followed by a 3-minute development in N-amyl-acetate and a 1-minute rinse in MIBK:IPA. 40 nm Cr films were then deposited at a rate of 2 A / s using a Temescal electron beam evaporation system. The pattern was lifted off in a water bath at 80 °C for 2 hours using 1165 remover (Mi croChem). To eliminate any potential contamination from residual metal particles, the sample was sonicated for several minutes in solvent before drying it with an N2gun. A chlorine-based plasma ICP-RIE recipe, involving Cl2and Ar gas, was used to etch the amorphous Si and create the meta-atoms. Finally, the sample was immersed in a Chromium etchant to remove the mask.

[0137] The imaging targets, featuring ‘PSU’ letters and a cartoon figure, were fabricated on a glass substrate. The substrate was thoroughly cleaned through sonication in acetone and IP A, each for 3 minutes. A 150 nm Al layer was then deposited at a rate of 2 A / s using a Temescal electron beam evaporation system. Subsequently, the SPR950 photoresist was spun at 3000 r.p.m. for 45 seconds and prebaked at 90 °C for 1 minute. The patterns were written using a Heidelberg MLA 150 Direct Write Exposure Tool, followed by a 1-minute development in CD26 and a 1-minute water rinse. The Al was then dry-etched with a resist mask using a chlorine-based plasma ICP-RIE recipe, which involved Cl2and BC12gas. Finally, the sample was immersed in Remover PG to remove the resist.

[0138] Optical Characterization

[0139] The characterization setup for the encoding metasurfaces consists of three parts: a tunable source, a beam expander, and an HSP imaging system, as shown in FIG. 21. The tunable source comprises a Ti: sapphire pulsed laser, capable of tuning the wavelength across a broad range, and a half waveplate (HWP) and quarter waveplate (QWP) for full-Stokes polarization tuning. Two singlet lenses (fl = 50 mm and f2 = 150 mm) were employed to form a simple microscope setup, providing 3* magnification. The role of this microscope setup is twofold. In the calibration stage, it expands the input light beam and ensures the resulting beam size is sufficiently large to illuminate the entire area of the metasurface. In the measurement stage, it projects the image of the target onto the metasurface. Essentially, the metasurface can be directly integrated on the top of the camera sensor. However, in the current studies, to maintain the flexibility to iterate the metasurface design without damaging the camera each time, a microscope system was used after the metasurface to “virtually” attach the metasurface onto the camera sensor (FIG. 21).

[0140] To calibrate the system, the input wavelength and the polarization state were varied using the tunable light source and then the resultant images were captured using the HSP imaging system. An example of the calibration results is depicted in FIG. 9. Following the same procedure, illumination light with different wavelength and polarization state combinations was used to test the HSP imaging system. The captured raw images were used to retrieve the illumination spectra and polarization states (FIGS. 10-12) using the optimization method.

[0141] For acquiring HSP images, the target objects were placed at the front focal plane of the first singlet lens (fl = 50 mm). The letters ‘PSU’ and a cartoon figure were used as the test targets. The raw images captured by the HSP imaging system were then used to reconstruct the HSP images (FIG. 12). For acquiring the HSP images of the beetle, the setup was modified to measure the reflected light from the object instead of the transmitted light (FIG. 24). The light source was coupled into a single-mode fiber and passed through a diffuser to ensure the light beam had uniform intensity distribution. Next, it passed through a linear polarizer and became x (horizontally)-polarized. The beam illuminated the beetle sample, which was placed at the front focal plane of the first singlet lens. The reflected light from the beetle was finally collected by the HSP imaging system. The experimental setup for HSP imaging of beetles is depicted in FIG. 33.

[0142] Data Preprocessing in Machine Learning Process

[0143] The raw image taken by the camera is of the size of 1505 x 1764, which is relatively large for the input to the machine learning model. To simplify the model for high-speed imaging and future potential mobile deployment, the image is divided into super pixels, and one super pixel is 88x88. The image is then resized to 44x44 to reduce the image size, and to make sure there are still enough super pixels to cover the intensity information for wavelength and polarization retrieval. The 88x88 sliding window shifts 44 pixels to make another super pixel, and in total there are 42x50 image pixels from one-shot (FIGS. 26A-27B). All camera pixels within the super pixel are flattened to ID. Since there are 42x50 image pixels in total, there are 2100 unique image pixels. Therefore, there are 4036 numerical values as the input including the images pixel information and position information, and the output is the spectrum and polarization information for the image pixel.

[0144] EXAMPLE 2

[0145] From traditional wavelength encoding to spectrum and polarization encoding.

[0146] FIG. 19A illustrates the working principle of a narrow-band filter-based approach to distinguish spectral information. FIG. 19B illustrates the working principle of a broadband filter- based approach to distinguish spectral information. FIG 19C illustrates the working principle of metasurface-based multichannel encoders for distinguishing both spectral and polarimetric information.

[0147] Traditional filter-based spectrometers operate by using narrowband filters to distinguish wavelengths, allowing the estimated spectrum to be directly read from the intensity information of the detectors (FIG. 19 A). In this configuration, intensity information and wavelength information have a one-to-one mapping relationship. However, designing extremely narrowband filters poses a significant challenge, particularly in terms of the technical expertise required and the associated costs. Even when such filters are successfully designed, they are highly sensitive to detector noise and fabrication errors. Additionally, the one-to-one relationship necessitates multiple detectors and filters to achieve broadband detection. Once the device is fabricated, the resolution and bandwidth become fixed and cannot be adjusted.

[0148] To solve the issue, researchers have designed broadband color filters / encoders for wavelength multiplexing (see FIG. 19B). Instead of responding to one specific wavelength, these broadband filters have wide and distinctive spectrum, enabling the differentiation of the input spectrum through appropriate computer algorithms.

[0149] Consider an arbitrary incident spectrum f(λ) is transmitted through a set of broadband color filters, where is the wavelength. The transmission response of the i-th broadband color filter is denoted as hi(A) i=l ,2,3, . . ,,N, where N is the number of the color filters. The transmission intensity received by i-th detector behind the i-th color filter can be denoted as follows:Where Atand A2are the minimum and maximum wavelength of the incident spectrum respectively, P(A) represents the absorption efficiency of the color filter array, and P(A) is the dispersion effect of the system. nirepresents the measurement error. Here, the modified transmission spectral curve is defined as So the transmission equationcan be simplified and discretized as:

[0150] Mathematically, the reconstruction of spectrum f can be modeled as solving a system of linear equations. In principle, if the number of broadband fdters and the detectors sufficient, the spectrum can be reconstructed via matrix inversion. In practice, in order to save space and reduce cost, the number of filters (N) is often smaller than the number of wavelength channels desired (M). As a result, optimization algorithms are needed to reconstruct the spectra. Nonnegative least square, least mean squares, sparse optimization and other reconstruction algorithms are adopted to maximize the reconstruction accuracy and decrease the number of measurements.

[0151] Apart from the wavelength, the state of polarization is also an essential parameter for light. Thus, it is desirable to extend the one-shot spectrum measurement to one-shot spectrum and polarization measurement. The key to the broadband wavelength multiplexing is the design of the broadband color filters that have different transmission responses in different wavelengths. If there are filters that are sensitive to both polarization and wavelength, this method can be used to obtain the polarization information for different wavelengths at the same time. Most polarization measurements use the Stokes parameter to describe the fullpolarization information. The Stokes parameters are as follows:Where Ix, ly, Id, ladenote the intensity of light in linear polarization bases along x, y, 45 degrees (diagonal direction), and -45 degree (anti-diagonal direction). Irand / ; are the intensities of the RCP and LCP. For an optical device, the input and output Stokes parameters are linked by the Muller matrix:

[0152] The first Stokes parameter Soexactly represents the intensity information. The first row of the Muller matrix can be used to encode the polarization into an output intensity. Thetransmission intensity received by i-th detector behind the i-th color-polarization filter can be denoted as follows (considering noise):

[0153] With an appropriate reconstruction method and enough measurements, the intensity information received by the detectors can be used to reconstruct the spectrum information and the polarization intensity. Note that in Equation (S2), every wavelength is only denoted by one parameter while Equation S6 needs four parameters for each wavelength. In addition, the case is quite complex in Equation (6) because of the relationship between the Stokes parameters (So2> Si2+ S22+ S32), which makes it a Convex Optimization problem.

[0154] Design and optical characterization of the encoding metasurfaces.

[0155] FIG. 20 shows the tunable geometric parameters of split door 103a and split ring 103b meta-atoms and the 100 meta-atom designs 105 chosen as wavelength and polarization encoders.

[0156] To simultaneously encode polarization and wavelength, encoding metasurfaces were designed to break the symmetry in the x and y directions, as well as in the left-handedness and the right-handedness. Initially, common chiral shapes such as the letters 'S' and 'Z' were experimented with. While the spectrum varied for different input polarizations, the spectral and polarimetric features were not rich enough for distinguishing close wavelengths and polarization states. For improved wavelength and polarization resolutions, split door and split ring shaped metasurfaces were chosen as encoders, due to their rich spectra for different polarization inputs. Additionally, these shapes offer multiple geometric parameters for manipulation, enabling the creation of a large library of encoders. The geometric parameters, as shown in FIG. 20, were scanned for different designs, with the period and height fixed to 600 nm. These designs were simulated in COMSOL for the x and y polarization input and calculate the Jones matrix. To make the library more comprehensive, structures with the same geometric parameters but rotated by 90 degrees, mirrored, and both rotated and mirrored were also included. The Jones matrix for these transformed designs with same parameters can be calculated directly from the simulated Jones matrix. Then the transfer equation (Equation S7 below) can be used to get the Muller matrix and pick the first row as the transmission component for each wavelength to construct the transmission matrix, where J is the Jones matrix and M is the Muller matrix.

[0157] In total, the chiral-shape library used comprises 1936 split-door shapes and 1294 split- ring shapes. As mentioned in the main text, the performance of the encoders can be quantified by the condition number. Therefore, optimization methods were used to select 100 designs from the entire pool of 3232 designs, minimizing the condition number. The Surrogate Toolbox was utilized to iterate 5000 times, ultimately obtaining the final 100 designs with the minimal condition number. The chosen 100 designs (e.g., 105) are presented in FIG. 20 and its condition number is shown in FIG. 8.

[0158] Calibration

[0159] FIG. 21 illustrates an experimental setup used for characterization of encoding metasurfaces. The output of a Ti: sapphire pulsed laser, passing through a half waveplate (HWP) and quarter waveplate (QWP), serves as a source with tunable wavelength and polarization. The output beam is expanded by two lenses to ensure it covers the metasurfaces area. After passing through the encoding metasurfaces (MS), the light is captured by an objective lens (OL). The OL together with a tube lens forms an image system which projects the metasurface image on the camera. In some embodiments, the metasurface can be directly integrated to the active sensor surface of the camera, resulting in an ultra-compact device. In current feasibility studies, in order to maintain the flexibility to iterate the metasurface design, an imaging system (formed by an objective lens and a tube lens) was used to “virtually” attach the metasurface onto the camera sensor.

[0160] FIGS. 22A and 22B depict the Poincare sphere results for FIGS. 10 and 11, respectively. The results show the recovery results (blue points) are close to the ground truth (red points), which shows that the system is able to recover both linear polarization and circle polarization either for narrowband with high spectral resolution or broadband with low spectral resolution.

[0161] FIG. 23 illustrates the experimental setup used for HSP images of target (‘PSU or cartoon patterns). The artificial object (e g., target object) is made of tunable source and target, which is placed at the front focal plane of projections lens. The image of an artificial object is projected to the surface of metasurfaces and then projected to the camera. FIG. 24 depicts the ground truth images for FIG. 12. FIG. 25 illustrates a graphical representation of calculated structural similarity index measures (SSIMs) between retrieved images shown in FIG. 12 and their corresponding ground truth for different wavelengths and polarization states.

[0162] Characterization of trained model

[0163] FIG. 26A depicts a raw image captured by the CMOS camera, with the gray box indicating a super pixel of 88x88 camera sensor pixels. FIG. 26B depicts a down-sampled image of the raw image in FIG. 26A, with the gray box indicating a super pixel of 44x44 pixels. FIG. 27A depicts sliding of the window for a super pixel (gray box) along the horizontal and vertical directions, cutting out 50x42 super pixels in total. FIG. 27B depicts resulting super pixels, cut bythe sliding window of 44x44 pixel size, which are used as input for the machine learning. FIG. 28 illustrates a Poincare sphere showing the 14 polarization states used in calibration.

[0164] Upon completion of the ML model's training, its performance was initially evaluated on synthesized data. The results are displayed in FIGS. 29-31, which depict visualizations of the results of the trained machine learning model neural network, where the Y-axis is the sample number, the X-axis is the 12 elements that describe the spectrum and polarization. The first 3 elements of X are So, ranging from 0-1, and the last 9 elements are S^S-^ ranging from -1 to 1. Here there is an offset of 1 added mathematically in dataset preparation to make values positive. FIG. 29 depicts the ground truth, FIG. 30 depicts predicted values, and FIG. 31 depicts the absolute difference between FIGS. 29 and 30. As can be inferred from FIG. 31, the error between them is minimal. This suggests that the model is capable of distinguishing not only the intensity and spectrum but also the polarization information, even when wavelengths of light and polarization states are intermixed.

[0165] FIG. 34 depicts hyperspectro-polarimetric imaging of a glass / acrylic sample from 750nm to lOOOnm with 50-nm intervals. A glass substrate (left side of each HSP image) and an acrylic plate (right side of each HSP image) were placed side-by-side and imaged. In contrast to the glass, the acrylic plate has internal strains, resulting in rich polarization nonuniformity and complexity in the transmitted beam. Imaging results at six wavelengths ranging from 750nm to lOOOnm with 50-nm intervals are presented. The acrylic plate exhibits considerably more complex patterns in all channels, as expected.

[0166] FIG. 35 depicts hyperspectro-polarimetric imaging of the glass / acrylic sample of FIG. 34 from 700nm to 745nm with 5-nm intervals. The glass substrate is shown in the top half of each HSP image, and the acrylic plate is shown in the bottom half of each HSP image.

[0167] Recorded HSP videos of ‘PSU’ letters and a cartoon figure.

[0168] Movie SI Description of performed work.

[0169] The letters ‘PSU’ were used as a target object for HSP imaging. The target was slightly moved and both the wavelength and polarization of the illumination light was altered. These adjustments were controlled by rotating a half-wave plate and a quarter-wave plate, as well as by tuning the laser wavelength. The obtained HSP video demonstrates that the retrieved HSP imageof ‘PSU’ letters changed between different wavelength and polarization channels, matching well with the ground truth.

[0170] Move S2 Description of performed work.

[0171] A cartoon figure was used as a target object for HSP imaging. The target was slightly moved and both the wavelength and polarization of the illumination light was altered. These adjustments were controlled by rotating a half-wave plate and a quarter-wave plate, as well as by tuning the laser wavelength. The obtained HSP video demonstrates that the retrieved HSP image of the cartoon figure changed between different wavelength and polarization channels, matching well with the ground truth.

[0172] References.

[0173] The following references are incorporated herein by reference in their entirety.(1) Park, J.; Feng, X.; Liang, R.; Gao, L. Snapshot Multidimensional Photography through Active Optical Mapping. Nat Commun 2020, 11 (1), 5602. https: / / doi . org / 10.1038 / s41467-020- 19418-0.(2) Liu, S.; Liu, X.; Yuan, J.; Bao, J. Multidimensional Information Encryption and Storage: When the Input Is Light. Research (Wash D C) 2021, 2021, 7897849. https: / / doi.org / 10.34133 / 2021 / 7897849.(3) Cai, G.; Li, Y.; Zhang, Y.; Jiang, X.; Chen, Y.; Qu, G.; Zhang, X.; Xiao, S.; Han, J.; Yu, S.; Kivshar, Y.; Song, Q. Compact Angle-Resolved Metasurface Spectrometer. Nat. Mater. 2024, 23 (1), 71-78. https: / / doi.org / 10.1038 / s41563-023-01710-l.(4) Shen, Z.; Zhao, F.; Jin, C.; Wang, S.; Cao, L.; Yang, Y. Monocular Metasurface Camera for Passive Single-Shot 4D Imaging. Nat Commun 2023, 14 (1), 1035. https : / / doi . org / 10.1038 / s41467-023-36812-6.(5) Lu, G.; Fei, B. Medical Hyperspectral Imaging: A Review. J. Biomed. Opt 2014, 19 (1), 010901. https: / / doi.Org / 10.1117 / l.JBO.19. l .010901.(6) Khan, A.; Vibhute, A. D.; Mali, S.; Patil, C. H. A Systematic Review on Hyperspectral Imaging Technology with a Machine and Deep Learning Methodology for Agricultural Applications. Ecological Informatics 2022, 69, 101678. https: / / doi.Org / 10.1016 / j.ecoinf.2022.101678.(7) He, C.; He, H.; Chang, J.; Chen, B.; Ma, H.; Booth, M. J. Polarisation Optics for Biomedical and Clinical Applications: A Review. Light Sci Appl 2021, 10 (1), 194. https : / / doi . org / 10.1038 / s41377-021 -00639-x.(8) Sterzik, M. F.; Bagnulo, S.; Palle, E. Biosignatures as Revealed by Spectropolarimetry of Earthshine. Nature 2012, 483 (7387), 64-66. https: / / doi.org / 10.1038 / naturel0778.(9) Zhou, X.; Ma, L.; Brown, W.; Little, J. V.; Chen, A. Y.; Myers, L. L.; Sumer, B. D.; Fei, B. Automatic Detection of Head and Neck Squamous Cell Carcinoma on Pathologic Slides Using Polarized Hyperspectral Imaging and Machine Learning. Proc SPIE Int Soc Opt Eng 2021, 11603, 116030Q. https: / / doi.Org / 10.l 117 / 12.2582330.(10) Tua, D.; Liu, R.; Yang, W.; Zhou, L.; Song, H.; Ying, L.; Gan, Q. Imaging-Based Intelligent Spectrometer on a Plasmonic Rainbow Chip. Nat Common 2023, 14 (1), 1902. https: / / doi.org / 10.1038 / s41467-023-37628-0.(11) Zuo, J.; Bai, J.; Choi, S.; Basiri, A.; Chen, X.; Wang, C.; Yao, Y. Chip-Integrated Metasurface Full-Stokes Polarimetric Imaging Sensor. Light Sci Appl 2023, 12 (1), 218. https : / / doi . org / 10.1038 / s41377-023 -01260-w.(12) Altaqui, A.; Sen, P.; Schrickx, H.; Rech, J.; Lee, J.-W.; Escuti, M.; You, W.; Kim, B. J.; Kolbas, R.; O’Connor, B. T.; Kudenov, M. Mantis Shrimp-Inspired Organic Photodetector for Simultaneous Hyperspectral and Polarimetric Imaging. Science Advances 2021, 7 (10), eabe3196. https: / / doi.org / 10.1126 / sciadv.abe3196.(13) Tu, X.; Spires, O. J.; Tian, X.; Brock, N.; Liang, R.; Pau, S. Division of Amplitude RGB Full-Stokes Camera Using Micro-Polarizer Arrays. Opt. Express, OE 2017, 25 (26), 33160-33175. https: / / doi.org / 10.1364 / OE.25.033160.(14) Shinoda, K.; Ohtera, Y.; Hasegawa, M. Snapshot Multispectral Polarization Imaging Using a Photonic Crystal Filter Array. Opt. Express 2018, 26 (12), 15948. https: / / doi.org / 10.1364 / OE.26.015948.(15) Garcia, M.; Edmiston, C.; Marinov, R.; Vail, A.; Gruev, V. Bio-Inspired Color- Polarization Imager for Real-Time in Situ Imaging. Optica 2017, 4 (10), 1263. https: / / doi.Org / 10.1364 / OPTICA.4.001263.(16) Sattar, S.; Lapray, P.-J.; Aksas, L.; Foulonneau, A.; Bigue, L. Snapshot Spectropolarimetric Imaging Using a Pair of Filter Array Cameras. OE 2022, 61 (4), 043104. https: / / doi.Org / 10.1117 / l.OE.61.4.043104.(17) Sattar, S.; Lapray, P.-J.; Foulonneau, A.; Bigue, L. Review of Spectral and Polarization Imaging Systems. In Unconventional Optical Imaging II; International Society for Optics and Photonics, 2020; Vol. 11351, p 11351 IQ. https: / / doi.org / 10.1117 / 12.2555745.(18) Yu, N.; Genevet, P.; Kats, M. A.; Aieta, F.; Tetienne, J.-P.; Capasso, F.; Gaburro, Z. Light Propagation with Phase Discontinuities: Generalized Laws of Reflection and Refraction. Science 2011. https: / / doi.org / 10. ! 126 / science.1210713.(19) Ni, X.; Emani, N. K.; Kildishev, A. V.; Boltasseva, A.; Shalaev, V. M. Broadband Light Bending with Plasmonic Nanoantennas. Science 2012, 335 (6067), 427. https: / / doi.0rg / lO. l 126 / science.1214686.(20) Ding, F.; Pors, A.; Bozhevolnyi, S. I. Gradient Metasurfaces: A Review of Fundamentals and Applications. Rep. Prog. Phys. 2017, 81 (2), 026401. https: / / doi.org / 10.1088 / 1361- 6633 / aa8732.(21) Chang, S.; Guo, X.; Ni, X. Optical Metasurfaces: Progress and Applications. Annual Review of Materials Research 2018, 48 (1), 279-302. https: / / doi.org / 10. ! 146 / annurev- matsci-070616-124220.(22) Ni, X.; Wong, Z. J.; Mrejen, M.; Wang, Y.; Zhang, X. An Ultrathin Invisibility Skin Cloak for Visible Light. Science 2015, 349 (6254), 1310-1314. https : / / doi . org / 10.1126 / sci ence . aac9411.(23) Sun, T.; Hu, J.; Zhu, X.; Xu, F.; Wang, C. Broadband Single-Chip Full Stokes Polarization-Spectral Imaging Based on All-Dielectric Spatial Multiplexing Metalens. Laser & Photonics Reviews n / a (n / a), 2100650. https: / / doi.org / 10.1002 / lpor.202100650.(24) Wang, X.; Yang, Z ; Bao, F.; Sentz, T.; Jacob, Z. Spinning Meta-Cam: Spectro- Polarimetric Long-Wave Infrared Thermal Imaging Based on Spinning Metasurfaces. arXiv January 26, 2023. http: / / arxiv.org / abs / 230L l 1519 (accessed 2023-03-30).(25) Wang, Z.; Yi, S.; Chen, A.; Zhou, M.; Luk, T. S.; James, A.; Nogan, J.; Ross, W.; Joe,G ; Shahsafi, A.; Wang, K. X.; Kats, M. A.; Yu, Z. Single-Shot on-Chip Spectral SensorsBased on Photonic Crystal Slabs. Nature Communications 2019, 10 (1), 1020. https: / / doi . org / 10.1038 / s41467-019-08994-5.(26) Redding, B.; Liew, S. F.; Sanna, R.; Cao, H. Compact Spectrometer Based on a Disordered Photonic Chip. Nature Photon 2013, 7 (9), 746-751. https: / / doi.org / 10.1038 / nphoton.2013.190.(27) Yang, Z ; Albrow-Owen, T.; Cui, H.; Alexander-Webber, J.; Gu, F.; Wang, X.; Wu, T - C.; Zhuge, M.; Williams, C.; Wang, P.; Zayats, A. V.; Cai, W.; Dai, L.; Hofmann, S.; Overend, M.; Tong, L.; Yang, Q.; Sun, Z.; Hasan, T. Single-Nanowire Spectrometers. Science 2019, 365 (6457), 1017-1020. https: / / doi.org / 10.1126 / science.aax8814.(28) Xiong, J.; Cai, X.; Cui, K.; Cui, K.; Huang, Y.; Huang, Y.; Yang, J.; Zhu, H.; Li, W ; Hong, B.; Hong, B.; Rao, S.; Zheng, Z.; Xu, S.; He, Y.; Liu, F.; Liu, F.; Feng, X.; Zhang,W.; Zhang, W. Dynamic Brain Spectrum Acquired by a Real-Time Ultraspectral Imaging Chip with Reconfigurable Metasurfaces. Optica, OPTICA 2022, 9 (5), 461-468. https: / / doi.org / 10.1364 / OPTICA.440013.(29) Yang, J.; Cui, K.; Cai, X.; Xiong, J.; Zhu, H.; Rao, S.; Xu, S.; Huang, Y.; Liu, F.; Feng,X.; Zhang, W. Ultraspectral Imaging Based on Metasurfaces with Freeform Shaped Meta- Atoms. Laser & Photonics Reviews n / a (n / a), 2100663. https: / / doi.org / 10.1002 / lpor.202100663.(30) Chang, C.-C.; Lee, H.-N. On the Estimation of Target Spectrum for Filter- Array Based Spectrometers. Opt. Express, OE 2008, 16 (2), 1056-1061. https: / / doi.org / 10.1364 / OE.16.001056.(31) Ni, Y.; Chen, C.; Wen, S.; Xue, X.; Sun, L.; Yang, Y. Computational Spectropolarimetry with a Tunable Liquid Crystal Metasurface. eLight 2022, 2 (1), 23. https: / / doi.org / 10.1186 / s43593-022-00032-0.(32) Chen, W. T.; Torok, P.; Foreman, M. R.; Liao, C. Y.; Tsai, W.-Y.; Wu, P. R.; Tsai, D. P. Integrated Plasmonic Metasurfaces for Spectropolarimetry. Nanotechnology 2016, 27 (22), 224002. https: / / doi.org / 10.1088 / 0957-4484 / 27 / 22 / 224002.(33) Zhang, S.; Dong, Y.; Fu, H.; Huang, S.-L.; Zhang, L. A Spectral Reconstruction Algorithm of Miniature Spectrometer Based on Sparse Optimization and Dictionary Learning. Sensors 2018, 18 (2), 644. https: / / doi.org / 10.3390 / sl8020644.(34) Boyd, S. P.; Vandenberghe, L. Convex Optimization; Cambridge University Press, 2004.(35) Powell, M. J. D. A Fast Algorithm for Nonlinearly Constrained Optimization Calculations. In Numerical Analysis; Watson, G. A., Ed.; Lecture Notes in Mathematics; Springer: Berlin, Heidelberg, 1978; pp 144-157. https: / / doi.org / 10.1007 / BFb0067703.(36) Han, S. P. A Globally Convergent Method for Nonlinear Programming. J Optim Theory Appl 1977, 22 (3), 297-309. https: / / doi.org / 10.1007 / BF00932858.(37) Bonyadi, M. R.; Michalewicz, Z. Particle Swarm Optimization for Single Objective Continuous Space Problems: A Review. Evolutionary Computation 2017, 25 (1), 1-54. https: / / doi.org / 10.1162 / EVCO_r_00180.(38) Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, https: / / web.stanford.edu / ~boyd / papers / admm_distr_stats.html (accessed 2024-07-05).(39) Kang, L.; Wang, C.-Y.; Guo, X.; Ni, X.; Liu, Z.; Wemer, D. H. Nonlinear Chiral Meta- Mirrors: Enabling Technology for Ultrafast Switching of Light Polarization. Nano Lett. 2020, 20 (3), 2047-2055. https: / / doi.org / 10.1021 / acs.nanolett.0c00007.(40) Hancock, J. T.; Khoshgoftaar, T. M. Survey on Categorical Data for Neural Networks. Journal of Big Data 2020, 7 (1), 28. https: / / doi.org / 10.1186 / s40537-020-00305-w.

[0174] It should be understood that the disclosure of a range of values is a disclosure of every numerical value within that range, including the end points. It should also be appreciated that some components, features, and / or configurations may be described in connection with only one particular embodiment, but these same components, features, and / or configurations can be applied or used with many other embodiments and should be considered applicable to the other embodiments, unless stated otherwise or unless such a component, feature, and / or configuration is technically impossible to use with the other embodiment. Thus, the components, features, and / or configurations of the various embodiments can be combined together in any manner and such combinations are expressly contemplated and disclosed by this statement.

[0175] It will be apparent to those skilled in the art that numerous modifications and variations of the described examples and embodiments are possible considering the above teachings of the disclosure. The disclosed examples and embodiments are presented for purposes of illustration only. Other alternate embodiments may include some or all of the features disclosed herein.Therefore, it is the intent to cover all such modifications and alternate embodiments as may come within the true scope of this invention, which is to be given the full breadth thereof.

[0176] It should be understood that modifications to the embodiments disclosed herein can be made to meet a particular set of design criteria. Therefore, while certain exemplary embodiments of the compositions, materials, apparatuses, and methods of using and making the same disclosed herein have been discussed and illustrated, it is to be distinctly understood that the invention is not limited thereto but may be otherwise variously embodied and practiced within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A hyperspectro-polarimetric (HSP) image processing system comprising: a metasurface configured to encode spectral and polarization image data, the metasurface comprising: a plurality of super pixels disposed on the metasurface, wherein: each super pixel comprises a plurality of arrays of nanostructures, each array comprising a plurality of nanostructures; each array of nanostructures is configured such that each array of nanostructures comprises different spectral and polarization responses; and each super pixel is configured to encode the spectral and polarization image data into a plurality of distinct spatial intensity distributions, wherein each distinct spatial intensity distribution of the plurality corresponds to a predetermined wavelength and / or polarization state; an image capture device configured to capture each distinct spatial intensity distribution, the image capture device comprising: at least one image sensor comprising a plurality of subsets of pixels, wherein each subset of pixels corresponds to one super pixel such that each subset of pixels is configured to capture one distinct spatial intensity distribution of the plurality of distinct spatial intensity distributions, and wherein each subset of pixels is configured to generate pixel data corresponding to each distinct spatial intensity distribution captured by the subset of pixels; at least one processor in communicative connection with the image capture device, the at least one processor configured to:receive, from the at least one image sensor, the pixel data for each subset of pixels of the plurality of subsets of pixels; generate captured image input data based on the pixel data; access, from a non-transitory memory, at least one computational reconstruction model; input, to the at least one computational reconstruction model, the captured image input data; decode, via the at least one computational reconstruction model, the captured image input data; and generate, via the at least one computational reconstruction model, image output data based on the decoded captured image input data.

2. The system of claim 1, wherein the at least one computational reconstruction model comprises a machine learning model.

3. The system of claim 2, wherein the machine learning model comprises a trained neural network.

4. The system of claim 1, wherein the nanostructures comprise chiral meta-atoms.

5. The system of claim 4, wherein each of the chiral meta-atoms comprises at least one of a split-ring-shaped meta-atom or a split-door-shaped meta-atom.

6. The system of claim 4, wherein the metasurface comprises a predetermined number of chiral meta-atom designs based on a desired condition number.

7. The system of claim 1, wherein the at least one computational reconstruction model is configured to compensate, during the decoding, for aberrations and / or distortion caused by one or more super pixels, based on a known location of the one or more super pixels.

8. The system of claim 1, wherein the metasurface is disposed on a surface of the at least one image sensor.

9. The system of claim 1, wherein the system is further configured to generate, via the at least one processor, one or more HSP images based on the image output data.

10. The system of claim 1, wherein the at least one computational reconstruction model is further configured to generate image output data for each of one or more linear polarization states, one or more circular polarization states, and one or more elliptical polarization states.

11. A method for generating one or more hyperspectro-polarimetric (HSP) images, the method comprising: encoding, via a metasurface, spectral and polarization image data, wherein the metasurface comprises: a plurality of super pixels disposed on the metasurface, wherein:each super pixel comprises a plurality of arrays of nanostructures, each array comprising a plurality of nanostructures; each array of nanostructures is configured such that each array of nanostructures comprises different spectral and polarization responses; and each super pixel is configured to encode spectral and polarization image data into a plurality of distinct spatial intensity distributions, wherein each distinct spatial intensity distribution of the plurality corresponds to a predetermined wavelength and / or polarization states; capturing, via an image capture device, each distinct spatial intensity distribution, wherein the image capture device comprises: at least one image sensor comprising a plurality of subsets of pixels, wherein each subset of pixels corresponds to one super pixel such that each subset of pixels is configured to capture one distinct spatial intensity distribution of the plurality of distinct spatial intensity distributions, and wherein each subset of pixels is configured to generate pixel data corresponding to each distinct spatial intensity distribution captured by the subset of pixels; receiving, by at least one processor in communicative connection with the image capture device, the pixel data for each subset of pixels of the plurality of subsets of pixels from the at least one image sensor; generating, by the at least one processor, captured image input data based on the pixel data; accessing, by the at least one processor, at least one computational reconstruction model stored in a non-transitory memory;inputting, by the at least one processor to the at least one computational reconstruction model, the captured image input data; decoding, by the at least one processor via the at least one computational reconstruction model, the captured image input data; generating, by the at least one processor via the at least one computational reconstruction model, image output data based on the decoded captured image input data; and generating, by the at least one processor, the one or more hyperspectro- polarimetric images based on the image output data.

12. The method of claim 11, wherein the at least one computational reconstruction model comprises a machine learning model.

13. The method of claim 12, wherein the machine learning model comprises a trained neural network.

14. The method of claim 11, wherein the nanostructures comprise chiral meta-atoms.

15. The method of claim 14, wherein each of the chiral meta-atoms comprises at least one of a split-ring-shaped meta-atom or a split-door-shaped meta-atom.

16. The method of claim 14, wherein the metasurface comprises a predetermined number of chiral meta-atom designs based on a desired condition number.

17. The method of claim 11, wherein the at least one computational reconstruction model is configured to compensate, during the decoding, for aberrations and / or distortion caused by one or more super pixels, based on a known location of the one or more super pixels.

18. The method of claim 11, wherein the metasurface is disposed on a surface of the at least one image sensor.

19. The method of claim 11, wherein the at least one computational reconstruction model is further configured to generate image output data for each of one or more linear polarization states, one or more circular polarization states, and one or more elliptical polarization states.

20. The method of claim 11, wherein the metasurface is configured to encode spectral and polarization image data within a spectral range of 700 nanometers to 1150 nanometers.

Images