Multidimensional materials sensing system and method

Abstract

Multidimensional optical imaging, detection, and ranging (LIDAR) systems and methods optionally include a laser device configured to generate a plurality of light pulses emitted toward an object, a detector configured to receive a portion of the plurality of light pulses returning from the object, and a processor configured to generate a point cloud representing the object based on the plurality of light pulses received by the detector, the point cloud having a plurality of points, each point having three-dimensional position coordinates representing a location of the point on the object, and at least one additional value representing at least one of material information indicating the material of the object at the location of the point on the object, or optical information indicating at least one optical characteristic of the plurality of light pulses returning from the surface of the object from the location of the point on the object.

Description

[Technical field]

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 835,256, filed April 17, 2019, the disclosure of which is incorporated herein by reference in its entirety.

[0002] TECHNICAL FIELD This disclosure relates to materials sensing systems and methods, and more particularly to multi-dimensional materials sensing systems and methods for making and using the materials sensing systems and methods. [Background technology]

[0003] This section provides background information related to the present disclosure that is not necessarily prior art.

[0004] LIDAR is a surveying method that measures distance to objects by illuminating the object with pulsed laser light and measuring the reflected pulses with a sensor. The difference in laser return time and wavelength can then be used to create a digital 3D representation of the detected object. LIDAR can be used to generate high-resolution maps, with applications in geodesy, geomatics, archaeology, geography, geology, geomorphology, seismology, forestry, atmospheric physics, laser guidance, airborne laser swath mapping (ALSM), and laser altimetry. LIDAR technology can also be used for autonomous vehicle control and navigation.

[0005] A conventional lidar device may work as follows: A laser source produces pulses of polarized or unpolarized light at a particular wavelength. A time-of-flight sensor records the initial time when the light is first emitted. The time-of-flight is used to determine the total distance the light travels from the source to the detector by using the speed at which the light travels.

[0006] The emitted light is then "steered" at a given angle. This "steering" can also include splitting the light pulse into multiple pulse components oriented at different angles. The steering angle is changed over time to obtain a specific field of view for comprehensive mapping of the environment. After the light is steered, it may pass through linear polarization optics before and after emission. These types of lidars are known as polarized lidars and may use polarized optics in the alignment step.

[0007] Conventional lidar devices generally employ large and expensive optical lenses. Moreover, the optical lenses employed in conventional lidar devices are sensitive to moisture and therefore require extensive protective packaging, which increases the weight, size, and complexity of the lidar devices in which they are employed. One well-known problem of implementing lidar systems with rotating optics (e.g., Velodyne-HDL64™ model) in autonomous vehicles and robots is the large size and high cost of those lidar systems. The rotation of the entire device to steer the laser beam reduces reliability, limits miniaturization, and increases energy consumption. Lidar systems based on solid-state beam steering address this issue, but their implementation is hindered by insufficient accuracy and range. Another problem is the performance of lidar and all other sensors in adverse weather conditions. Currently utilized laser beams with wavelengths of about 900-940 nm can be strongly scattered by rain, fog, and snow, and thus the read-out of those laser beams can be highly uncertain under such conditions.

[0008] Furthermore, traditional lidar devices and their associated analysis systems have proven limited in their ability to accurately perform object recognition. For example, it is known that lidar point clouds are based solely on distance readouts from the laser source to the object. In this representation of the human world, a person resting on a bench is equivalent to an image of the person resting on the bench. The problem also applies when trying to distinguish a sleeping baby from a similarly sized plastic doll lying next to the baby, or a distant black car from the pavement. The burden of distinguishing these objects and interpreting the surroundings falls on the computational processing of these 3D maps.

[0009] Proper classification of objects based on their shape is not a trivial problem and requires complex algorithms and large computational power, especially considering the highly dynamic nature of various environments. Moreover, current beam steering methods cause clustering and banding of points in the lidar cloud, resulting in ambiguous interpretation of the 3D image and their individual points, making proper object recognition and classification even more difficult for typical lidar hardware. Thus, shape-based perception of the surroundings requires high computational cost, high energy consumption, and long processing times.

[0010] Therefore, improved lidar systems and methods are desired, particularly lidar systems and methods that provide the ability to identify the material from which an object is formed. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] U.S. Patent Application Serial No. 16 / 518,230 [Patent Document 2] U.S. Patent Application No. 15 / 092,885 (U.S. Publication No. 2016 / 0299270) [Non-patent literature]

[0012] [Non-Patent Document 1] XU et al., "Kirigami Nanocomposites as Wide-Angle Diffraction Gratings", ACS Nano, 10(6), pp. 6156-6162 (2016) [Non-Patent Document 2] Lizhi XU et al., “Origami and Kirigami Nanocomposites”, ACS Nano, 11(8) pp. 7587-7599 (2017) [Non-Patent Document 3] Jing Lyu et al., “Stretchable conductors by kirigami patterning of aramid-silver nanocomposites with zero conductance gradient”, Appl.Phys.Lett, 111, 161901 (2017) [Non-Patent Document 4] WJ Choi et al., “Chiroptical Kirigami Modulators for Terahertz Circular Dichroism Spectroscopy Biomaterials,” Nature Materials, 18, pp. 820–826 (2019) [Non-Patent Document 5] Kotov, NA et al., "Ultrathin graphite oxide-polyelectrolyte composites prepared by self-assembly: Transition between conductive and non-conductive states", Adv. Mater. 8, pp. 637-641 (1996) [Non-Patent Document 6] Kotov, NA et al., "Layer-by-Layer Self-assembly of Polyelectrolyte-Semiconductor Nanoparticle Composite FilMs", J.Phys.CheM., 99, pp. 13065-13069 (1995) Summary of the Invention [Problem to be solved by the invention]

[0013] This section provides a general overview of the disclosure, and is not an exhaustive disclosure of the complete scope or all of the features of the disclosure. [Means for solving the problem]

[0014] In some aspects, the disclosure provides a system comprising a laser device configured to generate a plurality of light pulses emitted towards an object, a detector configured to receive a portion of the plurality of light pulses returning from the object, and a processor configured to generate a point cloud representing the object based on the plurality of light pulses received by the detector, the point cloud having a plurality of points, each point having three-dimensional position coordinates representing a location of the point on the object and at least one additional value representing at least one of material information indicative of a material of the object at the location of the point on the object, or optical information indicative of at least one optical characteristic of the plurality of light pulses returning from a surface of the object from the location of the point on the object.

[0015] In one embodiment, the optical information includes at least one of polarization information, Raman scattering information, and circular dichroism information.

[0016] In one aspect, the at least one additional value represents material information indicative of a material on a surface of the object at the location of the point, and the processor is configured to generate the material information based on optical information indicative of at least one optical characteristic of a reflection among the multiple reflections returning from the surface of the object at the location of the point.

[0017] In a further aspect, the processor is configured to generate material information based on at least one of polarization information, Raman scattering information, and circular dichroism information of the reflection returning from the surface of the object at the location of the point.

[0018] In one aspect, the object is one of a plurality of objects, and the system further comprises a conveyor belt configured to transport the plurality of objects past the laser device and a sorter to sort the plurality of objects into a plurality of bins, wherein the at least one additional value represents material information indicative of a material of a surface of the object at the location of the point, and the processor is configured to control the sorter to sort the object into one of the plurality of bins based on the material information.

[0019] In one aspect, the system further comprises at least one additional laser device configured to generate a plurality of additional light pulses emitted toward the object, the at least one additional laser device located at a different location than the laser device, and at least one additional detector configured to receive a portion of the plurality of additional light pulses returning from the object, the at least one additional detector located at a different location than the detector, and the processor is configured to generate a point cloud representing the object further based on the plurality of additional light pulses received by the at least one additional detector.

[0020] In another variation, the disclosure provides a method including initializing, with at least one processor, parameters of a neural network based on physical laws of light / matter interaction; emitting, with a laser device, a plurality of light pulses toward an object having a predetermined material composition; receiving, with a detector, a plurality of reflections of the plurality of light pulses returning from the object; inputting, with the processor, optical properties of the plurality of reflections into the neural network; receiving, with the processor, a predicted material composition of the object from the neural network based on the input optical properties; comparing, with the processor, the predicted material composition to the predetermined material composition; and adjusting, with the processor, parameters of the neural network based on the comparison.

[0021] In one aspect, adjusting the parameters is performed using a machine learning algorithm.

[0022] In one aspect, the method further includes iteratively adjusting parameters of the neural network based on a comparison of the predicted material composition and the additional known material composition until a difference between one of the additional predicted material compositions for the additional object and a corresponding one of the additional known material compositions of the additional object is less than a predetermined error limit, the additional predicted material composition being generated by inputting optical properties for a plurality of additional reflections of a plurality of additional light pulses reflected from the additional object by the laser device.

[0023] In another variation, the present disclosure provides another system comprising a polarization camera configured to receive ambient light reflected from a surface of an object and generate image data related to the object and polarization data related to the ambient light reflected from the surface of the object. The system further comprises a processor configured to generate a point cloud representing the object based on the image data and the polarization data, the point cloud having a plurality of points, each point having three-dimensional position coordinates representing a location of the point on the object and material information indicative of a material of the object at the location of the point on the object. The processor is further configured to determine the material information based on the polarization of the ambient light reflected from the object from the location of the point on the object.

[0024] In one aspect, the processor is further configured to access a material database that stores correlation information between ambient light polarization and object material, and determine material information indicative of the material of the object at each point location on the object based on the material database and based on the polarization of ambient light reflected from the object from each point location on the object.

[0025] In one aspect, the processor is further configured to perform at least one of: determining a plurality of materials that make up the object based on the material information indicating the material of the object at the location of each point on the object; and classifying or identifying the object based on the plurality of materials that make up the object.

[0026] In a further aspect, the processor is further configured to access an object database that stores correlation information between a plurality of objects and materials that make up each object of the plurality of objects, and perform at least one of classifying or identifying the object based on the correlation information stored in the object database and the plurality of materials that make up the object.

[0027] In one aspect, the processor is further configured to determine an edge in the image data between the object and the additional object based on a difference between polarization data associated with ambient light reflected from a surface of the object and additional polarization data associated with ambient light reflected from a surface of the additional object detected by the polarization camera.

[0028] In one aspect, an additional light source may be utilized to image the object with a polarized camera, which may be a lidar laser utilized in the same perception system.

[0029] In one aspect, because the polarization signature of a material changes with the angle of reflection and scattering of light from an object, material identification from images acquired by a polarization camera can be achieved by using the time and distance progression of the images. As a result of AI / ML analysis of the time and distance dependence of the polarization images, the accuracy of material and object identification is improved.

[0030] In one aspect, the object is one of a plurality of objects, and the system further comprises a conveyor belt configured to transport the plurality of objects past the polarized camera and a sorter to sort the plurality of objects into a plurality of bins, The processor is configured to control the sorter to sort the object into one of the plurality of bins based on the material information.

[0031] In one aspect, the system further includes at least one additional polarized camera located at a different location than the polarized camera and configured to generate additional image data and additional polarization data related to the object, and the processor is configured to generate a point cloud representing the object further based on the additional image data and the additional polarization data generated by the at least one additional polarized camera.

[0032] In another variation, the disclosure provides another method including initializing, with at least one processor, parameters of a neural network based on physical laws of light / matter interaction and receiving, with a polarization camera, a plurality of reflections of ambient light reflected from an object. The method further includes determining, with the processor, polarization information for the plurality of reflections of ambient light reflected from the object, inputting, with the processor, the polarization information for the plurality of reflections into the neural network, and receiving, with the processor, a predicted material composition of the object from the neural network based on the input polarization information. The method further includes comparing, with the processor, the predicted material composition to a predetermined material composition and adjusting, with the processor, parameters of the neural network based on the comparison.

[0033] In one aspect, adjusting the parameters is performed using machine learning and / or artificial intelligence algorithms.

[0034] In one aspect, the method further includes iteratively adjusting parameters of the neural network based on a comparison of the predicted material composition and the additional known material composition until a difference between one of the additional predicted material compositions for the additional object and a corresponding one of the additional known material compositions of the additional object is less than a predetermined error limit, the additional predicted material composition being generated by inputting additional polarization information about multiple additional reflections of ambient light reflected from the additional object and received by the polarization camera.

[0035] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

[0036] The drawings described herein are only for purposes of illustrating selected embodiments, not all possible implementations, and are not intended to limit the scope of the present disclosure. [Brief description of the drawings]

[0037] [Figure 1] FIG. 1 is a functional diagram illustrating an M Lidar system, according to some embodiments of the present disclosure. [Figure 2a] 1 is a scanning electron microscope (SEM) image of a nano-kirigami nanocomposite sheet configured for use in an M-lidar system, according to some embodiments of the present disclosure. [Figure 2b] FIG. 1 is a scanning electron microscope (SEM) image of a nano-kirigami nanocomposite sheet configured for use in an M-lidar system, according to some embodiments of the present disclosure. [Figure 3a] 1 is an image showing a laser diffraction pattern from a nano-kirigami-based graphene composite for 0% strain level, according to some embodiments of the present disclosure. [Figure 3b] 1 is an image showing laser diffraction patterns from nanokirigami-based graphene composites for a strain level of 50%, according to some embodiments of the present disclosure. [Figure 3c] 1 is an image showing a laser diffraction pattern from a nanokirigami-based graphene composite for a strain level of 100%, according to some embodiments of the present disclosure. [Figure 4a] FIG. 1 illustrates an exemplary simplified process for fabricating nano-kirigami-based optical elements according to some aspects of the present disclosure. [Figure 4b] FIG. 1 illustrates an exemplary simplified process for fabricating nano-kirigami-based optical elements according to some aspects of the present disclosure. [Figure 4c] FIG. 1 illustrates an exemplary simplified process for fabricating nano-kirigami-based optical elements according to some aspects of the present disclosure. [Figure 4d] FIG. 1 illustrates an exemplary simplified process for fabricating nano-kirigami-based optical elements according to some aspects of the present disclosure. [Figure 5a]FIG. 1 shows a photograph of nano-kirigami nanocomposite optical elements fabricated on a wafer according to some aspects of the present disclosure. [Figure 5b] FIG. 5b shows an SEM image of the nano-kirigami nanocomposite optical element of FIG. 5a fabricated on a wafer with less than 0% distortion according to some embodiments of the present disclosure. [Figure 5c] FIG. 5b shows an SEM image of the nano-kirigami nanocomposite optical element of FIG. 5a fabricated on a wafer with less than 100% distortion according to some embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates a confusion matrix for materials and surface texture (MST) using an artificial intelligence algorithm and polarization information, in accordance with some aspects of the present disclosure. [Figure 7] FIG. 13 illustrates a confusion matrix for the detection of simulated black ice compared to other materials, in accordance with some embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates an example of a black ice detection unit incorporating an M Lidar system, according to some embodiments of the present disclosure. [Figure 9] 1 is a flowchart illustrating a method for performing object classification using an M LIDAR system, according to some embodiments of the present disclosure. [Figure 10] FIG. 2 is a schematic diagram of a flat composite material having a representative number of kirigami cuts formed therein as a linear pattern. [Figure 11] 1 is a diagram illustrating an M Lidar system for mounting on a vehicle, according to some embodiments of the present disclosure. [Figure 12] FIG. 2 illustrates a multi-dimensional point cloud of an object, in accordance with some aspects of the present disclosure. [Figure 13] 10 is a flowchart illustrating a method for calibrating a neural network to output predicted material information of an object based on input optical properties generated by a LIDAR system, according to some aspects of the present disclosure. [Figure 14]1 is a diagram of an exemplary recycling system according to some embodiments of the present disclosure. [Figure 15] FIG. 2 illustrates a multi-dimensional point cloud of an object, in accordance with some aspects of the present disclosure. [Figure 16] FIG. 1 illustrates a multi-dimensional point cloud of a patient's skin, in accordance with some aspects of the present disclosure. [Figure 17A] FIG. 13 is a diagram showing an image taken by a polarization camera. [Figure 17B] FIG. 13 is a diagram showing an image taken by a polarization camera. [Figure 17C] FIG. 13 is a diagram showing an image taken by a polarization camera. [Figure 18] FIG. 1 shows images of a group of objects using a non-polarized camera and a polarized camera. [Figure 19] FIG. 1 illustrates a confusion matrix for MST using an artificial intelligence algorithm and polarization information, in accordance with some aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.

[0039] Exemplary examples are given so that the disclosure is thorough and fully conveys the scope to those skilled in the art. Numerous specific details are described, such as examples of specific compositions, components, devices, and methods, to provide a thorough understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that the exemplary embodiments can be implemented in many different forms, and that none of these should be construed to limit the scope of the present disclosure. In some exemplary embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

[0040] The terms used herein are for the purpose of describing specific exemplary embodiments only and are not limiting. As used herein, the singular forms "a," "an," and "the" may include the plural unless the context clearly dictates otherwise. The terms "comprises," "comprising," "including," and "having" are inclusive and thus specify the presence of stated features, elements, compositions, steps, integers, operations, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The open-ended term "comprising" should in some aspects be understood as a non-limiting term used to describe and claim various embodiments described herein, although the term may alternatively be understood to be a more restrictive, restrictive term such as "consisting of" or "consisting essentially of." Thus, for any given embodiment reciting compositions, materials, components, elements, features, integers, operations, and / or process steps, the disclosure also specifically includes embodiments that consist of or consist essentially of such recited compositions, materials, components, elements, features, integers, operations, and / or process steps. In the case of "consisting of," alternative embodiments exclude additional compositions, materials, components, elements, features, integers, operations, and / or process steps that materially affect the basic and novel properties, whereas in the case of "consisting essentially of," additional compositions, materials, components, elements, features, integers, operations, and / or process steps that do not materially affect the basic and novel properties are excluded from such embodiment, but compositions, materials, components, elements, features, integers, operations, and / or process steps that do not materially affect the basic and novel properties may be included in the embodiment.

[0041] Any method steps, processes, and operations described herein should not be construed as necessarily requiring their performance in the particular order described or illustrated, unless specifically identified as an order of performance, and it should be understood that additional or alternative steps may be employed, unless otherwise indicated.

[0042] When a component, element, or layer is referred to as "on," "engaged with," "connected to," or "coupled to" another element or layer, it may be directly on, engaged, connected, or coupled to the other component, element, or layer, or there may be intervening elements or layers. In contrast, when an element is referred to as "directly on," "directly engaged," "directly connected," or "directly coupled to" another element or layer, there may not be intervening elements or layers. Other words used to describe relationships between elements should be construed in a similar manner (e.g., "between" vs. "directly between," "adjacent" vs. "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0043] Terms such as first, second, third, etc. may be used herein to describe various steps, elements, components, regions, layers, and / or sections, but these steps, elements, components, regions, layers, and / or sections should not be limited by these terms unless otherwise indicated. These terms may only be used to distinguish one step, element, component, region, layer, or section from another step, element, component, region, layer, or section. Terms such as "first", "second", and other numerals, when used herein, do not imply a sequence or order unless clearly indicated by the context. Thus, a first step, element, component, region, layer, or section described below may be referred to as a second step, element, component, region, layer, or section without departing from the teachings of the exemplary embodiments.

[0044] Spatially or temporally relative terms, such as "before," "after," "inner," "outer," "beneath," "below," "lower," "above," "upper," and the like, may be used herein for ease of description to describe the relationship of one element or feature to another element or feature as illustrated in the figures. Spatially or temporally relative terms may encompass different orientations of a device or system in use or operation in addition to the orientation illustrated in the figures.

[0045] Throughout this disclosure, numerical values ​​represent approximate measures or range limits to encompass small deviations from a given value, as well as examples having approximately the stated value, as well as examples having exactly the stated value. Except in the working examples provided at the end of the detailed description, all numerical values ​​of parameters (e.g., amounts or conditions) in this specification, including the appended claims, should be understood as being modified in all cases by the term "about", regardless of whether "about" actually appears before the numerical value. "About" indicates that the stated numerical value (along with some closeness to precision in the value, approximately or very close to the value, nearly) allows for some slight imprecision. Unless the imprecision provided by "about" is otherwise understood in the art using its ordinary meaning, "about", as used herein, indicates at least the variation that may result from ordinary methods of measuring and using such parameters. For example, "about" can include a variation of less than or equal to 5%, optionally less than or equal to 4%, optionally less than or equal to 3%, optionally less than or equal to 2%, optionally less than or equal to 1%, optionally less than or equal to 0.5%, and in some embodiments, optionally less than or equal to 0.1%.

[0046] Additionally, disclosure of a range includes disclosure of all values ​​and further divided ranges within the entire range, including the endpoints and subranges given for the range.

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0048] In some aspects, the present disclosure provides lidar systems and methods configured to detect not only the distance of an object, but also the material composition of the object. According to some examples, material composition classification can be achieved by polarization analysis processed using machine learning algorithms.

[0049] The optical elements of the systems described herein can be configured to bring all of the light emitted from a light source (e.g., a laser) into a known polarization state so that the shift in polarization can later be precisely measured. This light then travels until it reaches an object interface (an object is composed of one or more materials), at which point a portion of the light is diffusely reflected back.

[0050] This disclosure describes, among other things, a new method of surrounding perception by adding material and surface texture (MST) classification at each point of the lidar cloud to create a semantic map of the 3D space. MST classification, inferred from the polarization signature of the returning photons, may reduce ambiguity in the 3D point cloud and facilitate recognition of various objects (e.g., metal points, glass points, rough dielectric points, etc.). Polarization classification may precede surface tangent plane estimation and thus pre-identify objects by grouping points with similar polarization signatures. Lidar with MST classification is referred to herein as M-LIDAR.

[0051] According to one example of the present disclosure, M lidar technology can be configured to be lightweight and comfortable through the use of kirigami optics rather than traditional bulky optics, such as near-infrared optics. For example, a lidar system can employ kirigami composites as modulators, such as those described in U.S. Patent Application Serial No. 16 / 518,230, filed July 22, 2019, to Kotov et al., entitled "Kirigami Chiroptical Modulators for Circular Dichroism Measurements in Terahertz and Other Parts of Electromagnetic Spectrum." Such systems may include the use of periodically modified thin sheets of ultra-strong materials capable of diffracting light, such as the Kirigami-based systems described in XU et al., "Kirigami Nanocomposites as Wide-Angle Diffraction Gratings," ACS Nano, 10(6), 6156-6162 (2016), and Lizhi XU et al., "Origami and Kirigami Nanocomposites," ACS Nano, 11(8), 7587-7599 (2017). The relevant portions of all patents, patent applications, papers, and literature referenced or cited in this disclosure are hereby incorporated by reference. Varying the strain applied on the Kirigami sheet components changes the diffraction angle of the laser beam. Modulation can also be achieved by piezoelectric devices or by applying an electric field to conductive kirigami composites (Jing Lyu et al., "Stretchable conductors by kirigami patterning of aramid-silver nanocomposites with zero conductance gradient," Appl. Phys. Lett, 111, 161901 (2017)). Incorporation of such kirigami composite modulators can result in wide-angle, lightweight, and low-cost lidar.Furthermore, Kirigami modulators can be adapted to modulate a wide range of laser frequencies from optical frequencies to terahertz and beyond (W. J. Choi et al., "Chiroptical Kirigami Modulators for Terahertz Circular Dichroism Spectroscopy Biomaterials," Nature Materials, 18, 820-826 (2019)).

[0052] According to one example, the M-lidar systems and methods described herein may be used in black ice detection for vehicles with different degrees of automation.

[0053] Referring now to FIG. 1, a representative simplified M-lidar system is provided. The M-lidar system 100 may include a laser 102, a beam steerer 106, a first polarizer 114, a second polarizer 116, a first polarization detector 122, a second polarization detector 124, and a processor 126. Although FIG. 1 illustrates the first polarizer 114 and the second polarizer 116, and the first polarization detector 122 and the second polarization detector 124, according to some implementations, only a single polarizer (e.g., the first polarizer 114) and a single polarization detector (e.g., the first polarization detector 122) may be included as part of the system 100 without departing from the teachings of the present disclosure. Furthermore, according to some examples, three or more polarizers and / or three or more polarization detectors may be included as part of the system 100 without departing from the teachings herein.

[0054] For simplicity and purposes of illustration, throughout the remainder of this disclosure, the first polarizer 114 will be treated as an s-polarized linear polarizer 114. Similarly, for simplicity and purposes of illustration, the second polarizer 116 will be treated as a p-polarized linear polarizer 116. Additionally, the first polarization detector 122 will be treated as a p-polarized detector 122, and the second polarization detector will be treated as an s-polarized detector 124.

[0055] However, as will be appreciated by those skilled in the art, the polarizers 114, 116 may be configured for a variety of different types of polarization without departing from the teachings herein. For example, a given polarizer may be configured to perform linear polarization (e.g., s-type or p-type linear polarization), right-handed circular polarization, left-handed circular polarization, elliptically polarized light, or any other suitable type of polarization known in the art. Similarly, a given detector may be configured to detect linearly polarized light (e.g., s-type or p-type linearly polarized light), right-handed circular polarization, left-handed circular polarization, elliptically polarized light, or any other type of polarized light known in the art. According to some examples, the polarization of the light beam (i.e., a combination of two or more light pulses) may be modulated on a pulse-by-pulse basis to obtain additional information about one or more objects under consideration.

[0056] As described in further detail below, system 100 may be configured to detect one or more materials that make up object 110 and classify object 110 based at least in part on the detected materials. According to some examples, object classification may be performed using one or more artificial intelligence algorithms, including but not limited to neural network-based artificial intelligence.

[0057] In operation, system 100 may function as follows: Laser 102 may be configured to generate (i.e., emit) one or more polarized or unpolarized light pulses that collectively form polarized / unpolarized light beam 104. According to the example shown in Figure 1, each pulse includes an s-polarized component (represented in Figure 1 by dots along beam 104) and a transverse p-polarized component (represented in Figure 1 by a double-headed arrow traveling vertically through beam 104). Alternatively (and in conjunction with the previous discussion of different types of polarized light), the pulses may include, for example, left and right circularly polarized sequences, elliptically polarized sequences, any combination of the above, or any other suitable polarized light sequence.

[0058] According to some examples, the laser 102 may be configured to generate anywhere from one to over one million pulses per second. Furthermore, according to some implementations, the laser 102 may generate light having wavelengths within the range of visible light, ultraviolet light, infrared light, etc. Visible light has wavelengths ranging from about 390 nm to about 750 nm, and infrared radiation (IR) (including near infrared (NIR) ranging from about 0.75 μm to about 1.4 μm, and far infrared (FIR) ranging from about 15 μm to 1 mm). The far infrared (FIR) portion of the electromagnetic spectrum, also known as the terahertz (THz) diapason, has photon wavelengths from about 100 μm to about 1 mm, and energies from about 0.001 eV to about 0.01 eV. In some variations, the laser 102 may generate a 550 nanometer (nm), 808 nm, 905 nm, or 1550 nm pulsed laser, or any other suitable wavelength of laser, without departing from the teachings of this disclosure. For example, implementations for home robotics, autonomous vehicles, and machine vision may employ lasers with eye-safe frequencies above 800 nm. For outdoor applications, light beams in the water transparencies windows, for example, about 900 nm to 1550 nm, may be suitably employed. According to some implementations, upon generation of a given pulse by the laser 102, the processor 126 executing the executable instructions may record the initial time that the pulse was generated. This "time of flight" information may then be utilized to calculate the distance to the object 110 by using the speed of light.

[0059] The beam 104 may be directed by the laser 102 into the beam steerer 106. The beam steerer 106 may be configured to generate polarization-modulated light pulses. In some embodiments, the polarization of each polarized / unpolarized pulse of the polarized / unpolarized light beam 104 is modulated by the beam steerer 106. As used herein, modulating the polarization may include imparting a polarization or changing a polarization. Thus, the beam steerer 106 may modulate the polarization of each polarized / unpolarized pulse of the polarized / unpolarized light beam 104 to generate one or more linearly polarized light pulses (linearly polarized light pulses that collectively form the linearly polarized light beam a). Although the above examples contemplate linear polarization, the beam steerer 106 may circularly (e.g., left or right) or elliptically polarize the beam 104 according to some examples. According to another example, the beam steerer 106 may not apply any polarization to the beam. For example, if the beam 104 is already polarized when it enters the beam steerer 106, the beam steerer 106 may further modify the properties of the generated polarization-adjusted light pulses (e.g., split or modulate the pulses), but may not need to adjust the polarity of the previously polarized light pulses. Furthermore, according to some implementations, the beam steerer 106 may polarize a first pulse of the beam according to a first type of polarization and polarize a second pulse of the same beam according to a second, different type of polarization. In addition to or as an alternative to performing polarization of the beam 104, the beam steerer 106 may also control the direction of any beam (e.g., beam 108) emitted from the beam steerer 106. Furthermore, the beam steerer 106 may split a beam (e.g., beam 104) into several different beams, whereby one or more of the beams are emitted at a defined angle in order to steer multiple beams at once. This concept is illustrated in FIG. 1 with respect to the many branched arrows emanating from the beam steerer 106.

[0060] Additionally or alternatively, in some instances, the beam steerer 106 may be configured to modulate the linearly polarized light beam 108. In one instance, the beam steerer 106 may include a kirigami nanocomposite beam steerer or the like. According to this instance, as described in more detail below, the beam steerer 106 may be configured to linearly polarize and / or modulate the linearly polarized light beam 108 by increasing or decreasing the amount of strain applied to the kirigami nanocomposite beam steerer.

[0061] Further, according to one embodiment, the beam steerer 106 may be configured to linearly polarize each unpolarized pulse of the unpolarized light beam 104 by linearly polarizing each unpolarized pulse of the unpolarized light beam 104 for p-polarized light. This embodiment is shown in FIG. 1, where it can be seen that the beam 104 no longer contains an s-polarized component after passing through the beam steerer 106 (i.e., the "dot" component shown in the beam 104 is not present in the linearly polarized light beam 108). In an alternative embodiment, the linearly polarized light beam 108 may instead be p-polarized. Furthermore, in some embodiments, the beam steerer 106 may modify, control, and steer the linearly polarized light beam 108 emitted toward the object 110, as further described herein. The beam steerer 106 may enable dynamic wavelength-dependent beam steering and amplitude modulation of electromagnetic waves.

[0062] Continuing with FIG. 1, the linearly polarized light beam 108 may be diffusely reflected from the object 110. The one or more pulses of light collectively form a beam 112 that constitutes a reflected version of the linearly polarized light beam 108. According to some examples, the reflected linearly polarized light beam 112 may have a different polarization than the linearly polarized light beam 108 (i.e., the beam before reflection from the object 110). This difference in state is illustrated by the beam 112 including both p-polarized and s-polarized components (reflected by a dot and a double-headed arrow along the path of the beam 112, respectively), whereas the beam 108 is shown to include only a p-polarized component. Furthermore, the object 110 may include any suitable object (or target) of one or more different materials, the detection of which is desired. Although described above and in the following sections as being a reflected "linearly" polarized light beam 112, according to some examples, the reflected beam 112 may be polarized in a variety of different ways, including circularly or elliptically, without departing from the teachings herein.

[0063] The reflected linearly polarized light beam 112, diffusely reflected or scattered from or otherwise emitted by the object 112, may pass through an s-polarization linear polarizer 114 and / or a p-polarization linear polarizer 116 of the system 100. In some embodiments, each portion of the reflected or scattered or otherwise emitted linearly polarized light beam 112 passes through both an s-polarization linear polarizer 114 and / or a p-polarization linear polarizer 116 of the system 100. The s-polarization linear polarizer 114 is configured to linearly polarize one or more light pulses constituting the beam 112 for s-polarization to generate one or more reflected s-polarized light pulses (one or more reflected s-polarized light pulses that collectively form a reflected s-polarized light beam 118). Similarly, p-polarized linear polarizer 116 is configured to linearly polarize one or more light pulses constituting beam 112 for p-polarization to generate one or more reflected p-polarized light pulses (one or more reflected p-polarized light pulses that collectively form reflected p-polarized light beam 120). According to some examples, s-polarized linear polarizer 114 and / or p-polarized linear polarizer 116 may include kirigami nanocomposites, such as kirigami nanocomposites of the type described above with respect to beam steerer 106 and / or the type described below with respect to Figures 4a-4d and 5a-5c. However, those skilled in the art will recognize that according to some examples, non-kirigami nanocomposites or other optical devices may be employed as part of system 100 without departing from the teachings herein.

[0064] According to some examples, a similar configuration of polarizers 114, 116 may be utilized for polarization of left and right circularly polarized or elliptically polarized light reflected or scattered or otherwise emitted from object 110.

[0065] The s-polarized light detector 122 may be configured to detect an intensity of each of the one or more reflected s-polarized light pulses forming the reflected s-polarized light beam 118. Additionally, according to some implementations, the s-polarized light detector 122 may be configured to detect an angle of incidence associated with the reflected s-polarized light beam 118. The detected intensity of the one or more reflected s-polarized light pulses forming the reflected s-polarized light beam 118 and / or the detected angle of incidence associated with the reflected s-polarized light beam 118 may be utilized by the processor 126 to perform material type detection (e.g., using MST classification), as described in more detail below.

[0066] Similarly, the p-polarized light detector 124 may be configured to detect an intensity of each of the one or more reflected p-polarized light pulses forming the reflected p-polarized light beam 120. Additionally, according to some implementations, the p-polarized light detector 124 may be configured to detect an angle of incidence associated with the reflected p-polarized light beam 120. The detected intensity of the one or more reflected p-polarized light pulses forming the reflected p-polarized light beam 120 and / or the detected angle of incidence associated with the reflected p-polarized light beam 120 may also be utilized by the processor 126 to perform material type detection, as described in further detail below.

[0067] The processor 126 is configured to detect at least one material of the object 110 based on (i) the detected intensity of one or more light pulses forming the beams 118 and / or 120, and / or (ii) the detected angle of incidence associated with the reflected s-polarized light beam 118 and / or the reflected p-polarized light beam 120. More specifically, according to some examples, the processor 126 is configured to apply a machine learning algorithm to detect one or more materials comprising the object 110. As used herein, "applying a machine learning learning algorithm" may include, but is not limited to, executing executable instructions stored in a memory and accessible by the processor. Furthermore, according to one example, a particular machine learning algorithm used for material detection may include an artificial neural network. However, other machine learning algorithms known in the art may be suitably employed without departing from the teachings of the present disclosure.

[0068] Further, according to some examples, the processor 126 may be configured to classify the object 110 based on the detected material of the object 110 by applying a machine learning algorithm. Again, the machine learning algorithm used for object classification may include an artificial neural network. However, other machine learning algorithms known in the art may be suitably employed without departing from the teachings of the present disclosure.

[0069] Before proceeding to Figure 2, below is an overview of a process for detecting the material of an object utilizing an M LIDAR system, such as system 100 shown in Figure 1.

[0070] As noted above, one objective of the present disclosure is to enable object material detection and reduce the data processing required for modern lidar devices by acquiring more data at each point in the point cloud. This additional polarimetric data, when combined with machine learning algorithms, enables material detection, simplifying object recognition for a variety of applications, including, but not limited to, autonomous vehicles, machine vision, medical applications (e.g., devices to assist the visually impaired), and advanced robotics.

[0071] An M-lidar system according to an exemplary implementation of the present disclosure may operate as follows: A pair of detectors (e.g., detectors 122, 124) with vertically oriented linear polarizers (e.g., polarizers 114, 116) may be used to measure the returning light (e.g., one or more pulses of light constituting a reflected version of the linearly polarized light beam 112). A portion of the diffuse backscattered light (e.g., reflected light 112) may be directed to the detectors (e.g., detectors 122, 124) and pass through narrow-band interference filters (e.g., linear polarizers 114, 116) placed in front of each detector pair (e.g., detector pair 122 / 124). The narrow-band interference filters may only allow a small range of wavelengths (e.g., 1-2 nm) to pass, thereby reducing unwanted noise from ambient illumination or external light sources.

[0072] According to other examples of the above system, the system may be configured to detect circularly and / or elliptically polarized light reflected, scattered or otherwise emitted by an object and perform machine learning processing on the light.

[0073] This selectivity allows an M-lidar system (e.g., system 100) according to the present disclosure to be configured to simultaneously measure multiple wavelengths completely independently. The coherent light can then be polarized, for example, using a co-polarizer and / or a cross-polarizer. Depending on the shift in polarization upon reflection from an object (e.g., object 110), the intensity of the light may be reduced by some amount as it travels through the polarizer. Beam focusing optics (e.g., polarizers 114, 116) direct the coherent polarized light (e.g., beams 118, 120) toward a detection surface (e.g., the surface of detectors 122, 124), and based on the location where the light hits the detection screen, the angle from which the returning light traveled (i.e., the angle of incidence) can be detected.

[0074] Once the detector identifies the light, a time-of-flight sensor (e.g., a time-of-flight sensor implemented in processor 126) may record the travel time of that light pulse. Each light pulse may have its intensity measured for both the co-polarized and cross-polarized detectors, and combining these two values ​​allows for the quantification of the polarization effects caused during reflection.

[0075] Following this process, the following parameters can be detected: (i) the initial angle at which the beam was steered, (ii) the angle from which the backscattered light returned, (iii) the time of flight from emission to detection, and (iv) the intensity at each detector.

[0076] Note that due to the detectors being in different locations, a single pulse of light may take a slightly different amount of time to reach each detector. By understanding the system geometry and the relationship between intensity and distance, this difference can be compensated for and the intensity at one detector can be precisely adjusted. The time-of-flight data can be used to determine the distance between the light source (e.g., laser 102) and the object (e.g., object 110), and along with the initial and return angles, the specific location of that point in space relative to the M-lidar system can be determined. These compensated intensity values ​​can contain information indicative of the material from which the light pulse was reflected. Utilizing these values, machine learning algorithms can provide robust and comprehensive material recognition capabilities.

[0077] The process described above of emitting a pulse of light, the light being diffusely reflected off an object, the reflected light being measured by a detector, and the location of the object relative to the light source being determined can be repeated anywhere from once to millions of times per second, generating a point each time, and these points are then mapped onto the same coordinate system to create a point cloud.

[0078] Once the point cloud is generated, one or more machine learning algorithms may be used to cluster the points into objects, and ultimately to characterize the respective materials of each object (e.g., if multiple objects are detected). The points may be clustered, for example, based on one or more values ​​of measured intensity, and in some instances, based on the proximity of similar points.

[0079] Once a cluster has been determined using the intensity values, machine learning or artificial intelligence algorithms can be used to correlate the measured values ​​to a database of known materials in order to classify the material of that cluster of points. This process can be repeated for all clusters in the system. Understanding the surrounding materials allows the system (e.g. implemented in a car, robot, drone, etc.) to make faster and more reasoned decisions about what the object itself may be. From there, factors such as the hazards involved can be evaluated and a decision made thereafter (e.g. in the case of a system that detects black ice in front of a vehicle). As the process continues over time, more information can be extracted from the perceived changes and an even better understanding of the surrounding situation that has occurred.

[0080] The MST classification technique is also applicable to the detection of objects with surfaces modified to improve detection, for example, painted or textured with macro-, micro-, nano-, or molecular patterns to generate a reflected beam with a specific optical response adapted for high-speed MST classification by lidar. Examples of such surface treatments include paints containing additives that generate reflected, scattered, or otherwise emitted light with a specific linear, circular, or elliptical polarization. In one case, metal nano / micro / macro wires or axial carbon nanomaterials are added to a base paint. The array pattern can be random, linear, spiral, herring-bone, or any other pattern that generates a specific polarization signature that allows for high-speed identification of a specific object. As a non-limiting example, this can be used to create road markers, road signs, barriers, pylons, guard rails, vehicles, bicycles, clothing, and other objects.

[0081] Another implementation of surface treatments to facilitate MST classification may include the addition of chiral inorganic nanoparticles to the base paint used to coat such objects as road markers, vehicles, bicycles, clothing, etc., as described above. Chiral nanoparticles may display a specific, extremely strong circular polarization response to the beams used by LIDAR. Chiral nanoparticles may be mixed in the paint in specific ratios to generate a polarization signature (e.g., a "barcode") for a particular object.

[0082] Another example of polarization tagging of objects can include using surface texturing to generate a specific polarization response. One example of such texturing can include creating nanoscale patterns of metallic, semiconducting, insulating or ceramic nanoparticles with specific geometric features that result in a defined polarization response to the laser in the LIDAR. Two examples of such patterns include (a) linear nanoscale or microscale surface features that result in a linear polarization of the reflected, scattered or emitted light from the object, and (b) out-of-plane protruding chiral patterns on metal surfaces that result in a specific chirality and therefore a circular polarization of the reflected, scattered or emitted light from the object.

[0083] According to some examples, the above systems and methods may be employed to accurately recognize materials for use in autonomous vehicles, machine learning, medical applications, and advanced robotics.

[0084] Existing conventional lidar systems work primarily through the measurement of the distance between the object and the laser source, typically using either time-of-flight data or phase shift. In such cases, objects are classified based on the shape and pattern in the arrangement of points in the cloud. Some more advanced lidar point cloud classification methods utilize an additional parameter, namely the overall intensity.

[0085] Based on how strong the signal of the returning light pulse is, the system can effectively detect color differences. This additional piece of data makes it easier to recognize object boundaries in the point cloud and reduces the amount of processing required to classify all the points. However, applications such as autonomous vehicles may require a higher degree of confidence that total intensity cannot achieve. Additionally, detection of objects at long ranges can be achieved using single point detection utilizing MST classification, rather than the type of multipoint detection and processing employed in traditional lidar systems.

[0086] Thus, the method described herein changes the approach that machine vision currently takes for object recognition. Instead of relying only on shape, motion, and color to determine the identity of an object, the system described herein also takes into account another parameter: polarization. Upon reflection from a material interface, light undergoes some change in polarization. This change in polarization is quantified by measuring the intensity of the light after passing through both co-polarizing and cross-polarizing filters. This additional data can be paired with machine learning techniques to significantly improve clustering and therefore object recognition capabilities. Older object recognition methods are quite computationally expensive. The method described herein can significantly reduce the processing power required by lidar by using a material-based approach rather than the current shape-based approach.

[0087] In addition to traditional distance measurements, the polarized light data collected by this system allows machine learning algorithms to determine the material an object is made of. Current lidar systems lack awareness or information about materials in the surrounding environment. When obtained, such information provides context for better situational understanding and smarter decisions. For autonomous vehicles, this increased understanding of the environment can lead to improved occupant safety, as accurate and timely detection of potential hazards improves decision-making capabilities.

[0088] Turning now to Figures 2a-b, scanning electron microscope (SEM) images of nano-kirigami sheets are shown that may be used to form nano-kirigami nanocomposite optical components that are incorporated into M-lidar systems. Optically active kirigami sheets, such as those shown in Figures 2a-b, may be fabricated from ultra-strong nanoscale composites with cut patterns of 0.5-5 μm in length, according to some examples of the present disclosure. In some embodiments, composites (including highly conductive composites) may be modified by using concepts from the ancient Japanese paper-cutting technique known as "kirigami." See Lizhi Xu et al., "Kirigami Nanocomposites as Wide-Angle Diffraction Gratings," ACS Nano, 10(6), pp. 6156-6162 (2016); Lizhi Xu et al., "Origami and Kirigami Nanocomposites," ACS Nano, 11(8), pp. 7587-7599 (2017). Kirigami techniques can be used to engineer elasticity by using multiple cuts or notches that create a network on a flat polymeric material, such as a composite or nanocomposite material. Such cuts (e.g., extending from one side of the material to the other in a polymer or composite material) can be made by top-down patterning techniques, such as photolithography, to distribute stresses uniformly within the polymer or nanocomposite material and suppress uncontrolled high stress singularities. This technique prevents unpredictable localized fracture and, as a non-limiting example, can increase the ultimate strain of a rigid sheet from 4% to 370%.

[0089] By using microscale kirigami patterning, stiff nanocomposite sheets can achieve high extensibility. Moreover, kirigami-cut patterned composite sheets maintain their electrical conductivity throughout the strain regime, in stark contrast to most stretchy conductive materials. Kirigami structures can include composites, such as nanocomposites. In some embodiments, kirigami structures can be multilayer structures having at least two layers, at least one of which is a polymeric material. The polymeric material can be a composite or nanocomposite material. Composite materials include a matrix material, such as a polymer, polyelectrolyte, or other matrix (e.g., cellulose paper), and at least one reinforcing material dispersed therein. In some embodiments, nanocomposite materials are particularly suitable for use in kirigami structures, which are composites that include reinforcing nanomaterials, such as nanoparticles. The composites can take the form of a sheet or film, in some variations.

[0090] A "nanoparticle" is a solid or semi-solid material that can have a variety of shapes or forms, but which are generally understood by those skilled in the art to mean that the particle has at least one spatial dimension that is about 10 μm (10,000 nm) or less. In some embodiments, the nanoparticles have a relatively low aspect ratio (AR) (defined as the length of the longest axis of the component divided by the diameter) of about 100 or less, optionally about 50 or less, optionally about 25 or less, optionally about 20 or less, optionally about 15 or less, optionally about 10 or less, optionally about 5 or less, and in some variations, an aspect ratio equal to about 1. In other embodiments, the nanoparticles have a tube or fiber shape with a relatively high aspect ratio (AR) of about 100 or more, optionally about 1000 or more, and in some variations, optionally about 10,000 or more.

[0091] In some variations, the longest dimension of the nanoparticles is about 100 nm or less. In some examples, the nanoparticles selected for inclusion in the nanocomposite are conductive nanoparticles that produce a conductive nanocomposite material. The nanoparticles can be substantially circular nanoparticles with low aspect ratios as defined above, having morphologies or shapes including spherical, spheroidal, hemispherical, discoidal, spherical, annular, toroidal, cylindrical, diskoid, dome-shaped, oval, elliptical, orb-shaped, oval, and the like. In some preferred variations, the nanoparticle morphology has a spherical shape. Alternatively, the nanoparticles can have alternative shapes, such as filaments, fibers, rods, nanotubes, nanostars, or nanoshells. The nanocomposite can also include combinations of any such nanoparticles.

[0092] Further, in some embodiments, nanoparticles particularly suitable for use according to the present teachings have a particle size (average diameter of the nanoparticles present) of about 10 nm or more to about 100 nm or less. Conductive nanoparticles can be formed from a variety of conductive materials, including metallic, semiconducting, ceramic, and / or polymeric nanoscale particulates having a number of shapes. The nanoparticles can have magnetic or paramagnetic properties. The nanoparticles can include conductive materials such as carbon, graphene / graphite, graphene oxide, gold, silver, copper, aluminum, nickel, iron, platinum, silicon, cadmium, mercury, lead, molybdenum, iron, and alloys or compounds thereof. Thus, suitable nanoparticles include, but are not limited to, graphene oxide, graphene, gold, silver, copper, nickel, iron, carbon, platinum, silicon, seed ring metals, CdTe, CdSe, CdS, HgTe, HgSe, HgS, PbTe, PbSe, PbS, MoS2, FeS2, FeS, FeSe, WO 3-x, and other similar materials known to those skilled in the art. Graphene oxide is a particularly suitable conductive material for use as a reinforcement in composites. In some variations, the nanoparticles can include carbon nanotubes, such as single walled nanotubes (SWNTs) or multi-walled nanotubes (MWNTs), e.g., SWNTs are formed from a single sheet of graphite or graphene, while MWNTs include multiple cylinders arranged in a concentric fashion. Typical diameters of SWNTs can range from about 0.8 nm to about 2 nm, while MWNTs can have diameters greater than 100 nm.

[0093] In some variations, the nanocomposite can have a total amount of the plurality of nanoparticles of about 1% or more to about 97% or less by weight, optionally about 3% or more to about 95% or less by weight, optionally about 5% or more to about 75% or less by weight, optionally about 7% or more to about 60% or less by weight, and optionally about 10% or more to about 50% or less by weight of the total amount of nanoparticles in the nanocomposite. Of course, the appropriate amount of nanoparticles in the composite will depend on the material properties, percolation threshold, and other parameters for a particular type of nanoparticle in a particular matrix material.

[0094] In some variations, the nanocomposite can have a total amount of polymer matrix material of about 1% to about 97% by weight, optionally about 10% to about 95% by weight, optionally about 15% to about 90% by weight, optionally about 25% to about 85% by weight, optionally about 35% to about 75% by weight, and optionally about 40% to about 70% by weight of the total amount of matrix material in the nanocomposite.

[0095] In some variations, the nanocomposite material comprises a plurality of conductive nanoparticles, and is about 1.5×10 3 In some other embodiments, the nanocomposite material may include a plurality of conductive nanoparticles as the reinforcing nanomaterial, thus having a conductivity of about 1×10 -4In some other variations, the impedance (Z) of a conductive nanocomposite including a plurality of nanoparticles can be about 1×10 (e.g., measured using an AC sine wave signal of amplitude 25 mV, with impedance values ​​measured at a frequency of 1 kHz). 4 It may be an ohm or less.

[0096] The polymer or nanocomposite material may be in a flat form, such as a sheet, in an initial state (before being cut), but may be folded or formed into a three-dimensional structure and thus used as a structural component after the cutting process. As an example, structure 220 is shown in FIG. 10, including a portion of an exemplary nanocomposite material sheet 230 having a surface with a tessellated cut pattern. The sheet 230 includes a first row 232 of first discontinuous cuts 242 (extending into the sheet 230 to generate openings) in a pattern that defines first uncut areas 252 between the first discontinuous cuts 242. A discontinuous cut is a partial or individual cut made in a sheet that keeps the entire sheet intact with its original dimensions, rather than dividing it into separate smaller sheets or portions. When multiple discontinuous cuts 242 are present, at least some of the multiple discontinuous cuts 242 are non-continuous and not connected to each other, such that at least one uncut area remains on the sheet as a bridge between the discontinuous sheets. While many cut patterns are possible, a simple paper cutting pattern of straight lines in a centered rectangular configuration as shown in Figure 10 is used herein as an exemplary pattern. The first uncut area 252 has a length "x". Each discontinuous cut 242 has a length "L".

[0097] In some embodiments, the length of each discrete cut (e.g., discrete cut 242) can be microscale, mesoscale, nanoscale, and / or macroscale. A macroscale is generally considered to have a dimension of about 500 μm (0.5 mm) or more, while a mesoscale is about 1 μm (1000 nm) or more to about 500 μm (0.5 mm) or less. A microscale is generally considered to be about 100 μm (0.5 mm) or less, while a nanoscale is generally about 1 μm (1000 nm) or less. Thus, the conventional mesoscale, microscale, and nanoscale dimensions may be considered to overlap. In some embodiments, the length of each discrete cut may be on the microscale, for example, a length that is less than about 100 μm (i.e., 100,000 nm), optionally less than about 50 μm (i.e., 50,000 nm), optionally less than about 10 μm (i.e., 10,000 nm), optionally less than about 5 μm (i.e., 5,000 nm), and in some embodiments, less than about 1 μm (i.e., 1,000 nm). In some embodiments, discrete cuts 42 may have a length that is less than about 50 μm (i.e., 50,000 nm), optionally less than about 10 μm (i.e., 10,000 nm), and optionally less than about 1 μm (i.e., less than about 1,000 nm).

[0098] In some other variations, these dimensions may be reduced by at least 100 times to the nanoscale, e.g., to cuts having lengths of about 1 μm (1000 nm) or less, optionally about 500 nm or less, and in some variations, optionally about 100 nm or less.

[0099] Note that "x" and "L" may vary within a row depending on the pattern being formed, but in the preferred embodiment, these dimensions remain constant.

[0100] A second row 234 of second discontinuous cuts 244 are also patterned on the sheet 230. The second discontinuous cuts 244 define second uncut areas 254 therebetween. A third row 236 of third discontinuous cuts 246 are also patterned on the sheet 230. The third discontinuous cuts 246 define third uncut areas 256 therebetween. Note that the first row 232, the second row 234, and the third row 236 are used for illustrative and nominative purposes, but as can be seen, the tessellated pattern on the surface of the sheet 230 has more than three distinct rows. The first row 232 is spaced apart from the second row 234, as indicated by the designation "y". The second row 234 is similarly spaced apart from the third row 236. Note that "y" can vary between rows, but in some embodiments, "y" remains constant between rows. Such spacing between rows can be microscale, mesoscale, nanoscale, and / or macroscale, as similarly described above.

[0101] In particular, the first discontinuous cut 242 in the first row 232 is offset laterally (along the dimension / axis designated as "x") from the second discontinuous cut 244 in the second row 234, thus forming a tessellated pattern. Similarly, the second discontinuous cut 244 in the second row 234 is offset laterally from the third discontinuous cut 246 in the third row 236. Thus, the first uncut region 252, the second uncut region 254, and the third uncut region 256 in each respective row cooperate to form a structural bridge 260 that extends from the first row 232 across the second row 234 to the third row 236.

[0102] In this regard, sheet 230 having a patterned tessellated surface with a plurality of discontinuous cuts (e.g., 242, 244, and 246) can be stretched in at least one direction (e.g., along a dimension / axis designated as "y" or "x"). Sheet 230 formed from nanocomposites thus exhibits several advantageous properties, including enhanced strain.

[0103] Various aspects contemplate optical devices incorporating stretchable multilayer polymer or composite materials formed by the kirigami process. "Stretchable" means that the materials, structures, components, and devices are capable of withstanding strain without fracturing or other mechanical failure. Stretchable materials are stretchable and therefore capable of stretching and / or compressing, at least to some extent, without damage, mechanical failure, or significant degradation in performance.

[0104] "Young's modulus" is a mechanical property that refers to the ratio of stress to strain for a given material. Young's modulus is given by the formula:

number

[0105] In some embodiments, the extensible composite materials, structures, components, and devices can undergo a maximum tensile strain of at least about 50% without fracture, optionally about 75% or more without fracture, optionally about 100% or more without fracture, optionally about 150% or more without fracture, optionally about 200% or more without fracture, optionally about 250% or more without fracture, optionally about 300% or more without fracture, optionally about 350% or more without fracture, and in some embodiments, about 370% or more without fracture.

[0106] In addition to being stretchable, an extensible material can also be flexible, and thus capable of large elongations, flexions, bending, or other deformations along one or more axes. The term "flexible" can refer to the ability of a material, structure, or component to be deformed (e.g., into a curved shape) without undergoing permanent deformation that results in large strains, such as strains that indicate the breaking point of the material, structure, or component.

[0107] Thus, the present disclosure provides in some aspects a tensile polymeric material. In a further aspect, the present disclosure provides a tensile composite material comprising a polymer and a plurality of nanoparticles or other reinforcing materials. The polymer can be an elastomeric or thermoplastic polymer. One suitable polymer includes, as a non-limiting example, polyvinyl alcohol (PVA).

[0108] For example, for some materials, creating a surface with patterned kirigami cuts according to some embodiments of the present disclosure can increase the ultimate strain of an initially rigid sheet by about 100% or more, optionally about 500% or more, optionally about 1000% or more, and in some variations, optionally about 9000% or more from the initial ultimate strain before any cuts.

[0109] In particular, a wide range of maximum achievable strain or extension levels can be achieved based on the geometry of the cut pattern used. The ultimate strain is therefore determined by the geometry. The ultimate strain (% strain) is a function of the original or initial length (L i ), to the final achievable length while the structure is stretched to the point before it breaks, i.e.

number

[0110] In some embodiments, kirigami nanocomposites can form tunable optical lattice structures that can maintain a stable period over macroscopic length scales even under 100% stretching. The lateral spacing in the diffraction pattern shows a negative correlation with the amount of stretching, which is consistent with an inverse relationship between the dimensions in the diffraction pattern and the spacing of the corresponding gratings. The longitudinal spacing in the diffraction pattern exhibits a smaller dependence on the amount of stretching, since the change in longitudinal period is relatively small with lateral stretching. The diffraction pattern also shows a large dependence on the wavelength of the incident laser. The polymer stretchable tunable optical lattice structures exhibit elastic behavior upon stretching and spontaneously return to a relaxed (i.e., unstretched) shape as the stretching is removed under cyclic mechanical actuation. The diffracted beam forms a clear pattern that changes according to the deformation of the polymer stretchable tunable optical lattice structures. This behavior indicates an excellent capability for dynamic wavelength-dependent beam steering.

[0111] Three-dimensional (3D) kirigami nanocomposites thus provide a new dimension to traditional reflective and refractive optical systems with out-of-plane surface features, as shown in FIGS. 2a-b. For example, reconfigurable fins and slits formed by the cuts shown in the nanokirigami sheets shown in FIGS. 2a-b allow efficient modulation of light by reversible expansion (or strain levels) of the kirigami cut sheets. Thus, nanokirigami sheets, such as the nanokirigami sheets shown in FIGS. 2a-b, may be incorporated into one or more optical components of the M-lidar systems described herein. More specifically, these lightweight, thin, and inexpensive optical components may be used for the red and infrared portions of the optical spectrum, for example, to achieve beam steering and / or polarization modulation. According to some implementations, kirigami nanocomposites of the type shown in FIGS. 2a-b may be utilized to form the beam steer 106, s-polarized linear polarizer 114, and / or p-polarized linear polarizer 116 of the system shown in FIG. 1.

[0112] In some variations, kirigami nanocomposites can form kirigami optical modules fabricated from ultra-strong layer-by-layer (LbL) assembled nanocomposites (Kotov, N. A. et al., "Ultrathin graphite oxide-polyelectrolyte composites prepared by self-assembly: Transition between conductive and non-conductive states," Adv. Mater., 8, 637-641 (1996); and Kotov, N. A. et al., "Layer-by-Layer Self-assembly of Polyelectrolyte- Semiconductor Nanoparticle Composite Films," J. Phys. Chem., 99, 13065-13069 (1995)). These nanocomposite materials have high strength, e.g., about 650 MPa, and elastic modulus (E) of about 350 GPa, imparting exceptional mechanical properties, environmental robustness, e.g., with a wide temperature range of operation (e.g., from -40°C to +40°C), and proven scalability. The high elasticity of LbL composites makes them reconfigurable, and the high temperature elasticity of LbL composites allows for integration with different types of actuators, and CMOS compatibility. In some embodiments, the nanocomposite materials can be coated with plasmonic films, such as titanium nitride, gold, etc., to improve interaction with the target wavelength of photons, e.g., 1550 nm photons, where a laser light source has a wavelength of 1550 nm.

[0113] In some other variations, the kirigami nanocomposite sheets can include magnetic materials dispersed therein or coated thereon. For example, a layer of nickel can be deposited on the superstrong composite. The nickel layer can act as a magnetic and reflective layer, thus giving a magentoactive kirigami element. The kirigami unit can thus be directly integrated with the lidar component and act as a beam steerer (e.g., using the 1st and 2nd order diffracted beams) or as a polarizer (e.g., using the 1st order diffracted beam).

[0114] Referring now to Figures 3a-c, images are shown illustrating the laser diffraction patterns from the nano-kirigami-based graphene composite. For reference, the scale bar shown in the upper right corner of Figures 3a-c represents 25 mm. Figure 3a shows the laser diffraction pattern for the nano-kirigami-based graphene composite at 0% strain (relaxed state). Figure 3b shows the laser diffraction pattern for the nano-kirigami-based graphene composite at 50% strain. Finally, Figure 3c shows the laser diffraction pattern for the nano-kirigami-based graphene composite at 100% strain.

[0115] Polarization modulation of the lidar beam and polarization analysis of the returning photons allows for the collection of information about object materials, which is currently missing, for example, in car safety and robotic vision devices. Machine learning (ML) algorithms can be trained to recognize different materials and MST classification based on the unique polarization signatures of those different materials can be achieved. MST classification of objects by material can accelerate object recognition and improve the accuracy of machine perception of the surroundings, among other benefits.

[0116] Before turning to the details of FIGS. 4a-4d, it is noted that system 100 of FIG. 1 and corresponding methods of material detection and object classification may be implemented utilizing non-nano-kirigami optical elements, according to some examples of the present disclosure. Indeed, such non-nano-kirigami optical element based M LIDAR systems may be preferred for some applications (e.g., where size and weight are not primary concerns). Thus, in implementations of the presently disclosed M LIDAR systems in which nano-kirigami optical elements are not utilized, other traditional optical components such as (i) infrared (IR) polarizers, (ii) beam splitters, (iii) lenses fabricated from CdS, ZnS, silicon, and / or (iv) similarly suitable optical elements may equally be employed without departing from the teachings herein. However, nano-kirigami optical elements are generally preferred for M LIDAR systems that benefit from being lightweight and small.

[0117] With that as background, Figures 4a-4d show a step-by-step lithography type process for manufacturing nano-kirigami-based optical elements, such as beam splitters or linear polarizers. According to one example, the process for manufacturing nano-kirigami-based optical elements described in Figures 4a-4d can include using vacuum assisted filtration (VAF), where a nanocomposite material can be deposited as a layer on a rigid (e.g., plastic) substrate suitable for lithographic patterning. As mentioned above, US Publication No. 2016 / 0299270 describes a method for making such nanocomposite materials, such as by vacuum assisted filtration (VAF) and layer-by-layer (LBL) deposition process techniques. Nanocomposites produced by this process are known to exhibit high toughness and strong optical absorption.

[0118] FIG. 4a is a simplified diagram of a first step of process 400 in which a nanocomposite layer 404a is deposited on a substrate 402 by VAF, layer-by-layer deposition (LBL), or any other suitable deposition method known in the art. FIG. 4b illustrates a second step in process 400 after the nanocomposite 404a of FIG. 4 is patterned, for example by a photolithographic cutting process, through selected areas of the nanocomposite layer 404a to generate a patterned kirigami nanocomposite 404b on the substrate 402. FIG. 4c illustrates a third step in process 400 in which the cut or patterned kirigami nanocomposite 404b is released (e.g., lifted) from the substrate 402. Finally, FIG. 4d illustrates a final step of process 400 in which at least a portion of the patterned kirigami nanocomposite 408 is incorporated into a subassembly configured for, among other things, beam steering and / or modulation.

[0119] The subassembly shown in Figure 4d includes a patterned kirigami nanocomposite portion 408, a micromachined silicon layer 406 housing, and one or more bending beam actuators 410. A double sided arrow 412 indicates the possible directions of movement of the actuators 410. As described in further detail below, the bending beam actuators 410 may be configured to apply a reversible strain on the kirigami nanocomposite portion 408, for example, to adjust the size and / or orientation of the various slits and / or fins that make up the pattern of the kirigami nanocomposite portion 408. The kirigami nanocomposite portion 408 may thus be reversibly stretched at strain levels ranging from 0% to 100%.

[0120] A more detailed description of the patterning aspect of the process 400 shown in Figures 4a-4d follows. The fabrication of the kirigami transmission optical module can follow the step-by-step diagrams shown in Figures 4a-4d. Modulation of the lidar laser beam in the visible and infrared (IR) range can require feature sizes at 3 μm, for example, created over a width of 0.1-1 cm. The feasibility of such patterns has already been demonstrated. The 2D shapes of the patterns can be selected based on computer simulations of their 2D-3D reconstruction when stretched or distorted. The 3D shapes can be modeled for optical properties, for example, polarization modulation in the desired wavelength range. Photolithography can be the primary patterning tool, enabled by the chemistry of the VAF compounds described above. The patterning protocols can be substantially similar to those currently used for large-scale microfabrication. For example, the VAF compounds on a glass substrate can be coated by standard SU8 photoresist after photopatterning using a commercial conventional mask aligner. Examples of constructed kirigami patterns are shown in Figures 5a-c, which are described in more detail below.

[0121] Kirigami optical elements can be fabricated, for example, by integrating kirigami nanocomposite sheets with commercial microelectromechanical actuators, as shown in FIG. 4d. Microelectromechanical system (MEMS) kirigami units can be directly integrated with lidar components and act as beam steerers (e.g., using the first and second order diffracted beams) and / or polarizers (e.g., using the first order diffracted beams). Considering the almost infinite number of kirigami patterns and the wide variety of 2D-3D reconstructions, kirigami optical elements with both beam steering and polarization functions, as well as other optical functions, are contemplated within the teachings herein.

[0122] With brief reference to Figures 5a-5c, various images of an exemplary kirigami optical element are shown. For example, Figure 5a is an image of a kirigami optical element, such as those described herein, fabricated on a wafer according to process 400 described above with respect to Figures 4a-4d. Figure 5b is a scanning electron microscope (SEM) image of the kirigami optical element of Figure 5a at less than 0% strain. Finally, Figure 5c is an SEM image of the kirigami optical element of Figure 5a at less than 100% strain. The scale bars shown in the upper right corners of Figures 5b-5c are 50 μm.

[0123] Photolithography techniques can be used to fabricate kirigami transmission or reflection optical modules / elements. As an example, modulation of a lidar laser beam having a wavelength of about 1550 nm can have feature sizes of about 1 μm or more to about 2 μm or less, created over widths ranging from about 0.1 cm or more to about 1 cm or less, as exemplified by the present patterns in Figures 5a-c. The two-dimensional (2D) shapes of the patterns can be selected based on computer simulations of their 2D to three-dimensional (3D) reconfiguration when stretched. The 3D shapes can be modeled for optical properties, e.g., polarization modulation in the desired wavelength range.

[0124] Photolithography is the primary patterning technique that can be used in combination with LbL compounding to form kirigami optical elements. In one example, the patterning protocol can include applying LbL compounding onto a glass substrate coated with standard SU-8 photoresist followed by photopatterning using a commercial mask aligner (UM Lurie Nanofabrication Facility, LNF). Such a process can form kirigami elements such as those shown in Figures 5a-c.

[0125] Another representative simplified compact M LIDAR system 300 for use in a vehicle, such as an autonomous vehicle, is given in FIG. 11. To the extent that the components in the M LIDAR system 300 are similar to those in the M LIDAR system 100 of FIG. 1, their functions will not be repeated here for brevity. The M LIDAR system 300 may include a laser 310, a beam steerer 312, one or more polarizers (not shown, but similar to the first polarizer 114 and the second polarizer 116 described in the context of FIG. 1), and a processor (not shown, but similar to the processor 126 shown in FIG. 1). In the M LIDAR system 300, a pulse generator 312 is connected to the laser 310 and generates a polarized or unpolarized first optical pulse 314 and a polarized or unpolarized second optical pulse 316. The pulse generator 312 is connected to an oscilloscope 324. The first light pulse 314 and the second light pulse 316 generated by the laser 310 are directed to a beam steerer 318, which in some aspects may be a kirigami-based beam steerer such as those previously described above. The beam steerer 318 is connected to and controlled by a servo motor / Arduino 352. The servo motor / Arduino 352 is connected to a controller 350, which may be, as a non-limiting example, a MATLAB® control. As previously described above, the beam steerer 318 may polarize, modify, split, and / or modulate one or both of the first light pulse 314 and the second light pulse 316, as a non-limiting example, as previously described above. The first light pulse 314 and the second light pulse 316 are then directed to an object 340 to be detected.

[0126] The first light pulse 314 and the second light pulse 316 may be diffusely reflected from the object 110. The one or more pulses of light collectively form a first reflected beam 342 and a second reflected beam 344 that constitute reflected versions of the first light pulse 314 and the second light pulse 316. According to some examples, the first reflected beam 342 and the second reflected beam 344 may have a different polarization than the first light pulse 314 and the second light pulse 316 (i.e., before reflection from the object 340). After reflecting from the object 340, the first reflected beam 342 and the second reflected beam 344 may be directed to an off-axis parabolic reflector / mirror 330, which redirects the first reflected beam 342 and the second reflected beam 344 towards a beam splitter 360.

[0127] The first reflected beam 342 is thus split and directed to both a first detector 362 and a second detector 364. The first detector 362 and the second detector 364 may be connected to the oscilloscope 324. The first detector 362 may be an s-polarization detector configured to detect the intensity of one or more reflected s-polarized light pulses forming the first reflected light beam 342. Similarly, the second detector 364 may be a p-polarization detector configured to detect the intensity of one or more reflected p-polarized light pulses forming the first reflected light beam 342. After passing through the beam splitter 360, the second reflected light beam 344 is directed to both the first detector 362 and the second detector 364, where the intensity of the s-polarized light pulses and / or the p-polarized light pulses may be detected from the second reflected light beam 344. The first detector 362 and the second detector 364 may be connected to a processor (not shown) that further analyzes information received therefrom, as previously described above. The M LIDAR system 300 is compact, having dimensions of, by way of non-limiting example, approximately 17.8 cm by 30.5 cm (approximately 7 inches by 12 inches), which may make it particularly suitable for mounting in a vehicle.

[0128] MST classification can be achieved by using light source based MST classification with a light polarization classifier added to the point cloud according to the examples of the present disclosure as introduced above. In one example, for each 3D area measurement of the point cloud, the linear / circular polarization of the returning photons can be collected. Furthermore, although the relationship between the surface properties and the polarization state can be noisy in some cases due to surface roughness, the local curvature and local scattering conditions can be made directly based on the polarization state of the returning photons.

[0129] Referring now to FIG. 6, an MST polarization analysis of the reflected laser light was performed using AI data processing with a neural network algorithm to generate the confusion matrix of FIG. 6. More specifically, the confusion matrix was generated based on the analysis of s-polarized and p-polarized beams (such as s-polarized light beam 118 and p-polarized light beam 120 shown in FIG. 1). Along the x-axis, the predicted type of material for the test object that is the subject of the M LIDAR system and processing methods described herein is identified. Along the y-axis, the true type of material for the test object is identified. The accuracy of the various predictions of the AI ​​algorithm for the various material types is reflected in the intersection of the predicted material type and the true material type. As shown, the material detection function of the M LIDAR system can be achieved with a high degree of accuracy (including 99% or more in some cases) using these polarized light beams.

[0130] Referring now to Figure 7, a confusion matrix for the detection of simulated black ice compared to other materials is shown. Again, the material detection capabilities of the M Lidar system can be achieved with a high degree of accuracy (including 100% in some cases).

[0131] FIG. 8 illustrates one example of an M-lidar device 800 for use in, for example, black ice detection (e.g., when installed in a vehicle or the like). While the example focuses on a black ice detection application, one of ordinary skill in the art will recognize that the device 800 is not limited to black ice detection and may be suitably employed for a wide range of material detection and object classification applications, including on autonomous vehicles. The device 800 includes a housing 802, an emitter 804 (i.e., an emitter for emitting light pulses that constitute a laser light beam), a first detector 806a, and a second detector 806b. According to one example, one or more of the detectors 806a, 806b include orthogonal polarization analyzers. Furthermore, according to one example, one or more of the emitter 804, detector 806a, and / or detector 806b may be fabricated using kirigami optical elements. While the primary example of the device is for use in automobiles, the device may also be used in aircraft, such as, for example, drones.

[0132] 9, a flow chart is provided illustrating a method 900 of performing object classification using an M LIDAR system. The method 900 begins at 902, where an unpolarized pulse is generated. At 904, the unpolarized pulse is linearly polarized to generate a linearly polarized light pulse. The linearly polarized pulse may be emitted toward and reflected from an object to generate a reflected linearly polarized light pulse. At 906, the reflected linearly polarized light pulse may be linearly polarized for s-polarization to generate a reflected s-polarized light pulse.

[0133] At 908, the reflected linearly polarized light pulse may be linearly polarized for p-polarization to generate a reflected p-polarized light pulse. At 910, an intensity of the reflected s-polarized light pulse may be detected. At 912, an intensity of the reflected p-polarized light pulse may be detected. At 914, at least one material of the object may be detected based on the intensity of the reflected s-polarized light pulse and the intensity of the reflected p-polarized light pulse. Finally, at 916, the object may be classified based on the at least one material detected. After 916, method 900 ends.

[0134] Finally, according to some examples, kirigami patterns can be used as MST tags for polarization-based detection of objects, and mass-produced kirigami components can be added to paint to give specific polarization responses in road signs, clothing, markers, vehicles, household items, or any other suitable object.

[0135] In some variations, the lidar system of the present disclosure can provide modulation of the transmitted and reflected beams. A kirigami-based optical element can be added to the emitter side of the lidar to act as a beam steerer, which can therefore replace a conventional large rotary beam steerer or a liquid crystal phase array beam steerer. A magnetic actuation module can be integrated with the 1550 nm laser source. To reduce the size of the beam steerer, optical fibers can be directly coupled to the module.

[0136] In some variations, the lidar systems provided by the present disclosure may provide improved detection in precipitation and / or humid atmospheric conditions. For example, the lidar systems contemplated by the present disclosure may be particularly suitable for use in low visibility conditions by employing lasers with wavelengths of about 1550 nm, providing improved detection and performance during adverse weather conditions, including, by way of non-limiting example, low visibility conditions associated with fog, rain, and snow. Such lidar systems may enable long-range warnings, for example, up to 200 meters, which is particularly useful for highway driving conditions. Conventional lidars use lasers with wavelengths of about 900 nm, which are convenient for silicon-based detectors. However, these conventional laser beams suffer from relatively strong scattering in humid atmospheric conditions. Lidars operating at 1550 nm can take advantage of the high penetration of humid air, which is advantageous for all different levels of autonomy, from proximity warning to assisted driving and fully autonomous riding modalities. However, such lidars may be large and expensive due to the high weight and cost of near-infrared optics. According to some aspects of the present disclosure, kirigami-based optical elements can solve this problem by taking advantage of possible space charge and sub-wavelength effects on patterned kirigami sheets, see, for example, FIG. 2a-b. As shown in FIG. 3a-b, such kirigami sheets can effectively modulate and beam steer near-infrared lasers using the reconfigurable out-of-plane patterns of the kirigami sheets. A 1550 nm beam steering device incorporating such kirigami-based optical elements can be used as a thin, lightweight, and inexpensive solid-state lidar. Furthermore, the versatility of kirigami technology allows the pattern to be potentially adapted for specific applications, e.g., customizing the lidar system to a specific vehicle, and / or adapting it to surfaces of different curvatures of automotive parts.

[0137] In some aspects, the present disclosure can provide relatively fast detection for lidar systems by using a two-stage object proposal and detection method without sacrificing accuracy for latency. For example, improving model accuracy and generalizability for classification models can include enhancing stationary object classifiers by adding material dimensions to the data. Objects with material fingerprints including plastic, wood, and brick are highly unlikely to be moving, while objects with metal or textile fingerprints are more likely to be pedestrians and vehicles. Moreover, material dimensions are more robust to scenario variations, so these models tend to generalize better to rare and complex cases, such as construction sites and streets with complex festival decorations. Thus, material fingerprints greatly enhance model accuracy for point cloud association models with implications for tracking and autonomous vehicle maps. For example, a pedestrian walking with a bicycle can be picked up as a point cloud with metal material on the underside along with some textile or skin features from the pedestrian, so the material dimensions of the point cloud can make detection and classification much more reliable. It would then be much easier for the autonomous driving system to distinguish it from a pure pedestrian. The material fingerprint of the object would also make it easier for the system to associate the point cloud with the correct object classification, helping to maintain a correct and consistent composite object classification.

[0138] The present disclosure therefore provides an inexpensive, compact lidar system with improved object recognition, including the ability to distinguish material types and identify objects within milliseconds, providing an early detection and warning system and high effectiveness in low visibility conditions, among other benefits.

[0139] In some variations, the present disclosure provides a system comprising a laser configured to generate light pulses, a beam steerer configured to generate polarization-tuned light pulses emitted towards an object, at least one polarizer configured to polarize light returning, reflected, scattered or emitted from the object, and a processor configured to detect at least one material of the object based on the intensity and polarization of the polarized light reflected or emitted from the object. In one aspect, the beam steerer comprises a kirigami nanocomposite. In one aspect, the at least one polarizer comprises a kirigami nanocomposite. In one aspect, the processor is further configured to classify the object based on the detected at least one material of the object.

[0140] In a further aspect, the processor is configured to classify the object based on the detected at least one material of the object by applying a machine learning algorithm. In a further aspect, the machine learning algorithm comprises an artificial neural network algorithm.

[0141] In one aspect, the beam steerer is configured to adjust the polarization of the light pulses to generate polarization-modulated light pulses. In one aspect, the beam steerer is configured to adjust the polarization of the light pulses by at least one of imparting polarization to an unpolarized light pulse and changing the polarization of a polarized light pulse. In another aspect, the beam steerer is configured to adjust the polarization of the light pulses by applying at least one of linear, circular, and elliptical polarization types of polarization. In a further aspect, applying the linear polarization includes applying at least one of s-type linear polarization and p-type linear polarization.

[0142] In one aspect, the at least one polarizer is configured to polarize the light returned, reflected, scattered, or emitted from the object by applying at least one of the following types of polarization: linear, circular, and elliptical. In a further aspect, the applying is applying linear polarization, including applying at least one of s-type linear polarization and p-type linear polarization. In one aspect, the at least one polarizer comprises a plurality of polarizers.

[0143] In one aspect, the system further comprises at least one polarization detector coupled to the at least one polarizer and a processor, the at least one polarization detector configured to detect an intensity of the polarized light reflected or emitted from the object. In a further aspect, the at least one polarization detector comprises a plurality of polarization detectors. In a further aspect, the at least one polarization detector is configured to detect an angle of incidence associated with the polarized light reflected, scattered, or emitted from the object.

[0144] In a further aspect, the processor is further configured to detect at least one material of the object based on an angle of incidence associated with polarized light reflected, scattered or emitted from the object.

[0145] In yet another variation, the present disclosure provides a method that includes generating a light pulse, adjusting the polarization of the light pulse to generate a polarization-adjusted light pulse that is emitted toward an object, polarizing light returned, reflected, scattered or emitted from the object, and detecting at least one material of the object based on the intensity and polarization of the polarized light reflected or emitted from the object.

[0146] In one embodiment, adjusting the polarization of the light pulse is performed by a beam steerer that includes a kirigami nanocomposite. In one embodiment, the kirigami nanocomposite is fabricated by a vacuum assisted filtration (VAF) process. In one embodiment, the kirigami nanocomposite is fabricated by a layer-by-layer (LBL) deposition process. In one embodiment, the method further includes classifying the object based on the detected at least one material of the object.

[0147] In one embodiment, classifying the object comprises classifying the object by applying a machine learning algorithm. In one embodiment, the machine learning algorithm comprises an artificial neural network algorithm.

[0148] In another aspect, the present disclosure provides lidar systems and methods configured to detect the relative distance or location and shape of an object along with additional information about the object, such as the material composition of the object, and / or additional information about the returning beam of the laser pulse, such as optical properties about the reflected beam. The additional information about the returning beam from the object can include, for example, polarization information, Raman scattering information, circular dichroism information, etc. In this manner, the additional information about the object and / or about the returning / reflected beam is stored and adds an additional dimension of information related to the distance or location and shape of the object. In other words, the lidar systems and methods of the present disclosure are multidimensional in the sense that the systems and methods are configured to detect information in addition to the three-dimensional information representing the relative distance or location and shape of the object. For example, the material composition of the object can add a fourth dimension to the information related to the object. Additionally or alternatively, the additional information about the returning beam from the object can include polarization information, Raman scattering information, circular dichroism information, etc., which can add one or more additional dimensions to the information related to the object.

[0149] For example, an object detected by the lidar systems and methods of the present disclosure may be represented by a point cloud in a three-dimensional XYZ coordinate system. The point cloud may include a plurality of points, each point representing a location on the surface of the object and having associated XYZ coordinates. As described in further detail above, generating the point cloud may include a process of repeatedly emitting pulses of light, the pulses being diffusely reflected from the object, the reflected light being measured by a detector, and determining the location of the points on the surface of the object relative to the source of the light. This process may then be repeated multiple times per second, such as on the order of millions of times per second, with a point being generated each time. The various points are then mapped onto the same XYZ coordinate system to create the point cloud.

[0150] For each point in the point cloud representing the object, the LIDAR systems and methods of the present disclosure can also determine a fourth dimension representing the composition of the material located at that particular point on the surface of the object. Additionally or alternatively, for each point in the point cloud representing the object, the LIDAR systems and methods of the present disclosure can also determine an additional dimension corresponding to additional information about the returning (e.g., reflected) beam of the laser pulse associated with that particular point on the surface of the object, such as optical properties for the reflected beam. As noted above, the additional information can include, for example, polarization information, Raman scattering information, circular dichroism information, etc., about the returning (e.g., reflected) beam associated with that particular point on the surface of the object. In this manner, the LIDAR systems and methods of the present disclosure can also determine a multi-dimensional point cloud representing the detected object, including XYZ coordinates for points representing the surface of the object, each point having, in addition to the XYZ coordinates, additional associated information, such as information representing the material composition of the surface of the object at that particular point, and / or additional information about the reflected beam from that particular point, such as polarization information, Raman scattering information, circular dichroism information, etc. As discussed above, the material classification methods of the present disclosure can determine material composition information from optical information about the reflected beam, such as polarization information, Raman scattering information, circular dichroism information, etc. In this manner, the LIDAR systems and methods of the present disclosure can generate and store any combination of material composition information and / or additional optical information associated with each point in a point cloud representing a detected object.

[0151] Referring to FIG. 12, a four-dimensional point cloud 1000 is shown generated by the lidar system and method of the present disclosure using a person on a bicycle as an example detected object. While only the X and Y axes of an XYZ coordinate system are shown in FIG. 12, it should be understood that the point cloud 1000 is three-dimensional, with the Z axis omitted for ease of illustration. The point cloud 1000 includes a number of points, each point representing a location on a detected object, for example, a location on the surface of the detected object, in this case a person on a bicycle. As further shown in FIG. 12, each XYZ point in the point cloud 1000 also includes associated material composition information. As indicated by key 1002, point 1004 of the point cloud 1000 corresponding to the person's head is indicated as having material composition information corresponding to skin. Point 1006 of the point cloud 1000 corresponding to the person's torso, arms, and legs is indicated as having material composition information corresponding to cotton. A point 1008 of the point cloud 1000 corresponding to a bicycle frame is shown as having material composition information corresponding to aluminum. A point 1010 of the point cloud 1000 corresponding to a bicycle tire is shown as having material composition information corresponding to rubber. The point cloud 1000 shown in FIG. 12 may be used, for example, by an autonomous vehicle system. For example, an autonomous vehicle system may use the point cloud 1000 shown in FIG. 12 to determine that a detected object is a person riding a bicycle. The autonomous vehicle system may also use the point cloud 1000 to distinguish the object as an actual person riding a bicycle from, for example, an image of a person riding a bicycle.

[0152] The exemplary point cloud 1000 includes material composition information as a fourth dimension associated with each point of the point cloud, as described above, although other additional information about the reflected beam may be stored at each point in addition to or in place of the material composition information. The additional information may include, for example, polarization information, Raman scattering information, circular dichroism information, etc.

[0153] The disclosed lidar systems and methods can generate and utilize a materials database that stores material composition information along with corresponding optical information, such as polarization information, Raman scattering information, and circular dichroism information. In this manner, the disclosed lidar systems and methods can compare optical information associated with a reflected beam from an object to the stored optical information to interrogate and determine the corresponding material composition of the object. The disclosed lidar systems and methods can use neural networks and machine learning techniques to generate and build the materials database and to calibrate relevant parameters used in determining the material composition of an object based on optical information associated with a reflected beam from the object.

[0154] For example, the neural network may be initialized with a neural network layer that includes parameter values ​​based on the physical laws of light / matter interaction. In other words, the neural network layer may be initialized to provide a predicted material composition for an object using input optical information associated with a reflected beam from the object, based on the physical laws of light / matter interaction. The system may then be trained using a large number of objects (such as thousands, millions, or billions of objects), each having a known composition. For example, a lidar system may scan each object, compare the determined material composition to the known composition of the object, and adjust the parameter values ​​of the neural network based on the comparison using machine learning techniques. This process may be repeated multiple times for each object in the large number, with each iteration providing greater accuracy for the determined material composition as the system calibrates and adjusts the parameter values ​​of the neural network over time. Because the optical information associated with the reflected beam may depend on the orientation, position, and shape of the object, the object may be rescanned multiple times using different orientations and positions to increase the accuracy of the system. Additionally, as described in more detail below, the system can include multiple lidar lasers and detectors positioned at different orientations relative to the object, and information from each of the multiple lidar lasers and detectors can be used in parallel to determine the material composition of the object and increase the accuracy of the system.

[0155] 13, a flow chart is provided illustrating a method 1100 of training a neural network used to identify material composition of an object, including individual points on a surface of the object, according to the present disclosure. The method 1100 begins at 1102. At 1104, the neural network is initialized based on the physical laws of light / matter interaction. For example, the initialized parameters of the neural network are initialized and set based on the physical laws of light / matter interaction. At 1106, an object having a predetermined or known material composition is detected using a lidar system, and the material composition for each point of a point cloud representing the object is determined based on optical information of a returning (e.g., reflected) beam and based on the neural network.

[0156] At 1108, the material composition determined by the system is compared to the known material composition of the object. At 1110, parameters of the neural network are adjusted, if necessary, based on the comparison using machine learning techniques.

[0157] At 1112, a determination is made as to whether the accuracy of the system is now acceptable. When the accuracy of the system is still not acceptable at 1112, the method loops back to 1106 and repeats the process by detecting another object and making further adjustments to the parameters. The process repeats until the accuracy of the system is determined to be acceptable at 1112. When the accuracy of the system is determined to be acceptable, the process ends at 1114. The accuracy of the system may be determined to be acceptable, for example, when the system accurately determines the material composition of a predetermined number of objects within a predetermined acceptable error limit.

[0158] In this manner, by scanning a large number of objects (e.g., thousands, millions, or even billions of objects), the disclosed systems and methods can build large databases of materials and associated neural networks that can be used to accurately determine the material composition of objects detected using the disclosed lidar systems.

[0159] The multi-dimensional lidar systems and methods of the present disclosure may be used in recycling systems to identify the material composition of objects to be recycled and to appropriately sort the objects based on the determined material composition. Referring to FIG. 14, an exemplary recycling system 1200 is shown and includes a conveyor belt 1202. The conveyor belt 1202 moves objects from a recycling bin 1204 that holds objects to be sorted and recycled to a sorter 1206. In the example of FIG. 14, a metal can 1220 is shown on the conveyor belt 1202 as it moves from the recycling bin 1204 toward the sorter. As shown in FIG. 14, as the object 1220 is transported on the conveyor belt toward the sorter 1206, the object 1220 is scanned by one or more lidar systems 1214a, 1214b, 1214c according to the present disclosure. Although three lidar systems 1214a, 1214b, and 1214c are shown in Figure 14, any number of lidar systems may be used. For example, only one lidar system may be used. As discussed above, however, the use of additional lidar systems capable of scanning the object 1220 from multiple different angles and perspectives can increase the accuracy of the material classification determination.

[0160] The recycling system 1200 includes a control module 1218 in communication with the lidar systems 1214a, 1214b, 1214c, a sorter 1206, and a materials database 1216. As the object 1220 passes the lidar systems 1214a, 1214b, 1214c, the object 1220 is scanned by the lidar systems 1214a, 1214b, 1214c, which then communicate obtained information about the reflected beams from the object 1220 to the control module 1218. In accordance with the above classification technique, the control module 1218 is configured to access the materials database 1216 and determine a multidimensional point cloud for the object 1220 based on the optical information of the reflected beams received by each of the lidar systems 1214a, 1214b, 1214c. The multi-dimensional point cloud can include, for example, material classification information for each point of the point cloud representing the object 1220. Based on the material classification information, the control module 1218 can then control the sorter 1206 to sort the object 1220 into an appropriate classification bin for recycling.

[0161] 14, recycling system 1200 includes a glass bin 1208 for receiving glass objects, a metal bin 1210 for receiving metal objects, and a plastic bin 1212 for receiving plastic objects. Sorter 1206 can include baffles, doors, slots, and / or levers that are movable and configurable to direct objects 1220 to the appropriate bins, for example, through chutes 1209, 1211, and 1213. For example, chute 1209 leads to glass bin 1208, chute 1211 leads to metal bin 1210, and chute 1213 leads to plastic bin 1212. Although a sorter 1206, recycling bins 1208, 1210, 1212, and chutes 1209, 1211, 1213 are shown as examples, any other sorting mechanism may be used and controlled by the control module 1218 to sort objects from the conveyor belt 1202 to the applicable recycling bin based on the material composition of the objects, as determined based on the systems and methods of the present disclosure.

[0162] In this manner, the lidar systems and methods disclosed herein can accurately determine the material composition of objects to be recycled and can be used in conjunction with a sorting system to appropriately and automatically sort objects to be recycled into material groups for recycling.

[0163] Referring to FIG. 15, four-dimensional point clouds 1300, 1302, 1304 are shown that are generated by the disclosed lidar systems and methods for recycling objects. While only the X and Y axes of an XYZ coordinate system are shown in FIG. 15, it should be understood that the point clouds 1300, 1302, 1304 are three-dimensional with the Z axis omitted for ease of illustration. Each point cloud 1300, 1302, 1304 includes a number of points, each point representing a location on the surface of a detected object. As further shown in FIG. 15, each XYZ point in the point clouds 1300, 1302, 1304 also includes associated material composition information. As indicated by legend 1306, the points of point cloud 1300 are indicated as having material composition information corresponding to plastic. The points of point cloud 1302 are indicated as having material composition information corresponding to aluminum. The points of point cloud 1304 are indicated as having material composition information corresponding to paper. The point clouds 1300, 1302, 1304 may be generated, for example, by the control module 1218 of the recycling system shown in FIG.

[0164] The LIDAR systems and other methods of the present disclosure may also be utilized for detection of the health of living organisms, such as animals, which in some embodiments may be humans.

[0165] The lidar systems and other methods disclosed herein may also be used in medical / cosmetic applications, for example, to identify areas of the skin exhibiting a condition and to diagnose skin diseases by mapping the skin surface and analyzing polarimetry data from points corresponding to scanned points on a patient's skin surface.

[0166] Referring to FIG. 16, four-dimensional point clouds 1400, 1402, 1404, 1406 are shown generated by the lidar systems and methods of the present disclosure for scanning a patient's skin surface. While only the X and Y axes of an XYZ coordinate system are shown in FIG. 16, it should be understood that the point clouds 1400, 1402, 1404, 1406 are three-dimensional with the Z axis omitted for ease of illustration. Each of the point clouds 1400, 1402, 1404, 1406 includes a number of points, each point representing a location on the patient's skin surface. As further shown in FIG. 16, each of the XYZ points in the point clouds 1400, 1402, 1404, 1406 also includes associated material composition information. As indicated by legend 1408, the points of the point cloud 1400 are shown as having a material composition corresponding to normal skin. The points of the point cloud 1402 are shown as having material composition information corresponding to acne. The points of point cloud 1404 are shown as having material composition information corresponding to a tumor. The points of point cloud 1406 are shown as having material composition information corresponding to dry skin. Point clouds 1300, 1302, 1304 may be generated, for example, by a lidar system utilizing a material database that stores information corresponding to the material composition of different skin conditions and diseases. In general, the four-dimensional point clouds may be used to distinguish biological materials, cells, tissues, etc., such as the location of malignant cells versus benign cells, or the identity of select biological materials. In this manner, the lidar systems and other methods of the present disclosure may be utilized for detection of the health status of an organism, such as an animal-like life form. The animal may be a human in some aspects.

[0167] Similarly, the lidar systems and methods disclosed herein may be utilized for facial recognition.

[0168] Additionally, the lidar systems and methods of the present disclosure may be incorporated into other devices, such as personal computers, laptops, portable devices, smartphones, tablet devices, etc. For example, the laser used for the laser pulses may be implemented using a diode laser, which may be included in other devices, such as personal computers, portable devices, smartphones, tablet devices, etc. For example, when incorporated into a smartphone, laptop, or tablet device, the lidar systems and methods of the present disclosure may be used for facial recognition, for example, to enable access to the smartphone, laptop, or tablet device.

[0169] Additionally, the lidar system of the present disclosure may be used to communicate with other lidar systems. For example, the lidar system may use the polarization of the transmitted laser pulses to communicate data. By way of illustration, the lidar system may be incorporated into a vehicle, and when the vehicle is turning left, the lidar system may transmit left-polarized light pulses to the lidar systems of the surrounding vehicles. Similarly, when the vehicle is turning right, the lidar system may transmit right-polarized light pulses to the lidar systems of the surrounding vehicles. Additionally, two lidar systems may transmit pulses with a specific sequence of polarization to communicate information, such as verification information to confirm and verify each other's identity and prevent the lidar system from being spoofed or hacked by a third party. For example, a first lidar system may request that the second lidar system transmit verification information by a specific sequence of polarization known to the two lidar systems.

[0170] The disclosed lidar systems and methods can be used to generate information about the scanned material that is sensed and collected through analysis of the polarization of the reflected and scattered lidar beam. The disclosed material sensing lidar systems and methods perform a computational analysis of the polarization ratio between s-polarized and p-polarized light from a modulated laser beam incident on the scanned item. Concomitantly, Raman scattering and other information can be collected that complements the polarization process.

[0171] The polarization of the incident lidar beam can be circularly polarized. The polarization of the reflected and scattered lidar beam can be analyzed for circular polarization. The chirality of a material or its composition or its shape become intrinsic material properties that affect the polarization signature of the detected item. Each XYZ data point acquired by the lidar is complemented by another property, such as a material detection property, i.e., a polarization ratio, or data acquired from a comparison of the corresponding computationally processed polarization ratios with a database containing different materials. Each XYZ point is given additional data such that the point becomes a four-dimensional point or more. When material attributes are included, the point becomes a four-dimensional point. When different Raman scattering or other attribute information is simultaneously given to the data point, the point becomes multidimensional, having five or more dimensions.

[0172] As explained above, the material information of each point can be assigned by deep learning neural network algorithms based on the material database. The material database can be collected by a polarimetric lidar setup. The lidar beam is modulated as linear or circular polarization. Polarimetric lidar measures the polarization difference upon reflection / scattering by analyzing the s-polarization and p-polarization ratios of the reflected / scattered light. The angle dependence of reflectance and scattering physical properties of each material are included in the database based on both measurements and field calculations. The color effect of each material is included in the database based on both measurements and field calculations in the CIE color vector space. Polarimetric Raman scattering and Fourier-transform infrared (FTIR) information can be collected and included in the material database. The optical activity of the material including linear dichroism and circular dichroism properties are included in the material database. The material database package gives a full understanding of the material regardless of texture, angle of incidence, and coating color.

[0173] Multidimensional material recognition lidar can also be realized by processing the intensity profile of the reflected and scattered light. In this case, the signal is processed by a machine learning algorithm that includes, as one of the neural network layers, the physical properties of the reflection and the evaluation of the reflected light profile based on the expected material parameters. Parameters for training such a network include not only the intensity maximum but also the intensity attenuation and noise frequency.

[0174] In some variations, the present disclosure provides a system that includes a polarization-enabled camera, also referred to as a polarization camera, configured to sense the polarization of the light it detects. In these variations, the system does not require the laser and beam steerer described in the above systems and methods, since the system utilizing the polarization camera detects the polarization of the ambient light reflected from the object. Instead, in these variations, the system can utilize unpolarized ambient light and detect the polarization of the ambient light reflected from the object. Ambient electromagnetic radiation may include visible light, ultraviolet light, infrared light, and the like. In particular, suitable visible and infrared electromagnetic radiation includes visible light having a wavelength ranging from about 390 nm to about 750 nm, and infrared radiation (IR) (including near infrared (NIR) ranging from about 0.75 μm to about 1.4 μm and far infrared (FIR) ranging from about 15 μm to 1 mm). The far infrared (FIR) portion of the electromagnetic spectrum, also known as the terahertz (THz) diaphanous region, has photon wavelengths from about 100 μm to about 1 mm and energies from about 0.001 eV to about 0.01 eV. Besides being beneficial to many areas of THz research, from astronomy and solid state physics to telecommunications, THz circular dichroism (TCD) can be used in the understanding and detection of astronomy, solid state physics, telecommunications, biomaterials, biomolecules, and medicine, as non-limiting examples.

[0175] 1, for example, a system utilizing a polarization camera according to these variations does not require laser 102 and beam steerer 106, and does not require the generation of beam 104 or beam 108. In other words, in a system utilizing a polarization camera, ambient light is used in place of beam 104 and beam 108 generated by laser 102.

[0176] In one aspect, similar to the systems and methods described above, the sensed polarization information about the ambient light reflected from the detected object is used by a processor to determine and sense at least one material of the detected object.

[0177] In other aspects, the systems and methods of the present disclosure may include both a lidar and a polarization camera in the same perception system. In a further aspect, the polarization camera is configured to sense polarization information about the ambient light reflected from the detected object as well as polarization information about the laser from the lidar returning from the detected object.

[0178] In another aspect, the processor is configured to classify the object based on at least one material of the detected object by applying a machine learning algorithm. In a further aspect, the machine learning algorithm comprises an artificial neural network algorithm.

[0179] In another aspect, the present disclosure provides a polarization camera system and method configured to detect the relative distance or location and shape of an object along with additional information about the object, such as polarization information about ambient light reflected from the object and detected by the polarization camera system. The polarization information can be used to determine the material composition of the object, as described above with respect to the system and method for sensing polarization information using a laser, a beam steerer, and at least one polarizer. In this manner, the additional information about the object and / or about the returning / reflected beam is stored and adds an additional dimension of information related to the distance or location and shape information about the object. In other words, the polarization camera system and method of the present disclosure are multi-dimensional in the sense that the system and method are configured to detect information in addition to the three-dimensional information representing the relative distance or location and shape of the object. For example, the material composition of the object can add a fourth dimension to the information related to the object. Additionally or alternatively, the additional information about the returning beam from the object can include polarization information and material composition information, etc., which can add one or more additional dimensions to the information related to the object.

[0180] For example, an object detected by the disclosed polarization camera system and method may be represented by a point cloud in a three-dimensional XYZ coordinate system. The point cloud may include a plurality of points, each point representing a location on the surface of the object and having associated XYZ coordinates. Similar to the lidar system described above, generating a point cloud using the polarization camera system and method may include a process of repeatedly detecting ambient light reflecting off the object, the reflected light being measured by the polarization camera, and determining the location of the point on the surface of the object. This process may then be repeated multiple times per second, such as on the order of millions of times per second, with a point being generated each time. The various points are then mapped onto the same XYZ coordinate system to create the point cloud.

[0181] For each point in the point cloud representing the object, the polarization camera system and method of the present disclosure can also determine a fourth dimension representing the composition of the material located at that particular point on the surface of the object. Additionally or alternatively, for each point in the point cloud representing the object, the polarization camera system and method of the present disclosure can also determine an additional dimension corresponding to additional information about the returning (e.g., reflected) ambient light associated with that particular point on the surface of the object, such as optical properties about the reflected light. As described above, the additional information can include, for example, polarization information associated with that particular point on the surface of the object. In this manner, the polarization camera system and method of the present disclosure can also determine a multi-dimensional point cloud representing the detected object, including XYZ coordinates for points representing the surface of the object, each point having, in addition to the XYZ coordinates, additional associated information, such as information representing the material composition of the surface of the object at that particular point, and / or additional information about the reflected light from that particular point, such as polarization information. As explained above, the material classification method of the present disclosure can determine material composition information from optical information, such as polarization information, about the reflected light. In this manner, the disclosed polarization camera systems and methods can generate and store any combination of material composition information and / or additional optical information, such as polarization information, associated with each point in a point cloud representing a detected object.

[0182] 17A-17C, a polarization camera 2000 is shown. In particular, FIG. 17A shows a perspective view of the polarization camera 2000. FIG. 17B shows a front view of the polarization camera 2000, and FIG. 17C shows a rear view of the polarization camera. The polarization camera 2000 may be, for example, a Triton Polarization Model polarization camera available from Lucid Vision Labs that utilizes Sony's Polarsens technology. The polarization camera 2000 may record RGB polarization video at, for example, 24 frames per second (FPS) and 5.0 megapixel (MP) resolution. Although the present disclosure describes systems and methods utilizing a Triton Polarization Model polarization camera, any other polarization camera that captures polarization information of reflected light from an object may be used in accordance with the present disclosure.

[0183] An image captured by a standard photographic camera contains RGB color data for each pixel in the image, but nothing more. With the polarization camera 2000, each pixel also stores data about the polarization of the detected light. When light reflects off the surface of an object, it is shifted in polarization based on the properties of the material of that surface before being sent to the lens of the polarization camera 2000. Due to these unique polarization signatures, if two objects are constructed from different materials, the polarization of the reflected light from each object will be different, and the two objects are distinguishable based on the different polarization of the reflected light. Without the polarization information, a standard camera and image recognition algorithms would have difficulty distinguishing between the two objects. For example, a road with a layer of black ice will have a dramatically different polarization image than a road without black ice. A standard camera would, however, produce RGB images of the two roads that would appear very similar, if not identical. Similarly, a black rubber tire will have a completely different polarization profile than a black bag due to the different textures and other properties of the construction materials. Thus, the polarized camera systems and methods disclosed herein are advantageously capable of distinguishing black rubber tires from black plastic bags and roads covered in black ice from roads without black ice.

[0184] The polarization camera 2000 is configured to sense the polarization of the ambient light reflected from the detected object. Polarization is a property of all electromagnetic radiation, including the light that the camera measures to sense or detect objects in the surrounding environment. All light, even if not fully polarized initially, enters a new polarization state when reflected from the surface of the object. This change in polarization depends on and corresponds to the specific composition of the material of the object from which the light is being reflected, as well as the angle at which the light reflects off the material of the object. The closer the light beam is to a normal to the reflecting surface, the greater the change that the polarization value of this light undergoes. Thus, the more distant the object, the stronger the material-based influence of the object's surface on the light may be. At longer distances, object detection may become more difficult due to reduced resolution, but the relative enhanced interaction of light with the surface for distant objects provides some relief for this difficulty. Polarization information may be fed to the perception layer of the systems and methods of the present disclosure and used to determine the material composition of the sensed object, even when the sensed object is farther away and the resulting image of the object has a lower resolution than a closer object. For example, the systems and methods of the present disclosure may access a materials database, such as materials database 1216 shown in FIG. 14, to determine the material of an object and / or a point on an object based on the polarization of ambient light reflected from the object and / or the point on the object.

[0185] A particular polarization state can be any angle between 0 degrees and 360 degrees. In a similar fashion, hues (color characteristics) are often represented in a circular fashion, with each hue corresponding to a degree of rotation. This can be used to create pseudocolor photographs that represent the kind of visual contrast provided by the polarization data generated by the polarization camera 2000. For example, FIG. 18 shows an image 2050 of a group of objects shown without the use of polarization data. Image 2050 was generated using a non-polarized camera. FIG. 18 also shows an image 2052 of the same group of objects, generated by the polarization camera 2000, and shown using pseudocolor polarization to show the additional contrast and feature visibility available using the polarization data captured by the polarization camera 1000. By training a machine learning system to take this polarization information into account in the same context as color, in accordance with the present disclosure, entirely new correlations can be formed that are stronger and more general than anything a standard camera vision system is capable of achieving.

[0186] Referring to FIG. 19, similar to FIG. 6 described above, MST polarization analysis of reflected light was performed using AI data processing with a neural network algorithm to generate the confusion matrix of FIG. 19. Along the x-axis, the predicted type of material for the test object that is the subject of the processing method described herein is identified. Along the y-axis, the true type of material for the test object is identified. The accuracy of the various predictions of the AI ​​algorithm for the various material types is shown at the intersection of the predicted material type and the true material type. As shown, the material detection capabilities of the disclosed systems and methods can be achieved with high accuracy (including in one case 96% or greater).

[0187] Furthermore, there are many possible edge-case situations where an object may be inherently camouflaged by its color, but is completely visible to the polarization camera system and method of the present disclosure using the polarization information of the object. For example, in an image of a child wearing a yellow raincoat standing in front of a yellow school bus, a traditional system may not be able to distinguish the difference between the child and the school bus. A system using the polarization information of the present disclosure, however, can recognize that the polarization of light reflected from the material of the yellow raincoat is distinguishable from the polarization of light reflected from the material of the yellow school bus. Thus, a system using the polarization camera 2000 of the present disclosure can distinguish between the two objects, classify different materials and objects, and recognize the boundaries and edges between the two objects. As a result, to detect distant or visually ambiguous objects, the system and method of the present disclosure reduces the amount of time required for classification, identification, etc., while providing a large improvement in the success rate of identifying materials and objects.

[0188] Object detection and computer vision systems can segment an image into dozens or hundreds of objects represented therein. The object detection data can then be fed into an object classification algorithm that describes the identity of each identified object. Adding polarization information to the perception layer of object detection and computer vision systems in accordance with the present disclosure improves the ability of such systems to distinguish between objects and classify such objects with reduced processing time compared to systems that do not utilize such polarization information.

[0189] According to the present disclosure, the polarization camera 2000 may be used in place of the LIDAR, beam steerer, and at least one polarizer described in the systems and methods above.

[0190] For example, the polarization camera 2000 may be used to generate the four-dimensional point clouds shown in Figures 12, 15, and 16. Additionally, the polarization camera 2000 may be used with the method 1100 of training a neural network used to identify the material composition of an object described above with reference to Figure 13 by substituting a LIDAR system with a polarization camera system having the polarization camera 2000 and using polarization information detected as reflected optical information, e.g., as described above with reference to reference numeral 1106 of Figure 13.

[0191] Additionally, one or more polarization cameras 2000 may be used with the exemplary recycling system 1200 described above with reference to FIG. 14. In each case, the described lidar, beam steerer, and at least one polarizer described above with respect to each of those systems and methods may be replaced by a polarization camera 2000 configured to capture polarization information of the ambient light reflected from the object. With reference to FIG. 1, the laser 102, beam steerer 106, s-polarized linear polarizer 114, p-polarized linear polarizer 116, s-polarized detector 122, and p-polarized detector 124 may be replaced with a polarization camera 2000 that detects the polarization of the ambient light reflected from the object 110.

[0192] With reference to FIG. 14, for example, one or more of the lidar systems 1214a, 1214b, 1214c may be replaced with one or more polarization cameras 2000. In such a case, the control module 1218 is configured to access the material database 1216 and determine a multidimensional point cloud for the object 1220 according to the classification techniques described above based on the polarization information of the ambient light reflected from the object 1220 and sensed by the one or more polarization cameras 2000. The multidimensional point cloud may include, for example, material classification information for each point of the point cloud representing the object 1220. Based on the material classification information, as described above, the control module 1218 may then control the sorter 1206 to sort the object 1220 into an appropriate classification bin for recycling. In this manner, the polarization camera system and method of the present disclosure may accurately determine the material composition of the object to be recycled and may be used in conjunction with a sorting system to appropriately and automatically sort the object to be recycled into material groups for recycling.

[0193] Additionally, the disclosed polarization camera systems and methods may be used in autonomous driving systems to sense objects in the vicinity of an autonomous vehicle and to generate four-dimensional point clouds of objects for object sensing, classification, and recognition. For example, the disclosed polarization camera systems and methods may be used to generate a four-dimensional point cloud of a person riding a bicycle, as shown in FIG.

[0194] Additionally, the polarization camera system and method of the present disclosure may be used in medical / cosmetic applications, for example, to identify areas of skin exhibiting a condition and for diagnosing skin diseases by mapping the skin surface and analyzing polarization information from points corresponding to the scanned skin surface of a patient. For example, the polarization camera system and method of the present disclosure may be used to generate the four-dimensional point clouds 1400, 1402, 1404, 1406 shown in FIG. 16. As discussed above, a materials database that stores information corresponding to the material composition of different skin conditions and diseases may be used to generate the four-dimensional point clouds, which may be used to differentiate biological materials, cells, tissues, etc. For example, the location of malignant versus benign cells, or the identity of selected biological materials may be determined.

[0195] Similarly, the polarization camera systems and methods of the present disclosure may be utilized for detection of the health or biological indicators of a living organism, such as a living organism, which may be an animal, such as a human.

[0196] The following table (Table 1) gives a comparison between different camera / sensor types, such as standard cameras, hyperspectral cameras, polarimetric cameras, lidar systems, and radar systems, in terms of (i) the specific wavelengths of light measured, (ii) whether additional material-related data is given, (iii) the number of polarization angles, (iv) the current estimated price range, (v) cycles per second, and (vi) the relative data processing requirements (relative data processing requirements for lidar and radar systems are not shown). [Table 1]

[0197] In other aspects, the systems and methods of the present disclosure may be used to classify and / or identify objects based on the different types of materials that make up the objects. For example, machine learning techniques and algorithms may be used to build an object database that includes correlation information that correlates listings of specific objects with the specific materials that make up each object. With reference to the four-dimensional point cloud shown in FIG. 12, for example, the object database may include a database entry for a bicyclist that indicates that the bicyclist object is composed of skin, cotton, aluminum, and rubber. In this manner, when the systems and methods of the present disclosure sense an object composed of skin, cotton, aluminum, and rubber, the object database may be queried based on the sensed materials. Based on the query, the systems and methods of the present disclosure may determine that the object is likely to be a bicyclist based on the material composition of the object. Additionally, the object database may include percentages for each type of material, or percentage ranges for each type of material, that indicate the approximate percentage or range of percentages that each type of material may be found in a particular object. In this manner, the systems and methods of the present disclosure may utilize material information to more accurately classify and identify objects. Object identification and classification may be used in conjunction with computer vision systems to identify and classify objects based on the size, shape, and / or other characteristics of the objects.

[0198] In this application, including the following definitions, the term "module" or "controller" may be interchanged with the term "circuitry." The term "module" may refer to, be a part of, or include an application specific integrated circuit (ASIC), a digital, analog, or mixed analog / digital discrete circuit, a digital integrated circuit, an analog integrated circuit, or a mixed analog / digital integrated circuit, a combinatorial logic circuit, a field programmable gate array (FPGA), a processor circuit (shared, dedicated, or group) that executes code, a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit, other suitable hardware components that provide the described functionality, or a combination of any or all of the above, such as in a system on a chip.

[0199] A module may include one or more interface circuits. In some instances, the interface circuits may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules connected via interface circuits. For example, multiple modules may enable load balancing. In further instances, a server module (also known as remote or cloud) may perform some functionality on behalf of a client module.

[0200] As used above, the term code may include software, firmware, and / or microcode and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses a processor circuit that executes some or all code from one or more modules in combination with additional processor circuits. References to multiple processor circuits encompass multiple processor circuits on a discrete die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or combinations of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses a memory circuit that stores some or all code from one or more modules in combination with additional memory.

[0201] The term memory circuit is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not encompass transitory electrical or electromagnetic signals propagating in a medium (such as on a carrier wave), and the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or masked read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as compact discs (CDs), digital versatile discs (DVDs), or Blu-ray discs).

[0202] The apparatus and methods described in this application may be implemented in part or in whole by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flowchart components, and other elements described above act as software specifications that can be converted into a computer program by the routine work of a skilled technician or programmer.

[0203] A computer program includes processor-executable instructions stored on at least one non-transitory, tangible computer-readable medium. A computer program may also include or rely on stored data. A computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, device drivers that interact with specific devices of a special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0204] A computer program may include (i) descriptive text to be parsed, such as hypertext markup language (HTML), extensible markup language (XML), or Java® Script Object Notation (JSON), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, or (v) source code for compilation and execution by a just-in-time compiler. By way of example only, the source code may be written using syntax from languages ​​including C, C++, C#, Objective C, Swift, Haskell, Go, SQL, R, Lisp, Java, Fortran, Perl, Pascal, Curl, Ocaml, Javascript, HTML5 (Hypertext Markup Language Fifth Revision), Ada, Active Server Pages (ASP), Hypertext Preprocessor (PHP), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash, Visual Basic, Lua, MATLAB, SIMULINK, and Python.

[0205] Unless an element is expressly recited using the phrase "means for," or in the case of a method claim using the phrase "act for" or "step for," no element recited in a claim is a means-plus-function element within the meaning of 35 U.S.C. §112(f).

[0206] The above description of the embodiments is given for the purpose of illustration and description. It is not exhaustive or limiting of the present disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, may be interchangeable and may be used in selected embodiments even if not specifically shown or described. They may also be modified in many ways. Such variations should not be considered as departures from the present disclosure, and all such modifications are intended to be included within the scope of the present disclosure.

Claims

1. a laser device configured to generate a plurality of light pulses emitted toward the object; a detector configured to receive a portion of the plurality of light pulses returning from the object; a processor configured to generate a point cloud representative of the object based on the plurality of light pulses received by the detector; 11. A system comprising: the point cloud having a plurality of points, each point having three-dimensional position coordinates representing a location of the point on the object, and at least one additional value representing a material of the object at the location of the point on the object, the processor being further configured to generate the at least one additional value representing the material of the object at the location of the point of the object based on polarization information, Raman scattering information, and circular dichroism information of a reflection returning from the surface of the object at the location of the point.

2. the object is one of a plurality of objects; a conveyor belt configured to transport the plurality of objects past the laser device; a sorter for sorting the plurality of objects into a plurality of bins; Further equipped with 2. The system of claim 1, wherein the at least one additional value represents material information indicative of the material of the surface of the object at the location of the point, and the processor is configured to control the sorter to sort the object into one of the plurality of bins based on the material information.

3. at least one additional laser device configured to generate a plurality of additional light pulses emitted towards the object, the at least one additional laser device being located at a different location than the laser device; at least one additional detector configured to receive a portion of the plurality of additional light pulses returning from the object, the at least one additional detector being located at a different location than the detector; Further equipped with 2. The system of claim 1, wherein the processor is configured to generate the point cloud representing the object further based on the plurality of additional light pulses received by the at least one additional detector.

4. The system of claim 1 , wherein the laser device generates the plurality of light pulses in the far infrared portion of the electromagnetic spectrum.

Images