Polarization guided deflectometry

Abstract

Methods and systems for three-dimensional reconstruction of objects with specular surfaces are described. One example method for measuring characteristics of an object includes illuminating the object with an unpolarized illumination pattern produced by a screen, and capturing a plurality of images of the object based on light that is specularly reflected from the object. The images are captured using a polarization sensitive detector and each image corresponds to a particular polarization state. The method further includes determining an estimate of a zenith angle associated with a screen-detector configuration based on polarization information associated with the plurality of images, where the estimate of the zenith angle has an ambiguity among at least two values. Using deflectometric information obtained from the images, a value for the zenith angle is determined. The deflectometric information includes a screen-detector correspondence that enables determination of the surface normal at the particular location.

Description

POLARIZATION GUIDED DEFLECTOMETRYCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to the provisional application with serial number 63 / 570,933 titled “Polarization Guided Deflectometry,” filed March 28, 2024. The entire contents of the above noted provisional application are incorporated by reference as part of the disclosure of this document.TECHNICAL FIELD

[0002] The technology described in this patent document relates to imaging methods and systems that utilize polarization principles.BACKGROUND

[0003] Accurate three-dimensional (3D) reconstruction of the specular surface plays a crucial role in industrial inspection, culture heritage preservation, novel view synthesis rendering, and other applications. However, such accurate 3D construction is a challenging task because the specular reflection is "invisible" to sensors in most cases (i.e., it requires the sensor to be positioned precisely where the reflected ray can be intercepted). Therefore, many well-established 3D imaging methods such as stereo triangulation and time-of-flight only work well for diffuse objects.SUMMARY

[0004] The disclosed embodiments relate to 3D imaging methods and systems that enable accurate 3D reconstruction of specular objects.

[0005] One example method for measuring a characteristic of an object includes illuminating the object with a known illumination pattern produced by a screen, wherein the known illumination pattern is produced using unpolarized light, and capturing a plurality of images of the object based on light that is specularly reflected from the object, wherein the plurality of images is captured using a polarization sensitive detector and each of the plurality of images corresponds to a particular polarization state. The method further includesdetermining an estimate of a zenith angle associated with a screen-detector configuration based on polarization information associated with the plurality of images. The zenith angle represents an angle between a light ray from the screen that is incident on the object at a particular location and a surface normal of the object at the particular location, and the estimate of the zenith angle has an ambiguity among at least two values. The above noted method also includes using deflectometric information obtained from one or more of the plurality of images to select one of the at least two values for the zenith angle, wherein the deflectometric information includes a screen-detector correspondence that enables determination of the surface normal at the particular location.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1A is an optical diagram illustrating the height-normal ambiguity in determining the surface normal in an imaging system that uses Deflectometry technique.

[0007] FIG. 1 B is an optical diagram illustrating the ambiguity in determination of a surface normal in an imaging system that uses shape from polarization (SfP) technique.

[0008] FIG. 1 C is an optical diagram illustrating using polarization techniques and deflectometry techniques to enable determination of the zenith angle and surface normal and depth in an imaging system.

[0009] FIG. 2A illustrates an experimental configuration used to demonstrate a 3D reconstruction of an object in accordance with an example embodiment.

[0010] FIG. 2B illustrates an image obtained using a polarization camera in the experimental configuration of FIG. 2A.

[0011] FIG. 2C illustrates a surface normal map obtained using the experimental configuration of FIG. 2A.

[0012] FIG. 2D illustrates a 3D reconstruction obtained using the experimental configuration of FIG. 2A.

[0013] FIG. 3 illustrates two objects and corresponding measurement results conducted in accordance example embodiments.

[0014] FIG. 4 illustrates a perspective projection camera model convention that includes a camera origin, a camera pixel, and a surface point that are located on the same ray.

[0015] FIG. 5 illustrates a set of operations that can be carried out to measure a characteristic of an object in accordance with an example embodimentDETAILED DESCRIPTION

[0016] Notwithstanding other applications, specular object reconstruction has drawn the attention of both the optical metrology community and the computer vision community. In optical metrology, phase measuring deflectometry (PMD) can achieve sub-micron precision with off-the-shelf components (display and camera). PMD has been widely applied in industrial inspection. Recently, it has been used in broader applications including medical diagnosis, mobile measurement, eye-tracking and cultural heritage preservation. However, PMD techniques suffer from a height-normal ambiguity issue in determining the surface normal with a single camera and display because the incident ray from the display is not able to be determined if we only know the emission pixel. This is illustrated in FIG. 1A, where a specular object is illuminated using a cross-sinusoidal pattern from a large screen. The potential surface point can be anywhere along the camera ray since we only know the emission point of the screen, but the incident ray is unknown. Therefore, in the depicted example in FIG. 1A, there is an ambiguity as to which one of Si , S2, S3, or S4 vectors is associated with the reflected ray, which results in an ambiguity of the proper height and normal for the surface point. This is often called height-normal ambiguity (or normal ambiguity for short).

[0017] A classic group of solutions to solve the normal ambiguity problem rely on a multi-view system: a second camera or a moving display is employed to provide additional information. However, these multi-view solutions increase the complexity of the setup and bring up additional calibration issues. Another group of methods use a single camera but rely on prior knowledge of the object, which may not be always available; these solutions also cannot directly obtain the absolute depth information.

[0018] In the computer vision community, Shape from Polarization (SfP) is a well- established 3D imaging method for specular as well as diffuse objects. The disclosed embodiments rely in-part on some aspects of SfP relevant to specular objects. SfP recovers the surface normal by exploiting the polarization cues of the reflected light. Its radiance encodes the information related to the surface normal. SfP is widely used because polarization is observed in different materials including dielectric and metallic surfaces. Also, SfP can work without active illumination. It only requires unpolarized incident light and a rotational linear polarizer to obtain images at different polarization angles. With the advanced polarization sensor in which there is an on-chip polarizer on top of each pixel, SfP is able to recover the surface normal in a single shot. However, there are some limitations of SfP. First, it makes an assumption based on the orthographic projection model in surface normal calculations, which can cause errors in the commonly used perspective camera model. In addition, when measuring a specular object, there are two potential values for both zenith and azimuth angles, leading to four surface normal ambiguities. This is illustrated in FIG. 1 B, where due to having two potential zenith angles and two potential azimuth angles for specular measurement using SfP, there are four potential surface normals, depicted as m to . To solve this problem, additional information is needed. For example, photometric cues can assist in determining the azimuth angle; further utilizing the coarse depth map from depth sensor or multi-view triangulation can help obtain the initial surface normal as guidance. However, these methods either require complicated known illumination information or are designed for diffuse objects which do not have a zenith angle ambiguity. Moreover, SfP only recovers the surface normal but no absolute depth.

[0019] The disclosed embodiments, leverage certain aspects of PMD and certain aspects of SfP, along with additional features, to develop a three-dimensional (3D) imaging technique that enables accurate 3D reconstruction of specular objects. The disclosed systems in some embodiments include an unpolarized display and a single polarization camera. We solve the height-normal ambiguity by exploiting the polarization cues. For example, the polarization camera provides images at four different polarizer angles to allow the calculation of the zenith angle (i.e. , the angle between the incident light and surfacenormal, referred as incident angle theta). This is illustrated in FIG. 1 C, where the incident angle is obtained from SfP, and given the camera-display correspondence obtained from PMD, the surface depth and surface normal can be uniquely determined. Notably, we can find the unique surface point which is located on the camera ray (reflection ray), and we can satisfy the requirement that the angle between the incident ray and reflection ray is 2X (twice) the value of the incident angle. Therefore, the surface depth and surface normal are obtained simultaneously and uniquely.

[0020] In classic SfP, under the assumption of orthographic projection, the reflection ray is aligned with the Z-axis of the camera space; thus, the zenith angle is simply considered a component of the surface normal in the spherical coordinate system. However, in the perspective model, this approximation produces errors in the recovered surface normal, especially when the object is away from the image center. This issue is not present in the embodiments disclosed herein. Another feature of the disclosed embodiments is that we take advantage of specular reflection's law that the incident angle is equal to the reflection angle.

[0021] Using the disclosed technology, we can achieve accurate 3D reconstruction of specular objects with an easy setup including an unpolarized display and a polarization camera. From a general perspective, the disclosed embodiments exploit the polarization cues, instead of employing an additional camera or display, or relying on prior knowledge of objects, to solve the normal ambiguity problem associated with PMD-like techniques. Furthermore, the disclosed embodiments do not rely on the assumption of orthographic projection made in classic SfP, which can be the source of errors. Instead, the surface 3D information can be calculated using the disclosed embodiments in the perspective camera model without approximation, which enables a more accurate determination of surface shape and other characteristics. Notably, the disclosed embodiments enable simultaneous recovery of the surface normals and absolute depth associated with the specular object, whereas classic SfP and single-camera PMD techniques, among other deficiencies, are only able recover the surface normal and relative depth.

[0022] As noted above, the disclosed embodiments, among other features and benefits, leverage the information obtained from PMD and SfP in a unique way to solve the deficiencies of each method, and to achieve accurate 3D reconstruction of specular objects. The following sections provide further details regarding some of the issues of PMD and SfP and how the disclosed 3D sensor systems leverage various features to solve those problems.Phase Measuring Deflectometry

[0023] Phase measuring deflectometry or PMD is a well-established optical metrology method for measuring specular surfaces. Because of its high accuracy and inexpensive setup, it is a competitor to the optical interferometry system. In a PMD setup, a display illuminates the object with a known pattern (e.g., a fringe pattern). After reflection from the object, the pattern is deformed, which helps identify the pixel correspondence between the camera and the display (i.e. , which pixel or set of pixels on the display that illuminates the object correspond to which pixel or set of pixels on the detector that captures the deformed image). Once the correspondence is known, the surface normals (and the shape) of the object can be determined. In a classic technique, the phase of the illumination pattern is shifted in a temporal sequence, and multiple images are captured. However, the sequential capture aspect of PMD does not satisfy the speed requirement of many applications, such as eye-tracking and fast industrial inspection. To overcome this problem, besides improving the hardware speed, single-shot PMD methods use a cross-sinusoidal pattern that encodes the display pixel location that replaces the conventional phase-shifted patterns. The phase can be extracted in the Fourier domain using a single shot.

[0024] For each pair of the correspondence, the camera ray is determined given the pixel position and camera parameters; however, the illumination ray from the display is unknown because the display pixel emits light in all directions. As shown in FIG. 1A, two filled circles (one at the display screen and one at the camera) form a correspondence pair, and the ray coming out from the camera in the direction of the object is unique by connecting the camera pixel and camera nodal point. But all points (e.g., filled circles associated with Si, S2, S3, and S4 vectors) along this ray are possible surface points, resulting in differentheights and normals. As noted earlier, one way to address this problem is to use some prior knowledge of the object shape; another approach is to develop a stereo system to solve the height-normal ambiguity. For example, by adding a second camera to the system, each potential point along the ray of the first camera, can be connected to the second camera to find its corresponding display pixel. We can then search along the camera ray until the normals calculated from the two cameras are identical to each other. Stereo PMD successfully solves the height-normal ambiguity issue but at the cost of adding additional hardware components and more computational power. This also results in a more complicated system calibration procedure, which can exacerbate the problems given that PMD is highly reliant on a precise calibration. Moreover, stereo PMD only measures the surface area that is covered by both cameras, and therefore, it significantly decreases the measuring volume, especially for large curvature objects.Shape from Polarization

[0025] Shape from Polarization (SfP) is a well-known 3D sensing method that works for both specular and diffuse objects. To facilitate the understanding of the disclosed embodiments, we focus on SfP’s specular object measurement. SfP exploits the polarization cues of reflected light to estimate the surface normal. The unpolarized incident light becomes partially polarized after being reflected by the object’s surface. The surface 3D information and polarization cues are bridged by the Fresnel equation and Snell's law. Therefore, by exploiting image intensities, we can decode the surface shape. An example SfP setup includes a rotational linear polarizer in front of the camera to capture images at multiple polarizer angles. The recorded intensity varies sinusoidally represented as:

[0026] In Equation (1 ), < / >olis the polarizer angle, cp is reflected light's polarization angle. By fitting the recorded intensities into a sinusoidal wave, we determine the maximum and minimum intensities, Imax and lmin, respectively, that happen when only perpendicular orparallel polarized light passes through the polarizer, which are used to define the degree of polarization (dop) as:

[0027] To further derive, dop is written as a function of incident angle:

[0028] In Equation (3), 6Jis the incident angle and n is object's refractive indexes. We assume light is incident from the air.

[0029] Equation (2) is dop of the dielectric surfaces. For non-dielectric objects, dop is approximated as:where K is the material's attenuation index.

[0030] SfP assumes the orthographic projection model, so the reflected ray is perpendicular to the image plane. The incident angle, 0, is equal to the angle between the surface normal and the camera’s z-axis, considered as the zenith angle of the surface normal in spherical coordinates system. The azimuth angle, a , is defined as the angle between the projection of normal on the camera x-y plane and the camera’s x-axis. Then the surface normal is determined as: n = (sin0cosa, sin0sina, cos0 (5)

[0031] However, from Equations (2) or Eq. (4), for the same dop value, we have two potential zenith angles, 0. And in specular reflection, a =(p±x / 2. Therefore, there are four normal ambiguities. To overcome the normal ambiguity issue, SfP is fused with other 3D imaging methods such as multispectral measurements, shape from shading, coarsedepth map from depth sensor of stereo view. Although they solve the normal ambiguity problem (albeit by increasing the complexity and cost), 3D sensing methods use the perspective model, while SfP uses the orthographic model which results in errors in determination of surface normals and further errors in estimating surface depths.

[0032] Therefore, developing an SfP approach that can be flexibly and cost effectively implemented, can solve normal ambiguity problem, and can avoid the assumption of the orthographic model can result in significant improvements in 3D imaging and 3D image reconstruction.Example 3D Sensing, Measuring and Reconstruction Implementations

[0033] The disclosed embodiments combine certain features of PMD and SfP in a unique way, augmented with additional features to develop a new and improved 3D sensing method for specular objects. The disclosed embodiments solve the limitations of PMD and SfP simultaneously and deliver accurate surface normals and absolute depths values based on specular measurements.

[0034] FIG. 1C illustrates an example setup based on the disclosed embodiments that includes an unpolarized display and a polarization camera. In this example setup, we display a cross-sinusoidal pattern, and the polarization camera provides four images at different polarizer angles in one shot. We extract the zenith angle, 0 , as described earlier, and, in some embodiments, randomly pick one of the four images to retrieve the displaycamera correspondence, as also described earlier. Due to specular reflection's law, the angle between the incident ray and the reflected ray is 26 . Let’s define the camera coordinate system as the world coordinate system; using this coordinate system, the camera nodal point is the origin 0(0, 0, 0). With precise system calibration, a pair of camera-display correspondence points’ 3D positions are known, as C, where subscript d denotes the display, and subscript c denotes the camera. The surface pointis along the camera ray, so we have the following equations:DS = (xd-xs,yd-ys,zd-zs) (7)

[0035] The left-hand side of Equation (6) represents the surface point’s 3D coordinates. We know that this surface point is on the camera ray (e.g., the line connecting the surface point and the camera in FIGS. 1 A to 1 C). The camera ray, OC is defined byC(Xc ^c ’Zc ) - 0(0, 0, 0) = (xc,yc,zc). We only know that surface point, S, is on the camera ray, which means ray OS is linearly proportional to ray OC, but we do not know exactly how far surface point, S, is from the origin, O. The parameter t thus represents this proportionality relationship: OS = t * OC, as also shown in Equation (6). The surface point can be determined by t which can be uniquely numerically solved.

[0036] In this patent document, the term “depth” is used to refer to the 3D location of the surface point, S, of the object which can be simultaneously determined the surface normals. Surface normal vector, n, can be obtained using:

[0037] In the above computations, perspective projection camera model is used because the camera origin, O, camera pixel, C, and the surface point, S, are located on the same ray, which can be understood as a ray coming out from camera origin, O, and passing through C and S, which is represented by Equation (6), and illustrated in FIG. 4.

[0038] It is evident that from the PMD side, we solve the height-normal ambiguity without assistance from additional displays or cameras. We keep the single-camera PMD setup but don't require prior knowledge of object shape.

[0039] From the SfP side, we calculate the surface normal and depth simultaneously without the assumption of the orthographic model. Also, we avoid the azimuth angle ambiguity because we only utilize the degree of polarization to assist in reconstructing the object's surface. But in specular reflection, we still have to deal with the zenith angle ambiguity. We first use our setup's geometric information as a constraint for the zenith angle. For each pair of correspondence points, we know the surface point is along the camera ray, but it should be within the imaging lens's working distance. The zenith angle reaches its potential maximum value when the surface point is at the minimum working distance. The minimum is 0 degree at an infinite distance. If both candidates are within this range, we utilize the incident plane as the second constraint. The surface normal should be within the incident plane determined by the reflected ray and display pixel. From classic SfP, we can obtain four potential surface normals under the assumption of orthogonal projection. Although we don't select one of them as the final surface normal, we can use them to select the zenith angle. Among four candidates, we find the normal that has the minimum angular distance from the incident plane. We use this normal's zenith angle as our final choice. Considering the potentially low SNR of polarization images, the directly calculated t value may be noisy. We select the trust region's depth as seed points to assist normal integration to reconstruct the final shape.

[0040] We built our first prototype with a polarization camera (FLIR:BFS-U3-51 S5PC- C) and a tablet with an unpolarized display (Model: BOOX Tablet Tab X, resolution: 2048* 1536 pixels). This experimental setup is shown in FIG. 2A, illustrating the screen (light source - Tablet), the camera and the object (bearing ball). The polarization camera's full resolution is 2448 by 2048 pixels. Under polarization mode, it outputs four images at different polarizer angles in a single capture. The polarization image resolution is 1224 by 1024 pixels. The disclosed 3D measurement methodology is quantitatively evaluated by measuring a bearing ball with a known size (Diameter = 25.4mm) and refractive index n = 2.75 + 3.79 / . A cross-sinusoidal pattern is displayed to illuminate the bearing ball (see FIG. 2A). The object's surface normal and depth are calculated using the method discussed above. A sample image and calculated normal map as well as the 3D reconstruction areshown in FIGS. 2B and 2D. Specifically, the sample image from four images at different polarizer angles are illustrated in FIG. 2B, and the retrieved normal map is shown in panel FIG. 2C. FIG. 2D illustrates the 3D reconstruction. We compared our reconstructed shape with the respective ground truth. The RMSE of normal was 0.6 degrees. The reconstructed bearing ball radius was 25.47 mm while the ground truth is 25.4 mm.

[0041] Besides being a potential candidate for inspecting precise parts, which is a hot application in the optical metrology community, our method also works for measuring complex-shaped objects for computer vision applications. In FIG. 3, we demonstrated the measurement of two mirror-like objects. The images from left to right illustrate the object, captured sample images, calculated surface normals, and screenshots of reconstructed 3D shape from two views. Due to the complexity of their shape, we used the phase-shifting sinusoidal patterns rather than the single-shot cross-sinusoidal pattern to facilitate the illustration of the concepts.

[0042] The disclosed technology can be implemented in various embodiments to produce a measurement, sensing and image reconstruction system that are especially suited for objects with specular reflection properties. An example system includes one unpolarized display and one polarization camera, both facing the object to be measured. An unpolarized display is a display that emits unpolarized light. Thus, in displays where light is provided using LCDs - which are typically polarized - an unpolarized display can be provided by, for example, covering a normal LCD with a scattering material or a scattering sheet. Alternatively, special display architectures (such as an e-ink display) can be used to provide the unpolarized illumination. Using the above system, the camera is calibrated, and the camera-display geometrical calibration is performed to determine the relative locations of the components. The cross-sinusoidal pattern, for example, then illuminates the object and four simultaneous images with different polarization characteristics are obtained. For example, images can be obtained based on polarization angles of 0, 45, 90, 135 degrees. It should be noted that the cross-sinusoidal pattern is one example of a light pattern that can be used for illuminating the object. Other non-exclusive example patterns are sinusoids, stripes, or even any know pattens, such as image of a person, a scene and so on. Usingthe captured polarization images, zenith angle, 0, which is the angle between the incident light (from display) and surface normal, can be determined using the methodology that was described earlier in this patent document. In some embodiments, the polarization camera includes internal components that allow different polarization images to be captured in a single shot. In some embodiments, the polarization camera may be implemented using a simple camera with polarizer(s) placed in front of the camera configured to allow light of particular polarizations to pass through. For example, the polarizers can be positioned on a rotating stage in from the camera.

[0043] The determined zenith angle has two potential values (zenith ambiguity), which can be resolved based on deflectometric information. For example, one of those four polarization images is selected to retrieve the phase map of the displayed pattern, which can be used to calculate camera-display correspondence. The phase map can be calculated, for example, by Fourier transform methods or 2D continuous wavelet transform techniques. Next, the zenith angle ambiguity is resolved with geometric deflectometry cues. Notably, if the zenith angle (angle between surface normal and incidence ray) is 0, then the angle between incidence ray and camera ray is 20, and the surface depth and normal can be uniquely calculated once the zenith angle is known.

[0044] FIG. 5 illustrates a set of operations that can be carried out to measure a characteristic of an object in accordance with an example embodiment. At 502, the object is illuminated with a known illumination pattern displayed by a screen, wherein the known illumination pattern is produced using unpolarized light. At 504, a plurality of images of the object is captured based on light that is specularly reflected from the object. The plurality of images is captured using a polarization sensitive detector and each of the one or more images corresponds to a particular polarization state. At 506, an estimate of a zenith angle associated with a screen-detector configuration is determined based on polarization information associated with the one or more images. The zenith angle represents an angle between a light ray from the screen that is incident on the object at a particular location and a surface normal of the object at the particular location, and the estimate of the zenith angle has an ambiguity among at least two values. At 508, using deflectometric informationobtained from at least one or the one or more images, one of the at least two values for the zenith angle is selected. The deflectometric information includes a screen-detector correspondence that enables determination of the surface normal at the particular location.

[0045] In one example embodiment, the known illumination pattern is a cross- sinusoidal pattern, and using the deflectometric information includes using one of the one or more images to retrieve a phase map of the cross-sinusoidal pattern and to determine the screen-detector correspondence. In another example embodiment, the phase map is obtained based on a Fourier transform or a wavelet transform. In still another example embodiment, determination of the surface normal includes using a perspective camera model for the polarization sensitive detector. In yet another example embodiment, the above noted method includes determining surface normals at multiple locations of the object based on repeating the operations 506 and 508 for the multiple locations of the object.

[0046] According to another example embodiment, the above noted method includes simultaneously determining the surface normal and absolute depth at the particular location for the object. In one example embodiment, the one or more images includes four images, wherein each image of the plurality of images corresponds to a unique polarization state. In still another example embodiment, the unique polarization states include four linear polarization states at 0, 45, 90 and 135 degrees. In yet another example embodiment, operations 504 to 508 of FIG. 5 are carried out using, or based on measurements obtained from, the polarization sensitive camera operating in a single-shot operation. In another example embodiment, selection of the one of the at least two values for the zenith angle includes randomly selecting one of two values of the estimated zenith angle, and using a constraint based on the deflectometric information to determine whether the selected value conforms to a predetermined condition or criteria associated with the deflectometric information. In another example embodiment, the characteristic of the object includes one or more of: a surface normal of at the particular location or at one or more additional locations, a depth of the object at the particular location or at one or more additional locations, a surface map of the object, or a 3-dimensional reconstruction of the object.

[0047] Another aspect of the disclosed embodiments relates to a system for measuring a characteristic of an object that includes a screen configured to illuminate the object with a known illumination pattern using unpolarized light, a polarization sensitive detector positioned to capture a plurality of images of the object based on light that is specularly reflected from the object, wherein each of the plurality of images correspond to a different polarization state, a processor and a memory including instructions stored thereon. The instructions upon execution by the processor configure the processor to determine an estimate of a zenith angle associated with a screen-detector configuration based on polarization information associated with the plurality of images, wherein the zenith angle represents an angle between a light ray from the screen that is incident on the object at a particular location and a surface normal of the object at the particular location, and the estimate of the zenith angle has an ambiguity among at least two values. The instructions upon execution by the processor also configure the processor to use deflectometric information obtained from one or more of the plurality of images to select one of the at least two values for the zenith angle, wherein the deflectometric information includes a screendetector correspondence that enables determination of the surface normal at the particular location.

[0048] In one example embodiment, the known illumination pattern is a cross- sinusoidal pattern, and the instructions upon execution by the processor configure the processor to, as part of using the deflectometric information, use one of the plurality of images to retrieve a phase map of the cross-sinusoidal pattern and to determine the screendetector correspondence. In another example embodiment, the instructions upon execution by the processor configure the processor to obtain the phase map based on a Fourier transform or a wavelet transform. In yet another example embodiment, the instructions upon execution by the processor configure the processor to, as part of determination of the surface normal, use a perspective camera model for the polarization sensitive detector.

[0049] According to another example embodiment, the instructions upon execution by the processor configure the processor to determine surface normals at multiple locations of the object based on repeated determinations of estimates of the zenith angle and selectionof the one of the at least two values of the zenith angle at the multiple locations of the object. In another example embodiment, the instructions upon execution by the processor configure the processor to simultaneously determine the surface normal and absolute depth at the particular location for the object. In still another example embodiment, the plurality of images includes four images, wherein each image corresponds to a unique polarization state. In yet another example embodiment, the screen and the polarization sensitive camera are the only screen and the only detector in the system to obtain the estimate of the zenith angle and to select the one of the at least two values. In one example embodiment, the instructions upon execution by the processor configure the processor to, as part of selection of the one of the at least two values for the zenith angle, randomly select one of two values of the estimated zenith angle, and use a constraint based on the deflectometric information to determine whether the selected value conforms to a predetermined condition or criteria associated with the deflectometric information. In another example embodiment, the characteristic of the object includes one or more of: a surface normal of at the particular location or at one or more additional locations, a depth of the object at the particular location or at one or more additional locations, a surface map of the object, or a 3-dimensional reconstruction of the object. In one example embodiment, the polarization sensitive detector is operable in a single-shot mode, configured to capture the plurality of images in a single frame

[0050] As noted earlier, PMD and SfP are top candidates for specular surface 3D reconstruction from optical metrology and computer vision respectively. However, they all have normal ambiguity issues. Moreover, SfP's accuracy suffers from the assumption of the orthographic projection model and it can't obtain the absolute surface depth. The example embodiments disclosed herein use sensor fusion to bridge PMD and SfP. Notably, polarization cues are used to solve single-camera PMD's height-normal ambiguity. With geometric cues from PMD, SfP's normal ambiguity is resolved, and orthographic projection assumption is not needed. Instead, the commonly used perspective model is used, which can uniquely calculate the surface normal and depth simultaneously. Our experiments deliver sub-millimeter accuracy and successfully reconstruct the complex surface shape.

[0051] Various operations disclosed herein can be implemented using a processor / controller is configured to include, or be couple to, a memory that stores processor executable code that causes the processor / controller carry out various computations and processing of information. The processor / controller can further generate and transmit / receive suitable information to / from the various system components, as well as suitable input / output (IO) capabilities (e.g., wired or wireless) to transmit and receive commands and / or data. The processor / controller may receive the information associated with optical rays and material parameters, and further process that information to simulate or trace rays throughout an optical system.

[0052] Various information and data processing operations described herein may be implemented in one embodiment by a computer program product, embodied in a computer- readable medium, including computer-executable instructions, such as program code, executed by computers in networked environments. A computer-readable medium may include removable and non-removable storage devices including, but not limited to, Read Only Memory (ROM), Random Access Memory (RAM), compact discs (CDs), digital versatile discs (DVD), etc. Therefore, the computer-readable media that is described in the present application comprises non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0053] From the foregoing, it will be appreciated that specific embodiments of the invention have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the invention. Accordingly, the invention is not limited except as by the appended claims.

Claims

CLAIMSWHAT IS CLAIMED IS:1 . A method for measuring a characteristic of an object, comprising:(a) illuminating the object with a known illumination pattern displayed by a screen, wherein the known illumination pattern is produced using unpolarized light;(b) capturing one or more images of the object based on light that is specularly reflected from the object, wherein the one or more images are captured using a polarization sensitive detector and each of the one or more images corresponds to a particular polarization state;(c) determining an estimate of a zenith angle associated with a screen-detector configuration based on polarization information associated with the one or more images, wherein: the zenith angle represents an angle between a light ray from the screen that is incident on the object at a particular location and a surface normal of the object at the particular location, and the estimate of the zenith angle has an ambiguity among at least two values; and(d) using deflectometric information obtained from at least one of the one or more images to select one of the at least two values for the zenith angle, wherein: the deflectometric information includes a screen-detector correspondence that enables determination of the surface normal at the particular location.

2. The method of claim 1 , wherein the known illumination pattern is a cross-sinusoidal pattern, and using the deflectometric information includes using one of the one or more images to retrieve a phase map of the cross-sinusoidal pattern and to determine the screendetector correspondence.

3. The method of claim 2, wherein the phase map is obtained based on a Fourier transform or a wavelet transform.

4. The method of claim 1 , wherein determination of the surface normal includes using a perspective camera model for the polarization sensitive detector.

5. The method of claim 1 , comprising determining surface normals at multiple locations of the object based on repeating the operations (c) and (d) for the multiple locations of the object.

6. The method of claim 1 , comprising simultaneously determining the surface normal and absolute depth at the particular location for the object.

7. The method of claim 1 , wherein the one or more images includes four images, wherein each image of the plurality of images corresponds to a unique polarization state.

8. The method of claim 7, wherein the unique polarization states include four linear polarization states at 0, 45, 90 and 135 degrees.

9. The method of claim 1 , wherein operations (b) to (d) are carried out using, or based on measurements obtained from, the polarization sensitive detector operating in a singleshot operation.

10. The method of claim 1 , wherein selection of the one of the at least two values for the zenith angle includes randomly selecting one of two values of the estimated zenith angle, and using a constraint based on the deflectometric information to determine whether the selected value conforms to a predetermined condition or criteria associated with the deflectometric information.11 . The method of claim 1 , wherein the characteristic of the object includes one or more of: the surface normal at the particular location or at one or more additional locations,a depth of the object at the particular location or at one or more additional locations, a surface map of the object, or a 3-dimensional reconstruction of the object.

12. A system for measuring a characteristic of an object, comprising: a screen configured to illuminate the object with a known illumination pattern using unpolarized light; a polarization sensitive detector positioned to capture a plurality of images of the object based on light that is specularly reflected from the object, wherein each of the plurality of images correspond to a different polarization state; a processor and a memory including instructions stored thereon, wherein the instructions upon execution by the processor configure the processor to: determine an estimate of a zenith angle associated with a screen-detector configuration based on polarization information associated with the plurality of images, wherein the zenith angle represents an angle between a light ray from the screen that is incident on the object at a particular location and a surface normal of the object at the particular location, and the estimate of the zenith angle has an ambiguity among at least two values; and use deflectometric information obtained from one or more of the plurality of images to select one of the at least two values for the zenith angle, wherein the deflectometric information includes a screen-detector correspondence that enables determination of the surface normal at the particular location.

13. The system of claim 12, wherein the known illumination pattern is a cross-sinusoidal pattern, and the instructions upon execution by the processor configure the processor to, as part of using the deflectometric information, use one of the plurality of images to retrieve a phase map of the cross-sinusoidal pattern and to determine the screen-detector correspondence.

14. The system of claim 13, wherein the instructions upon execution by the processor configure the processor to obtain the phase map based on a Fourier transform or a wavelet transform.

15. The system of claim 12, wherein the instructions upon execution by the processor configure the processor to, as part of determination of the surface normal, use a perspective camera model for the polarization sensitive detector.

16. The system of claim 12, wherein the instructions upon execution by the processor configure the processor to determine surface normals at multiple locations of the object based on repeated determinations of estimates of the zenith angle and selection of the one of the at least two values of the zenith angle at the multiple locations of the object.

17. The system of claim 12, wherein the instructions upon execution by the processor configure the processor to simultaneously determine the surface normal and absolute depth at the particular location for the object.

18. The system of claim 12, wherein the plurality of images includes four images, wherein each image corresponds to a unique polarization state.

19. The system of claim 12, wherein the screen and the polarization sensitive detector are the only screen and the only detector in the system to obtain the estimate of the zenith angle and to select the one of the at least two values.

20. The system of claim 12, wherein the instructions upon execution by the processor configure the processor to, as part of selection of the one of the at least two values for the zenith angle, randomly select one of two values of the estimated zenith angle, and use a constraint based on the deflectometric information to determine whether the selected value conforms to a predetermined condition or criteria associated with the deflectometric information.21 . The system of claim 12, wherein the characteristic of the object includes one or more of: the surface normal at the particular location or at one or more additional locations, a depth of the object at the particular location or at one or more additional locations, a surface map of the object, or a 3-dimensional reconstruction of the object.

22. The system of claim 12, wherein the polarization sensitive detector is operable in a single-shot mode, configured to capture the plurality of images in a single frame.

Images