Method for correcting multipath light propagation in structured-light 3D scanners

Abstract

A method of measuring a 3D shape of an object comprising a target semi-transparent surface, the method comprising the steps of: with a projector, projecting a first light signal having a predetermined pattern and a predetermined frequency from at least one projector column towards at least one location on a scene; with an image capture device, receiving a second light signal from the at least one location of the scene at an image capture device pixel; measuring an intensity signal received at the image capture device pixel; with the measured intensity signal, generating a correspondence map between the at least one projector column illuminating the at least one location of the scene and the image capture device pixel with coordinates receiving the light signal from the at least one location of the scene; and with the correspondence map, determining 3D coordinates of the at least one location of the scene.

Description

METHOD FOR CORRECTING MULTIPATH LIGHT PROPAGATION IN STRUCTURED-LIGHT 3D SCANNERSFIELD

[0001] Aspects of the disclosure relate to methods and systems for measuring 3D shape of objects comprising semi-transparent surfaces and opaque surfaces with projector-camera 3D scanners.BACKGROUND

[0002] Measuring 3D shape of semi-transparent surfaces with projector-camera 3D scanners is a difficult task because these surfaces weakly reflect light in a diffuse manner, and transmit a large part of the incident light. The task is even harder in the presence of participating background surfaces.

[0003] State-of-the-art works related to 3D shape measurement of semi-transparent surfaces using a projector-camera system include methods addressing multipath light propagation, and methods for reconstructing transparent objects. The following is an overview of related works and adopt a taxonomy in six classes (intrusive, phase measurement, multipath correction, multi-frequency, shape from distortion and learning-based methods) [4], [7]- [9] .

[0004] 1) Intrusive methods'. Before measuring transparent surfaces, one can cover them with a thin layer of diffuse material (e.g. powder). However this surface treatment is not viable in industrial environment because it is time consuming, and can result in overestimating the size of the surface

[0010] , Scanning from heating is another intrusive method. A high -power laser is utilized to locally heat up the surface while it is captured by an infrared camera

[0011] , Triangulation principle is then used to compute the corresponding 3D point. However, the high power laser can pose security issues, and heating the surface can be long and destructive.

[0005] 2) Phase measurement profilometry methods'. Phase measurement profilometry consists in recovering the 3D shape through the per pixel phase computed using fringe patterns. It has demonstrated micrometric accuracy in measuring 3D shape of diffuse opaque surfaces [3], [4], [6] after robust phase unwrapping

[0012] , Phase measurement profilometry methods assume that there is a single path light propagation, i.e., the light from the projector reaches the camera after a single bounce on the scene. They generally fail to measure surfaces in the presence of multipath light propagation unless additional formulations and assumptions are considered

[0013] —

[0015] ,

[0006] 3) Multipath correction methods'. Multipath light propagation in projector-camera systems refers to scenarios where a camera pixel receives light originating from two or more projector pixels / columns. It is the case in the presence of interreflections or subsurface scatterings / reflections, when the camera observes a step edge, or when a semi-transparent surface has a participating background surface. One must extract the contribution of the front / target surface from intensity signal measured by each camera pixel. This has been achieved using unstructured light

[0016] , and using discrete Fourier transform to acquire transparent surfaces

[0017] , and opaque surfaces in the presence of interreflections

[0018] , High frequency patterns were used for scenes under global illumination

[0013] —

[0015] ,

[0019] , Multi-path correction was also achieved by analyzing how light paths interact as a function of the pattern frequency

[0020] , or by combining a sparse Bayesian learning framework with novel applicationspecific constraints

[0021] , Others authors designed patterns that are resilient to indirect light paths

[0013] ,

[0015] ,

[0016] ,

[0019] , or proposed iterative projection approaches consisting in several round of data acquisition

[0022] ,

[0023] , Multi-path correction methods relying on special hardware, e.g. polarizer

[0014] or a custom modification of the optics

[0024] , have also been proposed.

[0007] 4) Multi-frequency methods'. Multi-frequency illumination methods have been proposed for phase unwrapping in phase measurement profilometry

[0013] —

[0015] ,

[0025] , to acquire surfaces by multipath correction

[0013] —

[0015] ,

[0017] ,

[0018] ,

[0025] , to obtain the appearance of inner slices of translucent objects

[0026] , and in deflectometry techniques

[0027] ,

[0028] ,

[0008] 5) Methods for measuring 3D shape from distortion'. Methods relying on distortion to measure the 3D shape of transparent objects operate by scrutinizing, either from a single camera viewpoint (

[0017] ,

[0029] ,

[0030] ) or from several camera viewpoints (

[0031] —

[0033] ), the distortion of known patterns display ed / projected behind or near the target transparent object. These distortions are due to the refraction

[0030] ,

[0031] ,

[0034] , or reflection of light

[0017] , Shape from refraction methods include model-based reconstruction methods which require a parametrization of the measured surface

[0030] , direct ray measurement methods where it is assumed that no more than two refractions occur

[0034] , and intrusive methods where the objects is immersed in a liquid to simplify the light transport model

[0035] , A position normal consistency constraint is used in

[0036] , while

[0032] and

[0033] use surface and refraction normal consistency, surface projection and silhouette consistency, and surface smoothness to refine a coarse 3D model through a computationally expensive optimization process. In the reflection -based method proposed in

[0017] , patterns of different frequencies are projected on the transparent surface. The surface points are determined by exploiting the discrete Fourier transform. Indeflectometry techniques, the specular reflection of fringe patterns on the target surface are captured by the camera. The computed phase is used to recover the surface 3D shape

[0027] ,

[0028] , It has also been proposed to use the polarization state of the reflected light to acquire (semi) transparent surfaces

[0014] ,

[0037] ,6) Learning-based methods'. Learning-based methods utilize neural networks to reconstruct transparent objects [9], Existing training datasets are constituted in a large proportion by synthetic data capturing different (shape, texture, material) transparent objects under various environments

[0038] ,

[0039] , In

[0040] , a convolutional neural network uses a single input image to estimate the 3D shape and normals of a transparent surface. Depth completion is performed by exploiting a local implicit neural representation built on ray-voxel pairs

[0039] , using deep convolutional network

[0038] , or using a generative adversarial network (GAN)

[0041] , Multi-view methods operate with several images of the transparent object taken from different viewpoints. They are less efficient (computational cost) but more robust (poor lighting, cluttered objects) than single view approaches. The methods proposed in

[0042] refines the visual hull of the object based on the predictions of normals. By representing a 3D scene as a continuous function, neural radiance field can be exploited to reconstruct transparent objects

[0043] , One of the strengths of learning-based methods is that they can be formulated without constraints and assumptions. Their main drawback, in addition to the fact that their need large, annotated datasets, is their current inability to achieve accuracy required in metrological industrial applications where reliability and consistency of 3D measurements are of great importance [8],SUMMARY

[0009] In one example, a method of measuring a 3D shape of an obj ect comprising a target semi-transparent surface, the method comprising the steps of: with a projector, projecting a first light signal having a predetermined pattern and a predetermined frequency from at least one projector column towards at least one location on a scene; with an image capture device, receiving the second light signal from the at least one location of the scene at an image capture device pixel; measuring an intensity of the second signal received at the image capture device pixel; with the measured second light signal, generating a correspondence map between the at least one projector column illuminating the at least one location of the scene and the imagecapture device pixel with coordinates receiving the light signal from the at least one location of the scene; and with the correspondence map, determining 3D coordinates of the at least one location of the scene by triangulation.

[0010] In another example, a method of isolating a desired illumination from a plurality of illumination components from a target surface of an object to determine a shape of the target surface, the method comprising: projecting a first light signal having predetermined patterns and predetermined frequencies from a projector towards at least one location on a scene on the target surface; receiving a second light signal from the at least one location of the scene at an at least one image capture device pixel, wherein the second light signal comprises a plurality of illumination components; measuring an intensity of the second light signal to quantify each of the plurality of illumination components measured at each of the at least one image capture device pixel; inventorying the plurality of illumination components to extract the desired illumination from a plurality of illumination components based on the measured second light signal; with the measured second light signal, with an optimization model generating a plurality of correspondence maps between at least one projector column illuminating the at least one location of the scene and the at least one image capture device pixel receiving the second light signal from the at least one location of the scene, wherein the plurality of correspondence maps comprises raw 3D point clouds comprising the at least one location associated with each of the plurality of illumination components; from the raw 3D point clouds, determining sets of 3D points having a predefined surface constraint of continuity with neighbour 3D points; and based on the predefined surface constraint of continuity extracting a shape of the target surface.

[0011] In another example, a system for measuring a 3D shape of an object comprising a target semi-transparent surface, the system comprising:a projector to projecting a light signal having a predetermined pattern and a predetermined frequency from at least one projector column towards at least one location on a scene; an image capture device for receiving the light signal from the at least one location of the scene at an image capture device pixel, and for measuring an intensity signal received at the image capture device pixel; a processing unit for executing instructions stored in a computer readable medium to at least perform the steps of: using the measured intensity signal, generate a k correspondence map constructed using the at least one projector column linked to the k'hhighest peak at each image capture device pixel, wherein the k correspondence maps obtained are used to compute k 3D point clouds using triangulation, wherein the k correspondence maps and corresponding 3D point clouds comprise raw data with information related to at least one of the target semi-transparent surface and a background surface. applying a surface continuity constraint to the raw data to extract the correspondence map and the 3D point cloud representing the target semi-transparent surface only.

[0012] In other examples, there are provided methods of using multi -frequency patterns and curve fitting to measure the 3D shape of semi-transparent surfaces or opaque surfaces. In one example, sinusoidal patterns with frequencies selected in the frequency range allowed by the projection optics of the system are projected on to a scene, and discrete Fourier transform (DFT) performed on the intensity signal measured at a camera pixel to inventory the projector columns illuminating directly and indirectly the scene point imaged at that pixel. In another example, sinusoidal patterns with frequencies selected in the frequency range allowed by the projection optics of the system are projected on to a scene, and curve fitting (CF)-based method is employed to fitting an analytical model to the measured intensity signal. Both methods produce a one camera pixel) to many (projector columns) correspondence between a camera pixel and projectors columns illuminating, directly or indirectly, the scene point imaged at that pixel. This results in several camera-projector correspondence maps, each of them yielding a 3D point cloud. A surface continuity constraint is applied to extract the 3D point cloud of the target surface.

[0013] The DFT-based and curve fitting-based methods described herein are not intrusive and rely on the reflection of light to operate. These methods are fundamentally different fromlaser scanning approaches

[0044] , and from single pixel imaging methods

[0045] , The methods use multi-frequency sinusoidal patterns to explicitly address multipath light propagation in three steps: light paths captured by each camera pixel are inventoried, then the direct illumination received by each camera pixel is determined, and finally the 3D coordinates of the semitransparent surface points are determined. Furthermore, these methods do not rely on phase measurement; rather, they use a nonlinear formulation, making them generic and versatile. In contrast, the prior art methods for measuring the 3D shape of transparent surfaces require either special hardware or are built under restrictive assumptions (known refractive index and number of refraction, known 3D shape estimate; require the use of high frequency patterns; simple or parametrizable surface geometry, and linear modulation transfer function of the projectorcamera system, etc.). Furthermore, although learning-based methods show promising results for applications requiring only coarse 3D models (e.g. computer graphics, robot navigation and grasping), they are not yet suitable for metrological applications in industrial inspection. One of the advantages of the reflection-based methods described herein is that they use a projectorcamera pair without additional hardware, and achieve metrological accuracy in measuring the 3D shape of semi-transparent surfaces characterized by weak reflected light and perturbing background illuminations. Preferably, the semi-transparent surface reflects enough light for the camera to measure a signal above its noise level. The DFT-based method goes further than

[0017] by considering the perturbing illumination from background surfaces, by providing an in-deep theoretical fundamental (frequency selection, accuracy expected, influence of the projection optics), and by elaborating on practical implementations (projection hardware, acquisition speed and processing speed). The curve fitting based method described herein goes beyond DFT and achieves comparable accuracy with a significantly reduced number of patterns.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Several exemplary embodiments of the present disclosure will now be described, by way of example only, with reference to the appended drawings in which:

[0015] Figure la shows a projector-camera configuration 3D shape measurement of semitransparent surfaces with no participating background surfaces (i.e., either no background surface or background surfaces at infinity using a projector-camera system, and Figure lb shows a geometric equivalent;

[0016] Figure 1c shows a projector-camera configuration 3D shape measurement of semitransparent surfaces in the presence of participating background surfaces, in which point Q isalso illuminated by the perturbing light beam Lb originating from projector column Cb and reflected by a background surface, and Figure Id shows a geometric equivalent;

[0017] Figure le shows a projector-camera configuration 3D shape measurement of semitransparent surfaces with no participating background surfaces (i.e., either no background surface or background surfaces at infinity using a projector-camera system, in which light beam Lsoriginating from projector column csreaches point Q after subsurface scatterings / refl ections within the semi-transparent object, and Figure If shows a geometric equivalent;

[0018] Figure 1g shows a projector-camera configuration 3D shape measurement of semitransparent surfaces in scenarios involving both participating background surfaces and subsurface reflections / scatterings , in which the light beam L arriving at camera pixel v is a combination of the direct illumination Z* received by point Q, and indirect illuminations Lsand Lb received by Q, and Figure Ih shows a geometric equivalent;

[0019] Figure 2a shows an empirical model for amplitude offset A (i) and amplitude contrast B (ii) for a projector-camera system: projector resolution: 1024 x 768, and camera resolution: (1920 x 988, 16to), BL = 80cm, SoD = 200cm, H = 180cm, W = 100cm, DoF = 150cm;

[0020] Figure 2b shows an empirical model for amplitude offset A (i) and amplitude contrast B (ii) for a projector-camera system: projector resolution: 1024 x 768, and camera resolution: camera (3400 x 2200, ibit), BL = 15cm, SoD = 22cm, H = 23mm, W= 33mm, DoF = 13mm;

[0021] Figure 2c shows an empirical model for amplitude offset A (i) and amplitude contrast B (ii) for a projector-camera system: projector resolution: 1024 x 768, and camera resolution: (3400 x 2200, \Obit), BL = 26cm, SoD = 36cm, H = 33cm, W= 22cm, DoF= 16cm;

[0022] Figure 3a shows a flowchart comprising example acquisition and processing steps for a DFT-based method;

[0023] Figure 3b shows a flowchart comprising example acquisition and processing steps for a curve fitting-based method;

[0024] Figure 4a shows a simulation of a DFT based method in estimating the values c±and c2of projector columns illuminating a given scene point for different values of the spacing 8c = | Ci — c21, and intensity signal (i) and DFT (ii); be = 15 > Ac: two peaks are present in the DFT and their positions are close to the true values.

[0025] Figure 4b shows a simulation of a DFT based method in estimating the values c and c2of projector columns illuminating a given scene point for different values of the spacing 8c = I — c2\, and intensity signal (i) and DFT (ii); 5c = 3 < Ac: Only one peak is present. DFT fails in determining the two components c and c2present in the signal.

[0026] Figure 4c shows a simulation of a curve fitting method in estimating the values c and c2of projector columns illuminating a given scene point for different values of the spacing 8c = | Ci — c21, Intensity signal (i) and the result of the fitting for a suitable initialization (ii); 5c = 15 > Ac; the determined cl and c2 are close to the true values;

[0027] Figure 4d shows a simulation of a curve fitting method in estimating the values c and c2of projector columns illuminating a given scene point for different values of the spacing 8c = | Ci — c21, Intensity signal (i) and the result of the fitting for a suitable initialization (ii); 5c = 3 < Ac; the determined cl and c2 are close to the true values;

[0028] Figure 5a shows a scene captured by the camera when all projector columns are turned on;

[0029] Figure 5b shows the scene of Figure 5a when a sinusoidal pattern is projected;

[0030] Figure 5c shows intensity signals at point Qi (i) and Q2 (ii), of a DFT-based method using 768 samples;

[0031] Figure 5d shows the DFT magnitude of the intensity signals of Figure 5c at point Qi (i) and Q2 (ii); in which point Qi is illuminated by one projector columns (one peak); point Q2 is illuminated by two projector columns (two peaks);

[0032] Figures 5e-f shows two correspondence maps of the DFT-based method;

[0033] Figures 5g-h shows two 3D point clouds of the DFT-based method;

[0034] Figure 5i shows intensity signals at point Qi (i) and Q2 (ii), of a curve fitting-based method using 32 samples;

[0035] Figure 5j shows results of fitting model (3) to the intensity signals of figure 5i, in which point Qi is illuminated by one projector columns (one component) and point Q2 is illuminated by two projector columns (two components);

[0036] Figures 5k-l show correspondence maps of the curve fitting-based method;

[0037] Figures 5m-n show 3D point clouds of the curve fitting-based method;

[0038] Figure 5o shows an outcome after a surface continuity constraint is applied to the raw results to extract the correspondence map and 3D point cloud of the target semi-transparent surface;

[0039] Figure 6 shows results for some semi-transparent surfaces with a first column (a) showing an image of picture of a scene (i) and camera viewpoint (ii); column (b) shows a 3D point cloud (i), Z-map (ii), and ID profile along the black line in the Z-map (iii) for a PSbased method and a line scanning method; and column (c) shows a 3D point cloud (i), Z-map (ii), and ID profile along the black line in the Z-map (iii) for a DFT-based method and a curve fitting method;

[0040] Figure 7 shows quantitative results (rmsd, n): the root-mean-square deviation rmsd is computed using n 3D points belonging to the region bounded by the dashed lines;

[0041] Figure 8 shows results using state-of-the-art methods versus the novel methods described herein; and

[0042] Figure 9 shows an overview of a computing environment of components configured to facilitate the systems and methods described herein.DETAILED DESCRIPTION

[0043] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar elements. While embodiments of the disclosure may be described, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the elements illustrated in the drawings, and the methods described herein may be modified by substituting, reordering, or adding stages to the disclosed methods. Accordingly, the following detailed description does not limit the disclosure. Instead, the proper scope of the disclosure is defined by the appended claims.

[0044] Moreover, it should be appreciated that the particular implementations shown and described herein are illustrative of the invention and are not intended to otherwise limit the scope of the present invention in any way. Indeed, for the sake of brevity, certain subcomponents of the individual operating components, conventional data networking, application development and other functional aspects of the systems may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and / or physical couplings between the variouselements. It should be noted that many alternative or additional functional relationships or physical connections may be present in a practical system.

[0045] In one example, a projector camera system 10 measures a 3D shape of an object 11 comprising a target surface 12 as shown in Figures la-h. Light patterns are generated using a specific coding strategy and sequentially projected on a scene 13 by the projector 14. An image capture device 15, such as a camera, having a different viewpoint than the projector 14, sequentially captures the appearance of the illuminated scene 13. Images recorded by the camera 15 are decoded (processed) to establish the camera-projector correspondence map which serves to triangulate the surface points and calculate their 3D coordinates by making use of a prior geometrical calibration of the projector-camera pair. An intensity signal measured at a camera pixel 24 to inventory the projector columns illuminating directly and indirectly the scene point 22 imaged at that pixel 24, to generate a one camera pixel 24) to many (projector columns 20) correspondence between a camera pixel 24 and the projectors columns 20 illuminating, directly or indirectly, the scene point 22 imaged at that pixel 24. This results in several camera-projector correspondence maps, each of them yielding a 3D point cloud. A surface continuity constraint is applied to extract the 3D point cloud of the target semitransparent surface 12.

[0046] Looking at Figure la, there is shown a projector-camera system 10 for 3D shape measurement of an object 11 with a target surface 12 with no participating background surfaces (i.e., either no background surface or background surfaces at infinity, such as an opaque surface) using a projector 14 and camera 15, and Figure lb shows a geometric equivalent of the projector-camera configuration 10 of Figure la. Light L* 18 with light patterns generated using a specific coding strategy and are sequentially projected on the scene 11 by the projector 14. Sinusoidal patterns are used to establish the correspondence between projector columns c* 21 directly illuminating point Q 22, and camera pixel v 24 with coordinates (x, y) receiving light 26 from point Q 22. An intensity signal measured at a camera pixel 24 to inventory the projector columns 20 illuminating directly and indirectly the scene point 22 imaged at that pixel 24. This correspondence is utilized to determine the 3D coordinates of point Q 22 by triangulation. In this case, state-of-the-art methods should provide accurate results as long as the surface reflects enough light. Point Q 22 is illuminated by only one light beam L* 18 coming from projector column c* 21 (L* 18 is the direct illumination of point Q 22), and light L 26 received from point Q 22 is imaged by camera pixel v 24.

[0047] Figure 1c shows a projector-camera configuration for 3D shape measurement of a target surface 12, such as a semi-transparent surface 12 in the presence of participating background surfaces 30, in which point Q 22 is also illuminated by the perturbing light beam Lb 32 originating from projector column Cb 34 and reflected by a background surface 30, and light L 26 received from point Q 22 is imaged by camera pixel 24. Figure Id shows a geometric equivalent of the projector-camera configuration 10 of Figure 1c..

[0048] Figure le shows a projector-camera configuration 3D shape for measurement of semi-transparent surfaces 12 with no participating background surfaces (i.e., either no background surface or background surfaces at infinity) using a projector-camera system 10, in which light beam Ls40 originating from projector column cs42 reaches point Q 22 after subsurface scatterings / reflections 44 within the semi-transparent object 11, and Figure If shows a geometric equivalent of the projector-camera configuration 10 of Figure 1c.

[0049] Figure 1g shows a projector-camera configuration 3D shape for measurement of semi-transparent surfaces 12 in scenarios involving both participating background surfaces 30 and subsurface reflections / scatterings 44, 46, 48, and also light due to interreflections (not shown) in which the light beam / . 26 arriving at camera pixel v 24 is a combination of the direct illumination Z* 18 received by point Q 22, and indirect illuminations Ls40 and Lb 50 received by Q 22, and Figure Ih shows a geometric equivalent.

[0050] In one example, a discrete Fourier transform (DFT)-based method is used to measure a 3D shape of an object 11 with a semi-transparent surface. The DFT-based method comprises similar formulation as the Baseline Frequency shifted method was proposed in two previous works on 3D shape measurement in the presence of interreflections

[0018] ,

[0047] , and comprises acquisition and processing steps. A coding strategy of the method is first formulated, then the decoding strategy utilizing DFT is implemented.

[0051] Looking at Figure 3a, there is shown a flow chart 100 with example steps for measuring a 3D shape of a target semi-transparent surface 12 of a scene using a DFT-based method. The method comprises an acquisition stage, a processing stage, and an output stage. In one example, in the acquisition stage, sinusoidal patterns (1) are generated using frequencies chosen to satisfy the Shannon-Nyquist sampling condition, as part of the coding strategy (step 102).

[0052] The method utilizes sinusoidal patterns (1) with frequencies fm(m = 1, 2, ... , AL) given by:

[0053] fm= A + (m - 1) A , fmE [0; fmax] (4)

[0054] In (4), f is the minimum frequency. The maximum frequency allowed by the projection optics of the projector-camera system 10 is fmax. In theory, fmaxis equal to 0.5pixels-1(minimum period = pixels). In one example, fmaxis determined empirically since it is influenced by the modulation transfer function of the projector 14 and the limited capacity of a digital projector 14 to approximate high frequency sinusoidal signals. Afis the sampling (frequency) step size, and is chosen such that the Shannon-Nyquist sampling condition is satisfied. For a projector 14 with cmaxcolumns sampling (frequency) step size Afis defined as:

[0056] The width fwof the sampling interval [0; fnax\ may be used to also express the sampling step size as A = fw / (M — 1). Then we obtain the number of patterns as follows:

[0057] M > 1 + 2fwcmax(6)

[0058] In step 104, a projector 14 projects the sinusoidal patterns from one or more projector pixels / columns towards a point Q 22 of the scene 13. At point Q 22, the incident light 18 is reflected by semi-transparent surface 12 and a participating background surface 30, and the reflected light 26 comprising light interreflections or subsurface scatterings and reflections 44, 46, 48 is received by camera 15 (step 106). In step 108, the direct illumination received by each camera pixel 24 is determined by measuring an intensity signal at the camera pixel 24.

[0059] As part of the decoding strategy, DFT is used to establish the one (camera pixel 24) to many (projector columns 20) correspondence.

[0060] The intensity signal measured by the camera 15 of a projector-camera system 10 after projecting multi -frequency sinusoidal patterns may be modeled as follows. A sinusoidal pattern of frequency f can be generated using the following equation:

[0061] P c, f) = PA+ PBcos(27r c) (1)

[0062] where PA and s are constants. The projector column is c 6 [1; cmax], and cmaxis the total number of projector columns. The frequency f (unit: (projector pixel)-1) takes a set of discrete values Jly = {f,f2■such that fmE [0; fnax], fmfmax is the maximum frequency allowed by the projection optics of the projector-camera system 10

[0046] , When sinusoidal patterns (1) are projected on the scene, the camera 15 captures intensity signal I(v, f) at camera pixel v 24:

[0064] where A is the amplitude offset (including camera noise) and B is the amplitude contrast. These amplitudes are defined for each camera pixel v 24 and vary with pattern frequency f-. A = A v, f) andK refers to the number of projector columns 20 illuminating, directly or indirectly, scene point Q 22 imaged at camera pixel v 24 (Figure Id). In (2), the illumination component k originates from the projector column Ck.are subsets of Jly such that Ji5U= 0. These subsets are defined for each camera pixel 24: the measured intensity) at camera pixel v 24 reaches the saturation value Bat for each frequency in th .

[0065] The relationship between amplitudes A and B and the pattern frequency f can be empirically modelled

[0047] , For that the scene is an opaque surface with no global / indirect illumination (K = 1= Jly in (2)). For each frequency in Jly, a Phase Shift algorithm is applied, and amplitudes A and B are computed. Figures 2a-c show the evolution of amplitudes A and B with respect to the pattern frequency f for three projector-camera systems with different specifications. These plots suggest that the relationship between the amplitude offset A and the pattern frequency can be described by a constant for f < 0.2, and by a polynomial Pp(f) of degree p (typically p = 3) for f >= 0. These plots also indicate that the amplitude contrast B behaves as a Gaussian function with respect to the pattern frequency f.

[0066] By imposing fmax= 0.2, intensity model (2) becomes:

[0068] Parameters a, bk, Ui< > 0, and Ok > 0 are real constants all linked to camera pixel v 24: a = a(v), bk= bkv / zk= / zk(v) and crk(v) = crk(v).

[0069] Next, a discrete Fourier transform (DFT) is performed on the intensity signal measured at the camera pixel 24 to inventory the projector columns 20 illuminating directly and indirectly the scene point 22 imaged at that pixel 24. The discrete Fourier transform is used to detect peaks and the position of relevant peaks present in the DFT magnitude is obtained (step 110).

[0070] Among the K projector columns 20 participating in the illumination of scene point Q 22 imaged at camera pixel v 24, at most one, referred to as c* 21, is responsible for the direct illumination received by Q 22. Other projector columns 20 are responsible for the indirect illumination of scene point Q 22. To compute the 3D coordinates of point Q 22 bytriangulation, only c* 21 needs to be determined. A discrete Fourier transform can be used to determine projector columns Ck 20, k = 1, 2, . . . , K as the peak positions.

[0071] After projecting M sinusoidal patterns, the measured intensity at each camera pixel v 24 is given by model (3). It is the sum of the individual contributions of the K projector columns 20 participating directly and indirectly to the illumination of the scene point imaged at camera pixel v 24. To determine the K projector columns c*20, k = 1, 2, ... , K, model (3), a sum of exponentially damped sines, suggests to compute the DFT and determine ck as the position of relevant peaks present in the DFT magnitude obtained. These positions are determined with subpixel precision using well-established peak detector methods

[0048] , Thus,K is equal to the number of peaks present in the DFT magnitude, and c / < is the position of the k'hhighest peak along the x-axis which is actually the projector column axis (c-axis) (see Figure 5d).

[0072] At the end of this process a one camera pixel v 24 to K projector columns 20 k = KO -> {ck} ) correspondence is established. As result, up to K camera-projector k = l correspondence maps can be built (step 112). Typically, the first correspondence map is formed by projector columns 20 related to the highest peak at each camera pixel 24; the second correspondence map is constructed using projector columns 20 linked to the second highest peak at each camera pixel 24, and so on. Instead of sorting peaks based on their amplitude, one can also consider sorting them in an increasing order along the c-axis. In step 114, the K correspondence maps obtained are used to compute K 3D point clouds by using the geometrical calibration of the projector-camera pair (triangulation). These data (correspondence maps and corresponding 3D point clouds) represent the raw results. It is most likely that the k!ilcorrespondence map (3D point cloud) will contains information related to the target semitransparent surface!2, but also information related to participating background surfaces 30.

[0073] The DFT-based method produces a one (camera pixel 24) to many (projector columns) correspondence between a camera pixel 24 and projector columns 20 illuminating the scene point imaged by that camera pixel 24. np camera-projector correspondence maps are obtained, and thus np 3D point clouds Pi, i = 1, 2, ... , np are obtained under the epipolar constraint. Each of these 3D point clouds is a mixture between 3D points belonging to the semitransparent surface 12 e.g. Q and 3D points, either real (e.g., belonging to background surfaces e.g. Qb) or fictitious e.g. Qs, not belonging to the semi-transparent surface 12. Accordingly, the 3D point cloud of the target semi-transparent surface 12 from these np 3D point clouds will need to be extracted. As such, in step 116, a surface continuity constraint is applied on theseraw results to extract the correspondence map and 3D point cloud representing the target semitransparent surface 12 only, as will be described in more detailed below.

[0074] In step 118, a final camera-projector correspondence map is outputted and a final 3D point cloud is outputted (step 120).

[0075] In one example, step 114 may be bypassed.

[0076] The accuracy in locating peaks in the DFT magnitude depends on the step size Ac between points along the c-axis. This step size is related to the number of samples (patterns) M and the sampling step A / by:[°077l

[0078] It may be desirable to maintain Ac as small as possible. In general, this can be achieved by increasing either the number of patterns M (preferably, as few patterns as possible are projected) or maintain the sampling step Af below the Nyquist rate). Both possibilities require increasing the sampling interval. However, due to the projection optics, the frequency sampling range [0 fmax] of a proj ector-camera system 10 is bounded, as such Ac has a minimum bound: Ac > Hfmax. The immediate consequence of this is that the DFT-based approach is unable to inventory (number of peaks in the DFT magnitude) the projector columns 20 illuminating a given scene point when these projector columns 20 are close to each other. Formally, de is defined as the spacing between two projector columns ci and C220 illuminating a given scene point, i.e., de = \ci - C2|. The DFT magnitude will show only one peak (instead of two) if Ac > bc / 2

[0018] ,

[0079] A discrete Fourier transform has some inherent requirements that can negatively impact its performance. First, due to sampling theory, DFT requires a large number of samples / pattems to ensure minimum accuracy; this results in a low acquisition speed. Second, accuracy of DFT typically increases when Ac, the spacing between points along the c-axis, decreases. However, this spacing is inversely proportional to the sampling step A / and the number of samples M. Increasing either parameters (with the aim of decreasing Ac) is not possible in projector-camera systems because they have a limited frequency sampling range [0 fmax] (typically fmca< 0.2). Therefore, Ac has a minimum bound equal to )lfmca(typically Ac > 5 pixels'). Moreover, increasing the sampling step will reduce the range of projector columns 20 identifiable using DFT; it is not an option. Thus, if two very close projector columns 20 illuminate a given scene point 22, it may be challenging to identify them individually using DFT, and an incorrect correspondence may be obtained, resulting in an inaccurate 3D point.

[0080] To improve on the accuracy of the DFT-based method, in another example, a curvefitting based method is employed to measure a 3D shape of a target semi-transparent surface 12 of a scene 13, using a significantly reduced number of patterns. Looking at Figure 3b, there is shown a flow chart 200 with example steps for measuring a 3D shape of a target semitransparent surface 12 of a scene using a curve-fitting based method. The method comprises an acquisition stage, a processing stage and an output stage. In the acquisition stage, light signals having a predetermined pattern, such as sinusoidal patterns at chosen frequencies, are generated (step 202). The coding strategy for the curve fitting-based method comprises using sinusoidal patterns (1) with frequencies (fm6 9? / . m = 1, 2, ... , AL)) randomly selected in the range of frequencies allowed by the projector-camera system. In one example, the selected frequencies satisfy the Shannon-Nyquist sampling condition. Formally, the sampling range [0; fmax\ is discretized at the Nyquist rate to yield AT frequencies, as described above. Then, AL frequencies are randomly selected among the AL' available frequencies. The curve fitting-based method works with much fewer number of patterns (AL « AL) than the DFT-based method.

[0081] In step 204, a projector 14 projects the sinusoidal patterns from one or more projector pixels / columns 20, 34, 42, towards a point Q 22 of the scene. At point Q 22, the incident light is reflected by semi-transparent surface 12 and a participating background surface, and the reflected light 26 comprising light interreflections or subsurface scatterings and reflections 44, 46, 48 is received by a camera 15 (step 206). The direct illumination received by each camera pixel 24 is determined by measuring an intensity signal at the camera pixel 24.

[0082] As part of the decoding strategy, curve fitting is used to establish the one (camera pixel 24) to many (projector columns) correspondence. After projecting AL sinusoidal patterns, intensity signal measured at each camera pixel v 24 is analytically described by model (3). This model is a sum of K exponentially damped sinusoids, each sinusoid being the contribution of a projector column 20 illuminating directly or indirectly the scene point 22 imaged at camera pixel v 24. The model has 4K +1 parameters: a, bi, it, a,, and , z = 1, ... , K. These parameters are determined, in particular the projector columns Ck 20, k = 1, 2, . . . , K, by fitting model (3) on measured data. Since model (3) is a nonlinear function of its parameters, the curve fitting problem may be solved iteratively preferably when the following four conditions are met: (i) parameter estimation problem is formulated, including the cost function to be optimized; (b) the model is preferably sensitive enough to each parameter such that all the parameters are identifiable; (c) for the stability of the solver, the problem is initialized so that the convergenceto the global optimum is substantially guaranteed; and (d) the nonlinear optimization process is preferably fast and reliable.

[0083] In step 208, a parameter estimation problem is formulated and initialized using a component subtraction approach to determine a number of informative components in the measured intensity signal at the image capture device pixel 24, to inventory the one or more projector columns 20 illuminating the scene point 22 imaged at the image capture device pixel 24.

[0084] In more detail, 1) to formulate the parameter estimation problem, vector 0 defined as vector 0 = [a, bt, ..., .K, OI, ... , OK, CI, ... , CK] of size N = 4K +1 as the parameter vector associated to model (3), which takes the following general form:

[0086] Intensity I(v, f) measured at camera pixel v 24 is written as

[0088] Due to the presence of noise N(v) at camera pixel v 24 [3], / (v, ) =7(v, / )< +N(v). H is desirable to estimate parameter vector 0 that minimizes the error

[0090] i.e., 0 should minimizes cost function :

[0092] Starting with an initial guess, parameter vector 0 is iteratively updated until cost function (11) is minimized up to a desired precision. The speed of convergence to the global minimum strongly depends on the choice of the initial guess for the parameter vector 0. Accuracy achieved can be related to whether or not parameter vector 0 is identifiable, and model (8) is sensitive to its parameters.

[0093] 2) Parameter sensitivity and identifiability. The sensitivity S, of model (8) to the ithelement & of parameter vector 0 is given by

[0049] ,

[0050] :

[0095] which captures the influence of parameter & on model (8). This influence can be quantified using the importance index

[0051] : the greater the importance index of parameter &, the higher the sensitivity of model (8) to parameter &. Parameter identifiability analysis allows ensuring that no more than one parameter vector can minimize the cost function (11); and aims to determine if there is a unique combination of the parameters of model (3) that minimizes cost function (11).

[0096] Numerical simulations were carried out for the parameter sensitivity and identifiability analysis. Results are summarized as follows: (i) model (3) is sensitive to each of its parameters; (ii) model (3) is more sensitive to parameter c than to other parameters; (iii) parameter a, b and c are fully identifiable, (iv) parameter p and a are not identifiable (their sensitivities are fairly linearly dependent).

[0097] 3) Parameter Initialization using component subtraction'. Numerical simulation also revealed that: (v) the initial value of parameter c must deviates by less than 2 from its true value; (vi) parameters a, b and p can all be initialized to 0; (vii) the initial value for parameter a should be different from 0 and less than 0.2. The convergence of the minimization process to the global minimum is guaranteed if (v), (vi) and (vii) are adopted. Motivated by (ii), a component subtraction method, a brute search approach, is introduced to initialize parameter c. The idea is to approximate model (3) by a sum of K pure sinusoidal functions rk sm(2nckf), k= 1, 2, ... , K. The first component {H; c?i} is determined as the values of {r c} that minimize the deviation between the original signal and the sinusoidal function r sin(27ic / ). This is achieved through an exhaustive search on r and c. Once {H; c?i} is found, the original signal is updated by subtracting the component risin(2ncif). The new obtained signal is then used to determine the second component {rp c } and so on. This process is repeated until the standard deviation of the updated signal is smaller than a desired threshold e. Algorithm 1 summarizes the proposed component subtraction method which determines the number of informative components in the measured signal at camera pixel v 24. It allows to inventory projector columns 20 illuminating the scene point imaged at camera pixel v 24.

[0098] Algorithm 1 : Component subtraction methodInitialize the deviation Et: Et■ = || Iy,f)\\iWhile Et> edoE = EtFor c = 1 to cmaxFor a = 0 to 1E = \\ / (v,y) - a cos(27tc / )||2If E < E then Deviation decreasesE = E, at = a,' Ci = cEndlfEndForEndForC = C U CtRemove component:Compute the deviation Et: Et■ = || I( ,f)\\2EndWhileOutput: C

[0099] 4) Levenberg Marquardt algorithm'. In this paper, we adopt the LevenbergMarquardt (LM) algorithm to estimate parameter vector 0 that minimizes cost function (11)

[0052] ,

[0053] , The LM behaves like a gradient-descent method when the updated parameter vectorotherwise, it acts like the Gauss-Newton method

[0054] , In this work we implemented the adapted Levenberg Marquardt algorithm described in Algorithm 2.

[0100] Algorithm 2 : Levenberg Marquardt algorithm implemented0 = 0;Compute error using (10) with 0 = 0Compute cost function using (11) with 0 = 0Initialize Z = 10'3While 2, > eLMdoCompute Sensitivities 5 using (12) with 0 = 0Assemble the Jacobian J using (13)Assemble matrices J* J andSolve ( / fJ + AA)<50=for 6QUpdate Qupdt= 0 + 8QCompute errorpdtusing (10) with 0 = GupdtCompute cost function jupdtusing (11) with 0 = QupdtIf jupdt< <; thenA = 103; 0 =ElseX = 10XEndlfEndWhileOutput: 0

[0101] Computations are performed on a per pixel basis with intensity signal I(v, f), selected frequencies J? / , initial parameter vector 0;„;, and error threshold ELM as main inputs. We remind that M is the number of patterns and N is the number of parameters of model (3). In each iteration, sensitivities St, i = 1, 2, ... , JVare computed using (12) and the M x N Jacobian matrix Jis assembled. Element Jm,i of the Jacobian is given by:

[0102] 1,2, ... , A (13)

[0103] The N x N matrix J J and N x 1 vectorare also assembled. Defining as a tunable scalable parameter initialized to a small value (e.g., I O3). and identity matrix A, the step size 8@ utilized to update parameter vector 0 is determined by solving the following system of equations:

[0105] Parameter vector is then updated as Qupdt= &+8e, and cost function ^updt is computed using (11). if ^updt has decreased compared to the previous iteration (convergence speed is similar to Gauss-Newton method), parameter vector 0 is set to &updt. If uPdt has increased (convergence speed decreases as a gradient-descent method), Z is increased as well and no update is done. Iterations stop as soon as cost function is less than a desired error threshold ELM.

[0106] Next, a curve fitting model is used to establish the one (camera pixel 24) to many (projector columns) correspondence (step 210). This model is a sum of K exponentially damped sinusoids, each sinusoid being the contribution of a projector column illuminating directly or indirectly the scene point imaged at the camera pixel 24. The model is a nonlinear function of its parameters, the curve fitting problem is solved iteratively to determine these parameters.

[0107] In step 212, an algorithm is applied to generate k correspondence maps between the one or more projector columns 20 illuminating the scene point and the camera pixel 24, and the k correspondence maps are used to generate k 3D point clouds (step 214). The correspondence maps and corresponding 3D point clouds are raw results containing information related to the target semi-transparent surface, but also information related to participating background surfaces.

[0108] In step 216, a surface continuity constraint is applied on these raw results to extract the correspondence map and 3D point cloud representing the target semi-transparent surface 12 only, as will be described in more detail below.

[0109] In step 218, a final camera-projector correspondence map is outputted and a final 3D point cloud is outputted (step 220).

[0110] Compared to the DFT-based method, the curve fitting-based method has some advantages, as it uses fewer samples and is reliable for inventorying the number of components in a multi-sine signal such as model (3). However, fitting data to a nonlinear model is an iterative process that can be time consuming. Convergence towards the solution as well as the speed of convergence strongly depend on the initialization of the problem. Figure 4 comparesDFT and curve fiting in estimating the values cj and C2 of projector columns 20 illuminating a given scene point for different values of the spacing de = \ci - c?2|. In one example, the DFT- based method using 768 samples can not separate ci from C2 for de < Ac = pixels, whereas the curve fiting-based method working with only 32 samples provides an accurate estimate of ci and C2.

[0111] In more detail, the surface continuity constraint implemented above exploits the fact that any real surface is continuous. Therefore, their digitized version is constituted by a set of 3D points having a strong constraint of continuity with their neighbors. Locally, neighbor 3D points belonging to the semi-transparent surface 12 should be very close to each other, and globally, they will represent the actual semi-transparent surface. Furthermore, as schematized in Figure 1, the semi-transparent surface 12 is the surface closest to the projector-camera system; a light ray traced from any camera pixel 24 will hit the semi-transparent surface 12 before any other surface in the scene. Unlike filtering methods, the surface continuity constraint does not recover a smoother surface from a combination of measurements. Rather, since there is more than one 3D point per camera pixel 24, the surface continuity constraint allows sorting 3D points belonging to the np 3D point clouds. Only one 3D point is kept for each camera pixel 24. This 3D point is the one with a strong constraint of continuity with its neighborhood (it belongs to a surface), and the point closest to the camera (the semi-transparent surface 12 is the surface closest to the camera 15).

[0112] The core of the surface continuity constraint is the surface growing process described in Algorithm 3.

[0113] Algorithm 3 : Surface growing algorithmInput: pixel seed Seed, 3D point cloud seed Pseed, candidate point cloud P, and threshold zscns= -1While nBuild Nseed that contains the neighbouring pixels of SeedNvc : neighbor pixels of vcthat belong to SeedIf d< Zsc then Good continuitySeed Seed U Vc311(1 P seed Pseed U P(rfcEndlf EndFor EndWhileOutput: Seed and Pseed

[0114] It is built on the region growing image segmentation algorithm

[0055] , and describes the semi-transparent surface 12 by two sets of data: the group of pixels Seed linked to the surface, and the group of 3D points Pseed belonging to the surface. The group Nseed of the neighboring pixels of Seed is determined. A pixel is a neighbor of Seed if it does not belong to Seed and have at least one of its neighbors (8-neighborhood connectivity) belonging to Seed. For each candidate neighboring pixel vcG Nseed the group Nvcof its neighbors belonging to Seed is built. Next, the mean Euclidean distance between the 3D point P(yc) at pixel vcand 3D points at pixels in Nvcis calculated. Candidate pixel vcis added to the group Seed of pixels linked to the semi-transparent surface 12 pixel (3D point P(yc) is added to Pseed) if the mean Euclidean distance is less than a chosen surface continuity threshold eSc. This process iterates until the size of Seed no longer increases.

[0115] Algorithm 4 implements the surface continuity constraint.

[0116] Algorithm 4 : Surface Continuity constraintInput: Pixel vseed, point clouds {P;}-Z”p, threshold £scArrange (per pixel) {P;}-Z”pin ascending order of distanceInitialize (Pseed)For i = 1Apply Algorithm 3 with inputs Surf> Psurf> andPi Surf See(nd Psurf PseedEndForOutput: Surf and Psurf

[0117] First, the np 3D point clouds are rearranged in an increasing order of distance from the camera. At each pixel, 3D point cloud Pi contains the 3D point closest to the camera. A seed pixel vseed randomly chosen and the 3D point cloud Pi are used to initialize the group of pixels SM, / and the group of 3D points Psurf) inked to the semi-transparent surface 12. By looping over the np rearranged 3D point clouds, the semi-transparent surface 12 is progressively extracted by providing the necessary inputs to the surface growing Algorithm 3.

[0118] Looking at Figure 5, there is shown an illustrative example for the proposed methods. The scene is constituted by a planar semi-transparent surface 12 placed in front of anopaque white cup acting as a participating background. Figure 5a shows the scene from the camera 15 viewpoint when all projector columns 20 are turned on, and Figure 5b shows the scene when a sinusoidal pattern is projected. Point Q\ belongs to a region of the semitransparent surface 12 where background surfaces are at infinity, whereas point Qi belongs to a region with a participating background surface (the opaque white cup).

[0119] As described above, 768 frequencies were selected to generate the sinusoidal patterns. After projecting these patterns on the scene, the measured intensity signal at point Q\ and ?2 are presented in Figure 5c. The DFT magnitude for these signals is given in Figure 5d. Point Q\ is illuminated by only one projector column (peak at ci = 602.51) , and point QT. receives light from two projector columns 20 (peaks at ci = 431.79 and C2 = 356.47). In this example there is a one (camera pixel 24) to two (projector columns) correspondence. Figure 5e presents the first camera-projector correspondence map obtained by aggregating projector columns 20 related to the highest peaks at each camera pixel 24. Similarly, Figure 5f is the second correspondence map obtained. The corresponding 3D point clouds are shown in Figure 5g and Figure 5h respectively.

[0120] For the curve fitting-based method, 32 frequencies were selected as described above. The intensity signals measured at points Q\ and QT. are plotted in Figure 5i. Figure 5j show the results obtained by fitting model (3) to these signals using Levenberg Marquardt algorithm. Point Q\ is illuminated by one projector column ci = 601.77, while projector columns ci = 432.53 and C2 = 356.47 illuminate point ?2. The two resulting camera-projector correspondence maps and corresponding 3D point clouds are shown in Figure 5k, Figure 51, Figure 5m, and Figure 5n, respectively. The information relating to the target semi-transparent surface 12 is distributed among the two correspondence maps obtained (see the region containing the cup in Figure 5e-f and in Figure 5k-l). This leads to discontinuities in the 3D point clouds (see Figure 5g and Figure 5m). Figure 5p presents the results obtained by applying the surface continuity constraint to extract the target semi-transparent surface.

[0121] Details of the practical implementation of the DFT-based and curve-fitting based methods, and the experimental results obtained with various types of semi-transparent surfaces, are discussed below.

[0122] A. Practical implementation

[0123] To evaluate the effectiveness of the DFT-based and curve-fitting based methods, an example projector-camera system 10 comprising aHR-12000-M camera (10 bits, 4096 x 2200 pixels) from Emergent Vision Technologies, British Columbia, Canada, and aML750ST consumer-grade projector (resolution 768 x 1024) from Optoma, Fremont, CA, U.S.A., was implemented. The system 10 baseline and standoff distance were 26cm and 36cm, respectively. The height, width, and depth of field of its measuring volume were 33cm, 22cm, and 16cm, respectively. The projection optics included a cylindrical lens and allowed projecting sinusoidal patterns with a maximum frequency of fmax = 0.2. Acquisition speed of the system 10 was 120Hz.

[0124] In this example, all the above-noted algorithms operate on a per pixel basis and are parallelizable. These algorithms were implemented on graphics processing units (GPU) using OpenCL™ from The Khronos Group, Inc. A clFFT library was used for parallel DFT computation, and original GPU implementations were made for the component subtraction algorithm, the Levenberg Marquardt algorithm, and the surface continuity constraint. A consumer-grade computer running Windows™ 10 (64— bit) from Microsoft Corporation, Redmond, Washington, U.S.A., and equipped with a 4.00GHz 720 Intel® Core™ i7-6700K CPU, a 48GB core memory and a GTX-1070Ti GPU from Nvidia Corporation, Santa Clara, California, U.S.A. Overall, for HD images (1920 x 1080 pixels), processing time was around 35s for the DFT-based method. The curve fitting-based method required more than three times as much time as the DFT-based method.

[0125] To compare the results, the gold standard Phase Shift (PS) method [3], and the line scanning (LS) method inspired from the laser scanning method

[0044] were implemented. The PS method summarizes well the results that can be expected with phase profilometry methods when measuring semi-transparent surfaces 12 [3],

[0056] , The LS method operates by turning on projector columns 20 one after the other and applies a threshold to establish cameraprojector correspondences. 32 patterns (period of 15 pixels) were used for the PS method, and 1024 patterns were used for the LS method. The DFT-based method was configured to use 768 patterns, and the number of patterns for the curve fitting-based method was adjusted from scene to scene between 32 and 96 patterns. The settings of the systems, including camera gain, camera lens aperture, projector power, were adjusted from scene to scene to minimize pixel saturation and avoid measuring tow level signals. However, for a given scene, the same settings were used for all implemented (PS, LS, DFT-based, Curve fitting-based) methods.

[0126] Semi-transparent surfaces 12 with various shapes, thickness and materials were considered. The tested scenes included semi-transparent surfaces 12 with participating background surfaces, and semi-transparent objects 11 having a rear part acting as a participating background or having internal reflections / scatterings 44, 46, 48. Figure 6summarizes results obtained with state-of-the-art PS and LS methods, and proposed DFT-based and curve fitting-based methods. For each methods, we provide one pose of the 3D point cloud, the 2D Z-map, and a ID profile taken on the Z-map.

[0127] The first three scenes involve a semi-transparent surface 12 placed in front of a participating background surface. The Planar sheet and the ball in its background have similar reflectances. The Mold cover is in front of an opaque ice mold. The Plastic film covers an opaque mannequin head. PS method fails in reconstructing these semi-transparent surfaces. LS performs pretty well on these scenes, but suffers from the threshold used when background reflections dominate the signal measured by a camera pixel 24 (see missing points at the bottom-right in 3D point cloud and Z-map of Planar sheet). The LS, DFT-based, curve fittingbased methods achieved good results. With just 32 patterns, the curve fitting-based method produced results with no artifacts caused by pixel saturation (see bottom-left in 3D point cloud and Z-map of Planar sheet). The Cone, Bowl in bowl, and Ball are semi-transparent objects 15 whose rear part acts as a participating background. PS also fails in these cases. LS provides satisfying results only for Bowl in bowl, but fails when the camera pixel 24 measures a low- level signal or when the measured signal is dominated by contributions from the background (see Cone and Ball). The DFT-based method performs well, except when there is pixel saturation (see Bowl in bowl). The curve fitting-based method also provides good results. In the Box and Silicone rubber bags scenes, background surfaces include the rear part of the object 11 and another object 11 (the spheroid bowl inside Box and fruits inside Silicone rubber bags). PS (32 patterns) yields the worst reconstruction. LS (1024 patterns) is affected by the choice of the threshold (see the right region of Box). DFT-based (768 patterns) and curve fitting-based (64 patterns) methods show reliable results. Subsurface reflections / scatterings 44, 46, 48 may exist in the liquid-filled Tube and in the Printed Duck. Except for PS, all tested methods are successful in reconstructing these semi-transparent surfaces, but with different number of patterns.

[0128] A quadric surface was fitted on the Planar sheet, Paper inside Cup, Silicone rubber bags and Cone to quantitatively evaluate the PS, LS, DFT-based, curve fitting-based methods. The root-mean-square deviation obtained was less than 0.26mm for both the DFT and curve fitting methods (See Figure 7). This confirms the qualitative observations in Figure 6, and the potential offered by the proposed methods for metrological applications. Nonetheless, the overall quantitative performance expected for PS, LS, DFT-based, curvefitting-based methods depend on the reflectance of the semi-transparent surface 12 as well as the sensitivity and inherent noise level of the camera 15.

[0129] While providing similar performance, both proposed methods have different specifications in terms of acquisition time and processing time. The DFT-based method has a long acquisition time due to the high number of frame required, and a linearithmic (M logM) time complexity. Data acquisition for the curve fitting-based method is more than ten times faster than acquisition time for the DFT-based method. However, its computing time (> 2min for HD images on GTX-1070Ti) is more than three time that of the DFT-based method (~ 35s for HD images on GTX-1070Ti). Note that these computing times are far less than computing time reported in literature on 3D shape measurement of (semi-) transparent surfaces [9], Generally, around 75% of the computing time for the curve fitting-based method is spent to initialize (component subtraction method) the parameter estimation problem, which is a key and mandatory step to ensure convergence to the global minimum of the cost function. The exhaustive search approach adopted for the parameter initialization is computationally stable, and allows on the fly inventorying of all components in the received illumination. In another example, on regression neural networks are employed to initialize the problem, which promises fast inferring time, however, this approach may require setting the maximum number of components in advance to reduce the size of the training data to be generated.

[0130] While the methods described herein extracted the target semi-transparent surface 12 as the first interface in front of the projector-camera system, the extraction process (surface continuity constraint) may be iterated, and background surfaces may be measured whenever possible.

[0131] The underlying hypothesis of the DFT-based, curve fitting-based methods is that the light reflected by the semi-transparent surface 12 is strong enough to be captured with a signal to noise ratio above the noise level of the camera. The reliability, robustness and accuracy provided by these methods can successfully be exploited for applications not requiring real time operations.

[0132] Figure 8 shows results using state-of-the-art methods versus the novel methods described herein for measuring 3D shapes of semi-transparent surfaces.

[0133] Looking at Figure 9, there is shown an overview of a computing environment 310 of components configured to facilitate the systems and methods described herein. It should be appreciated that the computing environment 310 is merely an example and that alternative or additional components are envisioned. Computing environment 310 comprises computingmeans with computing system 312, such as a server, comprising at least one processor such as processor 314 or graphics processing units (GPU) 315, at least one memory device such as memory 316, input / output (I / O) module 318 and communications interface 320, which are in communication with each other via centralized circuit system 322. Although computing system 312 is depicted to include only one processor 314, computing system 312 may include a number of processors therein. In an embodiment, memory 316 is capable of storing machine executable instructions, data models and process models. Database 323 is coupled to computing system 312 and stores pre-processed data, model output data and audit data. Further, the processor 314 is capable of executing the instructions in memory 316 to implement aspects of processes described herein. For example, processor 314 may be embodied as an executor of software instructions, wherein the software instructions may specifically configure processor 314 to perform algorithms and / or operations described herein when the software instructions are executed. Alternatively, processor 314 may be execute hard-coded functionality. Computing environment 310 may be software (e.g., code segments compiled into machine code), hardware, embedded firmware, or a combination of software and hardware, according to various embodiments.

[0134] In one implementation, processor 314 may be embodied as a multi-core processor, a single core processor, or a combination of one or more multi-core processors and one or more single core processors. For example, processor 314 may be embodied as one or more of various processing devices, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing circuitry with or without an accompanying DSP, or various other processing devices including integrated circuits such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, Application-Specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), Programmable Logic Controllers (PLC), Graphics Processing Units (GPUs), and the like. For example, some or all of the device functionality or method sequences may be performed by one or more hardware logic components.

[0135] Memory 316 may be embodied as one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, memory 316 may be embodied as magnetic storage devices (such as hard disk drives, floppy disks, magnetic tapes, etc.), optical magnetic storage devices (e.g., magneto-optical disks), CD-ROM (compact disc read onlymemory), CD-R (compact disc recordable), CD-R / W (compact disc rewritable), DVD (Digital Versatile Disc), BD (BLU-RAY™ Disc), and semiconductor memories (such as mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.).

[0136] I / O module 318 facilitates provisioning of an output to a user of computing system 312 and / or for receiving an input from the user of computing system 312, and send / receive communications to / from the various sensors, components, and actuators of computing environment 310. I / O module 318 may be in communication with processor 314 and memory 316. Examples of the I / O module 318 include, but are not limited to, an input interface and / or an output interface. Some examples of the input interface may include, but are not limited to, a keyboard, a mouse, a joystick, a keypad, a touch screen, soft keys, a microphone, and the like. Some examples of the output interface may include, but are not limited to, a microphone, a speaker, a ringer, a light emitting diode display, a thin-film transistor (TFT) display, a liquid crystal display, an active-matrix organic light-emitting diode (AMOLED) display, and the like. In an example embodiment, processor 314 may include I / O circuitry for controlling at least some functions of one or more elements of I / O module 318, such as, for example, a speaker, a microphone, a display, and / or the like. Processor 314 and / or the I / O circuitry may control one or more functions of the one or more elements of I / O module 318 through computer program instructions, for example, software and / or firmware, stored on a memory, for example, the memory 316, and / or the like, accessible to the processor 314.

[0137] In an embodiment, various components of computing system 312, such as processor 314, memory 316, I / O module 318 and communications interface 320 may communicate with each other via or through a centralized circuit system 322. Centralized circuit system 322 provides or enables communication between the components (314-320) of computing system 312. In certain embodiments, centralized circuit system 322 may be a central printed circuit board (PCB) such as a motherboard, a main board, a system board, or a logic board. Centralized circuit system 322 may also, or alternatively, include other printed circuit assemblies (PCAs) or communication channel media.

[0138] Communications interface 320 enables computing system 312 to communicate with other entities over various types of wired, wireless or combinations of wired and wireless networks, such as for example, the Internet. In at least one example embodiment, communications interface 320 includes a transceiver circuitry for enabling transmission and reception of data signals over the various types of communication networks. In someembodiments, communications interface 320 may include appropriate data compression and encoding mechanisms for securely transmitting and receiving data over the communication networks. Communications interface 320 facilitates communication between computing system 312 and I / O peripherals.

[0139] Centralized circuit system 322 may be various devices for providing or enabling communication between the components (312-320) of computing system 312. In certain embodiments, centralized circuit system 322 may be a central printed circuit board (PCB) such as a motherboard, a main board, a system board, or a logic board. Centralized circuit system 322 may also, or alternatively, include other printed circuit assemblies (PCAs), communication channel media or bus.

[0140] A plurality of user computing devices 324 and data sources 326 are coupled to computing system 312 with communication network 328.

[0141] It is noted that various example embodiments as described herein may be implemented in a wide variety of devices, network configurations and applications.

[0142] Those of skill in the art will appreciate that other embodiments of the disclosure may be practiced in network computing environments with many types of computer system configurations, including personal computers (PCs), industrial PCs, desktop PCs), hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, server computers, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0143] In another implementation, computing environment 310 follows a cloud computing model, by providing an on-demand network access to a shared pool of configurable computing resources (e.g., servers, storage, applications, and / or services) that can be rapidly provisioned and released with minimal or nor resource management effort, including interaction with a service provider, by a user (operator of a thin client).

[0144] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular implementations of particular inventions. Certain features that are described in thisspecification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0145] Accordingly, the above description of example implementations does not define or constrain this disclosure. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this disclosure.

[0146] REFERENCES[1] M.-A. Drouin and J. -A. Beraldin, Active Triangulation 3D Imaging Systems for Industrial Inspection. Cham: Springer International Publishing, 2020, pp. 109-165.[2] M.-A. Drouin and A. Tahan, 3D Imaging Systems for Optical Metrology. Singapore: Springer Nature Singapore, 2023, pp. 1293-1318.[3] C. Zuo, S. Feng, L. Huang, T. Tao, W. Yin, and Q. Chen, “Phase shifting algorithms for fringe projection profilometry: A review,” Optics and Lasers in Engineering, vol. 109, pp. 23-59, 2018.[4] J. Xu and S. Zhang, “Status, challenges, and future perspectives of fringe projection profilometry,” Optics and Lasers in Engineering, vol. 135, p. 106193, 2020.[5] S. Feng, C. Zuo, L. Zhang, T. Tao, Y. Hu, W. Yin, J. Qian, and Q. Chen, “Calibration of fringe projection profilometry: A comparative review,” Optics and Lasers in Engineering, vol. 143, p. 106622, 2021.[6] S. Zhang, “High-speed 3D shape measurement with structured light methods: a review,” Optics and Lasers in Engineering, vol. 106, pp. 119-131, 2018.[7] I. Ihrke, K. N. Kutulakos, H. P. A. Lensch, M. Magnor, and W. Heidrich, “Transparent and Specular Object Reconstruction,” Computer Graphics Forum, 2010.[8] A. Karami, R. Battisti, F. Menna, and F. Remondino, “3d digitization of transparent and glass surfaces: State of the art and analysis of some methods,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLIII-B2- 2022, pp. 695-702, 2022.[9] J. Jiang, G. Cao, J. Deng, T.-T. Do, and S. Luo, “Robotic perception of transparent objects: A review,” IEEE Transactions on Artificial Intelligence, pp. 1-21, 2023.

[0010] D. Palousek, M. Omasta, D. Koutny, J. Bednar, T. Koutecky, and F. Dokoupil, “Effect of matte coating on 3d optical measurement accuracy,” Optical Materials, vol. 40, pp. 1-9, 2015.

[0011] M. Landmann, H. Speck, P. Dietrich, S. Heist, P. Kuhmstedt, A. Tunnermann, and G. Notni, “High-resolution sequential thermal fringe projection technique for fast and accurate 3d shape measurement of transparent objects,” Appl. Opt., vol. 60, no. 8, pp. 2362-2371, Mar 2021.

[0012] C. Zuo, L. Huang, M. Zhang, Q. Chen, and A. Asundi, “Temporal phase unwrapping algorithms for fringe projection profilometiy: A comparative review,” Optics and Lasers in Engineering, vol. 85, pp. 84-103, 2016.

[0013] M. Gupta and S. K. Nayar, “Micro phase shifting,” in Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), ser. CVPR ’12, 2012, pp. 813-820.

[0014] T. Chen, H. P. A. Lensch, C. Fuchs, and H.-P. Seidel, “Polarization and phase-shifting for 3d scanning of translucent objects,” in 2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007, pp. 1-8.

[0015] D. Moreno, K. Son, and G. Taubin, “Embedded phase shifting: Robust phase shifting with embedded signals,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.

[0016] V. Couture, N. Martin, and S. Roy, “Unstructured light scanning robust to indirect illumination and depth discontinuities,” International Journal of Computer Vision, vol. 108, no. 3, pp. 204-221, Jul. 2014.

[0017] D. Liu, X. Chen, and Y.-H. Yang, “Frequency -based 3D reconstruction of transparent and specular objects,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2014, pp. 660-667.

[0018] F. B. Djupkep Dizeu, J. Boisvert, M.-A. Drouin, G. Godin, M. Rivard, and G. Lamouche, “Frequency shift method: A technique for 3-d shape acquisition in the presence of strong interreflections,” IEEE Transactions on Instrumentation and Measurement , vol. 71, pp. 1-19, 2022.

[0019] M. Gupta, A. Agrawal, A. Veeraraghavan, and S. G. Narasimhan, “A practical approach to 3D scanning in the presence of interreflections, subsurface scattering and defocus,” International Journal of Computer Vision, vol. 102, no. 1, pp. 33-55, Mar 2013.

[0020] Y. Zhang, D. L. Lau, and Y. Yu, “Causes and corrections for bimodal multi-path scanning with structured light,” in 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4426-4434.

[0021] Y. Zhang, D. Lau, and D. Wipf, “Sparse multi-path corrections in fringe projection profilometry,” in 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 13339-13348.

[0022] X. Chen and Y.-H. Yang, “Scene adaptive structured light using error detection and correction,” Pattern Recognition, vol. 48, no. 1, pp. 220-230, 2015.

[0023] H. Zhao, Y. Xu, H. Jiang, and X. Li, “3D shape measurement in the presence of strong interreflections by epipolar imaging and regional fringe projection,” Optics Express, vol. 26, no. 6, pp. 7117 - 7131, 2018.

[0024] M. O’Toole, J. Mather, and K. N. Kutulakos, “3d shape and indirect appearance by structured light transport,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3246-3253.

[0025] Y. Xu, H. Zhao, H. Jiang, and X. Li, “High-accuracy 3d shape measurement of translucent objects by fringe projection profilometry,” Opt. Express, vol. 27, no. 13, pp. 18421- 18434, Jun 2019.

[0026] K. Tanaka, Y. Mukaigawa, H. Kubo, Y. Matsushita, and Y. Yagi, “Recovering inner slices of translucent objects by multi -frequency illumination,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 5464-5472.

[0027] W. Zheng, D. Li, R. Wang, X. Zhang, R. Ge, and L. Yu, “Front and back surface measurement of the transparent planar element based on multi -frequency fringe deflectometiy,” Opt. Express, vol. 30, no. 20, pp. 35409-35430, Sep 2022.

[0028] T. Siwei, Y. Huimin, C. Hongli, W. Tianhe, C. Jiawei, W. Yuxiang, and L. Yong, “Elimination of parasitic reflections for objects with high transparency in phase measuring deflectometiy,” Results in Physics, vol. 15, p. 102734, 2019.

[0029] G. Wetzstein, D. Roodnick, W. Heidrich, and R. Raskar, “Refractive shape from light field distortion,” in 2011 International Conference on Computer Vision, 2011, pp. 1180-1186.

[0030] J. Kim, I. Reshetouski, and A. Ghosh, “Acquiring axially-symmetric transparent objects using single- view transmission imaging,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1484-1492.

[0031] N. J. W. Morris and K. N. Kutulakos, “Reconstructing the surface of inhomogeneous transparent scenes by scatter-trace photography,” in 2007 IEEE 11th International Conference on Computer Vision, 2007, pp. 1-8.

[0032] B. Wu, Y. Zhou, Y. Qian, M. Cong, and H. Huang, “Full 3d reconstruction of transparent objects,” A CM Trans. Graph., vol. 37, no. 4, Jul 2018.

[0033] J. Lyu, B. Wu, D. Lischinski, D. Cohen-Or, and H. Huang, “Differentiable refractiontracing for mesh reconstruction of transparent objects,” ACM Trans. Graph., vol. 39, no. 6, Nov 2020.

[0034] K. Kutulakos and E. Steger, “A theory of refractive and specular 3d shape by light-path triangulation,” vol. 76, p. 13-29, 2008.

[0035] K. Han, K.-Y. K. Wong, and M. Liu, “Dense reconstruction of transparent objects by altering incident light paths through refraction,” International Journal of Computer Vision, vol. 126, p. 460-475, 2018.

[0036] Y. Qian, M. Gong, and Y.-H. Yang, “3d reconstruction of transparent objects with position-normal consistency,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4369-4377.

[0037] X. Xu, Y. Qiao, and B. Qiu, “Reconstructing the surface of transparent objects by polarized light measurements,” Opt. Express, vol. 25, no. 21, pp. 26296-26309, Oct 2017.

[0038] S. Sajjan, M. Moore, M. Pan, G. Nagaraja, J. Lee, A. Zeng, and S. Song, “Clear grasp: 3d shape estimation of transparent objects for manipulation,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 3634-3642.

[0039] L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. v. Eenbergen, S. Debnath, and D. Fox, “Rgb-d local implicit function for depth completion of transparent objects,” in 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 4647-4656.

[0040] J. Stets, Z. Li, J. R. Frisvad, and M. Chandraker, “Single-shot analysis of refractive shape using convolutional neural networks,” in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 2019, pp. 995-1003.

[0041] Y. Tang, J. Chen, Z. Yang, Z. Lin, Q. Li, and W. Liu, “Depthgrasp: Depth completion of transparent objects using self-attentive adversarial network with spectral residual for grasping,” in 2021 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 5710-5716.

[0042] Z. Li, Y.-Y. Yeh, and M. Chandraker, “Through the looking glass: Neural 3d reconstruction of transparent shapes,” in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1259-1268.

[0043] J. Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, J. Ichnowski, A. Kanazawa, and K. Goldberg, “Evo-nerf: Evolving nerf for sequential robot grasping of transparent objects,” in Proceedings of The 6th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, 14-18 Dec 2023, pp. 353- 367.

[0044] K. He, C. Sui, T. Huang, R. Dai, C. Lyu, and Y.-H. Liu, “3d surface reconstruction of transparent objects using laser scanning with Itftf method,” Optics and Lasers in Engineering, vol. 148, p. 106774, 2022.

[0045] W. Huang, S. Jiao, H. Miao, H. Guan, T. Li, and C. Xiao, “A single-pixel imaging based stereo deflectometry for 3d reconstruction of free-form transparent objects with parasitic reflections,” IEEE Transactions on Computational Imaging, vol. 9, pp. 1201-1212, 2023.

[0046] M.-A. Drouin, F. Blais, M. Picard, J. Boisvert, and J. -A. Beraldin, “Characterizing the impact of optically induced blurring of a high-resolution phase-shift 3D scanner,” Machine Vision and Applications , vol. 28, no. 8, pp. 903-915, Nov 2017.

[0047] F. B. Djupkep Dizeu, J. Boisvert, M.-A. Drouin, G. Godin, M. Rivard, and G. Lamouche, “Frequency shift method: a robust fringe projection technique for 3D shape acquisition in the presence of strong interreflections,” in International Conference on 3D Vision, 2019, pp. 194-203.

[0048] R. B. Fisher and D. K. Naidu, A Comparison of Algorithms for Subpixel Peak Detection. Berlin, Heidelberg: Springer Berlin Heidelberg, 1996, pp. 385-404. [Online]. Available: https: / / doi.org / 10.1007 / 978-3-642-58288-2_15

[0049] B. J. V. and K. J. Arnold, Parameter Estimation in Engineering and Science. New- York: John Wiley & Sons, 1977.

[0050] J. H. Guillaume, J. D. Jakeman, S. Marsili-Libelli, M. Asher, P. Brunner, B. Croke, M. C. Hill, A. J. Jakeman, K. J. Keesman, S. Razavi, and J. D. Stigter, “Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose,” Environmental Modelling & Software, vol. 119, pp. 418-432, 2019.

[0051] R. Brun, P. Reichert, and H. R. Kunsch, “Practical identifiability analysis of large environmental simulation models,” Water Resources Research, vol. 37, no. 4, pp. 1015-1030, 2001.

[0052] K. Levengert, “A method for the solution of certain problems in least squares,” Quart. Appl. Math, vol. 2, pp. 164-168, 1944.

[0053] D. Marquardt, “An algorithm for least squares estimation on nonlinear parameters,” SIAM J. APPL. MATH., vol. 11, pp. 431-441, 1963.

[0054] H. P. Gavin, “The Levenberg-Marquardt method for nonlinear least squares curvefitting problems,” 2013.

[0055] J. Fan, G. Zeng, M. Body, and M.-S. Hacid, “Seeded region growing: an extensive and comparative study ,” Pattern Recognition Letters, vol. 26, no. 8, pp. 1139-1156, 2005.

[0056] E. Medeiros, H. Doraiswamy, M. Berger, and C. T. Silva, “Using physically based rendering to benchmark structured light scanners,” Computer Graphics Forum, vol. 33, no. 7, pp. 71-80, 2014.

Claims

CLAIMS:

1. A method of measuring a 3D shape of an object comprising a target semi-transparent surface, the method comprising the steps of: with a projector, projecting a first light signal having a predetermined pattern and a predetermined frequency from at least one projector column towards at least one location on a scene; with an image capture device, receiving the second light signal from the at least one location of the scene at an image capture device pixel; measuring an intensity of the second signal received at the image capture device pixel; with the measured second light signal, generating a correspondence map between the at least one projector column illuminating the at least one location of the scene and the image capture device pixel with coordinates receiving the light signal from the at least one location of the scene; and with the correspondence map, determining 3D coordinates of the at least one location of the scene by triangulation.

2. The method of claim 1, wherein the predetermined pattern comprises sinusoidal patterns.

3. The method of claim 2, further comprising a step of performing a discrete Fourier transform (DFT) on the measured second light signal measured at the image capture device pixel to inventory the at least projector column illuminating directly and indirectly the at least one location of the scene imaged at the image capture device pixel.

4. The method of claim 3, further comprising a step of generating a plurality of correspondence maps, wherein each of the plurality of correspondence maps yields a 3D point cloud associated with the at least one location of the scene.

5. The method of claim 4, wherein a first correspondence map of the plurality of correspondence maps is formed by the at least one projector column related to a highest peak at each camera pixel; a second correspondence map is constructed using the at least one projector column linked to the second highest peak at each image capture device pixel; and a k correspondence map is constructed using the at least one projector column linked to the k'hhighest peak at each image capture device pixel.

6. The method of claim 5, wherein the k correspondence maps obtained are used to compute k 3D point clouds using geometrical calibration of the projector-image capture device pair using triangulation.

7. The method of claim 6, wherein the k correspondence maps and the corresponding k 3D point clouds comprise raw data.

8. The method of claim 7, wherein the k correspondence maps comprise information related to at least one of the target semi-transparent surface and a background surface.

9. The method of claim 8, wherein a surface continuity constraint is applied on the raw data to extract the correspondence map and the 3D point cloud representing the target semitransparent surface only.

10. The method of claim 2, comprising further steps of formulating a parameter estimation problem and fitting a model to the measured second light signal measured at the image capture device pixel.

11. The method of claim 10, comprising a further step of initializing the parameter estimation problem using a component subtraction approach12. The method of claim 11, wherein the component subtraction approach determines a number of informative components in the measured second light signal at the image capture device pixel, to inventory the at least one projector column illuminating the scene point imaged at the image capture device pixel.

13. The method of claim 12, with a processing unit for executing instructions stored in a computer readable medium, performing the steps of generating k correspondence maps between the at least one projector column illuminating the at least one location of the scene and the image capture device pixel, wherein the k correspondence maps are used to compute k 3D point clouds.

14. The method of claim 13, wherein the k correspondence maps and corresponding k 3D point clouds comprise raw data.

15. The method of claim 14, wherein the k correspondence maps comprise information related to at least one of the target semi-transparent surface and a background surface.

16. The method of claim 15, wherein the instructions comprise a Levenberg Marquardt (LM) algorithm to estimate a parameter vector 0 for minimizing a cost function.

17. The method of claim 16, wherein a surface continuity constraint is applied to the raw data to extract the correspondence map and the 3D point cloud representing the target semitransparent surface only.

18. The method of claim 17, further comprising a step of determining a set of 3D points having a predefined constraint of continuity with 3D neighbors, wherein the neighbor 3D points belonging to the semi-transparent surface are close to each other, and globally represent the semi-transparent surface.

19. A method of isolating a desired illumination from a plurality of illumination components from a target surface of an object to determine a shape of the target surface, the method comprising: projecting a first light signal having predetermined patterns and predetermined frequencies from a projector towards at least one location on a scene on the target surface; receiving a second light signal from the at least one location of the scene at an at least one image capture device pixel, wherein the second light signal comprises a plurality of illumination components; measuring an intensity of the second light signal to quantify each of the plurality of illumination components measured at each of the at least one image capture device pixel; inventorying the plurality of illumination components to extract the desired illumination from a plurality of illumination components based on the measured second light signal; with the measured second light signal, with an optimization model generating a plurality of correspondence maps between at least one projector column illuminating the at least one location of the scene and the at least one image capture device pixel receiving the second light signal from the at least one location of the scene, wherein the plurality of correspondence maps comprises raw 3D point clouds comprising the at least one location associated with each of the plurality of illumination components; from the raw 3D point clouds, determining sets of 3D points having a predefined surface constraint of continuity with neighbour 3D points; and based on the predefined surface constraint of continuity extracting a shape of the target surface.

20. The method of claim 19, wherein the predetermined patterns comprise sinusoidal patterns.

21. The method of claim 19, further comprising a step of performing a discrete Fourier transform (DFT) on the measured second light signal measured at the image capture device pixel to inventory the at least projector column illuminating directly and indirectly the at least one location of the scene imaged at the image capture device pixel.

22. The method of claim 21, wherein a first correspondence map of the plurality of correspondence maps is formed by the at least one projector column related to a highest peak at each camera pixel; a second correspondence map is constructed using the at least one projector column linked to the second highest peak at each image capture device pixel; and a k correspondence map is constructed using the at least one projector column linked to the k'hhighest peak at each image capture device pixel.

23. The method of claim 22, wherein the k correspondence maps obtained are used to compute k 3D point clouds using geometrical calibration of the projector-image capture device pair using triangulation.

24. The method of claim 19, comprising further steps of formulating a parameter estimation problem and fitting a model to the measured second light signal measured at the image capture device pixel.

25. The method of claim 24, comprising a further step of initializing the parameter estimation problem using a component subtraction approach to determine a number of informative components in the measured second light signal at the image capture device pixel, to inventory the at least one projector column illuminating the scene point imaged at the image capture device pixel.

26. The method of claim 27, comprising a further step of initializing the parameter estimation problem using a component subtraction approach.

27. The method of claim 26, with a processing unit for executing instructions stored in a computer readable medium, performing the steps of generating k correspondence maps.

28. The method of claim 27, wherein the instructions comprise a Levenberg Marquardt (LM) algorithm to estimate a parameter vector 0 for minimizing a cost function.

29. The method of claim 28, wherein the plurality of illumination components comprises at least one of surface reflections, target semi-transparent surface reflections, interreflections, subsurface scatterings, subsurface reflections and background reflections.

30. The method of claim 29, wherein the predefined surface constraint of continuity extracts the correspondence map representing the target semi-transparent surface only from the 3D point cloud.

31. A system for measuring a 3D shape of an object comprising a target semi-transparent surface, the system comprising: a projector to projecting a light signal having a predetermined pattern and a predetermined frequency from at least one projector column towards at least one location on a scene; an image capture device for receiving the light signal from the at least one location of the scene at an image capture device pixel, and for measuring an intensity signal received at the image capture device pixel; a processing unit for executing instructions stored in a computer readable medium to at least perform the steps of: using the measured intensity signal, generate a k correspondence map constructed using the at least one projector column linked to the k"' highest peak at each image capture device pixel, wherein the k correspondence maps obtained are used to compute k 3D point clouds using triangulation, wherein the k correspondence maps and corresponding 3D point clouds comprise raw data with information related to at least one of the target semi-transparent surface and a background surface. applying a surface continuity constraint to the raw data to extract the correspondence map and the 3D point cloud representing the target semi-transparent surface only.

32. The system of claim 31, wherein the predetermined pattern comprises sinusoidal patterns.

33. The system of claim 32, further comprising a step of performing a discrete Fourier transform (DFT) on the intensity signal measured at the image capture device pixel to inventory the at least projector column illuminating directly and indirectly the at least one location of the scene imaged at the image capture device pixel.

34. The system of claim 32, comprising further steps of formulating a parameter estimation problem and fitting a model to the intensity signal measured at the image capture device pixel.

35. The system of claim 34, comprising a further step of initializing the parameter estimation problem using a component subtraction approach to determine a number of informative components in the measured intensity signal at the image capture device pixel, to inventory the at least one projector column illuminating the scene point imaged at the image capture device pixel.

36. The system of claim 35, comprising a further step of initializing the parameter estimation problem using a component subtraction approach.

37. The system of claim 36, wherein the instructions comprise a Levenberg Marquardt (LM) algorithm to estimate a parameter vector 0 for minimizing a cost function.

Images