A 3D reconstruction method for highly reflective surfaces based on multi-frequency phase fusion

Abstract

The present invention relates to the technical field of optical three-dimensional imaging, and particularly to a three-dimensional reconstruction method for highly reflective surfaces based on multi-frequency phase fusion. The method uses a three-frequency heterodyne time phase unwrapping technique to calculate the unwrapped phase from a high-frequency fringe group, establishes a plane constraint based on a random sampling consistency algorithm, finds noise points caused by highly reflective surfaces, and uses a plane constraint-based discrete point removal technique to segment phase points affected by highly reflective surfaces. Simultaneously, based on the different sensitivities of low-frequency and high-frequency fringes to reflection, a low-frequency phase solution is used to obtain the phases of outliers. The phases obtained at various frequencies are fused to obtain a complete phase map of the highly reflective surface. Finally, three-dimensional reconstruction of the highly reflective surface is achieved based on calibration parameters, without having to search for exposure areas in advance. This solves the technical problem of overexposure areas being prone to appearing on highly reflective surfaces during three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of optical three-dimensional imaging, and in particular to a three-dimensional reconstruction method of a high-reflective surface based on multi-frequency phase fusion. Background Art

[0002] Three-dimensional imaging technology is an important means of recording and analyzing the real world and is one of the important research topics in the field of computational imaging and geometric measurement. Sinusoidal grating fringe projection profilometry based on surface structured light has the advantages of large field of view, non-contact, and high precision, and has broad application prospects in both daily consumer and industrial professional inspection fields. However, due to the inconsistency of the reflectivity of the object surface, overexposed areas are prone to appear on highly reflective surfaces, causing the stripes in this area to lose their sinusoidal nature and making high-quality three-dimensional reconstruction impossible. Existing technologies include the following, but each has different technical problems: First, high-dynamic images are created through multiple exposures, but multiple exposures require strict control of exposure time, which is difficult to operate; second, the high-reflective area is first determined, and then the fringe brightness distribution of the projector is corrected, which obviously affects the convenience of projecting and collecting fringe stripes; third, adding filters to the hardware system increases the complexity of system structure and adjustment. Summary of the Invention

[0003] The purpose of the present invention is to provide a three-dimensional reconstruction method for highly reflective surfaces based on multi-frequency phase fusion, which improves the three-dimensional reconstruction quality of grayscale saturated areas on the object surface and avoids the technical problem of overexposure areas on highly reflective surfaces during three-dimensional reconstruction.

[0004] To achieve the above objectives, the present invention provides a method for three-dimensional reconstruction of highly reflective surfaces based on multi-frequency phase fusion, comprising the following steps:

[0005] The projector projects multiple sets of triple-frequency heterodyne sinusoidal fringes;

[0006] The camera collects a plurality of fringe images of the triple-frequency heterodyne sinusoidal fringes;

[0007] Solving the folding phase of each group of fringe images based on a phase shift method, and obtaining the unfolding phase based on a triple-frequency heterodyne;

[0008] Find the noise points in the high-frequency group unfolding phase based on RANSAC three-dimensional point plane fitting;

[0009] Calculating the unwrapped phase of the noise point in the low-frequency group, and calculating the equivalent phase according to the frequency relationship;

[0010] The unwrapped phases of the high-frequency group and the low-frequency group are fused, and three-dimensional reconstruction is performed based on the calibration parameters.

[0011] The triple-frequency heterodyne sinusoidal stripes are vertical stripes, and the light intensity values of pixels in each column of the projected stripes are the same.

[0012] The calculation process of the folding phase is specifically to calculate the folding phase of the pixel point based on the least squares method.

[0013] Among them, the phase unwrapping method based on triple-frequency heterodyne is adopted, and the calculation between each point is independent of each other.

[0014] The noise points are estimated from a set of observed data containing outliers using RANSAC three-dimensional point plane fitting. The criterion for judging outlier noise points is to look at the distance from each point to the plane. If the distance is greater than a threshold, it is an outlier.

[0015] Among them, in the process of fusing the unfolded phase of the high-frequency group and the low-frequency group, the high-frequency three-frequency heterodyne of the first group is used as the basis, and the high-reflective area uses the low-frequency three-frequency heterodyne to calculate the equivalent phase, thereby realizing phase fusion.

[0016] The present invention provides a three-dimensional reconstruction method for highly reflective surfaces based on multi-frequency phase fusion. The method uses a three-frequency heterodyne time phase unwrapping technique to calculate the unwrapped phase from a high-frequency fringe group, establishes a plane constraint based on a random sampling consensus (RANSAC) algorithm, finds noise points caused by highly reflective surfaces, and uses a plane constraint-based discrete point removal technique to segment the phase points affected by highly reflective surfaces. Simultaneously, based on the different sensitivities of low-frequency and high-frequency fringes to reflection, a low-frequency phase solution is used to obtain the phase of the outlier. The phases obtained at various frequencies are fused to obtain a complete phase map of the highly reflective surface. Finally, three-dimensional reconstruction of the highly reflective surface is achieved based on calibration parameters, without having to search for the exposure area in advance. This solves the technical problem of overexposure areas easily appearing on highly reflective surfaces during three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flow chart of a method for three-dimensional reconstruction of a highly reflective surface based on multi-frequency phase fusion according to the present invention.

[0019] Figure 2 It is a schematic diagram of the phase fusion strategy of the present invention.

[0020] Figure 3 2 is a schematic diagram of an unfolded phase according to a specific embodiment of the present invention.

[0021] Figure 4 3D reconstruction of coins using two methods according to a specific embodiment of the present invention is compared. DETAILED DESCRIPTION

[0022] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0023] See also Figure 1 The present invention proposes a three-dimensional reconstruction method for highly reflective surfaces based on multi-frequency phase fusion, comprising the following steps:

[0024] S1: The projector projects multiple sets of triple-frequency heterodyne sinusoidal fringes;

[0025] S2: a camera collects multiple sets of fringe images of the triple-frequency heterodyne sinusoidal fringes;

[0026] S3: solving the folding phase of each group of fringe images based on a phase shift method, and obtaining the unfolding phase based on a triple-frequency heterodyne;

[0027] S4: Find the noise points in the high-frequency group unfolding phase based on RANSAC three-dimensional point plane fitting;

[0028] S5: calculating the unwrapped phase of the noise point in the low-frequency group, and calculating the equivalent phase according to the frequency relationship;

[0029] S6: The unwrapped phases of the high-frequency group and the low-frequency group are fused and three-dimensional reconstruction is performed based on the calibration parameters.

[0030] The present invention is further described below in combination with the principles and specific implementation steps:

[0031] 1. Analysis of system calibration principles

[0032] The theory on which the present invention is based is that the high frequency of grating stripes makes it easier to achieve high-precision three-dimensional reconstruction, but high-frequency stripes are more likely to cause grayscale saturation than low-frequency stripes. To this end, based on the three-frequency heterodyne time phase unwrapping technology, the unwrapped phase is calculated from the high-frequency stripe group, and a plane constraint is established based on the random sampling consensus (RANSAC) algorithm to find the noise points caused by high reflectivity. The phase of the noise points is solved in the low-frequency group, and the equivalent phase consistent with the high-frequency group is calculated, thereby realizing the fusion of the unwrapped phase. Finally, based on the system calibration parameters, the three-dimensional reconstruction of the highly reflective surface is achieved. The specific operations are:

[0033] 1) Generate multiple sets of triple-frequency heterodyne fringes; operate according to the following formula:

[0034]

[0035] Where, I n (x,y) is the light intensity at pixel (x,y), n = 1, 2, ... N-1, is the nth phase shift at a given frequency. A and B are the background term and modulation term, respectively. f is the frequency, which adjusts the density of the stripes. W is the image width, and N is the number of phase shift steps, which is 4 for a four-step phase shift. This formula generates vertical stripes, meaning that the light intensity of the pixels within each column is the same.

[0036] 2) Calculate the folding phase. Calculate the folding phase Φ(x,y) of the pixel point (x,y) based on the least squares method, as shown in the formula.

[0037]

[0038] Where, I i (x, y) is the light intensity distribution of pixel (x, y) in the i-th fringe pattern, δ i is the phase shift of the i-th fringe pattern, and N is the number of grating fringe patterns. The phase calculated by the formula is the folded phase, which requires phase unwrapping.

[0039] 3) There are many methods for phase unwrapping. Among them, the triple-frequency heterodyne method is a time-based phase unwrapping method. The calculations between each point are independent of each other, which can effectively avoid error propagation. Based on the three folded phases Φ1(x,y), Φ2(x,y), and Φ3(x,y) calculated in the previous step, the unwrapped phase is obtained:

[0040]

[0041] Where int(·) is a rounding operation, and R is a constant determined by the frequency. The specific calculation is:

[0042]

[0043] Among them, f1, f2, f3 is one of the multiple frequency groups, Φ 123 (x,y) is the composite phase, calculated according to the following formula:

[0044] Φ 123 (x,y)=(Φ1(x,y)-Φ2(x,y))-(Φ2(x,y)-Φ3(x,y)) (5)

[0045] 4) Outlier noise point judgment based on RANSAC algorithm

[0046] The Random Sample Consensus (RANSAC) algorithm uses an iterative approach to estimate the parameters of a mathematical model from a set of observed data that contains outliers. Here, the RANSAC algorithm ensures that outliers do not participate in the plane fitting. The criterion for determining outlier noise points is to look at the distance from each point to the plane. Points are considered outliers if their distance is greater than a certain threshold:

[0047]

[0048] 5) Calculation of equivalent phase of outliers:

[0049] The phase of the outlier is calculated using the low-frequency group and is equivalent to the first group of frequencies. The calculation formula for the equivalent phase is:

[0050]

[0051] In the formula, Up e (x,y) is the equivalent unfolding phase of the outlier point, Up i (x,y) is the unwrapped phase calculated for the i-th group of frequencies, f i1 is the first frequency in the i-th group of frequencies, f 11 is the first frequency in the first group of frequencies.

[0052] 6) Obtain the fused phase, replace the outlier phase of the first group of expanded phase Up1 with the equivalent phase, and repeat step 4) to determine the outliers until there are no outliers or the frequency group is fully used. Phase fusion is:

[0053] Up1(x,y)=Up e (x,y) (8)

[0054] The expression for solving the three-dimensional reconstruction is:

[0055]

[0056]

[0057] Among them, a1-a8 are the system parameters to be calibrated, (X c ,Y c ,Z c ) are the camera coordinates of the object point. ρ is the scale factor, (u, v) are the pixel coordinates, and (u0, v0) are the principal point coordinates. M is the camera intrinsic parameter matrix, which can be obtained after camera calibration. After system calibration, knowing the unwrapped phase corresponding to a pixel point allows the world coordinates of that point to be determined, enabling 3D reconstruction.

[0058] 2. Phase fusion strategy based on multiple sets of triple-frequency heterodynes

[0059] The phase fusion strategy based on multiple sets of triple-frequency heterodynes is as follows: Figure 2 As shown. Based on the high-frequency triple-frequency heterodyne of the first group, the high-reflectivity area uses the low-frequency triple-frequency heterodyne to calculate the equivalent phase, thereby achieving phase fusion. The judgment of outliers is based on the Random Sample Consensus (RANSAC) algorithm. The present invention is implemented based on two basic facts:

[0060] 1) Based on the fact that over-exposure points lose their sinusoidal properties and their unwrapped phase deviates far from the ideal phase, plane constraints are used to screen out over-exposure points.

[0061] 2) Under the same conditions, low-frequency grating stripes have a higher grayscale saturation dynamic range than high-frequency grating stripes, which makes it possible for overexposure points that appear at high frequencies to not have grayscale saturation at low frequencies.

[0062] Furthermore, the present invention also proposes a specific embodiment for result comparison and verification:

[0063] 1. Experimental steps

[0064] Step 1: Create multiple sets of triple-frequency heterodyne four-step phase-shifted grating fringe images and burn them into the projector. The projected patterns are captured by a camera. The four sets of grating fringe frequencies are {[706459], [433630], [211612],

[1396] }, which modulate the coin image.

[0065] Step 2: Based on the least squares method, the folded phase of the first set of triple-frequency heterodyne grating fringe images is calculated, and the unfolded phase UP0 is calculated based on the triple-frequency heterodyne algorithm;

[0066] Step 3: Based on the RANSAC algorithm, the unfolded phase is plane-constrained and out-of-plane points under a certain distance threshold are given. These out-of-plane points are outliers.

[0067] Step 4: Based on the next set of triple-frequency heterodyne grating images, solve the unwrapped phase of the outlier point in step 3, and calculate the equivalent unwrapped phase of the outlier point based on the relationship between this set of frequencies and the first set of frequencies;

[0068] Step 5: Replace the outlier phase at the corresponding position in UP0 with the equivalent unwrapped phase;

[0069] Step 6: Repeat steps 3-5 until there are no out-of-plane outliers in the unwrapped phase UP0, or the frequencies of all groups have been calculated

[0070] 2. Experimental results and analysis

[0071] The expanded phase diagram is as follows Figure 3As shown, there are fewer noise points. After obtaining the unfolded phase, three-dimensional reconstruction can be achieved according to the formula and the reconstruction effect of the coin is as follows Figure 4 The figure on the left shows (a) the 3D reconstruction effect based on high-frequency triple-frequency heterodyning, and the figure on the right shows (b) the 3D reconstruction effect based on the fusion phase proposed by the present invention. The number of point clouds in (a) and (b) are 463595 and 471044 respectively. The multi-frequency phase fusion technology used in the present invention can obtain better unwrapped phase and achieve better 3D reconstruction quality.

[0072] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for 3D reconstruction of highly reflective surfaces based on multi-frequency phase fusion, characterized in that: The following steps are involved: The projector projects multiple sets of triple-frequency heterodyne sinusoidal fringes; The camera collects a plurality of fringe images of the triple-frequency heterodyne sinusoidal fringes; Solving the folding phase of each group of fringe images based on a phase shift method, and obtaining the unfolding phase based on a triple-frequency heterodyne; Find the noise points in the high-frequency group unfolding phase based on RANSAC three-dimensional point plane fitting; Calculating the unwrapped phase of the noise point in the low-frequency group, and calculating the equivalent phase according to the frequency relationship; The unwrapped phases of the high-frequency group and the low-frequency group are fused and three-dimensional reconstruction is performed based on the calibration parameters; In the process of fusing the unfolded phases of the high-frequency group and the low-frequency group, the high-frequency triple-frequency heterodyne of the first group is used as the basis, and the low-frequency triple-frequency heterodyne is used to calculate the equivalent phase in the high-reflective area, thereby achieving phase fusion.

2. The method for 3D reconstruction of highly reflective surfaces based on multi-frequency phase fusion according to claim 1, wherein: The sinusoidal stripes of the triple-frequency heterodyne are vertical stripes, and the light intensity values of the pixels in each column of the projected stripes are the same.

3. The method for 3D reconstruction of highly reflective surfaces based on multi-frequency phase fusion according to claim 1, wherein: The calculation process of the folding phase is specifically to calculate the folding phase of the pixel point based on the least squares method.

4. The method for 3D reconstruction of highly reflective surfaces based on multi-frequency phase fusion according to claim 1, wherein: The phase unwrapping method based on triple-frequency heterodyne is adopted, and the calculation between each point is independent of each other.

5. The method for 3D reconstruction of highly reflective surfaces based on multi-frequency phase fusion according to claim 1, wherein: The noise points are estimated from a set of observed data containing outliers using RANSAC-based three-dimensional point plane fitting. The criterion for judging outlier noise points is to look at the distance from each point to the plane. If the distance is greater than a threshold, it is an outlier point.

Images