A dual-frequency correction method to eliminate the effects of indirect lighting and random noise

Abstract

The present invention discloses a dual-frequency correction method for eliminating the influence of indirect illumination and random noise; the steps are as follows: first, the errors caused by indirect illumination are analyzed, including the errors caused by the indirect illumination itself and the random noise errors amplified by the decrease in pattern contrast caused by the indirect illumination. An error model and an indirect illumination severity index are established. On this basis, a dual-frequency correction method is designed to correct the indirect illumination error. Phase optimization can be achieved by fully utilizing the additionally projected fringe pattern in the heterodyne unwrapped phase. The dual-frequency correction method designed by the present invention can effectively fit the indirect illumination error while avoiding the interference of random noise, thereby improving the reconstruction quality of the point cloud.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional reconstruction in computer vision, and in particular relates to a dual-frequency correction method for eliminating the effects of indirect lighting and random noise. Background Art

[0002] Three-dimensional metrology plays an important role in intelligent manufacturing, visual navigation, face recognition and other fields. Optical three-dimensional ranging methods, including stereo vision, laser triangulation, structured light, etc., have the characteristics of non-contact and non-destructive. Structured light and triangulation principles have been widely used in three-dimensional optical measurement. Methods based on structured light usually use simple processing algorithms and have good real-time performance. Fringe projection profilometry (FPP) has the advantages of low cost, strong robustness, and high spatial resolution, and has always been in a leading position in various research. The rapid development of digital light processing (DLP) technology has further promoted the research of FPP.

[0003] Structured light methods typically assume that the object being measured is illuminated only by the direct light source from the projection device. However, in many real-world measurement situations, such as with marble, skin, and fruit, this assumption often fails. Light emitted by the light source further illuminates the object being measured through interreflections, subsurface scattering, and volume scattering. These are collectively referred to as indirect illumination, also known as global illumination. Different types of indirect illumination have different impacts on the range of illumination. Interreflections and volume scattering have a wide range of influence and are therefore referred to as long-range indirect illumination. In contrast, subsurface scattering and lens defocus have a narrower range of influence and are referred to as short-range indirect illumination.

[0004] Although short-range indirect illumination has a small range, it is a ubiquitous phenomenon that often results in severe phase distortion after accumulation. Indirect illumination contaminates the projected pattern, causing severe measurement errors and ultimately leading to complex depth offsets. Indirect illumination, especially subsurface scattering, can also severely reduce the intensity modulation of the captured pattern. This reduction in intensity modulation amplifies the effects of random noise, resulting in even more severe random errors. Furthermore, the uncertainty of random noise hinders the algorithm from eliminating indirect illumination errors. The main focus of this patent is on how to accurately and stably eliminate indirect illumination errors while preventing random noise from interfering with the algorithm and ensuring its stability. Summary of the Invention

[0005] To address these issues, the present invention provides a dual-frequency correction method that eliminates the effects of indirect lighting and random noise. This method uses two sets of absolute phases to calculate a phase correction coefficient, scales it to suppress random noise interference, and then uses this coefficient to correct the absolute phase. This method can suppress the effects of indirect lighting and exhibits excellent robustness.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] The dual-frequency correction method for eliminating the effects of indirect lighting and random noise includes the following steps:

[0008] Step 1: Based on the basic formula of the phase shift method, calculate the phase error caused by indirect lighting and random noise.

[0009] Step 2: Further study its amplification effect on random noise and derive an indicator to judge the severity of the indirect lighting impact.

[0010] Step 3: Use a projector to project the three required phase-shifted fringe patterns onto the object with a complex surface texture. Use a camera to capture the projected patterns and calculate the absolute phases of two of the different periods.

[0011] Step 4: Design a dual-frequency correction method. For the two sets of absolute phases calculated in step 3, calculate the required correction coefficients.

[0012] Step 5: Scale the correction factor appropriately and use it to correct the absolute phase calculated in step 3.

[0013] The dual-frequency correction method for eliminating the effects of indirect illumination and random noise is characterized by first deriving the phase error caused by indirect illumination when implementing three-dimensional measurement using the phase shift method, and the formula is:

[0014]

[0015] Where Φ(x,y) is the phase value corresponding to the projected fringe image I at row x and column y, obtained by the phase shift method. ΔΦ(x,y) is the phase error caused by indirect illumination, and Δθ is the phase represented by a single pixel on the projected pattern, which is related to the fringe period p. j (x,y) is the sum of the indirect illumination transfer coefficients of the pixel whose wrapping phase differs from the wrapping phase at (x,y) by jΔθ. The relevant formula is as follows:

[0016] Φ(x,y)=2πx / p,

[0017] Δθ=2π / p,

[0018] G j (x,y)=∑ i T i (x,y,x i ,y i )x i -x=kp+j,

[0019] Where T i (x,y,x i ,y i ) is (x i ,y i) point to (x,y) point indirect lighting transfer coefficient, k∈Z.

[0020] The dual-frequency correction method for eliminating the influence of indirect lighting and random noise is characterized in that the error model calculation method is:

[0021] Step 1.1: When (x, y) is fixed, ΔΦ(x, y) is only related to Δθ. Since Δθ>0, after odd-number extension of ΔΦ(x, y), the Fourier expansion is transformed into a sine series, which is the error of indirect lighting:

[0022]

[0023] Among them, β j is the coefficient of the jth term in the error model, which is called the indirect lighting coefficient in this scheme.

[0024] Step 1.2: Based on existing research on random noise errors, the phase error produced by random noise under the influence of indirect lighting is further derived, and the formula is as follows:

[0025]

[0026] Where b is the intensity modulation designed when the computer generates the fringe pattern, b' is the intensity modulation after being contaminated by indirect lighting, is (x i ,y i ) at the wrapping phase, δ n is the phase shift amount in the phase shift method, N is the number of phase shift steps, Δa n is the random noise of the nth image.

[0027] The dual-frequency correction method for eliminating the effects of indirect lighting and random noise is characterized in that the indirect lighting severity determination indicator in step 2 is: assuming that a = b when the computer generates the fringe pattern, where a is the average intensity, and the standard deviation of the random noise is σ, the variance of the error caused by the random noise is calculated as follows:

[0028]

[0029] Where a′ is the average intensity after being polluted by indirect lighting. Since the more severe the indirect lighting, the lower the contrast of the pattern and the more severe the random noise. Therefore, α is used as the criterion for judging the severity of indirect lighting:

[0030]

[0031] The larger the value, the more serious the indirect lighting error.

[0032] The dual-frequency correction method for eliminating the effects of indirect illumination and random noise is characterized in that the specific method for designing the projection pattern in step 3 is: designing an N-step phase shift pattern with a period of p1, then designing an N-step phase shift pattern with a period of p2. Furthermore, a 3-step phase shift pattern with a period of p3 is designed. The three sets of corresponding wrapped phases must ensure that the phase can be solved using the heterodyne method.

[0033] The dual-frequency correction method for eliminating the influence of indirect lighting and random noise is characterized in that the specific method for calculating the phase in step 3 is: respectively calculating the wrapping phase corresponding to the three cycles and Then use the multi-frequency heterodyne method to solve The period is p1, and The period is p2, and the corresponding absolute phases are Φ1 and Φ2.

[0034] The dual-frequency correction method for eliminating the effects of indirect lighting and random noise is characterized in that the specific method for calculating the correction coefficient β1 in step 4 is: assuming that the indirect lighting coefficients of all high-frequency items except β1 are 0, β1 can be calculated using the following formula:

[0035]

[0036] Here, x1 is the coordinate calculated using Φ1, and x2 is the coordinate calculated using Φ2.

[0037] The dual-frequency correction method for eliminating the influence of indirect lighting and random noise is characterized in that the specific method of scaling the correction coefficient in step 5 is: considering that sin(Δθ1)-sin(Δθ2) is much less than 1, the phase error ΔΦ r The (x,y) part will be magnified, causing a huge error. β1 is appropriately scaled to balance this error:

[0038] β1(x,y)=4*β1(x,y) / max(β1),

[0039] Among them, max(β1) is the maximum value in β1.

[0040] The dual-frequency correction method for eliminating the influence of indirect lighting and random noise is characterized in that the specific method of correcting the phase in step 5 is: using the scaled indirect lighting coefficient β1, eliminating the phase error according to the error formula:

[0041] x o (x,y)=x1(x,y)-β1(x,y)sinΔθ1,

[0042] Among them, x o is the corrected coordinate, which can be converted into phase.

[0043] The corrected phase can greatly suppress the influence of indirect lighting, and can also play a role in defocusing and intensity diffusion. Combining the heterodyne method to solve the phase can make full use of the projected fringe pattern.

[0044] Beneficial Effects: This paper addresses the serious depth offset problem in traditional structured light projection 3D measurement systems when affected by indirect lighting. It proposes a dual-frequency correction method that eliminates the effects of indirect lighting and random noise, successfully correcting the resulting phase error. This method primarily includes three characteristic methods: indirect lighting effects, phase design, and dual-frequency correction. It offers the following advantages:

[0045] 1. In the context of indirect lighting effects, a simple formula for the phase error caused by indirect lighting in the phase shift method is derived. Furthermore, an in-depth analysis of the impact of indirect lighting on random noise is conducted, and an indicator for determining the severity of indirect lighting is provided.

[0046] 2. In the phase design method, according to the need of unwrapping the phase, two or more sets of phase shift patterns are designed based on the heterodyne method. This fully utilizes the projected fringes and avoids projecting too many additional patterns.

[0047] 3. In the dual-frequency correction method, the absolute phases of the two phase-shifted patterns with the higher period are calculated and used to calculate the correction coefficient. After eliminating the influence of random noise, the absolute phases are corrected.

[0048] Not only does it provide an error formula for indirect lighting and a criterion for judging its severity, it also successfully corrects the phase error. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the entire process of invention.

[0050] Figure 2 Three phase error simulations are shown: without random noise, with random noise, and using the dual-frequency correction method with random noise. The simulated random noise has a mean of 0 and a standard deviation of 0.005. TPC is the abbreviation for dual-frequency correction.

[0051] Figure 3 is the relationship between the maximum value of β1 and the RMSE of the phase after optimization by the dual-frequency correction method.

[0052] Figure 4 (a) is the average strength of soap.

[0053] Figure 4 (b) is the strength modulation of soap.

[0054] Figure 4 (c) is the severity of indirect lighting of soap, α.

[0055] Figure 5(a) is the error of phase Φ1 before soap correction.

[0056] Figure 5 (b) is the error of the phase Φ after the soap is optimized by the dual-frequency correction method.

[0057] Figure 6 (a) is the average strength of jade in complex environment.

[0058] Figure 6 (b) is the intensity modulation of jade.

[0059] Figure 6 (c) is the error of the phase Φ1 of the jade before correction.

[0060] Figure 6 (d) is the phase Φ error of the jade after being optimized by the dual-frequency correction network. DETAILED DESCRIPTION

[0061] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inward" and "outward" refer to directions toward or away from the geometric center of a particular component, respectively.

[0062] The indirect illumination error model and dual-frequency correction network of the present invention achieve three-dimensional reconstruction based on the FPP system. The system consists of a computer, a projector (DLP LightCrafter 4500; Texas Instruments, USA), and a camera (acA800-510um; Basler Vision Technology GmbH, Germany). The projector has a resolution of 912×1140 pixels and a maximum projection rate of 120Hz in 8-bit mode. The camera has a resolution of 800×600 pixels and a maximum frame rate of 393fps in normal sensor readout mode. The system is 0.4 to 0.5 meters away from the object to be measured, and the angle between the projector optical axis and the camera optical axis is approximately 25 degrees.

[0063] The present invention will be further described below with reference to the accompanying drawings.

[0064] Example 1:

[0065] Example 1 tests the effectiveness of the dual-frequency correction method in a simple environment. The object of measurement is soap. During the measurement, light enters the interior of the object and is then reflected back to the surface. Two sets of phase-shifted patterns of different frequencies are projected (8px and 9px). Existing research work has shown that when measuring translucent objects, the shorter the illumination wavelength, the smaller the error. Therefore, both sets of patterns measured are green and are projected only once. The color of the stripe pattern (8px) is then changed to blue, and the projection is repeated 10 times to suppress random noise. Φ1 (8px) and Φ2 (9px) are calculated from the green stripe pattern, and Φ' (8px) is calculated from the blue pattern. Φ' is regarded as the true value and is used to measure the accuracy of the dual-frequency correction method.

[0066] Figure 4 (a) is the average strength of soap, Figure 4 (b) is intensity modulation. Its intensity modulation is low, but there is no complex lighting environment, so the indirect lighting is not very serious. Calculating its indirect lighting severity, most pixels are around 0.1, such as Figure 4 (c). The final phase error is as follows Figure 5 (a) and Figure 5 (b) In this simple lighting environment, the dual-frequency correction method also performs well. Using MATLAB for comparison, the average error is reduced from 0.0756 to 0.0624, an improvement of approximately 17.5%. The RMSE is reduced from 0.0981 to 0.0861, an improvement of approximately 12.2%.

[0067] Example 2:

[0068] Example 2 tested the performance of the present invention under complex lighting conditions. The measurement was performed on a jade stone placed in a transparent storage box and labeled. The measurement method was consistent with that of Example 1. Since the jade stone was placed in a transparent, smooth storage box, this created a complex lighting environment. The jade stone was affected not only by its subsurface scattering but also by various other indirect lighting factors, such as specular reflection and volume scattering from the storage box, and diffuse reflection from the label. Figure 6 (b) shows the intensity modulation of jade, which is very low under indirect lighting. Calculating its index α shows that most pixels are above 1. This helps verify the performance of the dual-frequency correction method under strong indirect lighting.

[0069] After obtaining Φ1 and Φ2, the optimized absolute phase Φ is obtained through TPC. The absolute error between each phase and the true value Φ' is calculated. Figure 6 (c) shows the error corresponding to Φ1. Figure 6(d) also shows the error corresponding to Φ. It can be seen that the optimized phase error is significantly reduced. The dual-frequency correction method achieves excellent results. Compared with Φ1, the optimized average error decreases from 0.252 to 0.203, a decrease of approximately 19.4%. The RMSE decreases from 0.3543 to 0.2901, a decrease of approximately 18.1%. These results demonstrate that TPN can still perform effectively even in complex indirect lighting environments.

[0070] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above-mentioned embodiment, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A dual-frequency correction method for eliminating the effects of indirect lighting and random noise, characterized by: The following steps are involved: Step 1: Calculate the phase error caused by indirect lighting and random noise based on the basic formula of the phase shift method; Step 2: derive an index for judging the severity of the indirect lighting impact; the index for judging the severity of the indirect lighting impact in step 2 is specifically: assuming that when the computer generates the stripe pattern, , is the average intensity; The intensity modulation designed when the fringe pattern is generated by the computer; and the standard deviation of the random noise is , then the variance of the error caused by random noise is calculated as follows: in, is the average intensity after being polluted by indirect lighting; It is the intensity modulation after being polluted by indirect lighting; is the stripe period; is the number of phase shift steps; since the more serious the indirect illumination, the lower the contrast of the pattern and the more serious the random noise; As a criterion for judging the severity of indirect lighting: The larger the value, the more serious the indirect lighting error; Step 3: Use a projector to project the three phase-shifted fringe patterns required on the object to be tested, and use a camera to capture the projected patterns; calculate the absolute phases of two groups with different periods; the specific method for designing the projection pattern in step 3 is: design an N-step phase-shifted pattern with a period of , then design the N-step phase shift pattern, the period is ; In addition, a 3-step phase shift pattern is designed with a period of p3. The three sets of corresponding wrapped phases must ensure that the phase can be solved by the heterodyne method; calculate the wrapped phases corresponding to the three periods respectively 、 and ; Then use the multi-frequency heterodyne method to solve , the period is ,and , the period is ; The corresponding absolute phase is 、 ; Step 4: Design a dual-frequency correction method; For the two sets of absolute phases calculated in step 3, calculate the required correction coefficients; Calculate the correction coefficients in step 4 The specific method is: Assume that The indirect lighting coefficient of all high-frequency items except is 0, and is calculated using the following formula : in, For use The calculated coordinates, For use Calculated coordinates; is the phase represented by a single pixel on the projected pattern, and is related to the fringe period related; The coefficient of the Fourier expansion formula after the odd extension of the phase error formula caused by indirect illumination is the indirect illumination coefficient; Step 5: Scale the correction coefficient appropriately and use it to correct the absolute phase calculated in Step 3; The specific method for scaling the correction coefficient in step 5 is: in, for The specific method of correcting the phase in step 5 is: using the scaled indirect lighting coefficient , eliminate the phase error according to the error formula: in, is the corrected coordinate, converted into phase.

2. The dual-frequency correction method for eliminating the effects of indirect lighting and random noise according to claim 1, characterized in that: First, the phase error caused by indirect illumination when using the phase shift method to achieve three-dimensional measurement is derived. The formula is: in, is the projected fringe image In the Rank The corresponding phase value at the column is obtained by the phase shift method; Phase error caused by indirect lighting, is the phase represented by a single pixel on the projected pattern, and is related to the fringe period related; For the wrapped phase and The wrapping phase difference at The sum of the indirect lighting transfer coefficients of the pixel points; the relevant formula is as follows: in for Point to Indirect lighting transfer coefficient of the point, .

3. The dual-frequency correction method for eliminating the effects of indirect lighting and random noise according to claim 2, characterized in that: The specific steps for calculating the phase error caused by indirect illumination and random noise in step 1 are as follows: Step 1.1: When After fixing, Only with related; due to ,Will After singular extension, Fourier expansion becomes a sine series, which is the error of indirect lighting: in, The error model The coefficient of the term is called the indirect lighting coefficient in this scheme; Step 1.2: The phase error produced by random noise under the influence of indirect lighting is further derived, and the formula is as follows: in, Intensity modulation designed for computer-generated fringe patterns, is the intensity modulation after being polluted by indirect lighting, for The wrapping phase at is the phase shift in the phase shift method, is the phase shift step number, is the random noise of the nth image.

Images