A fast and accurate correction method for indirect lighting

Abstract

This invention discloses a fast and precise correction method for indirect illumination. The key concept is to design an optimal network structure based on an accurate error model to correct the error. The steps are as follows: First, the error caused by indirect illumination is converted into a sine series, reducing the dimension of the indirect illumination problem to a single-pixel problem. Based on this error model, a multilayer perceptron is designed to fit the error model. Only simulated data is used in network training, and real images are used for testing. The network designed in this invention can effectively fit the indirect illumination error, recover more shape details, and thus improve the reconstruction quality.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional reconstruction in computer vision, and in particular relates to constructing an indirect illumination error model and a dual-frequency correction network. Background Art

[0002] Recovering the dense 3D shape of an object from a 2D image is a fundamental problem in artificial intelligence, and numerous research results have been achieved in this area. 3D point clouds, also known as point sets, are considered one of the simplest 3D shape representations because they consist only of primitive coordinates in 3D space. Point clouds can be easily acquired using common sensors such as LiDAR and traditional cameras. The reconstructed point clouds can be used in a variety of fields, including object detection, object tracking, and 3D shape feature learning. Therefore, optimizing point cloud and reconstruction algorithms is a promising research direction.

[0003] Fringe projection profilometry is a highly efficient, non-contact method for acquiring 3D point clouds. The rapid development of digital light processing (FPP) has further fueled research in structured light. Structured light-based 3D measurement techniques assume that the object being measured is illuminated only by direct illumination from the projection device. However, in real-world measurement experiments inevitably involve semi-transparent objects such as skin, marble, and fruit. This assumption often fails in these measurement situations, as is the case with marble, skin, and fruit. Semi-transparent objects primarily introduce short-range indirect illumination. In addition, various long-range indirect illumination sources exist. Indirect illumination is a global illumination propagation problem, where light propagation becomes random after multiple reflections and refractions, making the problem extremely complex. Indirect illumination violates the fundamental principles of structured light technology and can therefore corrupt the point cloud reconstructed using structured light, resulting in complex depth migration. Numerous methods exist for designing optimization networks for phase shifting, but most papers blindly constrain the network using datasets. While this approach can yield effective network training, it is prohibitively time-consuming and complex due to the required functionality. Filtering is a simple solution for depth migration, but it often compromises point cloud detail. Furthermore, in indirect lighting, obtaining the true value of an object's shape is extremely difficult. These factors must be taken into account when using neural networks to optimize structured light algorithms. Designing an optimization network that accurately eliminates errors while preserving sufficient detail is the main focus of this patent. Summary of the Invention

[0004] Technical problem: In order to overcome the shortcomings of the existing technology, the present invention provides a fast and accurate correction method for indirect lighting. This method uses simulated data to train a neural network. By simply adding a set of phase-shifted stripe patterns, the influence of indirect lighting can be eliminated, and the robustness is extremely high.

[0005] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A fast and accurate correction method for indirect lighting includes the following steps:

[0007] Step 1: Based on the basic formula of the phase shift method, the phase error caused by indirect lighting is derived into a simple indirect lighting error model, successfully constraining the global error problem to a single-pixel error problem.

[0008] Step 2: Design a dual-frequency correction network whose input is only the phase value of a single pixel under different projection strategies;

[0009] Step 3: Design a method to simulate the effects of indirect lighting and simulate two sets of patterns affected by indirect lighting as training data for the multi-layer perceptron.

[0010] Step 4: Use a projector to project the required 2×N phase-shifted fringe patterns and M coding patterns onto the object with complex surface texture, and use a camera to capture the projected patterns.

[0011] Step 5: Use the phase-shift fringe pattern collected in step 3 to calculate the absolute phase, and use the dual-frequency correction network to convert it into the corrected phase.

[0012] The fast and accurate correction method for indirect illumination is characterized by first deriving the phase error caused by indirect illumination when using the phase shift method to achieve three-dimensional measurement, and the formula is:

[0013]

[0014] Where Φ(x,y) is the phase value corresponding to the projected fringe image I at the xth row and yth column, obtained by the phase shift method; ΔΦ(x,y) is the phase error caused by indirect illumination; Δθ is the phase represented by a single pixel on the projected pattern, which is related to the fringe period p; T i (x,y) is the indirect lighting transfer coefficient around the pixel (x,y); the relevant formula is as follows:

[0015] Φ(x,y)=2πx / p

[0016] Δθ=2π / p.

[0017] The fast and accurate correction method for indirect lighting is characterized by converting the error model as follows: when (x, y) is fixed, ΔΦ(x, y) is only related to Δθ; since Δθ>0, ΔΦ(x, y) is odd-numbered and Fourier expanded into a sine series, which is the error model for indirect lighting:

[0018]

[0019] Among them, β jis the coefficient of the jth term of the error model, which is the indirect lighting coefficient.

[0020] The method for fast and accurate correction of indirect lighting is characterized in that the specific method of problem constraint in step 1 is: when (x, y) is fixed, the indirect lighting coefficient β j is a constant, the error is only related to Δθ and has nothing to do with the surrounding pixels; that is, the global illumination problem is limited to a single pixel.

[0021] The fast and accurate correction method for indirect illumination is characterized in that the specific method of designing the dual-frequency correction network input in step 2 is: although the error model is an infinite series, the coefficient of the high-frequency part is small and is regarded as a finite series; that is, only the phase value obtained by the projection stripes of the same pixel point with different periods is required to calculate the indirect illumination coefficient, thereby correcting the error; therefore, the network structure input is 2 nodes, namely, the absolute phase Φ1 with a period of p1 and the absolute phase Φ2 with a period of p2; it is the absolute phase calculated for the same pixel point under different projection strategies, p1≠p2.

[0022] The method for rapid and accurate correction of indirect lighting is characterized in that the specific method of simulating data in step 3 is: generating two sets of phase-shift patterns with different periods, and setting the calculated absolute phases as true values; using filtering and randomly generated reflectivity to simulate indirect lighting to contaminate the phase-shift patterns, and then calculating the two absolute phases after contamination; this is the training data.

[0023] The fast and accurate correction method for indirect lighting is characterized in that the specific method of designing the projection pattern in step 4 is: designing an N-step phase-shift pattern with a period of p1 and a corresponding M-step Gray code pattern, and then designing an N-step phase-shift pattern with a period of p2.

[0024] The method for rapid and accurate correction of indirect illumination is characterized in that the specific method for calculating the phase in step 5 is as follows: using an N-step phase shift pattern with a period of p1 and a corresponding M-step Gray code pattern to calculate the absolute phase Φ1; then, based on the frequency doubling method, using Φ1 and the N-step phase shift pattern with a period of p2 to calculate the absolute phase Φ2:

[0025] The fast and accurate correction method for indirect illumination is characterized in that the specific method of repairing the phase of the dual-frequency correction network in step 5 is as follows: Φ1(x, y) and Φ2(x, y) are input into the network, and the coordinates of the output phase Φ are still set to (x, y); the period of the corrected phase Φ is the same as that of Φ1:

[0026] The corrected phase can greatly suppress the influence of indirect illumination, and can also play a role in defocusing and intensity diffusion; accurate three-dimensional reconstruction can be achieved by simply projecting two sets of phase-shifted fringe patterns with different periods.

[0027] Beneficial Effects: This paper addresses the severe depth offset problem in traditional structured light projection 3D measurement systems when affected by indirect lighting. By proposing an indirect lighting error model and a dual-frequency correction network, the paper successfully corrects the phase error caused. This method primarily includes three characteristic methods: the indirect lighting error model, the dual-frequency correction network, and the indirect lighting effect simulation method. It offers the following advantages:

[0028] 1. In the indirect illumination error model, a simple model of the phase error caused by indirect illumination in the phase shift method is derived, successfully reducing the problem from a global illumination problem to a single-pixel error problem.

[0029] 2. In the dual-frequency correction network, based on the conclusions of the error model, a simple MLP structure is used instead of a complex CNN structure. The network input is the value of the same pixel on two sets of phase patterns with different periods.

[0030] 3. In the indirect lighting simulation method, a phase shift pattern is first designed and the true phase is calculated on a computer. Then, the indirect lighting effect is added and the affected phase is calculated. This is combined with the training set required to build the network.

[0031] It not only provides an indirect illumination error model, but also successfully corrects the phase error. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the entire process of invention.

[0033] Figure 2 It is the structural diagram of the dual-frequency correction network.

[0034] Figure 3 (a) is a simulated stripe pattern contaminated by indirect lighting.

[0035] Figure 3 (b) Yes Figure 3 (a) The corresponding phase error.

[0036] Figure 4 (a) is the average intensity of bananas.

[0037] Figure 4 (b) is the intensity modulation of banana.

[0038] Figure 4 (c) is a fringe pattern projected onto a banana with a period of 8 pixels.

[0039] Figure 4 (d) is a fringe pattern projected onto a banana with a period of 9 pixels.

[0040] Figure 5(a) is the phase Φ of the banana before correction b1 error.

[0041] Figure 5 (b) is the phase Φ of the banana after being optimized by the dual-frequency correction network b error.

[0042] Figure 6 (a) is the average strength of jade in complex environment.

[0043] Figure 6 (b) is a fringe pattern projected onto the jade, with a period of 8 pixels.

[0044] Figure 6 (c) is the intensity modulation of jade.

[0045] Figure 7 (a) is the phase Φ of the jade before correction j1 roughness.

[0046] Figure 7 (b) is the phase Φ of the jade after being optimized by the dual-frequency correction network j roughness. DETAILED DESCRIPTION

[0047] The indirect illumination error model and dual-frequency correction network of the present invention achieve three-dimensional reconstruction based on the FPP system. The system consists of a computer, a projector (DLP LightCrafter 4500; Texas Instruments, USA), and a camera (acA800-510um; Basler Vision Technology GmbH, Germany). The projector has a resolution of 912×1140 pixels and a maximum projection rate of 120Hz in 8-bit mode. The camera has a resolution of 800×600 pixels and a maximum frame rate of 393fps in normal sensor readout mode. The system is 0.4 to 0.5 meters away from the object to be measured, and the angle between the projector optical axis and the camera optical axis is approximately 25 degrees.

[0048] The present invention will be further described below with reference to the accompanying drawings.

[0049] Example 1:

[0050] The first experiment aimed to verify the effectiveness of the dual-frequency correction network's phase correction and test its accuracy. The test set consisted of 179,569 pixels of data collected from real-world measurements. Two phase-shift patterns with different frequencies were projected onto a banana. Both patterns were green, with periods of 8 and 9 pixels, respectively. Existing research has shown that blue light produces lower measurement errors than green light. The object was again measured using a phase-shift pattern with an 8-pixel period, with the projected light set to blue. The projected patterns remained unchanged except for their color.

[0051] The captured fringe pattern is as follows Figure 4 As shown in (a). Figure 4 The contrast of the object in (b) is extremely low, which leads to serious phase error. First, the period of Φ is 8 pixels. b1 and a period of 9 pixels Φ b2 Calculated from the pattern projected by green light, the period of Φ is 8 pixels b 'Calculated by the pattern projected by blue light. Then, as in the training set, Φ b1 and Φ b2 After regularization, it is input into the dual-frequency correction network. Finally, the corrected phase Φ is obtained b . With Φ b 'In comparison, the calculated mean absolute error and RMSE are 0.0746 rad and 0.0991 rad respectively. Figure 5 This is the error between the phases before and after correction. The mean absolute error decreased from 0.0987 rad to 0.0746 rad, a decrease of approximately 24.4%. The RMSE decreased from 0.1256 rad to 0.0991 rad, a decrease of approximately 21.1%. As can be seen, the error is significantly reduced after optimization, and the error at the edges is also suppressed, proving that TPN is not a simple filtering method.

[0052] Example 2:

[0053] Example 2 tests the performance of the present invention in a complex lighting environment and analyzes its optimization results for point clouds. The experimental measurement object is a translucent jade, which is placed in a transparent storage box with an opaque label attached to it. Figure 6 As shown in (a), as in the first experiment, two sets of fringe patterns are projected to achieve dual-frequency correction. Figure 6 (b) shows one of the fringe patterns. The contrast of the fringe patterns on the jade is significantly lower than that on the label. Because the storage box is transparent and smooth, this creates a complex lighting environment. The jade is affected not only by subsurface scattering but also by various other indirect lighting sources, such as specular and volumetric scattering from the storage box, as well as diffuse reflections from the label.

[0054] The intensity modulation of jade is calculated using a stripe pattern with a period of 8 pixels, as Figure 6 (c) shows a clear distinction between the jade and the label. The contrast of the translucent jade, or intensity modulation, is approximately 7. This is much lower than the contrast of the opaque label, which is approximately 60. Due to the extremely low intensity modulation, the error is so large that the blue light is no longer suitable as the ground truth. Figure 6 (c) The contrast of the jade in the middle is higher, with an average of 12, and it is also a plane. Therefore, in this experiment, only the results of the plane part are retained to calculate the standard deviation of the fitted plane. In addition, Cloudcompare is used to calculate the roughness of the point cloud before and after correction, as shown in Figure 7 As shown in Figure 2, a plane fitting was performed on 30,580 points. After correction by the dual-frequency correction network, the standard deviation of the fitted plane decreased from 0.332 mm to 0.219 mm, a 34.04% reduction in error. These results demonstrate that TPN can still perform effectively even in complex indirect lighting environments.

Claims

1. A fast and accurate correction method for indirect lighting, characterized in that: The following steps are involved: Step 1: Based on the basic formula of the phase shift method, the phase error caused by indirect lighting is derived into a simple indirect lighting error model, successfully constraining the global error problem to a single-pixel error problem. Step 2: Design a dual-frequency correction network whose input is only the phase value of a single pixel under different projection strategies; Step 3: Design a method to simulate the effects of indirect lighting and simulate two sets of patterns affected by indirect lighting as training data for the multi-layer perceptron. Step 4: Use a projector to project the required 2×N phase-shifted fringe patterns and M coding patterns onto the object with complex surface texture, and use a camera to capture the projected patterns. Step 5: Calculate the absolute phase using the phase-shift fringe pattern acquired in step 4, and convert it back into the corrected phase using a dual-frequency correction network. First, the phase error caused by indirect illumination when using the phase shift method to achieve three-dimensional measurement is derived. The formula is: Where Φ(x,y) is the phase value corresponding to the projected fringe image I at the xth row and yth column, obtained by the phase shift method; ΔΦ(x,y) is the phase error caused by indirect illumination; Δθ is the phase represented by a single pixel on the projected pattern, which is related to the fringe period p; T i (x,y) is the indirect lighting transfer coefficient around the pixel (x,y); the relevant formula is as follows: Φ(x,y)=2πx / p Δθ=2π / p The error model transformation method is: when (x, y) is fixed, ΔΦ(x, y) is only related to Δθ; since Δθ>0, after odd-number extension of ΔΦ(x, y), Fourier expansion is carried out into a sine series, which is the error model of indirect lighting: Among them, β j is the coefficient of the jth term of the error model, which is the indirect lighting coefficient; The specific method of problem constraint in step 1 is: when (x, y) is fixed, the indirect lighting coefficient β j is a constant, and the error is only related to Δθ and has nothing to do with the surrounding pixels; that is, the global illumination problem is limited to a single pixel; The specific method for designing the dual-frequency correction network input in step 2 is: providing the phase value obtained by projecting fringes of different periods for the same pixel, the indirect illumination coefficient can be calculated to correct the error; therefore, the network structure input is two nodes, namely the absolute phase Φ1 with period p1 and the absolute phase Φ2 with period p2; the absolute phases Φ1 and Φ2 are the absolute phases calculated for the same pixel under different projection strategies, p1 ≠ p2; The specific method of simulating two sets of patterns affected by indirect lighting as training data for the multilayer perceptron in step 3 is as follows: generate two sets of phase-shift patterns with different periods, and set the calculated absolute phases as true values; use filtering and randomly generated reflectivity to simulate indirect lighting to contaminate the phase-shift patterns, and then calculate the two absolute phases after contamination as training data.

2. A fast and accurate correction method for indirect lighting according to claim 1, characterized in that: The specific method of designing the projection pattern in step 4 is: designing an N-step phase-shift pattern with a period of p1 and a corresponding M-step Gray code pattern, and then designing an N-step phase-shift pattern with a period of p2.

3. The method for rapid and accurate correction of indirect lighting according to claim 1, characterized in that: The specific method for calculating the phase in step 5 is: using the N-step phase shift pattern with a period of p1 and the corresponding M-step Gray code pattern to calculate the absolute phase Φ1; Then, based on the frequency doubling method, the absolute phase Φ2 is calculated using Φ1 and an N-step phase shift pattern with a period of p2.

4. A fast and accurate correction method for indirect lighting according to claim 1, characterized in that: The specific method for repairing the phase in the dual-frequency correction network in step 5 is as follows: Φ1(x, y) and Φ2(x, y) are input into the network, and the coordinates of the output phase Φ are still set to (x, y); the period of the corrected phase Φ is the same as Φ1: The corrected phase can greatly suppress the influence of indirect illumination, and can also play a role in defocusing and intensity diffusion; accurate three-dimensional reconstruction can be achieved by simply projecting two sets of phase-shifted fringe patterns with different periods.

Images