Deep learning fringe projection contour acquisition method based on geometric constraint
Through a deep learning method based on geometric constraints, the prediction of intermediate variables of the stripe pattern using a dual convolutional neural network is solved, and the phase calculation accuracy reduction caused by spectral aliasing in the prior art is achieved, and a high-precision single-frame three-dimensional measurement is achieved.
Patent Information
- Application Number
- CN202411552016.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2025-02-14
AI Technical Summary
When existing stripe projection contours deal with complex textures or highly jump scenes, the spectrum aliasing problem leads to a decrease in phase calculation accuracy, making it difficult to achieve high-precision single-frame three-dimensional measurements.
Deep learning method based on geometric constraints is adopted to predict the intermediate variables corresponding to the single-frame fringe mode through a dual convolutional neural network to restore high-quality absolute phases, thereby achieving high-precision single-frame three-dimensional measurements.
It realizes high-precision three-dimensional measurements that require only projection and acquisition of a single frequency, and is suitable for high-precision three-dimensional measurements without motion artifacts in dynamic measurement tasks.
Smart Images
Figure CN119444992A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of measurement technology, and in particular to a deep learning fringe projection profilometry based on geometric constraints. Background Art
[0002] Fringe Projection Profilometry (FPP) is a popular 3D measurement technology that obtains the 3D shape of an object by analyzing the fringe pattern projected onto the surface of the object. This technology is widely used in many fields such as industrial design, cultural relic restoration and entertainment due to its full-field measurement capability, non-contact operation and high precision.
[0003] In FPP, fringe analysis algorithm and phase unwrapping algorithm are key steps. The fringe analysis algorithm is responsible for extracting phase information from the fringe image captured by the camera, while the phase unwrapping algorithm is used to resolve the ambiguity of phase data to obtain accurate three-dimensional information. Traditional fringe analysis algorithms are mainly divided into phase shift method (PSA) and transformation method. PSA calculates the phase by collecting multiple fringe images with different phase shift amounts. Although this method has high accuracy, it requires multiple images, resulting in low measurement efficiency. The transformation method calculates the phase by analyzing a single-frame fringe image in the frequency domain. It is suitable for dynamic measurement, but when dealing with complex textures or highly jumpy scenes, the spectrum aliasing problem will cause the phase calculation accuracy to decrease.
[0004] No matter which fringe analysis algorithm is used, due to the use of the inverse tangent function in the complex amplitude field, there is always ambiguity in the phase information, that is, the wrapped phase. The phase unwrapping algorithm solves this problem by calculating the fringe order of the wrapped phase. The phase unwrapping algorithm can be divided into time domain phase unwrapping (TPU) and spatial domain phase unwrapping (SPU). TPU eliminates phase ambiguity pixel by pixel by analyzing the wrapped phase images of different frequencies, but this method requires additional fringe images, which reduces the measurement efficiency. SPU eliminates ambiguity by analyzing the wrapped phase information of multiple pixels in the spatial domain, but it requires the measurement scene to meet the continuity assumption, which is not always feasible in practical applications.
[0005] The physical model of FPP often requires a trade-off between measurement accuracy and speed in practical applications. In order to improve accuracy, multiple images are usually required, which naturally leads to a decrease in measurement speed. Color multiplexing technology can increase image acquisition speed, but this requires the use of a color camera and has low measurement accuracy for colored objects. In some dynamic measurement tasks, in order to reduce motion errors, researchers may reduce the number of images required to be collected. Ideally, researchers hope to obtain three-dimensional information with only one fringe image, but due to the limitations of the physical model of FPP, it is difficult to achieve high-precision single-shot measurement for complex scenes. Summary of the invention
[0006] In order to address the deficiencies in the prior art, the present invention provides a deep learning fringe projection profile acquisition method based on geometric constraints. By fully considering the geometric constraints of the measurement system and designing appropriate intermediate variables based on the physical model of FPP, it is proposed to use a dual convolutional neural network to predict the intermediate variables corresponding to a single-frame fringe pattern to restore the high-quality absolute phase, thereby achieving high-precision single-frame three-dimensional measurement.
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] A method for acquiring fringe projection profiles by deep learning based on geometric constraints comprises the following steps:
[0009] S1. Generate the reference phase corresponding to the measurement system
[0010] S2, calculate the data set required for training the dual convolutional neural network;
[0011] S3, training dual convolutional neural network model;
[0012] S4, using dual convolutional neural network model to predict intermediate variables M, D, To calculate the absolute phase This enables phase-height mapping to obtain three-dimensional information.
[0013] Furthermore, the reference phase corresponding to the measurement system is generated in S1 Use the following method:
[0014] S11. Establish a measurement system including an industrial camera and a projector, and obtain calibration parameters of the measurement system through system calibration;
[0015] S12. After the system calibration is completed, the camera pixel coordinates (x, y), the projector horizontal pixel coordinates x p , 3D coordinates (x w ,y w , z w ) form the following relationship:
[0016]
[0017] In formula (1), p is the calibration parameter, where the upper right superscripts c and p represent the camera and projector respectively, and the lower right subscript represents the position of the calibration parameter in its projection matrix;
[0018] S13, by defining the first calibration position as the zero depth plane, i.e. z w = 0, then the projector horizontal pixel coordinate x p It can be calculated by formula (2):
[0019]
[0020] S14, reference phase It can be calculated by formula (3):
[0021]
[0022] Where f is the fringe frequency and W is the width of the projector imaging plane.
[0023] Furthermore, the data set required for training the dual convolutional neural network is calculated in S2 using the following method:
[0024] S21, the measurement system is used to collect 360 sets of 3-frequency 12-step phase-shifted fringe pattern sequences corresponding to different measurement scenarios, and the fringe frequencies are f h =64,f m =8,f l =1;
[0025] S22, using the 12-step phase shift method shown in formula (4) and the three-frequency layered time domain phase unwrapping method shown in formula (5), the frequency f is calculated h The corresponding sinusoidal part M h , cosine part D h With absolute phase
[0026]
[0027] In formula (4), I is the fringe pattern, n is the phase shift index, and its value range is n∈[0, 11];
[0028] S23, Phase difference It can be calculated by formula (6):
[0029]
[0030] Furthermore, the dual convolutional neural network is trained in S3 using the following method:
[0031] S31. Use PyTorch to build two convolutional neural networks with the structure of Unet, named CNN1 and CNN2 respectively. CNN1 is used to predict M and D corresponding to the single-frame stripe pattern, and CNN2 is used to predict the corresponding
[0032] S32, the M in the data set calculated in S22 h With D h As the label value of CNN1, As the label value of CNN2, I h 0As the input value of CNN1 and CNN2;
[0033] S33, the data set is divided into a training set and a validation set in a ratio of 5:1. The training set is used to update the network weights, and the validation set is used to monitor the network training process and serve as an important criterion for selecting network weights;
[0034] S34, the training process uses a small batch gradient descent algorithm, batch size = 2, the convolution kernel parameters are initialized using the kaiming_normal method, the mean square error is used to calculate the loss between the network output and the label value to learn the network parameters, and the mean absolute error is used to evaluate the generalization performance of the network under the validation set;
[0035] S35, the initial learning rate is set to γ = 1 × 10 -4 , using the adaptive moment estimation algorithm to dynamically optimize the learning rate;
[0036] S36. During the training process, if the MAE calculated on the validation set does not decrease for 8 consecutive iterations, the learning rate will decay to 0.5 times the previous value; gradient clipping is used to enhance the stability of network training. CNN1 and CNN2 stop training after 300 iterations, and the network weight corresponding to the minimum MAE calculated on the validation set during training is selected as the final trained network model.
[0037] Furthermore, a dual convolutional neural network is used in S4 to predict the intermediate variables M, D, To calculate the absolute phase Used to achieve phase-height mapping to obtain three-dimensional information, specifically including:
[0038] S41, the frequency of the measurement system collecting unseen scenes is f h Single frame stripe pattern I h 0 , and input them into the trained CNN1 and CNN2 respectively. CNN1 outputs M and D, and the wrapped phase φ is calculated using formula (8):
[0039]
[0040] S42, CNN2 output The absolute phase It can be calculated by formula (9):
[0041]
[0042] In S2, the mask is calculated using formula (7) to filter out the area in the data where the phase calculation error is caused by the low fringe modulation:
[0043]
[0044] In formula (7), Tr is the mask threshold.
[0045] Compared with the prior art, the beneficial effects of the present invention are:
[0046] 1. The present invention only needs to project and collect a single frequency fringe pattern to achieve high-precision three-dimensional measurement of the measured scene;
[0047] 2. The present invention can be effectively used in dynamic measurement tasks to achieve high-precision three-dimensional measurement without motion artifacts. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 This is the implementation flow chart of the proposed method;
[0049] Figure 2 The calculation flow chart of the data set;
[0050] Figure 3 This is the loss curve of the dual convolutional neural network training process;
[0051] Figure 4 The intermediate variables predicted by the tested scene and the dual convolutional neural network;
[0052] Figure 5 This is the three-dimensional measurement result after phase-height mapping. DETAILED DESCRIPTION
[0053] In conjunction with specific embodiments, the present invention will be further described. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the application.
[0054] The present invention provides a deep learning fringe projection profile acquisition method based on geometric constraints. It only needs to project and collect a single frequency fringe pattern to achieve high-precision three-dimensional measurement of the measured scene; and it is effectively used in dynamic measurement tasks to achieve high-precision three-dimensional measurement without motion artifacts. The specific method is as follows.
[0055] S1. Generate the reference phase corresponding to the measurement system
[0056] Firstly, a grayscale fringe pattern without phase shift is projected onto the object to be measured, a measurement system including an industrial camera and a projector is established, and the calibration parameters of the measurement system are obtained through system calibration.
[0057] After the system calibration is completed, the camera pixel coordinates (x, y), the projector horizontal pixel coordinates x p , 3D coordinates (x w ,y w , z w ) form the following relationship:
[0058]
[0059] In formula (1), p is the calibration parameter, where the upper right superscripts c and p represent the camera and projector respectively, and the lower right subscript represents the position of the calibration parameter in its projection matrix;
[0060] By defining the first calibration position as the zero depth plane, i.e. z w = 0, then the projector horizontal pixel coordinate x p It can be calculated by formula (2):
[0061]
[0062] Reference Phase It can be calculated by formula (3):
[0063]
[0064] Where f is the fringe frequency and W is the width of the projector imaging plane.
[0065] S2. Calculate the dataset required for training dual convolutional neural networks
[0066] In order to enable the dual convolutional neural network to learn the ability to predict intermediate variables, a measurement system is used to collect 3-frequency 12-step phase-shifted fringe pattern sequences corresponding to multiple groups of different measurement scenarios, whose fringe frequencies are f h =64,f m =8,f l =1; the number of groups is positively correlated with the measurement accuracy of the proposed method, so in this embodiment, it is preferred to collect 360 groups of 3-frequency 12-step phase-shifted fringe pattern sequences corresponding to different measurement scenarios.
[0067] Figure 2 The calculation process of the data set is shown. The 12-step phase shift method shown in formula (4) and the three-frequency layered time domain phase unwrapping method shown in formula (5) are used to calculate the frequency f h The corresponding sinusoidal part M h , cosine part D h With absolute phase
[0068]
[0069] In formula (4), I is the fringe pattern, n is the phase shift index, and its value range is n∈[0, 11];
[0070] S23, Phase difference It can be calculated by formula (6):
[0071]
[0072] In order to ensure the stability of network training and improve performance, the mask is calculated using formula (7) to filter out the area in the data where the phase calculation error is caused by the low fringe modulation:
[0073]
[0074] In formula (7), Tr is the mask threshold.
[0075] S3. Training dual convolutional neural network model
[0076] PyTorch was used to build two convolutional neural networks with the Unet structure, named CNN1 and CNN2 respectively. CNN1 was used to predict the M and D corresponding to the single-frame stripe pattern, and CNN2 was used to predict the M and D corresponding to the single-frame stripe pattern.
[0077] The M in the calculated data set h With D h As the label value of CNN1, As the label value of CNN2, I h 0 As the input value of CNN1 and CNN2;
[0078] The data set is divided into a training set and a validation set in a ratio of 5:1. The training set is used to update the network weights, and the validation set is used to monitor the network training process and serve as an important criterion for selecting network weights.
[0079] The training process uses a small batch gradient descent algorithm with batch size = 2. The kaiming_normal method is used to initialize the convolution kernel parameters. The mean square error is used to calculate the loss between the network output and the label value to learn the network parameters. The mean absolute error is used to evaluate the generalization performance of the network under the validation set.
[0080] The initial learning rate is set to γ = 1 × 10 -4 , using the adaptive moment estimation algorithm to dynamically optimize the learning rate;
[0081] During the training process, if the MAE calculated on the validation set does not decrease for 8 consecutive iterations, the learning rate will decay to 0.5 times the previous value; gradient clipping is used to enhance the stability of network training. CNN1 and CNN2 stop training after 300 iterations, and the network weights corresponding to the minimum MAE calculated on the validation set during training are selected as the final trained network model.
[0082] S4, using dual convolutional neural network model to predict intermediate variables M, D, To calculate the absolute phase Phase-height mapping is then used to obtain three-dimensional information.
[0083] The frequency at which the measurement system collects unseen scenes is f h Single frame stripe pattern I h 0 , and input them into the trained CNN1 and CNN2 respectively. CNN1 outputs M and D, and the wrapped phase φ is calculated using formula (8):
[0084]
[0085] CNN2 Output The absolute phase It can be calculated by formula (9):
[0086]
[0087] Example 1: In order to demonstrate the performance of the method of the present invention, three tested scenarios were selected: Figure 4 (a) shows a single frame of the fringe pattern of the tested scene. Figure 4 (b) is the M, D, obtained by the proposed method. And the calculated φ and Figure 5 (a, e, i) show the three-dimensional morphology obtained by Fourier transform profilometry combined with the phase unwrapping method of the proposed method. Figure 5 (b, f, j) show the 3D shape obtained using an end-to-end method based on deep learning. Figure 5 (c, g, k) show the three-dimensional morphology obtained by the proposed method. Figure 5 (d, h, l) show the three-dimensional morphology obtained by using the 12-step phase shift method combined with time-domain phase unwrapping. Figure 5 (a, e, i) It can be seen that although effective phase unwrapping is achieved, the severe spectrum aliasing effect makes it difficult for Fourier transform profilometry to achieve high-quality fringe analysis, resulting in obvious irregular errors in the reconstructed surface, especially in the edge area. Figure 5(b,f,j) It can be seen that although single-frame measurement is achieved, the resolution of the end-to-end method is poor and the reconstructed surface ripples are obvious. This is because the end-to-end method relies entirely on the mapping of patterns to heights learned from a large amount of data, without utilizing the physical model and prior knowledge that controls the formation of height information. In contrast, the proposed method ( Figure 5 (c, g, k)) can achieve high-resolution single-shot 3D measurement, and the reconstruction results of the 12-step phase shift method combined with time domain phase unwrapping ( Figure 5 (d,h,l)) are equivalent.
Claims
1. A method for acquiring fringe projection profiles based on deep learning of geometric constraints, characterized in that: The following steps are involved: S1. Generate the reference phase corresponding to the measurement system S2, calculate the data set required for training the dual convolutional neural network; S3, training dual convolutional neural network model; S4, using the dual convolutional neural network model to predict the sine part M, cosine part D, phase difference To calculate the absolute phase This enables phase-height mapping to obtain three-dimensional information.
2. According to the method for acquiring fringe projection profile based on deep learning of geometric constraints in claim 1, it is characterized in that: S1 generates the reference phase corresponding to the measurement system Use the following method: S11. Establish a measurement system including an industrial camera and a projector, and obtain calibration parameters of the measurement system through system calibration; S12. After the system calibration is completed, the camera pixel coordinates (x, y), the projector horizontal pixel coordinates x p , 3D coordinates (x w ,y w , z w ) form the following relationship: In formula (1), p is the calibration parameter, where the upper right superscripts c and p represent the camera and projector respectively, and the lower right subscript represents the position of the calibration parameter in its projection matrix; S13, by defining the first calibration position as the zero depth plane, i.e. z w = 0, then the projector horizontal pixel coordinate x p It can be calculated by formula (2): S14, reference phase It can be calculated by formula (3): Where f is the fringe frequency and W is the width of the projector imaging plane.
3. The method for acquiring fringe projection profile based on deep learning of geometric constraints according to claim 1, characterized in that: In S2, the dataset required for training the dual convolutional neural network is calculated using the following method: S21, the measurement system is used to collect multiple sets of () 3-frequency 12-step phase-shifted fringe pattern sequences corresponding to different measurement scenarios, and the fringe frequencies are f h =64,f m =8, f l =1; S22, using the 12-step phase shift method shown in formula (4) and the three-frequency layered time domain phase unwrapping method shown in formula (5), the frequency f is calculated h The corresponding sinusoidal part M h , cosine part D h With absolute phase In formula (4), I is the fringe pattern, n is the phase shift index, and its value range is n∈[0, 11]. is the absolute phase, φ is the wrapped phase, and the subscripts h, m, and l represent the frequencies f respectively. h , f m , f l The corresponding variable, round is the rounding function; S23, Phase difference It can be calculated by formula (6):
4. The method for acquiring fringe projection profile based on deep learning of geometric constraints according to claim 3 is characterized in that: The dual convolutional neural network is trained in S3 using the following method: S31. Use PyTorch to build two convolutional neural networks with the Unet structure, named CNN1 and CNN2 respectively. CNN1 is used to predict the M and D corresponding to the single-frame stripe pattern, and CNN2 is used to predict the corresponding S32, the M in the data set calculated in S22 h With D h As the label value of CNN1, As the label value of CNN2, I h 0 As the input value of CNN1 and CNN2; S33, the data set is divided into a training set and a validation set in a ratio of 5:
1. The training set is used to update the network weights, and the validation set is used to monitor the network training process and serve as an important criterion for selecting network weights; S34, the training process uses a small batch gradient descent algorithm, batch size = 2, the convolution kernel parameters are initialized using the kaiming_normal method, the mean square error is used to calculate the loss between the network output and the label value to learn the network parameters, and the mean absolute error is used to evaluate the generalization performance of the network under the validation set; S35, the initial learning rate is set to γ = 1 × 10 -4 , using the adaptive moment estimation algorithm to dynamically optimize the learning rate; S36. During the training process, if the MAE calculated on the validation set does not decrease for 8 consecutive iterations, the learning rate will decay to 0.5 times the previous value; gradient clipping is used to enhance the stability of network training. CNN1 and CNN2 stop training after 300 iterations, and the network weight corresponding to the minimum MAE calculated on the validation set during training is selected as the final trained network model.
5. The method for acquiring fringe projection profile based on deep learning of geometric constraints according to claim 1, characterized in that: In S4, a dual convolutional neural network is used to predict the intermediate variables M, D, To calculate the absolute phase Used to achieve phase-height mapping to obtain three-dimensional information, specifically including: S41, the frequency of the measurement system collecting unseen scenes is f h Single frame stripe pattern I h 0 , and input them into the trained CNN1 and CNN2 respectively. CNN1 outputs the sine part M and the cosine part D. Formula (8) is used to calculate the wrapped phase φ: S42, CNN2 output The absolute phase It can be calculated by formula (9):
6. The method for acquiring fringe projection profile based on deep learning of geometric constraints according to claim 1, characterized in that: In S2, the mask is calculated using formula (7) to filter out the area in the data where the phase calculation error is caused by the low fringe modulation: In formula (7), Tr is the mask threshold.