3D Facial Shape Measurement Method Based on Single-Frame Color Fringe Projection of Deep Learning

Abstract

The present invention discloses a three-dimensional surface shape measurement method based on single-frame color fringe projection of deep learning, including a model CNN based on a convolutional neural network. The input contains three channels, which are grayscale fringe images in the red, green, and blue channels of the color fringe image. Three 12-step phase-shifted fringes with different frequencies are projected by a projector, and the phase-shift (PS) method and the projection minimum distance method (PDM) are used to generate the training data required by the CNN for training. When in use, the grayscale fringe images of the three channels of the color fringe image are input into the CNN to obtain the numerator term, the denominator term, and a low-precision absolute phase containing fringe order information. The numerator term and the denominator term are substituted into the arctangent function, and the high-precision absolute phase information is calculated in combination with the low-precision absolute phase. The present invention can provide more accurate phase information and more reliable phase unwrapping without any complex pre- / post-processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and particularly relates to a three-dimensional surface shape measurement method based on single-frame color fringe projection of deep learning. Background Technique

[0002] Fringe projection profilometry (FPP) has become one of the most widely used three-dimensional (3D) measurement techniques due to its simple hardware facilities, flexible implementation methods, and high measurement accuracy. In recent years, with the increasing requirements for 3D information acquisition in high-speed scenarios in applications such as online quality inspection and rapid reverse engineering, the high-speed 3D shape measurement technology based on FPP has become crucial (Robust dynamic 3-d measurements with motion-compensated phase-shifting profilometry, by author S Feng, etc.).

[0003] To achieve 3D imaging in high-speed scenarios, it is necessary to improve the measurement efficiency and reduce the number of fringe patterns required for a single 3D reconstruction. The ideal method is to recover the high-quality 3D absolute surface of an object from a single image. Color-coded projection technology (Review of single-shot 3d shape measurement by phase calculation-based fringe projection techniques, author Z Zhang) has great advantages in dynamic scene measurement because this technology can encode three independent fringe images in the red, blue, and green channels, and thus the imaging efficiency is increased by 2 times compared with the traditional monochromatic projection method. To make full use of the color image channels, many single-frame color-coded projection techniques have been proposed by scholars (Composite phase-shifting algorithm for three-dimensional shape compression, author N Karpinsky, etc.). However, these techniques are rarely applicable to the high-precision measurement of complex objects. On the one hand, to obtain high-precision phase information, the phase-shifting (PS) method with high measurement resolution (Phaseshifting algorithms for fringe projection profilometry: A review, author C Zuo, etc.) should be preferred. However, the PS method requires at least three fringe images, and these fringe images occupy all channels of the RGB image. Therefore, phase ambiguity can only be eliminated by the spatial phase unwrapping method (which will fail to unwrap when encountering isolated phases) (Color-encoded digital fringe projection technique for high-speed 3-d surface contouring, author P S Huang, etc.). On the other hand, to achieve stable phase unwrapping, a strategy of combining fringe patterns with Gray codes or multi-frequency fringe images is usually adopted. The former still cannot stably unwrap the phase because it is difficult to identify the edges of the Gray code patterns (Projected fringe profilometry using the area-encoded algorithm for spatially isolated and dynamic objects, author W H Su).The latter can recover the absolute phase through the three - fringe number selection method (Optical imaging of physical objects, by D Towers et al.), but due to the use of the Fourier Transform (FT) method (a single - frame imaging method, but the quality of this method is poor in discontinuous or isolated regions of the phase diagram), the phase accuracy is poor. In addition, the color - coded projection method also has some inherent defects, such as chromatic aberration between channels and color crosstalk, which will affect the quality of phase calculation. Although researchers have proposed some pre - processing methods to compensate for these defects, they can only reduce the impact of these defects on the measurement to a certain extent.

[0004] From the above analysis, it can be seen that although the color - coded projection technology has great potential for single - frame three - dimensional measurement, the only three color channels are not sufficient to encode the fringe images that can both meet the acquisition of high - quality phase information and stable phase unwrapping. In addition, the inherent chromatic aberration and color crosstalk problems of this technology are also very difficult to solve by traditional methods. Summary of the Invention

[0005] The purpose of the present invention is to provide a three - dimensional surface measurement method based on deep - learning single - frame color fringe projection.

[0006] The technical solution to achieve the purpose of the present invention is: a phase unwrapping method for single - frame color fringe projection based on deep learning, and the specific steps are as follows:

[0007] Step 1: Construct a Convolutional Neural Network - based model CNN;

[0008] Step 2: Generate CNN model training data and train the model CNN;

[0009] Step 3: Input the grayscale images in the three channels of the color composite fringe image of the object to be measured into the trained model CNN, obtain the numerator term, denominator term, and low - precision absolute phase, substitute the numerator term and denominator term into the arctangent function, and calculate the final absolute phase information in combination with the low - precision absolute phase.

[0010] Preferably, the model CNN includes five data - processing paths, connection layer 1, and convolutional layer 11, where:

[0011] The data - processing path 1 is set as: the input data sequentially passes through convolutional layer 1 and residual module 1, the data output from residual module 1 and the data output from convolutional layer 1 are input into convolutional layer 2 together, and the output data of convolutional layer 2 is input into connection layer 1;

[0012] The data processing path 2 is set as follows: the input data sequentially passes through a convolutional layer 3, a pooling layer 1, a residual module 2, and an upsampling layer 1. The data output by the upsampling layer 1 and the data output by the pooling layer 1 are input into a convolutional layer 4 together, and the data output by the convolutional layer 4 is input into a connection layer 1;

[0013] The data processing path 3 is set as follows: the input data sequentially passes through a convolutional layer 5, a pooling layer 2, a residual module 3, an upsampling layer 2, and an upsampling layer 3. The data output by the upsampling layer 3 and the data output by the pooling layer 2 are input into a convolutional layer 6 together, and the data output by the convolutional layer 6 is input into a connection layer 1;

[0014] The data processing path 4 is set as follows: the input data sequentially passes through a convolutional layer 7, a pooling layer 3, a residual module 4, an upsampling layer 4, an upsampling layer 5, and an upsampling layer 6. The data output by the upsampling layer 6 and the data output by the pooling layer 3 are input into a convolutional layer 8 together, and the data output by the convolutional layer 8 is input into a connection layer 1;

[0015] The data processing path 5 is set as follows: the input data sequentially passes through a convolutional layer 9, a pooling layer 4, a residual module 5, an upsampling layer 7, an upsampling layer 8, an upsampling layer 9, and an upsampling layer 10. The data output by the upsampling layer 10 and the data output by the pooling layer 4 are input into a convolutional layer 10 together, and the data output by the convolutional layer 10 is input into a connection layer 1;

[0016] The connection layer 1 is used to input the 5-way data into a convolutional layer 11 later to obtain a 3D tensor with 3 output channels.

[0017] Preferably, the pooling layer 1, the pooling layer 2, the pooling layer 3, the pooling layer 4, and the pooling layer 5 respectively perform downsampling of 1 / 2, 1 / 4, 1 / 8, 1 / 16 on the data.

[0018] Preferably, the specific method for generating CNN model training data is as follows:

[0019] Step 2.1: Use a projector to project 37 fringe images onto an object. The 37 fringe images include 12 green phase-shifted fringe images with a frequency of f R 12 green phase-shifted fringe images with a frequency of f 12 green phase-shifted fringe images with a frequency of f G 12 green phase-shifted fringe images with a frequency of f and 12 green phase-shifted fringe images with a frequency of f B 12 green phase-shifted fringe images with a frequency of f and 1 composite color fringe image I RGB whose red channel is a grayscale fringe image I with a frequency of f R a grayscale fringe image I with a frequency of f R whose green channel is a grayscale fringe image I with a frequency of f G a grayscale fringe image I with a frequency of f G and whose blue channel is a grayscale fringe image I with a frequency of f B a grayscale fringe image I with a frequency of fB ;

[0020] Step 2.2: Use a color camera to collect 37 fringe images modulated by the object and generate a set of input and output data required for training the CNN, specifically:

[0021] Step 2.2.1: For the first 36 green fringe images collected Use the phase-shifting (PS) method to obtain the wrapped phases with frequencies of f R , f G , f B ; obtain the absolute phase Φ

[0022] of frequency f G through the PDM method, and use the numerator term M G of frequency f G , the denominator term D G , and the absolute phase Φ G as a set of standard data for the model CNN. G

[0023] Step 2.2.2: Use the grayscale images I RGB in the three channels of the 37th composite color fringe image I collected R , I G , I B as a set of input data for the network CNN;

[0024] Step 2.3: Repeat Steps 2.1 and 2.2 to generate a set number of training data.

[0025] Preferably, the specific method for training the model CNN is:

[0026] Use the grayscale images I R , I G , I B in the three channels of the 37th composite color fringe image as the input data for the model CNN, and use the numerator term M G of frequency f G , the denominator term D G and the absolute phase Φ G as the standard data for the model CNN, calculate the difference between the standard data and the output value of the model CNN, and use the backpropagation method to iteratively optimize the internal parameters of the CNN until the loss function converges.

[0027] Preferably, substitute the numerator term and the denominator term into the arctangent function, and combine with the low-precision absolute phase to calculate the final absolute phase information, specifically:

[0028] Substitute the numerator term and the denominator term into the arctangent function to obtain the wrapped phase;

[0029] Combining the wrapped phase and the low-precision absolute phase to obtain the final absolute phase. The specific formula is as follows:

[0030]

[0031] In the formula, Round represents the rounding operation, and Φ G is the final absolute phase, is the wrapped phase, is the low-precision absolute phase output by the model CNN.

[0032] Compared with the prior art, the remarkable advantages of the present invention are as follows: (1) The present invention can simultaneously achieve high-precision phase information acquisition and stable phase unwrapping through a single color image; (2) The invention does not require any complex pre- / post-processing of the system and can automatically compensate for the color difference and color crosstalk problems between color channels.

[0033] The following further describes the present invention in detail with reference to the accompanying drawings. Description of the Drawings

[0034] Figure 1 is the flow chart of the present invention.

[0035] Figure 2 is the structure and schematic diagram of CNN.

[0036] Figure 3 is the result comparison diagram between the present invention and the traditional method. Detailed Embodiments

[0037] A three-dimensional surface measurement method based on single-frame color fringe projection of deep learning, which obtains high-precision absolute phase information through a single-frame color fringe image, includes the following steps:

[0038] Step 1: Construct a model CNN based on a convolutional neural network.

[0039] Specifically, the constructed model CNN is as Figure 2 shown, where H represents the height (pixels) of the image, W represents the width of the image, C represents the number of channels, and the number of channels is equal to the number of filters used. The input of the model CNN is a 3D tensor with three channels, and the output is also a 3D tensor with three channels. The model CNN includes five data processing paths, connection layer 1, and convolutional layer 11.

[0040] In a further embodiment, the data processing path 1 is set as follows: The input data sequentially passes through convolutional layer 1 and residual module 1, and the data output by residual module 1 and the data output by convolutional layer 1 are input into convolutional layer 2 together, and the output data of convolutional layer 2 is input into connection layer 1.

[0041] The data processing path 2 is set as follows: The input data sequentially passes through a convolutional layer 3, a pooling layer 1, a residual module 2, and an upsampling layer 1. The data output by the upsampling layer 1 and the data output by the pooling layer 1 are input into a convolutional layer 4 together, and the data output by the convolutional layer 4 is input into a connection layer 1.

[0042] The data processing path 3 is set as follows: The input data sequentially passes through a convolutional layer 5, a pooling layer 2, a residual module 3, an upsampling layer 2, and an upsampling layer 3. The data output by the upsampling layer 3 and the data output by the pooling layer 2 are input into a convolutional layer 6 together, and the data output by the convolutional layer 6 is input into a connection layer 1.

[0043] The data processing path 4 is set as follows: The input data sequentially passes through a convolutional layer 7, a pooling layer 3, a residual module 4, an upsampling layer 4, an upsampling layer 5, and an upsampling layer 6. The data output by the upsampling layer 6 and the data output by the pooling layer 3 are input into a convolutional layer 8 together, and the data output by the convolutional layer 8 is input into a connection layer 1.

[0044] The data processing path 5 is set as follows: The input data sequentially passes through a convolutional layer 9, a pooling layer 4, a residual module 5, an upsampling layer 7, an upsampling layer 8, an upsampling layer 9, and an upsampling layer 10. The data output by the upsampling layer 10 and the data output by the pooling layer 4 are input into a convolutional layer 10 together, and the data output by the convolutional layer 10 is input into a connection layer 1.

[0045] For the specific construction method of each residual module, refer to the literature "Deep residual learning for image recognition" by K He et al.

[0046] Specifically, the pooling layer 1, pooling layer 2, pooling layer 3, pooling layer 4, and pooling layer 5 respectively perform downsampling on the data by 1 / 2, 1 / 4, 1 / 8, 1 / 16 to improve the model's ability to recognize features while keeping the number of channels unchanged.

[0047] Specifically, the function of the upsampling layer 1 to the upsampling layer 10 is to perform upsampling on the resolution of the data, doubling the height and width of the data respectively, with the aim of restoring the original resolution of the image.

[0048] Subsequently, the connection layer 1 superimposes the five-way data. Finally, after passing through a convolutional layer 11, a 3D tensor with 3 output channels is output.

[0049] Step 2: Generate training data and train the model CNN. The specific steps are as follows:

[0050] Step 2.1: The projector projects 37 stripe images (including 36 monochromatic stripe images and one composite stripe image) onto the object.

[0051] Use a projector to project 37 fringe images onto an object. The 37 fringe images include 12 green phase-shifted fringe images with a frequency of f R 12 green phase-shifted fringe images with a frequency of f G and 12 green phase-shifted fringe images with a frequency of f B and 1 composite color fringe image I RGB whose red channel is a grayscale fringe image I with a frequency of f R R whose green channel is a grayscale fringe image I with a frequency of f G G whose blue channel is a grayscale fringe image I with a frequency of f B B .

[0052] Step 2.2: Use a color camera to collect the 37 fringe images modulated by the object and generate a set of input and output data required for training the CNN. Specifically:

[0053] Step 2.2.1. For the first 36 green fringe images collected Use the PS method to obtain the wrapped phases with frequencies of f R 、f G 、f B respectively

[0054]

[0055]

[0056]

[0057] where respectively represent the nth green fringe image with frequencies of f R 、f G 、f B , n = 1, 2,..., 12, M and D respectively represent the numerator term and denominator term of the arctangent function.

[0058] After obtaining the wrapped phases at three different frequencies then use the PDM method (Micro fouriertransform profilometry (mftp): 3d shape measurement at 10,000 frames per second, authors Zuo Chao, etc.) to obtain the absolute phase Φ G with a frequency of f G , and the absolute phase Φ G ​​​​​​There is no problem of color difference and color crosstalk between color channels because only monochromatic fringe images are used. The frequency f obtained from the above calculation G of the numerator term M G and the denominator term D G , as well as the absolute phase Φ G are used as a set of standard (ground truth) data for the CNN.

[0059] Step 2.2.2: For the 37th acquired composite color fringe image I RGB , the grayscale images I R , I G , I B in its three channels are used as a set of input data for the network CNN;

[0060] Step 2.3: Repeat Steps 2.1 and 2.2 to generate 1000 sets of training data.

[0061] Step 2.4: Train the CNN: The grayscale images I R , I G , I B in the three channels of the 37th composite color fringe image are used as input data, and M G , D G , Φ G are used as standard data and fed into the model CNN. The mean square error is used as the loss function to calculate the difference between the standard value and the output value of the CNN. Combining with the backpropagation method, the internal parameters of the CNN are repeatedly iteratively optimized until the loss function converges, at which point the training of the model CNN ends. During the training of the model, except for the convolutional layer 11, the activation function used in the remaining convolutional layers is the rectified linear unit (Relu). When iteratively optimizing the loss function, the Adam algorithm is used to find the minimum value of the loss function.

[0062] Step 3: Use the trained model CNN to perform three-dimensional measurement of the object to be measured, specifically as follows:

[0063] Step 3.1: Simultaneously obtain the information for calculating the high-precision wrapped phase and for unwrapping.

[0064] Input the grayscale images I R , I G , I B in the three channels of the color composite fringe image of the object to be measured into the trained CNN, and obtain the numerator term M G , the denominator term D G for calculating the high-precision wrapped phase information, and the low-precision absolute phase Φl G containing the fringe order information (whose error is between -π and π);

[0065] Step 3.2: Obtain the high-precision absolute phase

[0066] Step 3.2.1. According to the M obtained in Step 3.1 G and D G , obtain the high-precision wrapped phase through Formula (2) The reason why this strategy can provide high-precision phase information is that: predicting the structures of the numerator term and the denominator term corresponding to the arctangent function overcomes the difficulty of 2π phase wrapping in the reproduced wrapped phase.

[0067] Step 3.2.2. Obtain the high-precision absolute phase Φ through the following formula G :

[0068]

[0069] In the formula, Round represents the rounding operation.

[0070] After obtaining the absolute phase, three-dimensional reconstruction can be performed through the calibration parameters between the color camera and the projector (Calibration of fringe projection profilometry with bundle adjustment strategy, authors Peng Xiang, etc.).

[0071] The present invention only needs to project a single color fringe image to obtain the high-precision absolute phase, and then realizes the measurement of the three-dimensional surface shape of the object to be measured. The present invention first constructs a model based on a convolutional neural network. In the present invention, it is called CNN. The input of CNN contains three channels, and the three channels are the grayscale fringe images in the red, green, and blue channels of the color fringe image respectively, and the output data are the numerator term, the denominator term for calculating the high-precision phase information, and a low-precision absolute phase containing the fringe order information. During training, the projector projects 12-step phase-shifted fringes with three different frequencies, and the PS method and the projection minimum distance method (PDM) are used to generate the training data required by CNN. After the training is completed, the grayscale fringe images of the three channels of the color fringe image are input into CNN to obtain the numerator term, the denominator term for calculating the high-precision phase information, and a low-precision absolute phase containing the fringe order information. Substitute the numerator term and the denominator term into the arctangent function, combine with the low-precision absolute phase to calculate the high-precision absolute phase information, and finally perform three-dimensional reconstruction.

[0072] Example:

[0073] To verify the effectiveness of the present invention, a digital grating projection device was constructed based on a color camera (model acA640-750uc, Basler, resolution 640×480), a projector (model LightCrafter 4500, TI, resolution 912×1140), and a computer to collect color fringe images. The H, W, and C of the constructed CNN are 480, 640, and 64, and the three fringe frequencies f R , f G , f B are 9, 11, and 13 respectively. When training the data, a total of 1000 groups of data were collected. During the training process, 800 groups of data were used for training, and the remaining 200 groups of data were used for verification. After the training was completed, to verify the effectiveness of the present invention, 2 scenes that were not seen during training were selected for testing. To demonstrate the advantages of the present invention, the present invention was compared with a traditional color fringe coding method (Snapshot color fringe projection for absolute three-dimensional metrology of video sequences, authors Zhang Zonghua, etc.), and the results of the monochromatic 12-step PS method and PDM were selected as the benchmark results. Figure 3 The measurement results are shown. Among them, 3(a) and 3(e) are the corresponding composite color images of the two scenes, 3(b) and 3(f) are the results measured by the traditional color fringe coding method, 3(c) and 3(g) are the results of the present method, and 3(d) and 3(h) are the benchmark results. It can be seen from the results that the present invention can obtain more accurate absolute phase reconstruction, and the final three-dimensional reconstruction quality can even be comparable to the results obtained by the PS method and the PDM method. At the same time, it should be pointed out that the present invention only uses 1 color composite fringe image, while the method used for the benchmark results uses 36 fringe images.

Claims

1. A three-dimensional surface measurement method based on single-frame color fringe projection of deep learning, characterized in that, the specific steps are as follows: Step 1: Construct a model CNN based on a convolutional neural network. The model CNN includes five data processing paths, a connection layer 1, and a convolutional layer 11, where: The data processing path 1 is set as follows: The input data sequentially passes through a convolutional layer 1 and a residual module 1. The data output from the residual module 1 and the data output from the convolutional layer 1 are input into a convolutional layer 2 together. The output data of the convolutional layer 2 is input into the connection layer 1; The data processing path 2 is set as follows: The input data sequentially passes through a convolutional layer 3, a pooling layer 1, a residual module 2, and an upsampling layer 1. The data output from the upsampling layer 1 and the data output from the pooling layer 1 are input into a convolutional layer 4 together. The output data of the convolutional layer 4 is input into the connection layer 1; The data processing path 3 is set as follows: The input data sequentially passes through a convolutional layer 5, a pooling layer 2, a residual module 3, an upsampling layer 2, and an upsampling layer 3. The data output from the upsampling layer 3 and the data output from the pooling layer 2 are input into a convolutional layer 6 together. The output data of the convolutional layer 6 is input into the connection layer 1; The data processing path 4 is set as follows: The input data sequentially passes through a convolutional layer 7, a pooling layer 3, a residual module 4, an upsampling layer 4, an upsampling layer 5, and an upsampling layer 6. The data output from the upsampling layer 6 and the data output from the pooling layer 3 are input into a convolutional layer 8 together. The output data of the convolutional layer 8 is input into the connection layer 1; The data processing path 5 is set as follows: The input data sequentially passes through a convolutional layer 9, a pooling layer 4, a residual module 5, an upsampling layer 7, an upsampling layer 8, an upsampling layer 9, and an upsampling layer 10. The data output from the upsampling layer 10 and the data output from the pooling layer 4 are input into a convolutional layer 10 together. The output data of the convolutional layer 10 is input into the connection layer 1; The connection layer 1 is used to input the 5-way data and then input it into the convolutional layer 11 to obtain a 3D tensor with an output channel number of 3; Step 2: Generate CNN model training data and train the model CNN. The specific method for generating CNN model training data is as follows: Step 2.1: Project 37 fringe images onto the object using a projector. The 37 fringe images include 12 green phase-shifted fringe images with a frequency of f R 12 green phase-shifted fringe images with a frequency of f 12 green phase-shifted fringe images with a frequency of f G 12 green phase-shifted fringe images with a frequency of f and 12 green phase-shifted fringe images with a frequency of f B 12 green phase-shifted fringe images with a frequency of f as well as 1 composite color fringe image I RGB whose red channel is a grayscale fringe image I with a frequency of f R 16, the green channel is a grayscale fringe image I with a frequency of f R 18, and the blue channel is a grayscale fringe image I with a frequency of f G 20; G The blue channel is a grayscale fringe image I with a frequency of f B 24; B ; Step 2.2: Use a color camera to collect 37 fringe images modulated by an object and generate a set of input and output data required for training the CNN. Specifically: Step 2.2.

1. For the first 36 collected green stripe images respectively use the PS method to obtain the wrapped phases with frequencies of f R , f G , f B ​ Obtain the frequency f by the PDM method G of the absolute phase φ G , and use the numerator term M G of the frequency f G , the denominator term D G , and the absolute phase φ G as a set of standard data for the model CNN; Step 2.2.2: Use the 37th acquired composite color fringe image I RGB The grayscale images I R 、I G 、I B in the three channels as a set of input data for the network CNN; Step 2.3: Repeat steps 2.1 and 2.2 to generate a set number of training data; The specific method for training the model CNN is as follows: Take the grayscale images I R 、I G 、I B in the three channels of the 37th composite color stripe image as the input data of the model CNN, and take the numerator term M G 、the denominator term D G and the absolute phase φ G G as the standard data of the model CNN, calculate the difference between the standard data and the output value of the model CNN, and use the backpropagation method to iteratively optimize the internal parameters of the CNN until the loss function converges;​ Step 3: Input the grayscale images in the three channels of the color composite fringe image of the object to be measured into the trained model CNN to obtain the numerator term, the denominator term, and the low-precision absolute phase. Substitute the numerator term and the denominator term into the arctangent function, and combine the low-precision absolute phase to calculate the final absolute phase information.

2. The three-dimensional surface measurement method based on single-frame color fringe projection of deep learning according to claim 1, characterized in that, the pooling layer 1, the pooling layer 2, the pooling layer 3, the pooling layer 4, and the pooling layer 5 respectively perform downsampling of 1 / 2, 1 / 4, 1 / 8, 1 / 16 on the data.

3. The three-dimensional surface measurement method based on single-frame color fringe projection of deep learning according to claim 1, characterized in that, Substitute the numerator term and the denominator term into the arctangent function, and combine with the low-precision absolute phase calculation to obtain the final absolute phase information, specifically as follows: Substitute the numerator term and the denominator term into the arctangent function to obtain the wrapped phase; Combine the wrapped phase and the low-precision absolute phase to obtain the final absolute phase. The specific formula is as follows: where Round represents the rounding operation, φ G is the final absolute phase, is the wrapped phase, is the low-precision absolute phase output by the model CNN.

Images