A single-frame three-dimensional measurement method based on structured light and deep learning

Abstract

The present invention belongs to the technical field of structured light three-dimensional measurement, and specifically is a single-frame three-dimensional measurement method based on structured light and deep learning. The present invention performs fringe image analysis through a deep neural network obtained by unsupervised learning pre-training and supervised learning fine-tuning to achieve three-dimensional measurement; wherein: the unsupervised learning pre-training uses a single-frame fringe image collected by a structured light system as input, and the supervised learning uses the collected four-frequency 12-step phase-shifted fringe images as input, and the deep neural network used is composed of 3 U-Net modules, which are respectively used for phase calculation, unwrapping and tuning. The present invention completes single-frame three-dimensional measurement through a deep neural network, can be used to measure dynamic scenes, and has high measurement accuracy; the combination of unsupervised and supervised learning reduces the number of collected fringe images and improves the training efficiency of the deep learning method.

Description

Technical Field

[0001] The present invention belongs to the technical field of structured light three-dimensional measurement, and specifically is a single-frame three-dimensional measurement method based on structured light and deep learning. Background Art

[0002] 3D optical measurement technology is widely used in academia and industry due to its advantages of non-contact, high speed and high precision. Its application fields include biomedicine, computer vision, manufacturing, etc. 3D optical measurement technology is also used in commercial portable devices such as Intel RealSense and Microsoft Kinect. In recent years, due to the development of optical engineering and computer vision, 3D optical measurement technology has made great progress.

[0003] Traditional single-frame 3D measurement methods based on structured light use Fourier transform profilometry (FTP) or windowed Fourier transform profilometry (WFTP). These methods can recover phase images from a single fringe projection image through spatial phase modulation, but the accuracy and spatial resolution are low and the ability to resist noise is weak. After the deep learning algorithm was introduced into fringe pattern analysis, the accuracy of the single-frame 3D measurement method was improved and its application scope was expanded. The deep learning method uses network architectures such as U-Net and GAN based on convolutional neural networks to obtain a phase image with wrapping from a single fringe projection image, or directly obtain a depth map using an end-to-end network. Other work focuses on high dynamic range methods and solving the problem of the amount of data required for deep learning, and further develops single-frame 3D measurement methods. Unlike traditional machine learning methods, deep learning needs to be trained on a data set, and a large amount of real data values ​​are required to obtain better network performance. In the field of fringe projection, the real depth value needs to be obtained by a multi-step phase shift method, and it is very time-consuming to establish a large data set; and there is a difference in the distribution of simulated images and real fringe images. The network trained on simulated images will increase the error in real scenes.

[0004] Unsupervised learning algorithms can train neural networks without measuring the real depth value first. After the network inputs the stripe image and outputs the predicted height map, the virtual height map is re-projected, and the obtained stripe image is compared with the real stripe image to obtain the network loss. However, since reprojection uses computer graphics, it cannot simulate the noise in the real environment, which reduces the measurement accuracy. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a single-frame three-dimensional measurement method based on structured light and deep learning; the present invention completes single-frame three-dimensional measurement through a deep neural network, can be used to measure dynamic scenes, and has higher measurement accuracy than the traditional structured light single-frame measurement method. The combination of unsupervised and supervised learning reduces the number of collected fringe images and improves the training efficiency of the deep learning method. Each module of the fringe analysis network contributes to the improvement of measurement accuracy.

[0006] The technical solution of the present invention is specifically described as follows.

[0007] A single-frame three-dimensional measurement method based on structured light and deep learning, the specific steps are as follows:

[0008] Step 1: Train a deep neural network for stripe image analysis

[0009] Firstly, a single-frame fringe image collected by the structured light system is used as input to perform unsupervised learning on the deep neural network.

[0010] training; then the multi-frequency phase-shifted fringe image is used as input to fine-tune the deep neural network through supervised learning to obtain a trained deep neural network; wherein: the deep neural network is composed of three U-Net modules in cascade, module a is used for the calculation of the wrapped phase map, module b is used for phase unwrapping, and module c is used for the tuning of the absolute phase map. Module a takes the fringe image as input and outputs the wrapped phase map, module b takes the fringe image and the wrapped phase map as input and outputs the absolute phase map, module c takes the fringe image, the wrapped phase map and the absolute phase map as input and outputs the tuned absolute phase map; each U-Net module is divided into downsampling and upsampling parts, the downsampling part is composed of a downsampling layer and a convolution layer, the upsampling part is composed of an upsampling layer, a convolution layer and an output layer, the upsampling layer and the downsampling layer use the same number of convolution operations, at the same time, the downsampling part results and the upsampling part with the same image resolution are connected in series, and the number of filters is unified by the convolution layer, and the output layer uses convolution and sigmoid functions to convert the features to the output dimension and value range;

[0011] Step 2: Detect single-frame fringe image

[0012] The single-frame fringe image collected by the structured light system to be detected is input into the deep neural network trained in step (i), and the prediction results including the wrapped phase map, the absolute phase map and the tuned absolute phase map are output. The tuned absolute phase map is then used as the final result for three-dimensional measurement.

[0013] In the present invention, in step 1, in the deep neural network, in the downsampling part, first, 5 convolutional layers and 5 downsampling layers are arranged in sequence to realize feature extraction at different image resolutions, and then at the lowest resolution of 1 / 32, 4 convolutional layers with residual connections are used to form a residual block. The downsampling layer uses a convolution with a convolution kernel size of 2 and a step size of 2, batch normalization and Leaky RELU activation function, and the convolution layer uses a convolution kernel size of 3 and a step size of 1, batch normalization and LeakyRELU activation function. The downsampling layer and the convolution layer with the same image resolution use the same number of filters, and the number of filters is set to 16, 24, 24, 32, 32 and 48, corresponding to 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 resolutions; in the upsampling part, first, 5 upsampling layers and 5 convolutional layers are arranged in sequence, and then the output layer is connected at a resolution of 1. The upsampling layer uses deconvolution with a kernel size of 2 and a stride of 2, batch normalization and Leaky RELU activation function. The convolution layer also uses convolution with a kernel size of 3 and a stride of 1, batch normalization and Leaky RELU activation function. At the same image resolution, the number of filters in the upsampling layer and the convolution layer is the same, which are 16, 24, 24, 16 and 8, corresponding to 1 / 16, 1 / 8, 1 / 4, 1 / 2 and 1 resolutions. The output layer uses convolution with a kernel size of 1 and a stride of 1 and a sigmoid function, and the number of filters is 1.

[0014] In the present invention, in step one, in the downsampling part, the number of filters of the downsampling layer and the convolution layer with the same image resolution is the same, and the number of filters is set to 16, 24, 24, 32, 32 and 48 respectively, corresponding to 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 resolutions.

[0015] In the present invention, in step one, in the upsampling part, the upsampling layer with the same image resolution has the same number of filters as the convolution layer, and the number of filters is set to 16, 24, 24, 16 and 8 respectively, corresponding to 1 / 16, 1 / 8, 1 / 4, 1 / 2 and 1 resolutions; the number of filters in the output layer is 1.

[0016] In the present invention, during unsupervised learning pre-training, the L1 loss of the stripe image predicted by the deep neural network and the real stripe image is used as the network loss, and the single-frame images at two frequencies are used to calculate the loss, wherein the phase map of the high-frequency single-frame image is used for prediction; the predicted stripe image is obtained by stripe reprojection of the phase map output by the deep neural network prediction.

[0017] In the present invention, during supervised learning fine-tuning, four frequency 12-step phase-shifted fringe images are used as inputs of the deep neural network.

[0018] In the present invention, during supervised learning fine-tuning, the L1 loss between the phase image predicted by the deep neural network and the real phase image is used as the network loss, and the real phase image is obtained by using the phase shift method and time phase unwrapping.

[0019] In the present invention, the Adam optimizer is used for the training of the deep neural network, and the python deep learning framework is used to accelerate the training process.

[0020] The present invention uses a method combining unsupervised learning pre-training with supervised learning fine-tuning, which effectively solves the problem of large data demand and low training efficiency of deep learning algorithms in structured light single-frame three-dimensional measurement in the prior art; its beneficial effects also include:

[0021] 1. The structured light system used in the present invention is simple to install and only requires a projector and an industrial camera.

[0022] 2. After training, the present invention can achieve complete single-frame 3D measurement, and each measurement only requires one image at a single fringe frequency. Although some single-frame measurement methods do not require the acquisition of multi-step phase shift images, they require fringe images at different frequencies for time phase unwrapping.

[0023] 3. The present invention uses supervised learning fine-tuning after unsupervised learning pre-training to solve the reprojection error problem of unsupervised learning and improve the measurement accuracy, which is comparable to the supervised learning method that requires a large amount of data.

[0024] 4. After unsupervised learning pre-training, the network of the present invention has the ability to analyze fringes. It does not require a large amount of real data for supervised learning, but only requires a small amount of data set obtained by the 12-step phase shift method for training. This improves training efficiency, reduces the time used for data collection, and reduces the difficulty of application.

[0025] 5. The present invention uses a neural network composed of three U-Net modules for fringe analysis, which are respectively used to predict the included phase image, predict the absolute phase image and tune the absolute phase image, so as to further improve the measurement accuracy.

[0026] In summary, compared with the existing structured light single-frame three-dimensional measurement method, the structured light system used in the present invention is simple in device, can realize complete single-frame three-dimensional measurement, improves the measurement accuracy of unsupervised learning, reduces the data acquisition time of supervised learning, and improves training efficiency. The three U-Net modules of the network architecture further improve the measurement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flow chart of the method of the present invention.

[0028] Figure 2 Schematic diagram of the neural network architecture used for stripe image analysis in the present invention.

[0029] Figure 3 Schematic diagram of the U-Net module architecture of the neural network for stripe image analysis of the present invention.

[0030] Figure 4 It is a single-frame fringe image collected in the example of the present invention.

[0031] Figure 5 This is the wrapped phase diagram obtained by the 12-step phase shift method in the example of the present invention.

[0032] Figure 6 This is the absolute phase diagram obtained by the 12-step phase shift method in the example of the present invention.

[0033] Figure 7 This is the wrapped phase diagram of the network prediction in the example of the present invention.

[0034] Figure 8 This is the absolute phase diagram predicted by the network in the example of the present invention.

[0035] Fig. 9 It is the absolute phase diagram after optimization predicted by the network in the example of the present invention.

[0036] Fig.10 This is the error map containing the wrapped phase map predicted by the network in the example of the present invention.

[0037] Fig.11 This is the error diagram of the absolute phase diagram predicted by the network in the example of the present invention.

[0038] Fig.12 This is the error diagram of the absolute phase diagram after optimization predicted by the network in the example of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below through specific examples and related drawings.

[0040] Figure 1 Flow chart of the method of the present invention.

[0041] In the present invention, a projector projects monochromatic sinusoidal stripes onto the object to be measured, an industrial camera collects single-frame images during the unsupervised learning stage and measurement, and collects multi-frequency phase-shifted images during the supervised learning stage; a deep neural network uses a single-frame stripe image as input, and outputs a predicted wrapped phase map, an absolute phase map, and an optimized absolute phase map, and the optimized absolute phase map is used as the final result for three-dimensional measurement. The training of the neural network uses a two-stage training method that combines unsupervised learning pre-training with supervised learning fine-tuning, and is used for single-frame three-dimensional measurement after training.

[0042] The specific method of unsupervised learning is: using the single-frame stripe image collected by the structured light system, using the deep neural network to predict the phase map, including the wrapped phase map, the absolute phase map and the tuned absolute phase map, reprojecting the predicted stripe image respectively, and using the predicted and real stripe images to form the network loss. In order to achieve the purpose of unwrapping training, single-frame images at two frequencies are used for loss calculation, and the network only predicts the phase map of the high-frequency single-frame image. The network loss can be expressed as ,in represents the L1 loss function, represents a high-frequency fringe image, The predicted wrapped phase diagram for the network at high frequency, represents the high-frequency fringe image obtained by reprojecting the wrapped phase map predicted by the network, is the absolute phase map predicted by the network at high frequency, and represents the high-frequency fringe image obtained by reprojecting the absolute phase map predicted by the network. The absolute phase diagram after tuning is predicted by the network at high frequency. represents the high-frequency fringe image reprojected from the tuned absolute phase map predicted by the network, represents a low-frequency fringe image, is the absolute phase diagram predicted by the network at low frequencies, represents the low-frequency fringe image reprojected from the absolute phase map predicted by the network, The absolute phase diagram after tuning is predicted by the network at low frequencies. Represents the low-frequency fringe image reprojected from the fine-tuned absolute phase map predicted by the network.

[0043] The specific method of supervised learning is: using the four-frequency 12-step phase-shifted fringe images collected by the structured light system, the phase shift method and time phase unwrapping are used to obtain the real phase map, including the wrapped phase map and the absolute phase map. The phase map predicted by the network, including the wrapped phase map, the absolute phase map and the tuned absolute phase map, constitutes the network loss with the real phase map. The network loss can be expressed as ,in represents the L1 loss function, represents the true wrapped phase diagram, represents the wrapped phase map predicted by the network, represents the true absolute phase diagram, represents the absolute phase map predicted by the network, Represents the fine-tuned absolute phase map predicted by the network.

[0044] The deep neural network used for streak image analysis consists of three U-Net modules. The three U-Net modules are used to predict the wrapped phase image, predict the absolute phase image, and tune the absolute phase image. The U-Net module consists of downsampling and upsampling parts, mainly convolutional layers and deconvolution layers, and no pooling layers are used. The Adam optimizer is used for neural network training, and the python deep learning framework is used to accelerate the training process.

[0045] To demonstrate the practical effect of the present invention, a deep neural network for stripe analysis is written using the pytorch deep learning framework of python. The network architecture is as follows: Figure 2 The network consists of three U-Net modules, module a is used for the calculation of the wrapped phase map, module b is used for phase unwrapping, and module c is used for the tuning of the absolute phase map. The specific architecture of the U-Net module is shown in Figure 3 As shown in the figure. U-Net can be divided into downsampling and upsampling parts. In the downsampling part, firstly, 5 convolutional layers c1, c2, c3, c4, c5 and 5 downsampling layers d1, d2, d3, d4, d5 are arranged in sequence to achieve feature extraction at different image resolutions. Then, at the lowest resolution of 1 / 32, 4 convolutional layers c6, c7, c8, c9 with residual connections are used to form a residual block. The downsampling layer uses convolution with a kernel size of 2 and a stride of 2, batch normalization and Leaky RELU activation function. The convolution layer uses convolution with a kernel size of 3 and a stride of 1, batch normalization and Leaky RELU activation function. The downsampling layer and the convolution layer with the same image resolution use the same number of filters, and the number of filters is set to 16, 24, 24, 32, 32 and 48, corresponding to resolutions of 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32; in the upsampling part, first, 5 upsampling layers u1, u2, u3, u4, u5 and 5 convolution layers c10, c11, c12, c13, c14 are arranged in sequence, and then connected to the output layer o1 at resolution 1. The upsampling layer uses deconvolution with a kernel size of 2 and a stride of 2, batch normalization and Leaky RELU activation function. The convolution layer also uses convolution with a kernel size of 3 and a stride of 1, batch normalization and Leaky RELU activation function. At the same image resolution, the number of filters in the upsampling layer and the convolution layer is the same, which are 16, 24, 24, 16 and 8, corresponding to 1 / 16, 1 / 8, 1 / 4, 1 / 2 and 1 resolutions. The output layer uses convolution with a kernel size of 1 and a stride of 1 and a sigmoid function, and the number of filters is 1.

[0046] In the embodiment, a structured light measurement system was built, which uses a blue light projector with a resolution of . Only the left camera was used for the industrial camera, and its resolution was . The projector projects the vertical blue stripes onto the object to be tested. In the unsupervised training stage, a total of 2000 sets of stripe images were collected, of which the high frequency was set to 64 and the low frequency was set to 9. In the supervised training stage, a total of 120 sets of 12-step phase shift images were collected, including four frequencies of 64, 16, 4 and 1. The 12-step phase shift method and time phase unwrapping were used to obtain the wrapped phase map and the absolute phase map as the true value. 20 sets of images were used as the test set. Figure 4 is a stripe image with a frequency of 64 in the test set, Figure 5 is the wrapped phase diagram obtained by the 12-step phase shift method, Figure 6 This is the absolute phase diagram obtained by the 12-step phase shift method.

[0047] After collecting stripe images to establish a data set, a combination of unsupervised learning and supervised learning is used to train the network. In unsupervised learning, a single-frame stripe image collected by a structured light system is used to predict the phase map using a deep neural network, including a wrapped phase map, an absolute phase map, and an optimized absolute phase map. The predicted stripe images are reprojected separately, and the predicted and real stripe images form a network loss. In supervised learning, four frequencies and 12-step phase-shifted stripe images collected by a structured light system are used to obtain the real phase map using the phase shift method and time phase unwrapping, including a wrapped phase map and an absolute phase map. The phase maps predicted by the network, including the wrapped phase map, the absolute phase map, and the optimized absolute phase map, form a network loss with the real phase map.

[0048] After the network training is completed, a single-frame fringe image with a frequency of 64 in the test set is used as input to test the single-frame 3D measurement effect of the fringe analysis network. Figure 7 is the network predicted wrapped phase map, Figure 8 is the absolute phase map predicted by the network, Fig. 9 This is the absolute phase map after optimization predicted by the network. It can be seen that the network prediction result is close to the result of the 12-step phase shift method, and accurate single-frame 3D measurement can be achieved. With the three modules of the network working in sequence, the results are becoming more and more accurate, reflecting the contribution of the network modules to the measurement accuracy.

[0049] Visualize the error of the network prediction results, such as Figure 10-12 As shown. Fig.10 is the error map of the network prediction containing the wrapped phase map, Fig.11 is the error map of the absolute phase map predicted by the network, Fig.12 This is the error diagram of the absolute phase image after optimization predicted by the network. The error unit is phase radians, and the horizontal and vertical coordinates are pixel values. It can be seen that the network prediction result has a small error and can achieve accurate three-dimensional measurement. The combination of network modules can greatly reduce the measurement error and improve the measurement accuracy.

[0050] In summary, the present invention demonstrates a single-frame three-dimensional measurement method based on structured light and deep learning. The structured light system used is simple in device and can realize complete single-frame three-dimensional measurement, which improves the measurement accuracy of unsupervised learning, reduces the data acquisition time of supervised learning, and improves training efficiency. The three U-Net modules of the network architecture further improve the measurement accuracy.

Claims

1. A single-frame three-dimensional measurement method based on structured light and deep learning, characterized in that: The specific steps are as follows: Step 1: Train a deep neural network for stripe image analysis Firstly, a single-frame fringe image collected by a structured light system is used as input to pre-train the deep neural network through unsupervised learning. Then, a multi-frequency phase-shifted fringe image is used as input to fine-tune the deep neural network through supervised learning to obtain a trained deep neural network. The deep neural network is composed of three cascaded U-Net modules, where module a is used for calculating the wrapped phase map, module b is used for phase unwrapping, and module c is used for tuning the absolute phase map. Module a takes the fringe image as input and outputs the wrapped phase map, while module b takes the fringe image and the wrapped phase map as input and outputs the absolute phase map. Figure, module c takes the stripe image, the wrapped phase image and the absolute phase image as input, and outputs the tuned absolute phase image; each U-Net module is divided into downsampling and upsampling parts. The downsampling part consists of a downsampling layer and a convolution layer, and the upsampling part consists of an upsampling layer, a convolution layer and an output layer. The upsampling and downsampling parts use the same number of convolution operations. At the same time, the downsampling part results and the upsampling part with the same image resolution are connected in series, and the convolution layer is used to unify the number of filters. The output layer uses convolution and sigmoid functions to convert the features to the output dimension and value range; Step 2: Detect single-frame fringe image The single-frame fringe image collected by the structured light system to be detected is input into the deep neural network trained in step (i), and the prediction results including the wrapped phase map, the absolute phase map and the tuned absolute phase map are output, and the tuned absolute phase map is used as the final result for three-dimensional measurement; wherein: In step 1, the specific method of unsupervised learning is as follows: using the single-frame fringe image collected by the structured light system, the deep neural network is used to predict the phase map, including the wrapped phase map, the absolute phase map and the tuned absolute phase map, and the predicted fringe images are reprojected respectively, and the predicted and real fringe images are used to form the network loss; in order to achieve the purpose of unwrapping training, single-frame images at two frequencies are used for loss calculation, and the network only predicts the phase map of the high-frequency single-frame image; the network loss is expressed as Where L represents the L1 loss function, I h represents a high-frequency fringe image, The predicted wrapped phase diagram for the network at high frequency, represents the high-frequency fringe image obtained by reprojecting the wrapped phase map predicted by the network, is the absolute phase diagram predicted for the network at high frequencies, represents the high-frequency fringe image reprojected from the absolute phase map predicted by the network, The absolute phase diagram after tuning is predicted by the network at high frequency. represents the high-frequency fringe image reprojected from the tuned absolute phase map predicted by the network, I l represents a low-frequency fringe image, is the absolute phase diagram predicted by the network at low frequencies, represents the low-frequency fringe image reprojected from the absolute phase map predicted by the network, The absolute phase diagram after tuning is predicted by the network at low frequencies. represents the low-frequency fringe image obtained by reprojecting the tuned absolute phase map predicted by the network; The specific method of supervised learning is: using the 12-step phase-shifted fringe images of four frequencies collected by the structured light system, the phase shift method and time phase unwrapping are used to obtain the real phase map, including the wrapped phase map and the absolute phase map, and the phase map predicted by the network, including the wrapped phase map, the absolute phase map and the tuned absolute phase map, together with the real phase map, constitutes the network loss; the network loss is expressed as Where L represents the L1 loss function, represents the true wrapped phase diagram, represents the network predicted wrapped phase map, Φ represents the true absolute phase map, represents the absolute phase map predicted by the network, Represents the fine-tuned absolute phase map predicted by the network.

2. The single-frame three-dimensional measurement method according to claim 1, characterized in that: In step 1, in the deep neural network, in the downsampling part, first, 5 convolutional layers and 5 downsampling layers are arranged in sequence to realize feature extraction at different image resolutions, and then at the lowest resolution of 1 / 32, 4 convolutional layers with residual connections are used to form a residual block; the downsampling layer uses convolution with a kernel size of 2 and a step size of 2, batch normalization and Leaky RELU activation function, and the convolution layer uses convolution with a kernel size of 3 and a step size of 1, batch normalization and Leaky RELU activation function; in the upsampling part, first, 5 upsampling layers and 5 convolutional layers are arranged in sequence, and then the output layer is connected at resolution 1. The upsampling layer uses deconvolution with a kernel size of 2 and a step size of 2, batch normalization and Leaky RELU activation function, and the convolution layer also uses convolution with a kernel size of 3 and a step size of 1, batch normalization and Leaky RELU activation function. RELU activation function, the output layer uses convolution with a kernel size of 1 and a step size of 1 and a sigmoid function.

3. The single-frame three-dimensional measurement method according to claim 2, characterized in that: In step 1, in the downsampling part, the number of filters of the downsampling layer and the convolution layer with the same image resolution is the same, and the number of filters is set to 16, 24, 24, 32, 32 and 48 respectively, corresponding to 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 resolutions.

4. The single-frame three-dimensional measurement method according to claim 2, characterized in that: In step 1, in the upsampling part, the upsampling layer with the same image resolution has the same number of filters as the convolution layer, and the number of filters is set to 16, 24, 24, 16 and 8 respectively, corresponding to 1 / 16, 1 / 8, 1 / 4, 1 / 2 and 1 resolutions; the number of filters in the output layer is 1.

5. The single-frame three-dimensional measurement method according to claim 1, characterized in that: The Adam optimizer is used for deep neural network training, and the python deep learning framework is used to accelerate the training process.

Images