High-precision three-dimensional reconstruction method based on single-frame fringe projection

Abstract

The invention discloses a high-precision three-dimensional reconstruction method based on single-frame fringe projection, and the method comprises the steps: 1, researching a structured light three-dimensional reconstruction principle, and determining the parameters of a camera and a projector; 2, processing the data set to obtain a required data set; 3, performing deep learning training based on the data set in the step 2, and predicting a wrapped phase; 4, post-processing the wrapped phase predicted in the step 3, and predicting the fringe level; and 5, carrying out mathematical calculation on the wrapped phase and the fringe level predicted in the step 3 and the step 4 to obtain an absolute phase, and carrying out three-dimensional point cloud reconstruction by combining the parameters determined in the step 1.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a method for high-precision three-dimensional reconstruction based on single-frame fringe projection. Background Art

[0002] Three-dimensional measurement technology has been widely studied and applied in many fields such as industrial component defect detection, medical disease diagnosis, artificial intelligence and machine vision. Among them, structured light three-dimensional measurement technology has become the mainstream optical technology in three-dimensional measurement technology due to its advantages such as non-contact, high precision, flexible fringe setting and large measurement range. The two most common structured light three-dimensional measurement technologies are speckle projection measurement technology and fringe projection measurement technology. The basic processing process of fringe projection measurement technology is mainly divided into steps such as wrapping phase calculation, phase unwrapping and three-dimensional reconstruction.

[0003] Common methods for calculating the wrapped phase include the phase shift (PS) method and the Fourier transform (FT) method. The Fourier transform method only requires one fringe image to calculate the wrapped phase, but there are problems such as spectral aliasing and spectral leakage, which will affect its calculation accuracy, especially when the surface morphology of the object under test is discontinuous. The phase shift method has the advantages of high phase solution accuracy and strong anti-interference ability, so it has been more widely used. The wrapped phase obtained by the above method also needs to be obtained by the phase unwrapping method to obtain the correct absolute phase. Commonly used phase unwrapping methods include the time phase unwrapping (TPU) method and the space phase unwrapping (SPU) method. The time phase unwrapping method requires multiple auxiliary graphics (Gray code or multi-wavelength stripes, etc.) to be projected onto the surface of the object under test, and the phase unwrapping of each pixel does not affect each other. Therefore, when this method is used to measure objects with complex surface morphology, it performs well and has strong anti-noise ability. The spatial phase unwrapping method only requires an image to be integrated row by row and column by column according to adjacent phase differences. This requires that the surface of the object being measured satisfies the continuity assumption. However, the actual wrapped phase image obtained has problems such as edge jumps and local shadows due to undersampling, resulting in a large measurement error of this method.

[0004] Speckle projection measurement technology can achieve 3D measurement through a single speckle image, but traditional speckle projection measurement technology has problems such as slow measurement speed and low measurement accuracy, and the introduction of dual cameras will increase production costs and system complexity. The method of fusing speckles and fringes to produce a composite image has certain limitations in its application scenarios because it requires projecting multiple frames of images or dual cameras to achieve speckle matching. Summary of the invention

[0005] In view of the above problems, the present invention proposes a high-precision three-dimensional reconstruction method based on single-frame fringe projection, comprising:

[0006] Step 1: Camera-Projector Calibration

[0007] Study the principle of structured light 3D reconstruction. Keep the relative position of the camera and projector unchanged, use Zhang Zhengyou calibration method to calibrate the camera and projector, and obtain the internal and external parameters of the camera and projector for subsequent reconstruction.

[0008] Step 2: Dataset collection and preprocessing

[0009] The images captured by the acquisition system are used to calculate the required package phase and fringe order true value using the phase shift method to form fringe pattern-M&D data pairs and package phase-fringe order data pairs for subsequent training.

[0010] Step 3: Build a network model for parcel phase prediction

[0011] Based on the data set in step 2, a wrapped phase prediction network model is constructed. The improved U-Net is applied to first extract features at different levels in the image, and then learn the information of M and D from these features. In order to improve the accuracy of the prediction results, the fringe image is transformed by DCT to obtain the frequency domain information of the image, which is then fused with the spatial domain information after frequency domain enhancement. After feature extraction and restoration, the information of M and D can be predicted from a single stripe image.

[0012] Step 4: Build a network model for stripe-level prediction

[0013] A deep convolutional network VR-Net based on VGG16 and RefineNet is used to predict the fringe order of the wrapped phase.

[0014] Step 5: 3D point cloud recovery

[0015] The predicted wrapped phase and fringe order are used for mathematical calculation to obtain the absolute phase, which is then combined with the internal and external parameters of the camera projector obtained in step 1 to restore the three-dimensional point cloud.

[0016] Beneficial Effects of the Invention

[0017] (1) The constructed single-frame image wrapping phase prediction model can accurately predict the object’s wrapping phase

[0018] (2) The absolute phase and the internal and external parameters of the camera projector can be used to quickly and accurately restore the three-dimensional point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flowchart of high-precision 3D reconstruction based on single-frame fringe projection.

[0020] Figure 2 Diagram of the dataset collection device.

[0021] Figure 3 This is the F-UNet structure diagram

[0022] Figure 4 This is the structure diagram of VR-Net.

[0023] Figure 5 The result of 3D point cloud reconstruction. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0025] A method for high-precision three-dimensional reconstruction based on single-frame fringe projection comprises the following steps:

[0026] Step 1: Study the principle of structured light 3D reconstruction, calibrate the camera and projector, and obtain the internal and external parameters of the camera and projector

[0027] Step 2: Process the data set to obtain the required data set;

[0028] Step 3: Perform deep learning training based on the data set in step 2 to predict the package phase;

[0029] Step 4: Post-process the parcel phase predicted in step 3 to predict the fringe order;

[0030] Step 5: Perform mathematical calculations on the wrapped phase and fringe order predicted in steps 3 and 4 to obtain the absolute phase, and reconstruct the 3D point cloud based on the parameters determined in step 1.

[0031] like Figure 1 As shown, the method of the present invention specifically comprises the following steps:

[0032] Step 1): Figure 2 As shown in Figure 1, a typical SPU-based system consists of a projector and two cameras. The camera internal parameters refer to the focal length of the camera and the center coordinates of the pixel coordinate system. The camera external parameters can be obtained through the coordinate relationship from the three-dimensional space point to the two-dimensional image plane. Figure 3 The coordinate system in the figure includes the world coordinate system (Ow, Xw, Yw, Zw), the camera coordinate system (Oc, Xc, Yc, Zc), and the image coordinate system (O l ,x,y) and pixel coordinate system (Oo,u,v). The origin of the world coordinate system can be established at any point in space and is usually selected when calibrating the camera's external parameters. The origin of the camera coordinate system coincides with the optical center of the camera, while the Zc axis of the camera coordinate system coincides with the direction of the optical axis, and the Xc axis and Yc axis are parallel to the edges of the image plane. In the camera imaging plane, O l represents the origin of the image coordinate system, and O l That is, the origin of the principal point pixel coordinates of the camera image plane is in the upper left corner of the entire imaging plane.

[0033] The position of any point P in the world coordinate system is recorded as (Xw, Yw, Zw), and similarly, the coordinates of P in the camera coordinate system are recorded as (Xc, Yc, Zc). Figure 3 It can be seen that there is a rotation and translation relationship between the world coordinate system and the camera coordinate system, that is, the transformation relationship between the two coordinate systems can be established through a rotation matrix R and a translation vector T:

[0034]

[0035] It is expressed in homogeneous coordinates as:

[0036]

[0037] Where R is a 3×3 unit orthogonal matrix and T is a three-dimensional translation vector. This is the camera extrinsic parameter. When obtaining the projector extrinsic parameters, the projector can be regarded as an inverse camera.

[0038] Step 2): The fringe image is projected by a projector, modulated by the object, and finally captured by two cameras. For the n-step PS algorithm:

[0039]

[0040] In the formula, I i represents the i-th captured image, i = 1, 2, ..., N, (x, y) is the camera pixel coordinate, A is the average intensity map, B is the amplitude intensity, is the wrapping phase, and 2πn / N is the phase shift. The wrapping phase can be obtained by the least squares method.

[0041]

[0042] M and D represent the numerator and denominator of the arctan function respectively. The absolute phase and the wrapped phase satisfy the following relationship:

[0043]

[0044] Where k is the stripe order. The stripe order k can be obtained by using the SPU based on geometric constraints.

[0045] Through the above operations, the fringe pattern-M&D data pair and the wrapped phase-fringe order (k) data pair can be obtained.

[0046] Step 3-1): Build a single-frame image model. The model used is F-UNet (Frenquency U-Net). Its overall structure is as follows: Figure 3As shown. U-Net is an encoder-decoder structured network. The encoder stage of the network is a series of downsampling operations consisting of convolution and downsampling, which is called the contracting path. The compression path consists of 4 blocks, each of which uses 3 effective convolutions and 1 Max Pooling downsampling. The number of feature maps is multiplied by 2 after each downsampling; the decoder stage is also composed of 4 blocks. Before each block starts, the size of the Feature Map is multiplied by 2 through deconvolution, and its number is halved, and then merged with the Feature Map of the compression path symmetrical to the encoder stage, and finally outputs the two-channel feature map, namely M and D. The input of the network is a single stripe image taken. Since the directly predicted wrapping phase has a jump of 2π, it is difficult for U-Net to accurately predict the jump position, and M and D do not have a jump, so M and D can be used to obtain high-quality wrapping phase.

[0047] Step 3-2): Considering that the frequency domain information of the image is analyzed when solving the wrapping phase of a single stripe image by the traditional Fourier transform method, adding the frequency domain information of the image to the convolutional neural network can increase the prediction accuracy of M and D. DCT stands for Discrete Cosine Transform. DCT transform is a type of Fourier transform. The frequency information of the image is used to assist in predicting the wrapping phase. First, the input image is transformed by DCT to obtain the frequency domain information of the image; secondly, the coefficients of each frequency band are enhanced by the frequency enhancement module to extract the discrimination signal; further, the spatial domain (RGB) and the frequency domain (RGB) are aligned to obtain the fusion feature. Subsequently, the image needs to be sent to the convolutional neural network for training, and the image data needs to be resampled. The fusion features are sent to the encoder for feature extraction, and the image is restored in the decoder.

[0048] Step 3-3): The activation function used by F-UNet is ReLU, so DAN can avoid the problem of gradient vanishing in deep networks for training, but ReLU will cause the phenomenon of neuron "death", resulting in the weights of "dead" neurons no longer being updated. Therefore, the activation function of the DAN model is changed to a leaky rectified linear unit Leaky ReLU, which gives a non-zero slope in the negative interval, so that the output in the negative interval is not zero, effectively avoiding the phenomenon of neuron "death". Its function is shown in the following formula:

[0049]

[0050] Among them, α is 0.2. The loss function in the network is the L1 loss function, which is often used in regression problems to represent the square error between the predicted value and the true value. The function body of this loss function is as follows:

[0051]

[0052] Among them, M pre is the predicted value of M, M gt is the truth value of M; D pre is the predicted value of D, D gt The true value of D; H is the height of the image, and W is the width of the image.

[0053] Step 4-1): Since the prediction problem of the wrapped phase is a regression problem, the predicted background element pixel value is not 0 but close to 0. For the subsequent prediction of the fringe level, the predicted wrapped phase is post-processed and the background element pixel value is set to 0 to generate a wrapped phase-fringe level data pair.

[0054] Step 4-2): Construct a prediction network model from wrapped phase to stripe level. The encoder part of VRNet is designed with reference to VGG16 and is divided into five layers. Each layer consists of several convolutional layers and maximum pooling layers. The output feature map is not only used as the input of the next layer, but also as part of the decoder input after passing through a convolutional layer. Each convolutional layer uses a 3x3 convolution kernel with a step size of 1. During the convolution process, 0 is filled to the feature map boundary to ensure that the resolution of the feature map before and after the convolution remains consistent, and ReLU is used as the activation function after the convolution. The maximum pooling layer uses a 2x2 sampling kernel with a step size of 2. The resolution of the feature map is halved after the pooling layer.

[0055] The encoder part enriches the semantic information of the feature map, but it also causes a lot of spatial information to be lost. Therefore, VRNet uses five feature fusion modules to recursively restore the resolution of the feature map, and obtains accurate predictions through upsampling layers, residual modules, and convolutional layers. The above modules constitute the decoder part of VRNet.

[0056] The wrapped phase predicted in step 3 is input into the network and predicted to obtain the fringe order. We use MSE as the loss function, as shown in formula (8):

[0057]

[0058] where k pre is the predicted fringe level value, k gt is the true value of the stripe level; H is the height of the image, and W is the width of the image.

[0059] Step 5): Using the wrapped phase and fringe order obtained in steps 3 and 4, the absolute phase is obtained by mathematical calculation of formula (5). The world coordinates of each point of the object can be calculated using the triangular stereo model obtained by the internal and external parameters of the camera projector obtained in step 1, thus completing the three-dimensional shape restoration.

Claims

1. A method for high-precision three-dimensional reconstruction based on single-frame fringe projection, the method comprising the following steps: Step 1: Study the principle of structured light 3D reconstruction and determine the number and parameters of cameras and projectors; Step 2: Process the data set to obtain the required data set; Step 3: Perform deep learning training based on the data set in step 2 to predict the package phase; Step 4: Post-process the parcel phase predicted in step 3 to predict the fringe order; Step 5: Perform mathematical calculations on the wrapped phase and fringe order predicted in steps 3 and 4 to obtain the absolute phase, and reconstruct the 3D point cloud based on the parameters determined in step 1.

2. The method according to claim 1, characterized in that: The parameters described in step 1 are the intrinsic parameters of the camera, the intrinsic parameters of the projector, the extrinsic parameters of the camera, and the extrinsic parameters of the projector.

3. The method according to claim 1, characterized in that: The images captured by the acquisition system in step 1 are used to calculate the required package phase and fringe order true value using the phase shift method, forming fringe pattern-M&D data pairs and package phase-fringe order data pairs for subsequent training.

4. The method according to claim 1, characterized in that: The deep convolutional network described in step 3 is U-Net, which is an encoder-decoder structured network. The encoder stage of the network is a series of downsampling operations consisting of convolution and downsampling, which is called the contracting path. The contracting path consists of 4 blocks, each of which uses 3 effective convolutions and 1 MaxPooling downsampling. The number of feature maps is multiplied by 2 after each downsampling; the decoder stage also consists of 4 blocks. Before each block starts, the size of the FeatureMap is multiplied by 2 through deconvolution, and its number is halved, and then merged with the FeatureMap of the compression path symmetrical to the encoder stage, and finally the required feature map is output. The improved deep convolutional network optimizes the shortcomings of U-Net on the basis of U-Net, and changes the original ReLU activation function of U-Net to Leaky_ReLU activation function: LeakReLu is similar to ReLu, but it can solve the problem of neuron death. In the part where the input is less than 0, the value is negative, and LeakReLu has a small gradient. Frequency domain information is added to the feature extraction stage of U-Net.

5. The method according to claim 4, characterized in that: When adding frequency domain information, the fringe pattern is first converted from the spatial domain to the frequency domain, that is, the frequency domain is used as an additional clue to better predict the wrapped phase. In order to better incorporate frequency clues into the U-Net model, a network with two special components is used: the frequency enhancement module (FEM), which contains offline discrete cosine transform and then learnable enhancement; in order to further make full use of frequency information, the high-order relationship module (HOR) is used to process the rich fusion features. Finally, the RGB and frequency inputs are fed into the network as RGB streams and frequency streams respectively. Feature alignment (FA) is used to fuse the features of the RGB domain and the frequency domain, and effective information is output after feature extraction and restoration.

6. The method according to claim 1, characterized in that: Step 4 describes the prediction of the wrapping phase to the stripe level, which is trained and learned using VR-net. This network also has an encoder-decoder structure, where the encoder is composed of the encoder of vgg16 and the decoder is composed of the decoder of RefineNet. The wrapping phase predicted by one segment of the network is sent to the second segment of the network to predict the stripe level.

7. The method according to claim 1, characterized in that: The M and D predicted by the first-stage network are transformed by inverse tangent to obtain the wrapped phase, and the wrapped phase is sent to the second-stage network for prediction to obtain the fringe order. The wrapped phase and the fringe order are mathematically operated to obtain the absolute phase. Since the image has noise points during shooting, the prediction is inaccurate, so the calculated absolute phase is corrected:

1. If the phase difference is 2Π, add or subtract 2Π to make up for it; 2. Phase points with a phase difference of not 2π are treated as noise points and set to 0.

8. The method according to claim 1, characterized in that: The absolute phase obtained by post-processing is calculated in combination with the internal and external parameters of the camera projector to obtain the final three-dimensional point cloud of the object.

Images