Dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction

Abstract

The present invention relates to a dynamic fringe projection three-dimensional measurement method based on super-resolution reconstruction of a fringe pattern, and relates to the fields of digital reverse engineering and machine vision. The method comprises the following steps: designing and training a neural network for detail recovery and super-resolution reconstruction; building a three-dimensional measurement system and acquiring a fringe pattern; placing the acquired fringe pattern into a trained neural network for image super-resolution reconstruction; and performing three-dimensional reconstruction of the super-resolution fringe pattern. The precise dynamic three-dimensional measurement technology based on super-resolution reconstruction of a fringe pattern proposed by the present invention solves the problem of dynamic errors that are difficult to avoid during measurement by super-resolution reconstruction of the fringe pattern, and can also obtain accurate high-resolution images, thereby obtaining more accurate and dense three-dimensional data. This improves measurement speed and efficiency.

Description

Technical Field

[0001] The invention relates to a dynamic fringe projection three-dimensional measurement method based on fringe pattern super-resolution reconstruction, and belongs to the technical field of digital reverse engineering and machine vision. Background Art

[0002] Optical 3D measurement technology has been widely used in industrial inspection, intelligent manufacturing, and reverse engineering. Reconstruction methods include laser line scanning technology, stereo vision, fringe projection profilometry, and so on. Among them, fringe projection profilometry (FPP) is one of the most popular technologies in 3D measurement due to its low cost, high precision, and high speed. A classic FPP system usually consists of a projector and a camera. The former is used to project the encoded fringe pattern onto the measurement object, and the latter is used to synchronously capture the highly modulated fringe pattern. Due to the limitations of multi-frame fringe pattern reconstruction, errors are inevitable in dynamic measurement when the object moves within the time interval between frames.

[0003] However, when measuring the 3D shape of large and complex objects, a single point cloud acquisition can be incomplete due to factors like shadows and occlusions. Therefore, multiple measurements are required to reconstruct partial point clouds of the object, ultimately fusing them together through stitching. Therefore, the registration and stitching method directly impacts the final 3D accuracy.

[0004] To address this problem, the current mainstream solutions can be roughly divided into two categories: hardware-based methods and algorithm-based methods. As the name suggests, hardware-based methods solve dynamic problems by using advanced hardware configurations. The recent development of digital light processing (DLP) projection technology has enabled the defocused projection of binary patterns to be accelerated to 20kHz. However, in actual measurements, the speed of three-dimensional measurement is usually limited by the camera. High-frame-rate cameras are usually more expensive and bulky than projectors. In addition, for low-cost portable cameras, a common solution is to sacrifice acquisition resolution to match the speed of the projector, but the resulting low-resolution three-dimensional information cannot accurately reflect the details of the object.

[0005] Unlike hardware-based methods, algorithm-based approaches focus more on improving 3D reconstruction algorithms, such as deriving motion-induced error models and compensating for them, estimating motion-induced phase shifts, and reducing the fringe pattern required for reconstruction. However, these algorithm-based approaches suffer from limitations such as limited motion direction and speed, high requirements for texture features, low reconstruction accuracy, and high computational costs. Furthermore, algorithm-based methods require measurement equipment with high sampling speeds, an unavoidable issue in dynamic measurements.

[0006] In conventional camera reading methods, different resolutions correspond to different frame rates due to differences in pixel reading scales. Due to camera bandwidth limitations, an increase in frame rate means a decrease in imaging resolution. FPP calculates the three-dimensional shape pixel by pixel, and the spatial resolution of the reconstructed three-dimensional shape is determined by the pixel resolution of the two-dimensional stripes. In order to maintain the field of view (FOV), a pixel needs to capture more spatial information, and the spatial distance between different pixels will also become larger. Moreover, since the captured stripes are discrete sinusoidal signals, if the sampling frequency is insufficient, it will cause sinusoidal signal aliasing and loss of details. In addition, if the spatial resolution of the reconstructed three-dimensional shape is greater than the actual details of the object, the reconstruction result is generally considered inaccurate, which is not conducive to applications that use object details to guide subsequent operations, such as 3D printing, industrial quality control, and medical-assisted robotic operations. Therefore, in dynamic measurement, it is necessary to solve the problem between increasing sampling speed and ensuring sampling resolution. Summary of the Invention

[0007] Purpose of the Invention: In response to the aforementioned existing problems and deficiencies, the present invention aims to provide a dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction. This method utilizes the original low-resolution fringe image and image super-resolution reconstruction technology to improve acquisition speed, eliminate dynamic errors generated during actual measurement, and obtain accurate, high-resolution fringes, preserving detailed 3D information of the measured object. This method flexibly achieves precise dynamic 3D measurement with only simple hardware configuration. The designed detail recovery super-resolution neural network is capable of restoring detail information while maintaining the sinusoidal characteristics of the fringes, making it more suitable for the FPP field. The input fringe pattern resolution and super-resolution reconstruction scale can be flexibly selected to better meet practical application requirements.

[0008] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0009] A dynamic fringe projection three-dimensional measurement method based on fringe pattern super-resolution reconstruction includes the following steps:

[0010] Step 1: Build a neural network: Design a super-resolution reconstruction neural network that automatically restores details;

[0011] Step 2: Collect the high-resolution fringe pattern of the object, degenerate it to obtain a low-resolution fringe pattern, and obtain the estimated high-resolution image based on the high-resolution fringe pattern;

[0012] Step 3: Perform the training of step 2 on various objects, and collect the corresponding relationships obtained in the neural network in step 1 to obtain the trained neural network;

[0013] Step 4: Acquire fringe images: Build a three-dimensional measurement system, measure the object to be measured using the three-dimensional measurement system, and extract the original fringe image of the object to be measured;

[0014] Step 5: Input the original fringe image from step 4 into the neural network trained in step 3 to obtain a high-resolution image;

[0015] Step 6: Substitute each point in the high-resolution image obtained in step 5 into the following formula:

[0016]

[0017] Where (x, y) is the coordinate of the pixel point, N is the number of steps of a set of phase shift stripes, δ n is the phase shift, I n (x, y) is the pixel grayscale value of the nth phase shift image at point (x, y). The wrapped phase is unwrapped based on the Gray code method to obtain the absolute phase. The three-dimensional information of the object to be measured is reconstructed by combining the obtained calibration parameters and the absolute phase.

[0018] Furthermore, the neural network in step 1 is based on a residual network structure, and the neural network includes a first convolutional layer, several residual structure modules, a second convolutional layer, an upsampling module and a third convolutional layer with jump connections in sequence. The residual structure module includes a convolutional layer, a rectified linear unit, a convolutional layer and a constant scaling layer in sequence, and the upsampling module includes multiple groups of convolutional layers and pixel scrubbing layers arranged alternately.

[0019] Furthermore, the specific steps of obtaining the low-resolution fringe pattern in step 2 are: using a high-resolution camera to obtain a low-resolution fringe pattern from each high-resolution fringe pattern according to a degradation model, and obtaining a corresponding low-resolution image as a network input. The degradation model is as follows:

[0020]

[0021] in, represents the i-th high-resolution fringe pattern, P represents the number of phase-shifted sine waves combined with the gray code pattern, H(x,y) represents the optical blur of the fringe pattern, represents the convolution operator, v(x,y) represents random noise, and D represents the downsampling operator of the stripe resolution.

[0022] Furthermore, the specific steps of obtaining the inferred high-resolution image in step 2 are: after the original fringe image is collected, input it into the trained neural network, and optimize the parameters of the neural network using the following formula:

[0023]

[0024] in, represents the jth high-resolution fringe pattern, S Θ is a nonlinear mapping function, is the L1 norm, Θ is the trainable parameter (i.e., weight and bias), and batch training is performed. n is the size of the mini-batch training when optimizing Θ using the stochastic gradient descent algorithm. The black background in the stripe pattern is not involved in the training. m is the number of valid pixels. The trained neural network is based on the original stripe image and outputs the corresponding 2x, 4x, or 8x resolution image without adjusting other parameters.

[0025] Furthermore, in step 3, a low-resolution camera is used to collect the original fringe image, and the object to be measured is placed on a track and moved to collect image information.

[0026] Beneficial effects: Compared with the existing technology, this is a new dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction. The present invention has the following advantages:

[0027] (1) By utilizing the inverse relationship between camera sampling resolution and sampling speed, we introduce image super-resolution reconstruction technology using the original low-resolution fringe image. This not only improves acquisition speed and eliminates dynamic errors generated during actual measurement, but also obtains accurate and high-resolution fringe images, preserving the detailed three-dimensional information of the measured object. Only simple hardware configuration is required to flexibly achieve accurate dynamic three-dimensional measurement.

[0028] (2) Compared with the traditional image super-reconstruction method, the designed detail recovery super-resolution neural network can restore detail information while ensuring the sinusoidal characteristics of the stripes, and is more suitable for the FPP field.

[0029] (3) Compared with the traditional deep learning-based method, after the neural network training is completed, the resolution of the input fringe pattern and the super-resolution reconstruction scale can be flexibly selected during the test process, which is more in line with actual application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic flow diagram of the present invention;

[0031] Figure 2 is a schematic diagram of the neural network structure of the present invention,

[0032] In the figure: (a) is the overall structure of the network, (b) is the structure of the residual structure module, (c) is the structure of the upsampling module, and (d) is a schematic diagram of the components represented by each figure;

[0033] Figure 3 FIG. 3 is a schematic diagram of three-dimensional imaging reconstruction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0035] like Figure 1 The process shown is a dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction. It includes the following steps:

[0036] Step 1: Build a neural network: Design a super-resolution reconstruction neural network that automatically restores details. The neural network is based on a residual network structure, which includes a first convolutional layer with skip connections, several residual structure modules, a second convolutional layer, an upsampling module, and a third convolutional layer. The residual structure module sequentially includes a convolutional layer, a rectified linear unit, a convolutional layer, and a constant scaling layer. The upsampling module includes multiple alternating groups of convolutional layers and pixel scrubbing layers.

[0037] Step 2: Super-resolution the original fringe image: The detail recovery and super-resolution reconstruction neural network includes a training phase and a testing phase. The training phase minimizes the difference between the output fringe image and the true high-resolution image, and the training detail recovery and super-resolution reconstruction neural network learns to convert the fringe image from a low-resolution to a high-resolution image. To facilitate the acquisition of the dataset, a high-resolution camera is used to capture the object's high-resolution fringe image as the network's true value, which is then degraded to a low-resolution fringe image. Based on the high-resolution fringe image, an inferred high-resolution image is obtained. The corresponding low-resolution image is obtained as the network input, and the degradation model is as follows:

[0038]

[0039] in, represents the i-th high-resolution fringe pattern, P represents the number of phase-shifted sine waves combined with the gray code pattern, H(x,y) represents the optical blur of the fringe pattern, Denotes the convolution operator, v(x,y) denotes random noise, and D denotes the fringe resolution downsampling operator. The specific steps to obtain the inferred high-resolution image are as follows: After collecting the original fringe image, input it into the trained neural network and use the following formula to optimize the neural network parameters:

[0040]

[0041] in, represents the jth high-resolution fringe pattern, S Θ is a nonlinear mapping function, is the L1 norm, Θ is the trainable parameter, i.e., weight and bias. A batch training strategy is used to improve the training effect. T is the size of the mini-batch training when optimizing Θ using the stochastic gradient descent algorithm. The black background in the stripe pattern is not involved in the training. m is the number of valid pixels. The trained network can output the corresponding 2x, 4x, or 8x resolution images based on the initial stripe pattern as needed without adjusting other parameters.

[0042] Step 3: Perform the training in step 2 on various objects and collect the corresponding relationships obtained in the neural network in step 1 to obtain the trained neural network. In step 3, a low-resolution camera is used to capture the original fringe image. The object to be tested is placed on a track and moved to collect image information.

[0043] Step 4: Acquire fringe images: Build a three-dimensional measurement system, measure the object to be measured using the three-dimensional measurement system, and extract the original fringe image of the object to be measured.

[0044] Step 5: Input the original stripe image from step 4 into the neural network trained in step 3 to obtain a high-resolution image.

[0045] Step 6: 3D reconstruction: Substitute each point in the high-resolution image obtained in step 5 into the following formula:

[0046]

[0047] Where (x, y) is the coordinate of the pixel point, N is the number of steps of a set of phase shift stripes, δ n is the phase shift, I n (x, y) is the pixel grayscale value of the nth phase shift image at point (x, y). The wrapped phase is unwrapped based on the Gray code method to obtain the absolute phase. The three-dimensional information of the object to be measured is reconstructed by combining the obtained calibration parameters and the absolute phase.

[0048] Example

[0049] according to Figure 2As shown, the detail restoration super-resolution neural network designed by the present invention uses a residual network as its basic framework. The input is operated by a convolutional layer, followed by 16 residual blocks, an upsampling module, and a convolutional layer. For the first convolutional layer, the kernel size is 3×3, the kernel stride is single-pixel, and single-pixel padding is used to control the output size. The output is a three-dimensional shape tensor (H, W, C), where C = 50 represents the number of filters. Then, the two residual blocks include a convolutional layer with 50 filters with a kernel size of 3×3, a rectified linear unit, and a constant scaling layer. The batch normalization layer is removed to ensure image contrast and accelerate the training process. In addition, the constant scaling module between the residual blocks helps to stabilize the network's 125 convergence. The upsampling blocks with different super-resolution scales contain convolutional layers and pixel reshaping modules. The convolutional layer has two filters of 200 and 50, and the pixel reshaping module has a magnification factor of 2. Finally, two high-resolution stripe patterns are generated in the last convolutional layer, where one filter has a kernel size of 3×3. It is worth noting that the trained ×2 nonlinear mapping function can be used to accelerate the training of ×4 and ×8 super-resolution, avoiding entering the local optimal prior.

[0050] For the network training and validation sets, we used a high-resolution camera to capture the corresponding fringe images as the ground truth for the network. These high-resolution fringe images were then degraded using a degradation model to generate low-resolution fringe images, which served as input for network training. In this experiment, we set a downsampling factor of 4, reducing the resolution to 1 / 4 of the original image. For example, a 1024×1024 image was degraded to a 256×256 image.

[0051] An FPP 3D measurement system was constructed, using a low-resolution camera as the acquisition device. To simulate the dynamics of actual measurement, the object was placed on a guide rail and moved during measurement. The acquired fringe pattern was then fed into a trained neural network to generate a high-resolution fringe pattern with restored detail. In this experiment, the super-resolution factor was set to ×4, quadrupling the original image resolution.

[0052] The obtained high-resolution fringes are subjected to three-dimensional reconstruction processes such as phase deconvolution and unwrapping to obtain three-dimensional data of the object.

[0053] In order to verify the present invention, a qualitative and quantitative experiment was carried out using dolls and standard balls. The final experimental results are as follows: Figure 3 It can be proved that the present invention can effectively perform dynamic three-dimensional reconstruction.

Claims

1. A dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction, characterized by: The steps include: Step 1: Design and construct a super-resolution reconstruction neural network that automatically restores details; Step 2: Collect a high-resolution fringe image of the object, build a fringe degradation model, degrade the high-resolution fringe image of the object to obtain a low-resolution fringe image, and construct a data set of several low-resolution fringe images and corresponding high-resolution fringe images; Step 3: Apply the data set in step 2 to the neural network in step 1 to train and learn the mapping relationship of stripe degradation, and obtain the trained neural network; Step 4: Build a 3D measurement system, use a camera to quickly measure the object to be measured, and collect a low-resolution original fringe image of the object to be measured; Step 5: Input the original fringe image from step 4 into the neural network trained in step 3 to obtain a high-resolution image; Step 6: Substitute each point in the high-resolution image obtained in step 5 into the following formula: Where (x, y) is the coordinate of the pixel point, N is the number of steps of a set of phase shift stripes, δ n is the phase shift, I n (x, y) is the pixel grayscale value of the nth phase shift image at point (x, y). The wrapped phase is unwrapped based on the Gray code method to obtain the absolute phase. The three-dimensional information of the object to be measured is reconstructed by combining the obtained calibration parameters and the absolute phase.

2. The dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction according to claim 1, characterized in that: The neural network in step 1 is based on a residual network structure, and the neural network includes a first convolutional layer, several residual structure modules, a second convolutional layer, an upsampling module and a third convolutional layer with jump connections in sequence. The residual structure module includes a convolutional layer, a rectified linear unit, a convolutional layer and a constant scaling layer in sequence, and the upsampling module includes multiple groups of convolutional layers and pixel scrubbing layers arranged alternately.

3. The dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction according to claim 1, characterized in that: The specific steps of obtaining the low-resolution fringe image in step 2 are: using a high-resolution camera to obtain a low-resolution fringe image from each high-resolution fringe image according to a degradation model, and obtaining a corresponding low-resolution image as a network input. The degradation model is as follows: in, represents the i-th high-resolution fringe pattern, P represents the number of phase-shifted sine waves combined with the gray code pattern, H(x,y) represents the optical blur of the fringe pattern, represents the convolution operator, v(x,y) represents random noise, and D represents the downsampling operator of the stripe resolution.

4. The dynamic fringe projection 3D measurement method based on fringe pattern super-resolution reconstruction according to claim 1, characterized in that: The specific steps of obtaining the inferred high-resolution image in step 2 are: after collecting the original fringe image, input it into the trained neural network, and use the following formula to optimize the parameters of the neural network: in, represents the jth high-resolution fringe pattern, S Θ is a nonlinear mapping function, is the L1 norm, Θ is the trainable parameter, i.e., weight and bias, for batch training, T is the size of the mini-batch training when optimizing Θ using the stochastic gradient descent algorithm, the black background in the stripe pattern is not involved in the training, m is the number of valid pixels, and the trained neural network outputs the corresponding 2x, 4x, or 8x resolution image based on the original stripe image without adjusting other parameters.

Images