Efficient fringe projection absolute phase recovery method based on dynamic fringe convolution

Abstract

The invention discloses an efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, and belongs to the technical field of image processing. The problems that in the prior art, absolute phase recovery errors are concentrated on the edge of an object, and the final measurement precision requirement cannot be guaranteed are solved. According to the method of the invention, a collected single high-resolution image is segmented, and the segmented high-resolution image is used as the input of a dynamic stripe convolutional network DFC-Net; sequentially outputting molecular denominator terms M1D1 and M2D2 corresponding to phase and phase shift term arc tangent functions wrapped by a four-step phase shift method through the trained dynamic stripe convolutional network DFC-Net; obtaining a wrapping phase and a fringe order corresponding to the initial fringe image; and the wrapped phase is recovered into absolute phase information depending on the fringe order. According to the technical scheme, the robustness of the measurement process and the precision and efficiency of the measurement result can be improved, and accurate and efficient three-dimensional measurement is achieved.

Description

Technical Field

[0001] The present invention relates to an efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, belonging to the field of image processing. Background Art

[0002] Optical measurement technologies based on structured light are widely used in high-end precision manufacturing, intelligent manufacturing and other fields due to their advantages such as non-contact, high precision, high efficiency, and wide application range. As one of the most widely used optical non-contact active three-dimensional shape measurement methods, fringe projection profilometry has the advantages of small computational load, high measurement accuracy, strong robustness, and dense measurement point clouds. However, it is difficult to balance measurement accuracy and measurement efficiency in actual deployment applications. How to accurately and efficiently obtain the three-dimensional shape information of complex dynamic targets is still an urgent problem to be solved in the current optical measurement field.

[0003] Deep learning technology has broken through the limitation that traditional fringe projection profilometry is difficult to balance measurement efficiency and accuracy. Combining deep learning with color coding, complex network structures, and complex input-output strategies, single-frame unambiguous three-dimensional reconstruction has been initially achieved, and its application has improved various optical problems that have not been solved by fringe projection profilometry, such as additive noise, phase shift error, and intensity non-linearity. However, the absolute phase recovery error is mainly distributed in the object edge region and is difficult to avoid. In this field, deep learning strategies are based on regression prediction, and it is difficult to learn the network for inferring absolute phase information from fringe images, and the final measurement accuracy requirements cannot be guaranteed. Summary of the Invention

[0004] Aiming at the problems existing in the prior art, the present application proposes an efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, which strengthens the feature extraction ability of the deep learning network for fringe images and reduces the object edge error.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is an efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, including the following steps:

[0006] Build a physical measurement system and a simulation measurement system; prepare a data set in the physical measurement system and the simulation measurement system; build a dynamic fringe convolution network model; train the model with the data set prepared by the physical measurement system and the simulation measurement system; the subsequent steps include:

[0007] 1) Segment the collected single high-resolution image as the input of the dynamic fringe convolution network DFC-Net;

[0008] 2) The trained dynamic fringe convolution network DFC-Net sequentially outputs the wrapped phase of the four-step phase-shifting method and the numerator and denominator terms M1D1, M2D2 corresponding to the arctangent function of the phase-shift term;

[0009] 3) Obtain the wrapped phase and fringe order corresponding to the initial fringe image;

[0010] 4) Restore the wrapped phase to absolute phase information depending on the fringe order.

[0011] Optimized, for the above-mentioned efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, in step 1), dynamic fringe convolution is introduced into the U-Net network to replace the standard convolution module therein, forming a DFC-Block.

[0012] Optimized, for the above-mentioned efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, in step 2), a dynamic fringe convolution network is used to obtain the numerator and denominator M1D1 corresponding to the arctangent function of the wrapped phase and the numerator and denominator M2D2 corresponding to the arctangent function of the phase shift term. The four-step phase-shifting method is used to obtain the wrapped phase, fringe order, and absolute phase information.

[0013] Optimized, for the above-mentioned efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, in step 3), the wrapped phase Φ w and the phase shift term α are obtained by the following formula,

[0014]

[0015] where I0, I1, I2, and I3 are the light intensities of each pixel in the four projected phase-shifted fringe images.

[0016] Optimized, for the above-mentioned efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, the four-step phase-shifting method embeds the fringe order into the phase shift domain, and encodes the phase information and fringe order information together into four fringe images in a single mode. Its fringe pattern is expressed as

[0017]

[0018] where I n represents the (n + 1)-th fringe image, n ∈ [0, 3], f is the fringe frequency, k represents the fringe order, encoded as k = int(x / T), int represents the truncation and rounding function, x represents the pixel position, T represents the fringe period, and ρ represents the phase shift factor; when n = 0 / 1, ρ = 1; when n = 2 / 3, ρ = 2;

[0019] Let α be the phase shift term expressed as:

[0020]

[0021] Optimized, for the above-mentioned efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, the fringe order k is obtained by Obtained, where Round represents the rounding function, and α' represents the phase shift term after filtering correction.

[0022] Optimized, in the above-mentioned efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, in step 4), the wrapped phase Φ w Is expanded into a single-period unwrapped absolute phase Φ, Φ = Φw + 2πk.

[0023] The beneficial effects of this application are as follows:

[0024] Aiming at the shortcomings of the prior art, the technical solution of this application studies the characteristics of the coded fringe image in fringe projection profilometry. Applies deep learning technology to construct a three-dimensional measurement model that deeply integrates the physical model and the data model, and clarifies the mapping relationship between the coded fringe image, the deep network model, and the three-dimensional shape of the object to be measured. Improves the robustness of the measurement process, the accuracy and efficiency of the measurement results, and realizes accurate and efficient three-dimensional measurement.

[0025] The technical solution of this application builds a DFC-Net to convert a single fringe image into the numerator and denominator terms corresponding to the wrapped phase and the arctangent function of the phase shift term. The proposed strategy can realize the inference of the absolute phase from a single fringe image without the assistance of an additional mode. To solve the problems of high data set preparation cost and lack of pertinence in the construction of network models in the field of fringe projection by current deep learning technologies, and establish the mapping relationship between the coded fringe image, DFC-Net, and the three-dimensional shape of the object to be measured.

[0026] The technical solution of this application constructs a simulation measurement system relying on the physical measurement system, and the simulation measurement system reproduces the physical measurement system by calibrating parameters. The simulation measurement system simplifies the complex data acquisition process. The dynamic fringe convolution module is used to improve the feature extraction ability for fringe images. According to experimental verification, DFC-Net has a stronger learning ability for fringe images compared with existing deep learning network models, and the recovered absolute phase has higher accuracy. At the same time, it also has strong robustness and generalization ability to different ambient light interferences. Brief Description of the Drawings

[0027] Figure 1 Is the coordinate position map of the dynamic fringe convolution kernel of this application;

[0028] Figure 2 Is the overall process of the dynamic fringe convolution network (DFC-Net) of this application for recovering the absolute phase;

[0029] Figure 3 Is the schematic diagram of the dynamic fringe convolution network model of this application (Dynamic fringe convolution-Network, DFC-Net);

[0030] Figure 4 (a) Schematic diagram of the physical measurement system;

[0031] Figure 4 (b) Rendered image of the simulation measurement system of the present application;

[0032] Figure 5 It is a comparison chart of the MAE error results of the absolute phase recovery of each model. Detailed implementation manners

[0033] The technical features of the present invention will be further elaborated below in combination with specific embodiments.

[0034] The present application provides an efficient fringe projection absolute phase recovery method based on dynamic fringe convolution. The overall process of the dynamic fringe convolution network (DFC-Net) constructed to recover the absolute phase is as Figure 2 shown.

[0035] In the technical solution of the present application, first, a physical measurement system and a simulation measurement system are constructed; a data set is prepared in the physical measurement system and the simulation measurement system; a dynamic fringe convolution network model is constructed; and the model is trained with the data set prepared by the physical measurement system and the simulation measurement system.

[0036] The single high-resolution image collected is segmented and used as the input of the dynamic fringe convolution network DFC-Net.

[0037] The trained dynamic fringe convolution network DFC-Net sequentially outputs the wrapped phase of the four-step phase-shifting method and the numerator and denominator terms M1D1 and M2D2 corresponding to the arctangent function of the phase-shift term.

[0038] Obtain the wrapped phase and the fringe order corresponding to the initial fringe image.

[0039] Finally, the wrapped phase is restored to absolute phase information depending on the fringe order.

[0040] The architecture of the dynamic fringe convolution network constructed in the present application refers to U-Net, as Figure 3 shown. By introducing dynamic fringe convolution into the U-Net network and replacing the standard convolution module therein, a DFC-Block is formed. As a lightweight network, this network has a high feature extraction ability for fringe image features and a high prediction accuracy for fringe images.

[0041] The data set preparation and network training processes are as follows.

[0042] The dataset used for training the DFC-Net network is jointly prepared by a physical measurement system and a simulation measurement system. The physical measurement system includes a DLP 3010 digital projector (1280*720 pixels) and an MV-CA016-10UM camera (1440*1080 pixels). The simulation measurement system takes the calibration parameters of the physical measurement system as input, is built based on the Blender software, and fits the grating projection and camera shooting of the physical measurement system through texture projection and rendering generation. The scenes built by the physical measurement system and the simulation measurement system are as shown in Figure 4 shown. Select 200 3D models of arbitrary categories as the objects to be measured, project four fringe images in sequence, and use the four-step phase-shifting method to sequentially obtain the numerator and denominator of the arctangent function of the wrapped phase (M1D1), the wrapped phase, the numerator and denominator of the arctangent function of the phase-shift term (M2D2), the fringe order, and the absolute phase information, providing the target ground truth for network training and result evaluation.

[0043] In this embodiment, the dataset is divided into 1500, 250, and 250 groups according to training, validation, and testing. 125 groups in the test group are collected and prepared by the physical measurement system. Network training is carried out on a Dell tower workstation Precision 7920. The training is based on the Pytorch deep learning framework, with a mini-batch size of 2 and 200 training rounds. The Adam optimizer is used to optimize the network parameters during the backpropagation process, and the learning rate is initially set to 0.001. In this paper, the structural similarity of images (SSIM) and smooth L1 loss (Smooth L1) are comprehensively used as the loss function, and the foreground region of the fringe image is extracted according to the fringe modulation threshold. Thus, it is ensured that the network can be smoothly fitted during the training process, and the accuracy of the inference result of the network model is further improved.

[0044] Next, in combination with specific embodiments, the principle of the technical solution of the present application will be described in detail.

[0045] The four-step phase-shifting method embeds the fringe order into the phase-shift domain, and jointly encodes the phase information and the fringe order information into four fringe images in a single mode. Its fringe pattern can be expressed by formula (2):

[0046]

[0047] where, I n represents the (n + 1)-th fringe image, n ∈ [0, 3], f is the fringe frequency, k represents the fringe order, encoded as k = int(x / T), int represents the truncation and rounding function, x represents the pixel position, T represents the fringe period, ρ represents the phase-shift factor, when n = 0 / 1, ρ = 1, when n = 2 / 3, ρ = 2.

[0048] Let α be the phase shift term, which can be expressed by Equation (3):

[0049]

[0050] Therefore, the wrapped phase Φ w and the phase shift term α can be obtained from Equation (4) and Equation (5), and the fringe order k can be obtained from Equation (6):

[0051]

[0052] where Round represents the rounding function, and α′ represents the phase shift term after filtering correction. Finally, according to Equation (7), the wrapped phase Φ w is unwrapped into the single-period unwrapped absolute phase Φ.

[0053] Φ = Φ w + 2πk (7)

[0054] The principle of the Dynamic fringe convolution block is as follows.

[0055] The fringe image in fringe projection profilometry contains the edge information of the measured object and the fringe modulation information, which is continuous on the smooth object surface and has a large height jump at the object edge. The coordinate positions of the dynamic fringe convolution kernel proposed in this application are as Figure 1 shown, which can adaptively focus on the local structure of the fringe image and improve the prediction accuracy at the object edge.

[0056] For the standard convolution kernel K, its center coordinate is expressed as K(x i , y i ), and a 3×3 convolution kernel can be expressed by Equation (8):

[0057] K = { (x - 1, y - 1), (x, y), …, (x + 1, y + 1)} (8)

[0058] The dynamic fringe convolution structure is constructed inspired by dynamic convolution, deformable convolution, and dynamic serpentine convolution.

[0059] A deformation offset Δ is added to the standard convolution structure. To ensure that the perception field does not deviate from the target, an iterative method is used to sequentially select the next position of each target to be processed for observation at each position to ensure the continuity of the perception range.

[0060] At the same time, the standard convolution is linearized in the X-axis and Y-axis directions. Taking the convolution kernel with a size of 9 in the X-axis as an example, the specific position of the convolution kernel is expressed as K i±c = (x i±c , y i±c), where c = 0, 1, 2, 3, 4 represents the horizontal distance from the center of the convolution kernel. Starting from the center position K with c = 0 i K i+1 Compared with K i an offset Δ={δ|δ∈[-1,1]} is added to ensure that the linear structure of the convolution kernel accumulates the offset.

[0061] The changes of the dynamic stripe convolution kernel coordinates on the X-axis and Y-axis, K i±c K j±c can be successively expressed as Formulas (9)-(10):

[0062]

[0063] Since the value of the offset is random during the training phase, it usually takes floating-point numbers. Bilinear interpolation is used to adapt to integer coordinates. As shown in Formula (11), the coordinate position K containing floating-point numbers is converted into an integer coordinate position K'.

[0064] K = Σ K′ B(K′,K)·K′ (11)

[0065] Among them, B represents the bilinear interpolation kernel. The one-dimensional kernels decomposed into the X-axis and Y-axis directions can be expressed as Equation (12):

[0066] B(K,K′)=b(K x ,K′ x )·b(K′ y ,K y ) (12)

[0067] Finally, for the dynamic stripe convolution kernel with a size of 9, combining the two-dimensional changes of the X-axis and Y-axis, the optional receptive field range is within a 9×9 area, enhancing the ability to perceive key features in the stripe image.

[0068] In this embodiment, several commonly used network models in the art are selected to be compared with the proposed dynamic stripe convolution network. To verify that the proposed dynamic stripe convolution network in this application has higher prediction accuracy for the strategy of directly inferring both the numerator and denominator terms of the wrapped phase and the stripe order from a single stripe image. Table 1 shows the comparison results of the minimum loss results and the mean absolute error (MAE) of the recovered absolute phase during the training processes of UNet, MultiResUNet, DA-UNet, R2AttentionUNet, and DFC-Net. Further, to verify the anti-interference ability of the method proposed in this paper to ambient light, the light intensity is randomly changed in the test set, and the MAE errors of the absolute phase recovery of each model are compared as Figure 5 shown.

[0069] Comparison of Training Results and Absolute Phase Recovery Error in Table 1

[0070]

[0071] Of course, the above description is not a limitation of the present invention, nor is the present invention limited to the above examples. Any changes, modifications, additions or substitutions made by those of ordinary skill in the art within the scope of the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. An efficient fringe projection absolute phase recovery method based on dynamic fringe convolution, characterized in that: Including the following steps: 1) Segment the single high-resolution image collected as the input of the dynamic fringe convolution network DFC-Net; 2) The trained dynamic fringe convolution network DFC-Net sequentially outputs the wrapped phase of the four-step phase-shifting method and the numerator and denominator terms M1D1 and M2D2 corresponding to the arctangent function of the phase-shift term; 3) Obtain the wrapped phase and fringe order corresponding to the initial fringe image; 4) Restore the wrapped phase to absolute phase information depending on the fringe order.

2. The efficient fringe projection absolute phase recovery method based on dynamic fringe convolution according to claim 1, characterized in that: In step 1), the dynamic fringe convolution is introduced into the U-Net network to replace the standard convolution module therein to form a DFC-Block.

3. The efficient fringe projection absolute phase recovery method based on dynamic fringe convolution according to claim 1, characterized in that: In step 2), the dynamic fringe convolution network is adopted to obtain the numerator and denominator M1D1 corresponding to the arctangent function of the wrapped phase and the numerator and denominator M2D2 corresponding to the arctangent function of the phase-shift term, and the four-step phase-shifting method is adopted to obtain the wrapped phase, fringe order, and absolute phase information.

4. The efficient fringe projection absolute phase recovery method based on dynamic fringe convolution according to claim 1, characterized in that: In step 3), the wrapped phase Φ w corresponding to the initial fringe image and the phase shift term α are obtained by the following formula: Wherein, I0, I1, I2, and I3 are the light intensities of each pixel in the four projected phase-shifted fringe images.

5. The efficient fringe projection absolute phase recovery method based on dynamic fringe convolution according to claim 4, characterized in that: The four-step phase-shifting method embeds the fringe order into the phase-shift domain and jointly encodes the phase information and fringe order information into four fringe images in a single mode, and its fringe pattern is expressed as where I n represents the (n + 1)-th fringe image, n ∈ [0, 3], f is the fringe frequency, k represents the fringe order, encoded as k = int(x / T), int represents the truncation and rounding function, x represents the pixel position, T represents the fringe period, and ρ represents the phase shift factor; when n = 0 / 1, ρ = 1; when n = 2 / 3, ρ = 2; Let α be the phase-shift term expressed as:

6. The efficient fringe projection absolute phase recovery method based on dynamic fringe convolution according to claim 4, characterized in that: The fringe order k is obtained from , where Round represents the rounding function and α' represents the phase shift term after filtering correction.

7. The efficient fringe projection absolute phase recovery method based on dynamic fringe convolution according to claim 4, characterized in that: In step 4), the wrapped phase Φ w is unwrapped into the single-period unwrapped absolute phase Φ, Φ = Φ w + 2πk.

Images