A saturated fringe self-repairing three-dimensional measurement method based on a generative adversarial network

Abstract

The application belongs to the technical field of three-dimensional measurement, and proposes a saturated fringe self-repairing three-dimensional measurement method based on a generative adversarial network. The method is realized through the following steps: firstly, a image sample data set containing saturated fringe and corresponding real fringe is collected; secondly, a hybrid network architecture is designed, the architecture takes an improved U-Net as a generator and a convolutional neural network as a discriminator to constitute a generative adversarial network for adversarial training; then, the saturated fringe image is input into the pre-trained generative adversarial network model to output the recovered fringe image; finally, the absolute phase information is obtained by phase extraction based on the recovered fringe image, and three-dimensional reconstruction is completed.

Description

Technical Field

[0001] This application relates to the field of three-dimensional measurement technology and provides a three-dimensional measurement method for saturation stripe self-healing based on generative adversarial networks. Background Technology

[0002] In fringe projection 3D measurement, the limited dynamic range of the camera often leads to saturation of the fringes captured on the highlight surface of the measured object, resulting in 3D measurement errors. Traditional high dynamic range methods include hardware-assisted and image-assisted techniques, but these methods are not suitable for single-frame fringe dynamic 3D measurement.

[0003] Deep learning-based image inpainting methods are better suited for frame-by-frame restoration of saturation stripes. While the U-Net network structure performs well in image inpainting, its pooling operation is prone to losing detailed features and struggles to effectively maintain the global structure and phase continuity of the stripes. Generative adversarial networks can enhance the visual realism of images, but the training process is unstable, the discriminator is prone to gradient vanishing, and the restoration results often show errors at the phase level.

[0004] Therefore, developing a three-dimensional measurement method that can achieve high-quality, high-fidelity self-repair of single-frame saturation fringes without relying on additional hardware or multiple frames of images, while ensuring phase accuracy, has significant theoretical and engineering application value. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a 3D measurement method for saturation fringes self-healing based on generative adversarial networks, used to achieve high-precision 3D topography measurement of highly reflective surfaces. This method can accurately reconstruct the 3D structure of objects by performing high-quality repair of saturation fringes without the need for additional images or hardware assistance.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a three-dimensional measurement method for saturation stripe self-healing based on generative adversarial networks, comprising the following steps:

[0007] S1. Collect an image sample dataset containing saturation stripes and corresponding real stripes;

[0008] S2. Construct a hybrid model architecture GAN-U-Net that combines an improved U-Net with a generative adversarial network;

[0009] S3. Use the dataset to perform adversarial training on the hybrid model to obtain a trained stripe restoration model;

[0010] S4. Input the saturation stripes to be measured into the trained model and output the repaired stripe image;

[0011] S5. Extract phase information based on the repaired stripes and perform three-dimensional reconstruction.

[0012] Furthermore, in S1, an image sample dataset containing saturation stripes and corresponding real stripes is collected using the following method:

[0013] S11, Project N-step phase-shifted fringes, with the height and width of the fringe pattern being H and W respectively. The intensity of the fringes captured by the camera can be expressed as formula (1):

[0014] (1);

[0015] In the formula, (x, y) are the camera pixel coordinates, a(x, y) and b(x, y) represent the background and modulation intensity, respectively, and ϕ(x, y) is the phase to be solved. For an 8-bit grayscale camera, the maximum grayscale value is 255; exceeding this value indicates saturation.

[0016] S12. For the surface of the object under test with high reflectivity, collect the saturation stripe image caused by local overexposure as a saturation sample.

[0017] S13. Spray a developer onto the surface of the object being tested and collect an unexposed stripe image of the same scene as the corresponding real sample.

[0018] Furthermore, in S2, a hybrid model architecture GAN-U-Net combining an improved U-Net and a generative adversarial network is constructed using the following method:

[0019] S21. Design an improved U-Net generator. The generator's encoder uses convolutional layers with a stride of 2 for downsampling, avoiding pooling layers, and employs 4×4 convolutional kernels to expand the receptive field. The generator uses asymmetric skip connections to achieve the fusion of multi-scale features from the encoder and the decoder. A residual module is added at the end of the encoder.

[0020] Its output satisfies formula (2):

[0021] (2);

[0022] In the formula, the input feature is x, the output feature is y, F(•) is the forward mapping achieved by combining a 3×3 convolution kernel with the ReLU activation function, and δ(•) is the input feature transformation using a 1×1 convolution kernel with the same number of output channels.

[0023] S22. Constructing a Generative Adversarial Network (GAN) model consists of a generator G and a discriminator D. The generator G adopts an improved U-Net structure, and the discriminator D is a convolutional neural network. The optimization objective of GAN can be expressed as formula (3):

[0024] (3);

[0025] In the formula, I o For the input reflection stripes, I r For the label (true value) of the stripes, G(I) o ) represents the generated stripes, and E represents the mathematical expectation.

[0026] Furthermore, S3 uses the dataset to perform adversarial training on the hybrid model to obtain a trained stripe restoration model, using the following method:

[0027] S31. To optimize the model, set the training parameters and use an adaptive moment estimator optimizer for iterative training.

[0028] S32. Train the discriminator with fixed generator parameters, and then train the generator with fixed discriminator parameters again.

[0029] S32. Repeat step S32 until the model converges, and save the generator parameters with the best performance.

[0030] Furthermore, in S4, the saturation stripes to be measured are input into the trained model, and the repaired stripe image is output using the following method:

[0031] S41. Load the generator parameters obtained after training into the improved U-Net structure to form the final stripe repair model.

[0032] S42. Input the saturated stripe image to be tested into the stripe restoration model.

[0033] Furthermore, in S5, phase information is extracted based on the repaired fringes, and three-dimensional reconstruction is performed using the following method:

[0034] S51. Based on the N-step phase shift method, the repaired stripe image I' n The wrapping phase ϕ of (x,y) w (x,y) can be expressed as formula (4):

[0035] (4);

[0036] In the formula, n is the phase shift index, and n = 1, 2, ..., N. N is the sum of the number of phase shift steps.

[0037] S52. Using the time-domain phase expansion method based on three-frequency heterodyne, the wrapping phases ϕ1, ϕ2, and ϕ3 with fringe frequencies f1, f2, and f3 are obtained respectively, and the heterodyne wrapping phase is calculated according to the preset frequency relationship, as shown in formula (5):

[0038] (5);

[0039] In the formula, pixel coordinates (x, y) are ignored to simplify the writing of the formula.

[0040] S53. Calculate the high-frequency absolute phase with higher accuracy from the low-frequency absolute phase, as shown in formula (6):

[0041] (6);

[0042] The fringe order k is calculated using formula (7). m and k h :

[0043] (7);

[0044] In the formula, round is a rounding function.

[0045] S54. Combine the high-frequency absolute phase obtained above with the system calibration parameters, calculate the three-dimensional coordinates of the surface of the object being measured using the triangulation principle, and complete the three-dimensional reconstruction.

[0046] Compared with the prior art, the beneficial effects of this application are as follows:

[0047] This paper proposes a hybrid network architecture (GAN-U-Net) combining an improved U-Net and a generative adversarial network (GAN) for self-healing of saturation fringes. High-quality self-healing of single-frame saturation fringes is achieved without relying on additional auxiliary images or hardware adjustments, thus enabling high-precision 3D topography reconstruction of highly reflective surfaces. This solves the problem that traditional high dynamic range methods are unsuitable for single-frame, dynamic 3D measurement scenarios. The proposed method requires no additional image acquisition or hardware assistance and is suitable for dynamic 3D measurement of specular surfaces. Attached Figure Description

[0048] Figure 1 The dataset contains input samples and their corresponding ground truth labels.

[0049] Figure 2 The proposed network architecture.

[0050] Figure 3 This is a TPU-based tri-frequency heterodyne.

[0051] Figure 4 A comparison of stripe restoration and absolute phase reconstruction for a vase. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0053] This invention provides a 3D measurement method for saturation fringes self-healing based on generative adversarial networks (GANs), enabling high-precision 3D topography measurement of highly reflective surfaces. This method can accurately reconstruct the 3D shape of an object by performing high-quality self-healing of saturation fringes without the need for additional images or hardware assistance.

[0054] The method of the present invention includes the following steps:

[0055] 1. Collect image sample dataset:

[0056] First, a measurement system is established, consisting of a camera, a projector, and a computer. The projected fringe period numbers based on a three-frequency heterodyne TPU are {73, 64, 56}, with frequencies of {74 / W, 64 / W, 56 / W}, and the fringe pattern width W = 912 pixels. The camera captures the intensity of the fringe pattern projected onto the object surface by the projector, and the intensity of the fringe projection is calculated using formula (1):

[0057] (1);

[0058] In the formula, (x, y) are the camera pixel coordinates, a and b represent the background and modulation intensity, respectively, and ϕ is the phase to be solved. For an 8-bit grayscale camera, the maximum grayscale value is 255; exceeding this value indicates saturation.

[0059] This dataset uses Figure 1 The method shown is constructed where the input is a partially or fully saturated capture stripe, and the ground truth label is an unsaturated stripe after developer spraying. To maximize the network's generalization ability, we select various real-world objects and randomly combine them into 180 different scenes. This 180-scene dataset contains 120 training scenes, 30 validation scenes, and 30 test scenes, with each scene containing 12 saturated stripe images and 12 unsaturated stripe images. Representative samples are shown below. Figure 1 As shown.

[0060] 2. Construct a GAN-U-Net hybrid network model:

[0061] like Figure 2 As shown in (a), the GAN network model consists of a generator and a discriminator. The generator is responsible for generating images, while the discriminator is responsible for determining whether an image was generated by the generator or is the original image. There is an adversarial game relationship between the generator and the discriminator. GAN utilizes this characteristic to make the images generated by the generator more realistic. The optimization objective of GAN can be expressed as formula (2):

[0062] (2);

[0063] In the formula, I o For the input reflection stripes, I rFor the label (true value) of the stripes, G(I) o ) represents the generated stripes, and E represents the mathematical expectation.

[0064] The generator G of the GAN employs an improved U-Net structure. The encoder of the generator mainly consists of convolutional layers and ReLU activation functions. It uses convolutional layers with a stride of 2 for downsampling, avoids pooling layers, and uses 4×4 convolutional kernels to expand the receptive field. Figure 2 As shown in (b); the generator employs asymmetric skip connections to achieve the fusion of multi-scale features from the encoder and the decoder; a residual module is added to the end of the encoder of the generator, such as... Figure 2 As shown in (c), its output satisfies formula (3):

[0065] (3);

[0066] In the formula, the input feature is x, the output feature is y, F(•) is the forward mapping achieved by combining a 3×3 convolution kernel with the ReLU activation function, and δ(•) is the input feature transformation using a 1×1 convolution kernel with the same number of output channels.

[0067] 3. Adversarial Training of GAN-U-Net Models

[0068] The model was implemented using the PyTorch 1.13.0 deep learning framework and trained on an NVIDIA RTX 4070 graphics card (12GB VRAM). During training, the generator G and discriminator D were subjected to adversarial optimization according to formula (2), where the network structure parameters of G were kept to be minimized, while those of D were kept to be maximized. During a specific training process, when one parameter was trained, the other parameters remained fixed. As the adversarial training progressed, G and D reached a dynamic equilibrium (i.e., Nash equilibrium), ultimately achieving the goal of making the generated image infinitely close to the original image.

[0069] The Adam adaptive moment estimator was used for 300 iterations of training with an initial learning rate of 0.001. Mean squared error (MSE) was chosen as the loss function to measure the phase error between the network output and the label. Mini-batch gradient descent was employed during training, with a batch size of 2 to effectively improve training efficiency.

[0070] After 300 rounds of training, the optimal training result was selected as the network parameters for G and D.

[0071] 4. Saturation stripe repair

[0072] The generator parameters obtained after training are loaded into the improved U-Net structure to form the final stripe inpainting model. The saturated stripe image I to be inpainted is then processed. nThe model is input with (x, y). The generator fuses features from different levels of the encoder with the corresponding features of the decoder through asymmetric skip connections. The decoder gradually reconstructs the texture and modulation information of the stripes. The final output is a high-quality restored stripe image I'. n (x,y), this image will be used for subsequent phase extraction and 3D reconstruction, such as Figure 2 As shown in (a).

[0073] 5. Phase extraction and 3D reconstruction

[0074] Based on the four-step phase-shifting method, the restored fringe image I' n The wrapping phase ϕ of (x,y) w (x,y) can be expressed as formula (4):

[0075] (4);

[0076] In the formula, n represents the phase shift index, and n = 1, 2, ..., N. N is the sum of the phase shift steps. The phase range of this package is (-π, π], and there is a 2π periodic ambiguity, so phase unwrapping is required to obtain the absolute phase.

[0077] A time-domain phase expansion method based on three-frequency heterodyne is used to calculate the high-frequency absolute phase, such as... Figure 3 As shown. ϕ1, ϕ2, and ϕ3 correspond to the wrapping phases with fringe frequencies f1, f2, and f3, respectively. The fringe frequencies satisfy f1 > f2 > f3, f... 1-2 =f1-f2, f 2-3 =f2-f3 and f 1-2-3 =f 1-2 -f 2-3 =1. Therefore, we can obtain formula (5):

[0078] (5);

[0079] In the formula, pixel coordinates (x, y) are ignored to simplify the writing of the formula.

[0080] Using formula (6), the high-frequency absolute phase with higher accuracy is calculated from the low-frequency absolute phase:

[0081] (6);

[0082] The fringe order k is calculated using formula (7). m and k h :

[0083] (7);

[0084] In the formula, round is a rounding function.

[0085] After obtaining the high-frequency absolute phase Φ1, the camera pixel (x,y) and the projector pixel (x) are compared. p ,y p Establish a mapping relationship, as shown in formula (10):

[0086] (10);

[0087] Based on the system calibration parameters of the camera and projector, the three-dimensional coordinates of the surface of the object under test are calculated by the principle of triangulation, and the three-dimensional shape reconstruction is completed.

[0088] 5. Algorithm:

[0089] The proposed method is illustrated through experiments on vases, such as... Figure 4 As shown.

[0090] S1, Collect image sample dataset

[0091] The camera captures images of saturated stripes on a vase containing locally overexposed areas, which are used as saturation samples, such as... Figure 4 As shown in (a), a developer is uniformly sprayed onto its surface to form a diffuse reflection layer, resulting in unsaturated stripes, as shown in [a]. Figure 4 As shown in (b).

[0092] S2, Construct and train the GAN-U-Net hybrid network model

[0093] Design an improved U-Net as a generator, such as Figure 2 As shown in (b), the hybrid model is subjected to adversarial training to obtain a trained stripe repair model.

[0094] S3, Saturation Stripe Repair

[0095] For comparison, two different restoration methods were applied to the collected saturation stripes on the vase. Figure 4 (c)-(d) show the 3D measurement results after stripe restoration using the U-Net method and the proposed GAN-U-Net method. Figure 4 The cross-sectional comparison of (e)-(f) shows that the saturated region is highly consistent with the real data after being repaired, indicating that both the U-Net method and the GAN-U-Net method can effectively repair saturation stripes.

[0096] S4, Absolute Phase Calculation

[0097] The absolute phase is calculated using formula (6). Figure 4 (g)-(j) respectively show the saturation fringes, the true value, the fringe restoration effect based on U-Net, and the absolute phase result based on GAN-U-Net. Figure 4 (g) compared to Figure 4 The absolute phase accuracy of (i) and 4(j) is significantly improved. Figure 4 (h) In comparison, due to the prediction bias of U-Net and GAN-UNet in stripe restoration, its absolute phase is not as good as... Figure 4 (h) Smooth.

[0098] S5, Quantitative Assessment of Phase Error

[0099] To achieve quantitative comparison, we used Peak Signal-to-Noise Ratio (PSNR) to measure image similarity and Root Mean Square Error (RMSE) to evaluate the accuracy of absolute phase. The comparative results show that the proposed GAN-U-Net method improves the PSNR of stripes by 4.03% compared to the U-Net method. Correspondingly, the RMSE of the absolute phase obtained after stripe restoration by the proposed GAN-U-Net method is also superior to U-Net, reaching 64.15%. This demonstrates that the neural network model proposed in this invention has a greater advantage in restoring saturated stripes.

[0100] A 3D measurement method for self-healing saturation stripes based on generative adversarial networks includes a data acquisition unit, a data processing unit, and an output unit.

[0101] The data acquisition unit contains a dataset of image samples of saturated stripes and corresponding real stripes;

[0102] Data processing unit: Constructs a hybrid model architecture GAN-U-Net that combines an improved U-Net and a generative adversarial network; uses the dataset to perform adversarial training on the hybrid model to obtain a trained stripe restoration model; inputs the saturated stripes to be measured into the trained model and outputs the restored stripe image; extracts phase information based on the restored stripes and completes the 3D shape reconstruction of the target object.

[0103] Output unit: Visualizes the processing results.

Claims

1. A three-dimensional measurement method for self-healing saturation stripes based on generative adversarial networks, characterized in that, Includes the following steps: S1. Collect an image sample dataset containing saturation stripes and corresponding real stripes; S2. Construct a hybrid model architecture GAN-U-Net that combines an improved U-Net with a generative adversarial network; S3. Use the dataset to perform adversarial training on the hybrid model to obtain a trained stripe restoration model; S4. Input the saturation stripes to be measured into the trained model and output the repaired stripe image; S5. Extract phase information based on the repaired stripes and perform three-dimensional reconstruction.

2. The three-dimensional measurement method for saturation stripe self-healing based on generative adversarial networks according to claim 1, characterized in that, The specific steps for S1 are as follows: S11, Project N-step phase-shifted fringes, with the height and width of the fringe pattern being H and W respectively. The intensity of the fringes captured by the camera can be expressed as formula (1): （1）； In the formula, (x, y) are the camera pixel coordinates, a(x, y) and b(x, y) represent the background and modulation intensity, respectively, and ϕ(x, y) is the phase to be solved. For an 8-bit grayscale camera, the maximum grayscale value is 255; exceeding this value indicates saturation. S12. For the surface of the object under test with high reflectivity, collect the saturation stripe image caused by local overexposure as a saturation sample. S13. Spray a developer onto the surface of the object being tested and collect an unexposed stripe image of the same scene as the corresponding real sample.

3. The 3D measurement method for self-healing saturation stripes based on generative adversarial networks according to claim 1, characterized in that, The specific steps for S2 are as follows: S21. Design an improved U-Net generator. The generator's encoder uses convolutional layers with a stride of 2 for downsampling, avoiding pooling layers, and employs 4×4 convolutional kernels to expand the receptive field. The generator uses asymmetric skip connections to achieve the fusion of multi-scale features from the encoder and the decoder. A residual module is added at the end of the encoder. Its output satisfies formula (2): （2）； In the formula, the input feature is x, the output feature is y, F(•) is the forward mapping achieved by combining a 3×3 convolution kernel with the ReLU activation function, and δ(•) is the input feature transformation using a 1×1 convolution kernel with the same number of output channels. S22. The Generative Adversarial Network (GAN) model consists of a generator G and a discriminator D. The generator G adopts an improved U-Net structure, and the discriminator D is a convolutional neural network. The optimization objective V(G,D) of GAN can be expressed as formula (3): （3）； In the formula, I o For the input reflection stripes, I r For the label (true value) of the stripes, G(I) o ) represents the generated stripes, and E represents the mathematical expectation.

4. The three-dimensional measurement method for saturation stripe self-healing based on generative adversarial networks according to claim 1, characterized in that, The specific steps for S3 are as follows: S31. To optimize the model, set the training parameters and use an adaptive moment estimator optimizer for iterative training. S32. Train the discriminator with fixed generator parameters, and then train the generator with fixed discriminator parameters again. S33. Repeat step S32 until the model converges, and save the generator parameters with the best performance.

5. The three-dimensional measurement method for saturation stripe self-healing based on generative adversarial networks according to claim 1, characterized in that, The specific steps for S4 are as follows: S41. Load the generator parameters obtained after training into the improved U-Net structure to form the final stripe repair model. S42. Input the saturated stripe image into the stripe restoration model.

6. The three-dimensional measurement method for saturation stripe self-healing based on generative adversarial networks according to claim 1, characterized in that, The specific steps for S5 are as follows: S51. Based on the N-step phase shift method, the repaired stripe image I' n The wrapping phase ϕ of (x,y) w (x,y) can be expressed as formula (4): （4）； In the formula, n is the phase shift index, and n = 1, 2, ..., N. N is the sum of the number of phase shift steps. S52. Using the time-domain phase expansion method based on three-frequency heterodyne, the wrapping phases ϕ1, ϕ2, and ϕ3 with fringe frequencies f1, f2, and f3 are obtained respectively, and the heterodyne wrapping phase is calculated according to the preset frequency relationship, as shown in formula (5): （5）； In the formula, pixel coordinates (x, y) are ignored to simplify the writing of the formula. S53. Calculate the high-frequency absolute phase with higher accuracy from the low-frequency absolute phase, as shown in formula (6): （6）； The fringe order k is calculated using formula (7). m and k h : （7）； In the formula, round is a rounding function. S54. Combine the high-frequency absolute phase obtained above with the system calibration parameters, calculate the three-dimensional coordinates of the surface of the object being measured using the triangulation principle, and complete the three-dimensional reconstruction.

7. A three-dimensional measurement method for self-healing saturation stripes based on generative adversarial networks, employing the method described in any one of claims 1-6, characterized in that, It includes a data acquisition unit, a data processing unit, and an output unit; The data acquisition unit contains a dataset of image samples of saturated stripes and corresponding real stripes; The data processing unit: constructs a hybrid model architecture GAN-U-Net that combines an improved U-Net and a generative adversarial network; uses the dataset to perform adversarial training on the hybrid model to obtain a trained stripe restoration model; inputs the saturated stripes to be measured into the trained model and outputs the restored stripe image; extracts phase information based on the restored stripes and completes the three-dimensional shape reconstruction of the target object. Output unit: Visualizes the processing results.

Images