A neural network mapping method for degraded fringe enhancement
By collecting and processing the sinusoidal and cosine components of the phase-shifted fringe diagram in high dynamic range scenarios, the neural network mapping mode is constructed, and the measurement accuracy and robustness in fringe projection contour is solved, and the network is quickly converged and high precision is achieved. three-dimensional surface shape recovery.
Patent Information
- Application Number
- CN202311358111.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-10-18
AI Technical Summary
In the high dynamic range scenario, the stripe projection contour technique is difficult to effectively measure, resulting in overexposure or darkness of the collected stripes, which leads to the reconstructed three-dimensional surface shape error, and the network of deep learning methods is difficult to converge and measurement accuracy. and poor robustness.
The projector-camera system uses a projector-camera system to collect phase shift fringes in the degradation measurement scenario, extract the degraded sinusoidal component and cosine component, and obtain the true component through multiple exposure methods or increasing the phase shift step count to construct the sinusoidal component-positive linear component. Mapping mode, using the Back propagation algorithm to train the neural network, output the repaired cosine component.
Effectively reduce the burden of network training, promote network training convergence, improve model generalization ability and restore texture details, and improve measurement accuracy and robustness under different lighting conditions.
Smart Images

Figure CN117333394B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a neural network mapping method for degenerate fringe enhancement, belonging to the technical field of optical measurement. Background Art
[0002] In the process of digitalization of industrial production and detection, how to measure complex scenes such as high dynamic range (HDR), high speed, and large depth of field is a difficult problem for mainstream fringe projection profilometry. In order to demonstrate the advantages of the present invention, the following description is made for the problem of high dynamic range measurement. Specifically, fringe projection profilometry first projects a fringe pattern on the measured surface by the projection end, and then collects the deformed fringe pattern by the imaging end, and finally restores the surface information of the object. However, for objects with a high dynamic range, due to the limited dynamic range of the imaging device, the collected fringes will be overexposed or too dark, and ultimately an erroneous three-dimensional surface will be reconstructed.
[0003] As a data-driven technology, deep learning has shown strong ability to solve many challenging problems in the field of optical metrology. For the HDR problem, many technical solutions based on deep learning have been proposed. According to the different mapping paradigms of neural networks, they can be divided into two categories: fringe-fringe deep learning methods and fringe-phase deep learning methods.
[0004] 1. The steps of the fringe-fringe deep learning method are: (1) establish a data pair of "degraded fringe image-true fringe image" to construct a data set; (2) complete network training; (3) input the degraded fringe image into the network to obtain the corresponding high-quality fringe image; (4) extract the phase, and then combine it with the system calibration to obtain the object surface shape. This method uses the fringe intensity containing redundant information as the learning object, such as the background light intensity and modulation degree in the fringe. For surface shape data with complex reflectivity under different lighting conditions, there is a large scale difference between the degraded image and the corresponding true value, and due to the uneven surface reflectivity, the intensity of the degraded image and the corresponding true value varies dramatically, so this type of method is difficult for their network to converge, and thus exhibits poor measurement accuracy and robustness.
[0005] Second, the steps of the fringe-phase deep learning method are: (1) establish the data pair of "degraded fringe pattern-true value truncated phase numerator and denominator" to construct a data set; (2) complete network training; (3) input the degraded fringe pattern into the network to obtain the numerator and denominator of the corresponding high-quality truncated phase; (4) complete the phase calculation, and then combine the system calibration to obtain the object surface shape. This type of method needs to learn the modulation and phase in the fringes. Although the redundant background light intensity is removed, for surface shape data with complex reflectivity under different lighting conditions, their networks are still difficult to converge, and the measurement accuracy and robustness are also poor. Summary of the invention
[0006] In order to overcome the problems in the prior art, the present invention provides a neural network mapping method for degenerate stripe enhancement.
[0007] The present invention provides a technical solution to solve the above technical problem: a neural network mapping method for degenerate fringe enhancement, comprising the following steps:
[0008] Step S1, using a projector-camera system to complete data acquisition in a degradation measurement scenario to obtain a phase-shift fringe pattern;
[0009] Step S2: extracting the degenerate sinusoidal component S from the phase-shifted fringe pattern d and the degenerate cosine component C d ;
[0010] Step S3: using a multiple exposure method or a method of increasing the number of phase shift steps to obtain the degenerate sinusoidal component S d , degenerate cosine component C d The corresponding true value sine component S gt , true value cosine component C gt ;
[0011] Step S4: Degenerate the sinusoidal component S d , degenerate cosine component C d As the network input, the true value sinusoidal component S gt , true value cosine component C gt As the network learning target, a mapping model of sinusoidal component-positive linear component is constructed;
[0012] Step S5, using the Back propagation algorithm to complete the network parameter update, so as to obtain a trained neural network model;
[0013] Step S6: input the degraded sine and cosine components that have not participated in the training into the trained neural network model, and output the repaired sine and cosine components.
[0014] A further technical solution is that the degraded measurement scene in step S1 includes a high dynamic range scene, a low projection illumination and low camera exposure scene, a scene with complex scattering and refraction of light, a measurement scene that is not properly defocused, and a measurement scene with nonlinearity.
[0015] A further technical solution is that there are three ways to obtain the degenerate sine and cosine components in step S2; the first is to extract the truncated phase from the phase-shifted fringe pattern and then extract the sine and cosine components; the second is to extract the modulation degree of the fringes from the phase-shifted fringe pattern and then extract the sine and cosine components; the third is to extract the background light intensity and modulation degree from the phase-shifted fringe pattern and then extract the sine and cosine components.
[0016] A further technical solution is that the first extraction formula is:
[0017]
[0018] Where: is the i-th fringe pattern; is the degenerate truncated phase; δ i =2πi / N is the phase shift; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; N is the number of phase shift steps
[0019] A further technical solution is that the second extraction formula is:
[0020]
[0021] Where: B d is the modulation index; δ i =2πi / N is the phase shift; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; N is the number of phase shift steps; is the i-th fringe pattern.
[0022] A further technical solution is that the third extraction formula is:
[0023]
[0024] Where: A d is the background light intensity; B d For the modulation system; is the i-th fringe pattern; δ i =2πi / N is the phase shift; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; N is the number of phase shift steps; To find the derivative operator.
[0025] A further technical solution is to calculate the degenerate sinusoidal component S in the three extraction processes by the formula d , degenerate cosine component C d Or the degenerate sinusoidal component S can be calculated by the formula d , degenerate cosine component C d One of the degenerate sinusoidal components S d Or the degenerate cosine component C d Derivatively obtain the degenerate cosine component C d Or the degenerate sinusoidal component S d .
[0026] A further technical solution is that the multiple exposure method in step S3 includes a method of modulating the projection light intensity, imaging exposure time, and using a multi-channel imaging device.
[0027] A further technical solution is that the network mapping mode of the sinusoidal component-positive linear component is:
[0028]
[0029] Where: θ * is the optimized network parameter; G θ is the network mapping; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; S gt is the true value sine component; C gt is the true valued cosine component.
[0030] The present invention has the following beneficial effects: the present invention can effectively reduce the burden of network training, promote network training convergence, and has the advantages of strong model generalization ability and strong ability to restore texture details. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 This is a schematic diagram showing the comparison between the degenerate sinusoidal component and the true sinusoidal component;
[0032] Figure 2 The following are the neural network training results of the three techniques;
[0033] Figure 3 The following are the measurement results of the three methods under different lighting conditions;
[0034] Figure 4 Phase detail reconstruction results of three methods under low projection brightness conditions;
[0035] Figure 5 This is a measurement result of a standard part under low projection brightness conditions. DETAILED DESCRIPTION
[0036] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] A neural network mapping method for degenerate fringe enhancement of the present invention comprises the following steps:
[0038] Step S1, using a projector-camera system, in a degraded measurement scenario (including a high dynamic range scenario, a low projection illumination and low camera exposure scenario, a scenario with complex scattering and refraction of light, a measurement scenario without proper defocusing, a measurement scenario with nonlinearity, etc. Such scenarios may cause noise and non-sinusoidal distortion in the collected fringes), complete data acquisition to obtain a phase-shifted fringe pattern;
[0039] Its phase shift fringe diagram is expressed as:
[0040]
[0041] Where: represents the i-th fringe pattern; A d and B d are background light intensity and modulation degree respectively; is the degenerate unfolded phase; N is the number of phase shift steps;
[0042] Step S2: extracting the degenerate sinusoidal component S from the phase-shifted fringe pattern d and cosine component C d ;
[0043] There are three ways to extract the degenerate sinusoidal component S d and cosine component C d ;
[0044] The first is to extract the truncated phase from the phase-shifted fringe pattern, and then extract the sine and cosine components:
[0045]
[0046] Where: represents the i-th fringe pattern; is the degenerate truncated phase; δ i =2πi / N is the phase shift; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; N is the number of phase shift steps; i is the number of fringe patterns.
[0047] It should be noted that by differentiating the degenerate cosine component C d and the degenerate sinusoidal component S d can be converted to each other. And, the degenerate truncated phase and the true value truncated phase The following relationship exists: n is the phase error introduced by the degraded environment.
[0048] The second method is to extract the modulation degree of the fringe from the phase-shifted fringe pattern, and then extract the sine and cosine components:
[0049]
[0050] Where: B d is the modulation degree. Similarly, by differentiating the degenerate cosine component C d and the degenerate sinusoidal component S d Can be converted into each other.
[0051] The third method is to extract the background light intensity and modulation degree from the phase-shifted fringe pattern, and then extract the sine and cosine components:
[0052]
[0053] Where: A d is the background light intensity; is the derivative operator. Similarly, the degenerate cosine component C can also be obtained first d , and then derive to calculate the degenerate sine component S d .
[0054] Step S3, using a multiple exposure method such as modulating the projection light intensity, imaging exposure time, using a multi-channel imaging device, or increasing the number of phase shift steps to obtain the degenerate sinusoidal component S d and the degenerate cosine component C d The corresponding true value sine component S gt , true value cosine component C gt ;
[0055] Degenerate sinusoidal component S d and the true value sine component S gt The comparison results are as follows Figure 1 As shown. It can be seen that the degenerate sinusoidal component S d There is obvious noise, but the true value sinusoidal component S gt has a smooth distribution, then from the degenerate sinusoidal component S d To the true value sine component S gt The mapping can also be considered as a denoising process. In addition, after normalization using trigonometric functions, they are all unified to the interval [-1,1]. This operation can suppress the scale difference between stripe data, ultimately promoting the convergence of the network and improving the generalization ability of the network model;
[0056] Step S4: Degenerate the sinusoidal component S d , degenerate cosine component C d As the network input, the true value sinusoidal component S gt , true value cosine component C gt As a network learning target, a network mapping model of sinusoidal components-sinusoidal components is constructed;
[0057]
[0058] Where: θ * is the optimized network parameter; Gθ is the network mapping; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; S gt is the true value sine component; C gt is the true value cosine component;
[0059] Step S5, using the back propagation algorithm to complete the neural network parameter update, so as to obtain a trained neural network model;
[0060] Step S6: The degraded sinusoidal component S d , degenerate cosine component C d Input the trained neural network model and output the repaired sinusoidal component S r , cosine component C r .
[0061] Example
[0062] The advantages of the present invention (S2S) are illustrated by using the fringe-fringe deep learning method (F2F) and the fringe-phase deep learning method (F2P) as control technologies.
[0063] The specific advantages are reflected in the following three experimental effects:
[0064] (1) The network converges quickly. Figure 2 Neural network training results for three techniques are shown.
[0065] It can be seen that the MSE (root mean square error) of F2F and F2P is very large and does not converge. On the contrary, the MSE of the present invention converges quickly and eventually remains at a low level, which shows that the network mapping method of the present invention can effectively reduce the burden of network training, and the trained network has good generalization ability.
[0066] (2) Higher accuracy and greater robustness in measuring objects with complex reflectivity under different lighting conditions.
[0067] Under 5 low projection brightness conditions, the measurement results of the three methods are as follows: Figure 3 shown.
[0068] Figure 3 (a) is the object to be measured; Figure 3 (b) is the fringe pattern under 5 projection intensities; Figure 3(c)-3(e) are the phase results of the three methods F2F, F2P and S2S. It can be seen that there are obvious phase jump errors in the measured phases of F2F and F2P. Although their errors gradually decrease with the increase of projection brightness, there are still large phase errors at the edges of large discontinuous objects. However, the present invention can obtain high-quality phase results, and their STD (phase error standard deviation) data changes very little, which shows that for complex reflectivity objects under different low illumination conditions, the technology of the present invention has a high robustness measurement performance.
[0069] (3) Strong ability to restore fine lines. Figure 4 The measurement results of a complex sculpture-metal mixed scene under low projection brightness are shown. It can be seen that in the sculpture magnification, the technology of the present invention can reconstruct fine textures equivalent to the true value (multiple exposure method), and the invention has strong fidelity.
[0070] (4) High-precision measurement performance under different lighting conditions. Using a metal standard step as the measurement object (the height difference between different steps is 30 mm), the measurement accuracy of the present invention under different lighting brightness is demonstrated. Figure 5 The average height and average height difference (MHE, the average value of the difference between the height of Area1-Area2 and Area2-Area3 and the relative true value of 30 mm) of Area1-Area3 under five low projection brightness conditions are listed in Table 1.
[0071] Table 1. Average heights of three regions at five projection intensities (mm)
[0072]
[0073] Under five projection brightness conditions, the RMSE (root mean square error) of Area1-Area3 is listed in Table 2. It can be seen that under different brightness, the present invention has good measurement accuracy.
[0074] Table 2. Root mean square error of three planes at five projection intensities (mm)
[0075]
[0076] The above description is not intended to impose any form of limitation on the present invention. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.
Claims
1. A neural network mapping method for degraded stripe enhancement, Its characteristics are reflected in the following steps: Step S1, using a projector-camera system to complete data acquisition in a degradation measurement scenario to obtain a phase-shift fringe pattern; The degraded measurement scenes include high dynamic range scenes, low projection illumination and low camera exposure scenes, scenes with complex scattering and refraction of light, measurement scenes without proper defocusing, and measurement scenes with nonlinearity; Step S2: extracting the degenerate sinusoidal component S from the phase-shifted fringe pattern d and the degenerate cosine component C d ; There are three ways to obtain degenerate sine and cosine components. The first is to extract the truncated phase from the phase-shifted fringe pattern and then extract the sine and cosine components. The second is to extract the modulation of the fringe from the phase-shifted fringe pattern and then extract the sine and cosine components. The third method is to extract the background light intensity and modulation degree from the phase-shifted fringe pattern, and then extract the sine and cosine components; The first extraction formula is: Where: is the i-th fringe pattern; is the degenerate truncated phase; δ i =2πi / N is the phase shift; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; N is the number of phase shift steps; The second extraction formula is: Where: B d is the modulation degree; δ i =2πi / N is the phase shift; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; N is the number of phase shift steps; is the i-th fringe pattern; The third extraction formula is: Where: A d is the background light intensity; B d For the modulation system; is the i-th fringe pattern; δ i =2πi / N is the phase shift; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; N is the number of phase shift steps; f′(·) is the derivative operator; The degenerate sinusoidal component S is calculated by the formula in the three extraction processes: d , degenerate cosine component C d Or the degenerate sinusoidal component S can be calculated by the formula d , degenerate cosine component C d One of the degenerate sinusoidal components S d Or the degenerate cosine component C d Derivatively obtain the degenerate cosine component C d Or the degenerate sinusoidal component S d ; Step S3: using a multiple exposure method or a method of increasing the number of phase shift steps to obtain the degenerate sinusoidal component S d , degenerate cosine component C d The corresponding true value sine component S gt , true value cosine component C gt ; Step S4: Degenerate the sinusoidal component S d , degenerate cosine component C d As the network input, the true value sinusoidal component S gt , true value cosine component C gt As a network learning target, a sinusoidal component-sinusoidal component mapping model is constructed; Where: θ * is the optimized network parameter; G θ is the network mapping; S d is the degenerate sinusoidal component; C d is the degenerate cosine component; S gt is the true value sine component; C gt is the true value cosine component; Step S5: Using the Backpropagation algorithm to complete the network parameter update, so as to obtain a trained neural network model; Step S6: input the degraded sine and cosine components that have not participated in the training into the trained neural network model, and output the repaired sine and cosine components.
2. A neural network mapping method for degenerate fringe enhancement according to claim 1, It is characterized in that The multiple exposure method in step S3 includes a method of modulating the projection light intensity, imaging exposure time, and using a multi-channel imaging device.
