A 3D reconstruction method for highly reflective surfaces based on structured light

Abstract

The present invention discloses a 3D reconstruction algorithm for highly reflective surfaces based on structured light. Based on the multiple exposure method and the method for adjusting the intensity of projection stripes in high dynamic range technology, a method for selecting an exposure time sequence and a stripe brightness sequence is proposed. The required exposure time sequence and stripe brightness sequence are calculated by using the statistical histogram of the number of pixels on the surface of the object under test under different projection conditions and combining the ratio of the number of pixels in the image that are not overexposed. The image of the object under test is collected and fused using the sequence, and then reconstructed according to the projection grating phase method. The binocular system calibration, phase information extraction, and phase stereo matching are performed respectively, and finally the 3D point cloud data of the highly reflective surface can be obtained.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision and three-dimensional reconstruction, and in particular relates to a three-dimensional reconstruction method of a high-reflective surface based on structured light. Background Art

[0002] With the further development of scientific research, the two-dimensional information of objects has gradually failed to meet the measurement needs, and the three-dimensional shape information has attracted more and more attention. Among the three-dimensional shape measurement technologies, structured light three-dimensional reconstruction technology has been widely used in medical, military and industrial fields due to its advantages such as fast speed, high precision and non-contact. The shape measurement of non-reflective surfaces has become increasingly mature, but the reconstruction of highly reflective surfaces is still the focus and difficulty in research.

[0003] Domestic and foreign scholars have conducted many studies on the above-mentioned issues. The most relevant technology to the present invention is the "Projection Grating Phase Method for Measuring the Topography of Strong Reflective Surfaces" published by Jiang Hongzhi et al., which is published in the 9th issue of Volume 18 of Optical Precision Engineering, with pages 2002-2008. The article proposes an improved projection grating phase method, which uses the bright and dark stripe projection and multi-exposure time image acquisition technologies in high dynamic range technology to make the imaging grayscale consistent with the dynamic range of the camera sensor, which can basically solve the problem of overexposure of images collected on highly reflective surfaces. However, in the selection of exposure time sequence and stripe brightness sequence, it is only manually selected according to the characteristics of the measured surface, lacking the necessary evidence.

[0004] Therefore, the present invention proposes a method for 3D reconstruction of highly reflective surfaces based on structured light. Based on the multiple exposure method and the method for adjusting the intensity of projection stripes in high dynamic range technology, a method for selecting exposure time sequence and stripe brightness sequence is proposed. The required exposure time sequence and stripe brightness sequence are calculated by using the statistical histogram of the number of pixels on the surface of the object under test under different projection conditions and the ratio of the number of pixels in the image that are not overexposed. The image of the object under test is collected and fused using the sequence, and then reconstructed according to the projection grating phase method. The binocular system calibration, phase information extraction, and phase stereo matching are performed respectively, and finally the 3D point cloud data of the highly reflective surface can be obtained. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned technology, the present invention proposes a three-dimensional reconstruction method of highly reflective surfaces based on structured light and a method for selecting an exposure time sequence and a stripe brightness sequence, so as to achieve the purpose of obtaining an accurate exposure time sequence and a stripe brightness sequence, and then complete the three-dimensional reconstruction of the highly reflective surface, and achieve the purpose of obtaining a point cloud with higher precision.

[0006] The present invention adopts the following technical solutions:

[0007] A method for 3D reconstruction of a highly reflective surface based on structured light, the specific steps are as follows:

[0008] Step 1: The projector projects white light, and the camera collects white light images with different exposure times and fringe brightness. The number of measurements is n and m, respectively, to construct two small data sets with capacities of n and m.

[0009] Step 2, filter out the pixel matrices t_exp and b_exp that complete the transition from underexposure to overexposure during the acquisition process, and obtain the estimated exposure time t_pre and estimated stripe brightness b_pre of each pixel before overexposure by counting the above pixel points;

[0010] Step 3, using the exposure time or stripe brightness as the horizontal coordinate and the exposure time or stripe brightness as the number of pixels t_pre or b_pre as the vertical coordinate, to make two statistical histograms of the number of pixels near overexposure;

[0011] Step 4: derive the pixel overexposure ratio p based on the statistical histogram height change trend obtained in step 3. k and q l , combined with the specific number of pixels in each square in the histogram, the equation is listed to accurately calculate the required exposure time x k and the fringe brightness y l ;

[0012] Step 5, using the float glass dot calibration plate as the target image, the target image is collected by the camera, and the binocular system is calibrated using the Zhang Zhengyou calibration method. After collecting the image, feature points are extracted from the images obtained by the left and right cameras respectively, and then the extracted feature points are subjected to sub-pixel optimization processing. Then, the left and right cameras are calibrated with internal parameters to obtain the camera internal parameters, and finally the binocular joint calibration is performed to obtain the camera external parameters;

[0013] Step 6, based on the x obtained in step 4 k and l , perform image acquisition, respectively acquire white light image I and grating fringe image P;

[0014] Step 7, creating a mask image Mask according to the collected white light image I;

[0015] Step 8, fusing the grating fringe image P acquired in step 6 and the mask image Mask produced in step 7 to obtain a final image sequence H;

[0016] Step 9, perform phase demodulation and phase unwrapping on the final image sequence H to obtain left and right absolute phase images;

[0017] Step 10, stereo matching the left and right absolute phase images to obtain the three-dimensional coordinates of the object to be measured, and finally generate a point cloud.

[0018] Furthermore, the formula for calculating the estimated exposure time t_pre and the estimated stripe brightness b_pre in step 2 is as follows:

[0019]

[0020] The maximum exposure time is t_max, the minimum exposure time is t_min, the maximum stripe brightness is b_max, the minimum exposure time is b_min, t_exp and b_exp are the pixel matrix selected in step 2, i t 、j t are the horizontal and vertical coordinates in the t_exp matrix respectively. Similarly, i b 、j b They are the horizontal and vertical coordinates in the matrix b_exp respectively.

[0021] Furthermore, the step 4 accurately calculates the exposure time x k and the fringe brightness y l The formula is as follows:

[0022]

[0023] About the exposure time histogram f_t(t i ), about the stripe brightness histogram f_b(b i ), sum_t represents the total number of points in the exposure time histogram, sum_b represents the total number of points in the fringe brightness histogram, k_max is the number of exposure times to be calculated, l_max represents the number of fringe brightnesses to be calculated, p k Overexposure exposure time pixel ratio, q l That is, the proportion of pixels with brightness that are not overexposed, x k Indicates the exact exposure time to be sought, y l Indicates the exact fringe brightness to be sought.

[0024] Furthermore, the formula for making a mask image Mask according to the collected white light image I in step 7 is as follows:

[0025] When i=1,

[0026]

[0027] When i>1,

[0028]

[0029] i, j are image numbers, I i (x, y) is the i-th image I in the image sequence I i Gray value at coordinate (x, y), Mask i (x,y), Maskj (x,y) is the mask sequence I i ,I j The mask value of the corresponding mask image at the coordinate (x, y), I max is the mask grayscale threshold.

[0030] Furthermore, in step 8, the formula for obtaining the final image sequence H by fusing the grating fringe image P acquired in step 6 and the mask image Mask produced in step 7 is as follows:

[0031]

[0032] r represents the total number of times the stripes are photographed, P i (x, y) represents the grayscale value of the i-th image in the image sequence P at the point (x, y), and H(x, y) represents the grayscale value of the image H at the point (x, y).

[0033] Beneficial Effects

[0034] 1. The present invention can accurately calculate the exposure time sequence and fringe brightness sequence required for the experiment, avoiding blind manual selection;

[0035] 2. The present invention can improve the reconstruction accuracy of highly reflective surfaces and increase the number of effective pixels in the point cloud to 96%. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flowchart of the method for 3D reconstruction of highly reflective surfaces;

[0037] Figure 2 Schematic diagram of small dataset construction;

[0038] Figure 3 (a) Exposure time statistical histogram (b) Fringe brightness statistical histogram;

[0039] Figure 4 Dual-target targeting flow chart;

[0040] Figure 5 (a) Schematic diagram without fusion (b) Schematic diagram of data fusion effect after manual selection (c) Schematic diagram of data fusion effect after histogram calculation selection;

[0041] Figure 6 Multi-frequency heterodyne method flow chart;

[0042] Figure 7 (a) Unfused solution phase diagram (b) Manually selected data solution phase diagram (c) Histogram calculation of selected data solution phase diagram (d) Unfused point cloud (e) Manually selected data point cloud (f) Histogram calculation of selected data point cloud. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0044] The present invention proposes a method for 3D reconstruction of highly reflective surfaces based on structured light. The specific flow chart is as follows: Figure 1 As shown, the main steps are as follows:

[0045] Step 1: The projector projects white light, and the camera collects white light images with different exposure times and fringe brightness. The number of measurements is n and m, respectively. Two small data sets with capacity of n and m are constructed. In this example, n and m are both 50. The construction effect is as follows: Figure 2 As shown;

[0046] Step 2, filter out the pixels that have completed the transition from underexposure to overexposure during the acquisition process, and obtain the estimated exposure time t_pre and estimated stripe brightness b_pre of each pixel before overexposure by counting the above pixels;

[0047] Step 2: Calculate the estimated exposure time t_pre and estimated fringe brightness b_pre using the following formula:

[0048]

[0049] The maximum exposure time is t_max, the minimum exposure time is t_min, the maximum stripe brightness is b_max, the minimum exposure time is b_min, t_exp and b_exp are the pixel matrix selected in step 2, i t j t are the horizontal and vertical coordinates in the t_exp matrix respectively. Similarly, i b j b They are the horizontal and vertical coordinates in the matrix b_exp respectively.

[0050] Step 3, using the exposure time or stripe brightness as the horizontal coordinate and the number of pixels with the time or brightness as t_pre or b_pre as the vertical coordinate, make two statistical histograms of the number of pixels near overexposure, such as Figure 3 As shown;

[0051] Step 4: derive the pixel overexposure ratio p based on the statistical histogram height change trend obtained in step 3. k and q l , combined with the specific number of pixels in each square in the histogram, the equation is listed to accurately calculate the required exposure time x k and the fringe brightness y l ; The precise calculation results in this example are shown in Table 1, and the manually selected comparison data are shown in Table 2;

[0052] Table 1 Correspondence table of exposure time for measuring fringe brightness

[0053]

[0054] Table 2 Correspondence table of exposure time for manually selected fringe brightness

[0055]

[0056] Step 4: Accurately calculate the exposure time x k and the fringe brightness y l The formula is as follows:

[0057]

[0058] About the exposure time histogram f_t(t i ), about the stripe brightness histogram f_b(b i ), sum_t represents the total number of points in the exposure time histogram, sum_b represents the total number of points in the fringe brightness histogram, k_max is the number of exposure times to be calculated, l_max represents the number of fringe brightnesses to be calculated, p k Overexposure exposure time pixel ratio, q l That is, the proportion of pixels with brightness that are not overexposed, x k Indicates the exact exposure time to be sought, y l Indicates the exact fringe brightness to be sought.

[0059] Step 5, using the float glass dot calibration plate as the target image, the target image is collected by the camera, and the binocular system is calibrated using the Zhang Zhengyou calibration method. After collecting the image, feature points are extracted from the images obtained by the left and right cameras respectively, and then the extracted feature points are subjected to sub-pixel optimization processing. Then, the left and right cameras are calibrated with internal parameters to obtain the camera internal parameters, and finally the binocular joint calibration is performed to obtain the camera external parameters. The process is as follows: Figure 4 As shown;

[0060] Step 6, based on the x obtained in step 4 k and l , perform image acquisition, respectively acquire white light image I and grating fringe image P;

[0061] Step 7, creating a mask image Mask according to the collected white light image sequence I;

[0062] Step 7: The mask image Mask formula is as follows:

[0063] When i=1,

[0064]

[0065] When i>1,

[0066]

[0067] i, j are image numbers, I i (x, y) is the i-th image I in the image sequence I i Gray value at coordinate (x, y), Mask i (x,y), Mask j (x,y) is the mask sequence I i , I j The mask value of the corresponding mask image at the coordinate (x, y), I max is the mask grayscale threshold.

[0068] Step 8: Fuse the stripe sequence P and the mask sequence Mask image to obtain the final image sequence H. The fusion contrast effect is as follows: Figure 5 As shown;

[0069] In step 8, the fringe sequence P and the mask sequence Mask image are fused to obtain the final image sequence H as follows:

[0070]

[0071] r represents the total number of times the stripes are photographed, P i (x, y) represents the grayscale value of the i-th image in the image sequence P at the point (x, y), and H(x, y) represents the grayscale value of the image H at the point (x, y).

[0072] Step 9: perform phase demodulation and phase unwrapping on H to obtain the left and right absolute phase diagrams. The specific process is as follows: Figure 6 As shown;

[0073] Step 10: Stereo match the left and right absolute phase images to obtain the three-dimensional coordinates of the object to be measured, and finally generate a point cloud. The comparison effect is shown in the figure below. Figure 7 shown.

[0074] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A 3D reconstruction method for highly reflective surfaces based on structured light. It is characterized in that The specific steps are as follows: Step 1: The projector projects white light, and the camera collects white light images with different exposure times and fringe brightness. The number of measurements is n and m, respectively, to construct two small data sets with capacities of n and m. Step 2, filter out the pixel matrices t_exp and b_exp that complete the transition from underexposure to overexposure during the acquisition process, and obtain the estimated exposure time t_pre and estimated stripe brightness b_pre of each pixel before overexposure by counting the above pixel points; Step 3, using the exposure time or stripe brightness as the horizontal coordinate and the exposure time or stripe brightness as the number of pixels t_pre or b_pre as the vertical coordinate, to make two statistical histograms of the number of pixels near overexposure; Step 4: derive the pixel overexposure ratio p based on the statistical histogram height change trend obtained in step 3. k and q l , combined with the specific number of pixels in each square in the histogram, the equation is listed to accurately calculate the required exposure time x k and the fringe brightness y l ; Step 5, using the float glass dot calibration plate as the target image, the target image is collected by the camera, and the binocular system is calibrated using the Zhang Zhengyou calibration method. After collecting the image, feature points are extracted from the images obtained by the left and right cameras respectively, and then the extracted feature points are subjected to sub-pixel optimization processing. Then, the left and right cameras are calibrated with internal parameters to obtain the camera internal parameters, and finally the binocular joint calibration is performed to obtain the camera external parameters; Step 6, based on the x obtained in step 4 k and l , perform image acquisition, respectively acquire white light image I and grating fringe image P; Step 7, creating a mask image Mask according to the collected white light image I; Step 8, fusing the grating fringe image P acquired in step 6 and the mask image Mask produced in step 7 to obtain a final image sequence H; Step 9, perform phase demodulation and phase unwrapping on the final image sequence H to obtain left and right absolute phase images; Step 10, stereo matching the left and right absolute phase images to obtain the three-dimensional coordinates of the object to be measured, and finally generate a point cloud; The formula for calculating the estimated exposure time t_pre and the estimated stripe brightness b_pre in step 2 is as follows: The maximum exposure time is t_max, the minimum is t_min, the maximum stripe brightness is b_max, the minimum is b_min, t_exp and b_exp are the pixel matrix selected in step 2, i t 、j t are the horizontal and vertical coordinates in the t_exp matrix respectively. Similarly, i b 、j b They are the horizontal and vertical coordinates in the matrix b_exp respectively; The step 4 accurately calculates the exposure time x k and the fringe brightness y l The formula is as follows: About the exposure time histogram f_t(t i ), about the stripe brightness histogram f_b(b i ), sum_t represents the total number of points in the exposure time histogram, sum_b represents the total number of points in the fringe brightness histogram, k_max is the number of exposure times to be calculated, l_max represents the number of fringe brightnesses to be calculated, p k Overexposure exposure time pixel ratio, q l That is, the ratio of the brightness pixels of the stripes that are not overexposed, x k Indicates the exact exposure time to be sought, y l represents the exact fringe brightness to be sought; The formula for making a mask image Mask according to the collected white light image I in step 7 is as follows: When i=1, When i>1, i, j are image numbers, I i (x, y) is the i-th image I in the image sequence I i The gray value at coordinate (x,y), Mask i (x,y), Mask j (x,y) is the mask sequence I i ,I j The mask value of the corresponding mask image at the coordinate (x, y), I max is the mask grayscale threshold.

2. According to claim 1, a method for three-dimensional reconstruction of a highly reflective surface based on structured light, It is characterized in that In step 8, the formula for obtaining the final image sequence H by fusing the grating fringe image P acquired in step 6 and the mask image Mask produced in step 7 is as follows: r represents the total number of times the stripes are photographed, P i (x, y) represents the grayscale value of the i-th image in the image sequence P at the point (x, y), and H(x, y) represents the grayscale value of the image H at the point (x, y).

Images