Regional adaptive fringe projection three-dimensional measurement method for spectrum separation

Abstract

The invention relates to a spectral separation regional adaptive fringe projection three-dimensional measurement method, which comprises the following steps of: projecting transverse and longitudinal sine phase shift fringe patterns to the surfaces of a plurality of measured objects, and acquiring a fringe sequence by using a camera; respectively obtaining an object contour mask and a saturated mask; the coordinates of the white area of the contour mask are converted into the coordinates of a projector, and a projection mask is obtained; obtaining a single object projection mask image by using a connected domain labeling algorithm; setting two groups of mask regions by using the marked regions and the saturated mask to generate a longitudinal sine phase shift fringe pattern with adaptive intensity, sequentially projecting two groups of region fringes to the surface of a measured object, and simultaneously collecting region fringe images of spectrum separation; performing independent phase demodulation and phase fusion on the fringe patterns of different channel colors corresponding to the object, and performing three-dimensional reconstruction according to a fused phase result; according to the method, the measurement efficiency is remarkably improved on the premise of eliminating stripe aliasing interference.

Description

Technical Field

[0001] The invention relates to the technical field of structured light three-dimensional measurement, and in particular to a spectrally separated regional adaptive fringe projection three-dimensional measurement method. Background Art

[0002] In modern industry and scientific research, accurate three-dimensional measurement of objects is a crucial technology, especially in the fields of manufacturing, medical treatment, cultural relics protection, virtual reality, etc., where the requirements for measurement accuracy and efficiency are increasing. In the field of three-dimensional measurement, when there are both specular reflection objects and diffuse reflection objects in the scene, traditional structured light measurement technology will face the following two challenges: 1) Specular reflection objects will cause local light intensity saturation, resulting in the destruction of structured light fringes and difficulty in recovering accurate phase information; 2) Specular reflection objects may reflect light to the surface of diffuse reflection objects for a second time, resulting in the introduction of other interference light on the surface of diffuse reflection objects, showing fringing aliasing phenomenon, causing errors in measurement results. Some existing solutions reduce measurement errors through post-phase compensation, signal filtering and other methods, but cannot avoid the occurrence of this phenomenon from the source. There are also some deep learning-based methods that can achieve aliasing fringes unmixing and improve measurement accuracy. However, a large number of training data sets need to be produced, which is very cumbersome and not conducive to practical applications.

[0003] Therefore, there is an urgent need to develop a flexible and simple method that can avoid multiple reflections, fringe aliasing and other phenomena from the source, so as to achieve simultaneous high-precision three-dimensional measurement of specular and diffuse reflection objects. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a spectrally separated regional adaptive fringe projection three-dimensional measurement method. The method designs a regional adaptive projection strategy to alternately project red and blue fringe patterns with different spectra in different areas of the object being measured. During the time of projecting the two groups of red and blue stripes, a camera is used to simultaneously capture images of two spectral colors with one exposure. After color channel separation, phase extraction is performed independently, fundamentally avoiding the interference of reflected light between complex materials, thereby improving the accuracy and reliability of three-dimensional measurement of complex multi-material measurement scenes.

[0005] To achieve the above objectives, the technical solution provided by the present invention is:

[0006] A spectrally separated regional adaptive fringe projection three-dimensional measurement method, comprising:

[0007] Project the fringe images with transverse and longitudinal sinusoidal phase shifts onto the surface of the object to be measured, and obtain the transverse and longitudinal phase distribution results respectively;

[0008] According to the fringe image with transverse and longitudinal sinusoidal phase shifts, the image of the object to be tested is segmented using a YOLO network to obtain a contour mask Mc;

[0009] Detecting the light intensity saturation area in the fringe image with transverse and longitudinal sinusoidal phase shifts, and obtaining a saturation mask Ms by setting a threshold value;

[0010] Generate a projection mask image according to the contour mask Mc and the saturation mask Ms;

[0011] Generating a longitudinal sinusoidal phase-shifted fringe pattern in a projection mask area;

[0012] Regional fringe patterns are collected sequentially, and phase demodulation and phase fusion are performed, and three-dimensional reconstruction is performed using a phase-height mapping model.

[0013] The projecting of the fringe images with transverse and longitudinal sinusoidal phase shifts onto the surface of the object to be measured, and obtaining the transverse and longitudinal phase distribution results respectively, comprises:

[0014] The horizontal and vertical phase-shifted fringe images collected by the camera are expressed as follows:

[0015]

[0016] in, represents the transverse phase shift fringe pattern; represents the longitudinal phase shift fringe pattern; A and B represent the background light intensity and fringe modulation respectively; F is the fringe frequency; i is the phase shift sequence number, i = 1, 2, 3; (u c ,v c ) is the horizontal and vertical coordinates of the pixels of the camera image; respectively collect: horizontal phase shift fringe pattern sequence

[0017] and the longitudinal phase shift fringe pattern sequence

[0018] The transverse and longitudinal phase distribution results are calculated using the transverse and longitudinal fringe pattern sequences respectively.

[0019] According to the fringe image of the transverse and longitudinal sinusoidal phase shift, the YOLO network is used to perform contour segmentation on the image of the object to be tested to obtain the contour mask Mc, including:

[0020] Input the horizontal and vertical stripe images into the YOLO network, and the YOLO network outputs the object contour segmentation image;

[0021] The segmented image is a binary mask image, which serves as a contour mask Mc.

[0022] The light intensity saturation area is detected in the fringe image with the transverse and longitudinal sinusoidal phase shift, and the saturation mask Ms is obtained by setting a threshold value, including:

[0023] The texture image I of the object under test is obtained by averaging the collected lateral and longitudinal phase distribution results. bg ;

[0024] According to the preset grayscale threshold, the texture image I bg Pixel-by-pixel traversal: Pixels with grayscale values ​​greater than the preset threshold are considered saturated pixels until a saturated mask image M is obtained. s .

[0025] The step of generating a projection mask image according to the contour mask Mc and the saturation mask Ms comprises:

[0026] The contour mask M c With the saturation mask M s The coordinates of are converted into the projector coordinate system to generate a projection mask image, and the coordinate transformation is performed according to the following formula:

[0027] v p =Φ hor (u c ,v c )×P / (2π)

[0028] u p =Φ ver (u c ,v c )×P / (2π)

[0029] Among them, Φ hor (u c ,v c ), Φ ver (u c ,v c ) represents the lateral phase and longitudinal phase of the white area point in the mask image Mc or Ms, and P represents the fringe frequency.

[0030] The method of generating a longitudinal sinusoidal phase-shifted fringe pattern in a projection mask region comprises:

[0031] The projection mask image M p The objects in the image are marked with consecutive numbers, and the strong light projection mask M is placed according to the odd and even numbers. p The white area is divided into two groups, where the first group consists of masks composed of odd numbers, named M pB The second group is composed of even-numbered masks, named M pR ; respectively in M pB and M pRGenerate a three-step phase-shifted fringe sequence in and As shown below:

[0032]

[0033] Among them, (u p ,v p ) represents the projection pixel coordinates; A and B are both constants; F represents the fringe frequency; i represents the i-th fringe pattern in the three-step phase shift method, i = 1, 2, 3;

[0034] The regional fringe patterns are projected alternately in the following order: {P1 pB , P1 pR , P2 pB , P2 pR , P3 pB , P3 pR};

[0035] In the weak light mask M q Generate a low-intensity three-step phase-shifted fringe sequence P i q , where parameters A and B are both strong photomask M p Reduce by half;

[0036] will sequence and The stripes corresponding to the weak light mask area in the image are replaced by P i q , respectively get the updated adaptive intensity fringe sequence and

[0037] Adaptive intensity fringe sequence and All are loaded into the projector and projected onto the surface of the object being measured.

[0038] Projection sequence When using a blue light source, the sequence Use red light source.

[0039] The sequentially collecting regional fringe images and performing phase demodulation and phase fusion include:

[0040] The camera exposure time is set to be greater than or equal to the projection time of two adjacent regional fringe patterns, so that the camera can capture the sequence simultaneously during the acquisition of one image. and two regional fringe images;

[0041] The camera collects data at least once in sequence, and saves at least one spectrally separated phase-shift fringe image respectively;

[0042] The collected spectrally separated fringe pattern is separated into RGB channels to obtain high-quality phase results without interference from multiple reflected light;

[0043] The high-quality phase results are used to obtain complete phase results for three-dimensional reconstruction.

[0044] The collected spectral separation fringe image is separated into RGB channels, and the two images of the R channel and the B channel are taken to perform phase extraction independently as follows:

[0045]

[0046] Among them, Ф R and Φ B The phases obtained for the red and blue channels, I 1R -I 3R is the red channel stripe image, I 1B -I 3B This is the blue channel stripe image.

[0047] The three-dimensional reconstruction using the phase-height mapping model includes:

[0048] The three-dimensional reconstruction is performed using the following mapping formula:

[0049]

[0050] Where H(x,y) represents the actual depth value at the coordinate (x,y), a(x,y), b(x,y), and c(x,y) are phase-height mapping parameters obtained by calibration; is the phase distribution of the object.

[0051] Compared with the existing technology, this solution has the following advantages:

[0052] The method described in the present invention can fundamentally avoid the problem of stripe aliasing caused by secondary reflection between different materials. There are significant improvements in both measurement accuracy and robustness. At the same time, the adaptive intensity stripes can avoid light intensity saturation and further reduce the phase error. Moreover, the method described in the present invention makes full use of the ability of independent perception between different color channels, and uses the blue and red spectral channels that are far apart to alternately project regional stripes. At the same time, the camera exposure time is strictly coordinated with the stripe alternating projection time, so that the camera is exposed once, and two color stripe images are collected at the same time for independent processing. Under the premise of ensuring the elimination of stripe aliasing interference, the measurement efficiency is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a schematic diagram of the measuring method of the present invention;

[0055] Figure 2 is a flow chart of the method of the present invention;

[0056] Figure 3 A rough processing image for the whole process;

[0057] Figure 4 The actual process and the final result of the photos captured by the camera when the red and blue lights are alternating at different time periods;

[0058] Figure 5 is the collected color spectrum separated fringe image;

[0059] Figure 6 Take the grayscale image of the R channel for color stripe separation;

[0060] Figure 7 Take the grayscale image of the B channel for color stripe separation; DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with specific embodiments:

[0062] Prior to this, a three-dimensional measurement system used in a spectrally separated regional adaptive fringe projection three-dimensional measurement method according to an embodiment of the present invention is specifically described:

[0063] like Figure 1 As shown, the structured light projection 3D measurement system includes a computer 1, a regional projection stripe Figure 2 , projection machine 3, measured scene 4, color industrial camera 5, camera collects color fringes Figure 6 . The regional projection stripes are generated by computer 1 Figure 2 , transmitted to the projection light machine 3 and projected onto the surface of the measured scene 4. Then, the color industrial camera 5 collects the color fringes Figure 6 After that, it is transmitted to computer 1 for data processing.

[0064] The working principle of the structured light projection technology based on regional spectrum separation described in this embodiment is as follows:

[0065] S1, projecting transverse and longitudinal sinusoidal phase-shifted fringe patterns onto the surface of the object to be measured, and using a camera to collect a fringe image sequence, wherein the transverse and longitudinal three-step phase-shifted fringe images are collected, a total of 9 images, which can be expressed as:

[0066]

[0067] in represents the transverse phase shift fringe pattern, represents the longitudinal phase shift fringe pattern, A and B represent the background light intensity and fringe modulation, respectively, F is the fringe frequency, i is the phase shift sequence number, i = 1, 2, 3; (u c ,v c ) is the horizontal and vertical coordinates of the pixels of the image captured by the camera;

[0068] S2. Use the YOLO network to perform contour segmentation on the image of the object to obtain the contour mask Mc; pass the three-step image through the YOLO network to obtain the binary mask contour segmentation map of the object, that is, the contour mask M c ;

[0069] S3, detect the light intensity saturation area in the original image, and obtain the saturation mask M through appropriate threshold division s ;

[0070] S4, the contour mask M c With saturation mask M s The coordinates of the projector are converted to the projector coordinate system to generate a projection mask image. The horizontal and vertical phases obtained in step S1 are used to perform coordinate conversion according to the following formula:

[0071] v p =Φ hor (u c ,v c )×P / (2π)

[0072] u p =Φ ver (u c ,v c )×P / (2π);

[0073] Among them, Φ hor (u c ,v c ), Φ ver (u c ,v c ) represents the mask image M c or M s The horizontal and vertical phases of the white area points in the middle, and P represents the fringe frequency.

[0074] S5, generating a longitudinal sinusoidal phase-shifted fringe pattern in the projection mask area;

[0075] S501, using the connected domain labeling algorithm, the projection mask map M p The objects in the image are marked with consecutive numbers. p The white area is divided into two groups, where the first group consists of masks composed of odd numbers, named M pB The second group is composed of even-numbered masks, named M pR . Respectively in M pB and M pR Generate a three-step phase-shifted fringe sequence in and As shown below:

[0076]

[0077] Where P i Can be and (u p ,v p ) represents the projection pixel coordinates, A and B are both set to 127.5, F represents the fringe frequency, i represents the i-th fringe pattern in the three-step phase shift method, i = 1, 2, 3.

[0078] S502, Projection Sequence using blue light source, projecting Use red light source in sequence. Project two sets of regional fringe patterns alternately in the following order: {P1 pB , P1 pR , P2 pB , P2 pR , P3 pB , P3 pR}.

[0079] S503, in the weak light mask M q Generate a low-intensity three-step phase-shifted fringe sequence P i q , where parameters A and B are both reduced by half;

[0080] S504, will and The stripes corresponding to the weak light mask area in the image are replaced by P i q , and get the updated adaptive intensity fringe sequence and Finally, and All are loaded into the projector and projected onto the surface of the object being measured;

[0081] S6, the camera collects regional fringe images in sequence, and performs phase demodulation and phase fusion;

[0082] S601, set the camera exposure time to be greater than or equal to the projection time of two adjacent regional fringe patterns. For example, if the projection time of two adjacent regional fringe patterns is 15ms each, the camera exposure time is set to be greater than or equal to 30ms. In this way, the camera captures and Two regional stripe images, to achieve the simultaneous presence of blue and red spectrally separated stripe images in the camera captured image;

[0083] S602, the camera collects data three times in sequence and saves three spectrally separated phase-shift fringe images;

[0084] S603, separate the three collected spectral separation fringe images into RGB channels, and take two images of the R channel and the B channel to perform the following phase extraction independently:

[0085]

[0086] Among them, Ф R and Φ B The phases obtained for the red and blue channels, I 1R ~I 3R is the red channel stripe image, I 1B ~I 3B This is the blue channel stripe image.

[0087] S604, Ф R and Φ B The complete phase result is added and used for three-dimensional reconstruction;

[0088] S7. Perform three-dimensional reconstruction using the phase-height mapping model.

[0089] The mapping formula is as follows:

[0090]

[0091] Where H(x,y) represents the actual depth value at the coordinate (x,y), and a(x,y), b(x,y), and c(x,y) are phase-height mapping parameters obtained through calibration. is the phase distribution of the object.

[0092] In summary, the present invention discloses a spectrally separated regional adaptive fringe projection three-dimensional measurement method. First, the transverse and longitudinal sinusoidal phase shift fringe patterns are projected onto the surfaces of multiple objects to be measured, and the fringe sequence is collected by a camera. Then, the object contour mask and saturation mask are obtained by the YOLO network and the saturation detection algorithm respectively. Then, the white area coordinates of the contour mask are converted into the projector coordinates to obtain the projection mask. Then, each independent object in the projection mask is marked using the connected domain marking algorithm to obtain a single object projection mask image. Then, using the marked area and saturation mask, two groups of mask areas are set to generate longitudinal sinusoidal phase shift fringe patterns with adaptive intensity, and the two groups of regional stripes are sequentially projected onto the surface of the object with different color light sources, and the camera exposure time is set to be greater than the projection time of the two groups of different color stripes, so as to realize the simultaneous acquisition of spectrally separated regional stripe images. Finally, the fringe patterns of different channel colors corresponding to the object are taken for independent phase demodulation and phase fusion, and three-dimensional reconstruction is performed according to the fused phase results; the present invention makes full use of the independent perception ability between different color channels, and uses the blue and red spectral channels that are far apart to alternately project regional stripes. At the same time, the camera exposure time is strictly coordinated with the alternating projection time of the stripes, so that the camera can be exposed once and two color stripe images can be collected and processed independently. Under the premise of eliminating the interference of stripe aliasing, the measurement efficiency is significantly improved.

[0093] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A spectrally separated regional adaptive fringe projection three-dimensional measurement method, characterized in that: include: Project the fringe images with transverse and longitudinal sinusoidal phase shifts onto the surface of the object to be measured, and obtain the transverse and longitudinal phase distribution results respectively; According to the fringe image with transverse and longitudinal sinusoidal phase shifts, the image of the object to be tested is segmented using a YOLO network to obtain a contour mask Mc; Detecting the light intensity saturation area in the fringe image with transverse and longitudinal sinusoidal phase shifts, and obtaining a saturation mask Ms by setting a threshold value; Generate a projection mask image according to the contour mask Mc and the saturation mask Ms; Generating a longitudinal sinusoidal phase-shifted fringe pattern in a projection mask area; Regional fringe patterns are collected sequentially, and phase demodulation and phase fusion are performed, and three-dimensional reconstruction is performed using a phase-height mapping model.

2. The method for three-dimensional measurement based on spectral separation and regional adaptive fringe projection according to claim 1, characterized in that: The projecting of the fringe images with transverse and longitudinal sinusoidal phase shifts onto the surface of the object to be measured, and obtaining the transverse and longitudinal phase distribution results respectively, comprises: The horizontal and vertical phase-shifted fringe images collected by the camera are expressed as follows: in, represents the transverse phase shift fringe pattern; represents the longitudinal phase shift fringe pattern; A and B represent the background light intensity and fringe modulation degree respectively; F is the fringe frequency; i is the phase shift sequence number, i = 1, 2, 3; (u c ,v c ) is the horizontal and vertical coordinates of the pixels of the camera image; respectively collect: horizontal phase shift fringe pattern sequence and the longitudinal phase shift fringe pattern sequence The transverse and longitudinal phase distribution results are calculated using the transverse and longitudinal fringe pattern sequences respectively.

3. The method for three-dimensional measurement based on spectral separation and regional adaptive fringe projection according to claim 1, characterized in that: According to the fringe image of the transverse and longitudinal sinusoidal phase shift, the YOLO network is used to perform contour segmentation on the image of the object to be tested to obtain the contour mask Mc, including: Input the horizontal and vertical stripe images into the YOLO network, and the YOLO network outputs the object contour segmentation image; The segmented image is a binary mask image, which serves as a contour mask Mc.

4. The method for three-dimensional measurement by regional adaptive fringe projection of spectrum separation according to claim 1, characterized in that: The light intensity saturation area is detected in the fringe image with the transverse and longitudinal sinusoidal phase shift, and the saturation mask Ms is obtained by setting a threshold value, including: The texture image I of the object under test is obtained by averaging the collected lateral and longitudinal phase distribution results. bg ; According to the preset grayscale threshold, the texture image I bg Pixel-by-pixel traversal: Pixels with grayscale values ​​greater than the preset threshold are considered saturated pixels until a saturated mask image M is obtained. s .

5. The method for three-dimensional measurement by regional adaptive fringe projection of spectrum separation according to claim 1, characterized in that: The step of generating a projection mask image according to the contour mask Mc and the saturation mask Ms comprises: The contour mask M c With the saturation mask M s The coordinates of are converted into the projector coordinate system to generate a projection mask image, and the coordinate transformation is performed according to the following formula: v p =Φ hor (u c ,v c )×P / (2π) u p =Φ ver (u c ,v c )×P / (2π); Among them, Φ hor (u c ,v c ), Φ ver (u c ,v c ) represents the lateral phase and longitudinal phase of the white area point in the mask image Mc or Ms, and P represents the fringe frequency.

6. The method for three-dimensional measurement by regional adaptive fringe projection of spectrum separation according to claim 1, characterized in that: The method of generating a longitudinal sinusoidal phase-shifted fringe pattern in a projection mask region comprises: The projection mask image M p The objects in the image are marked with consecutive numbers, and the strong light projection mask M is placed according to the odd and even numbers. p The white area is divided into two groups, where the first group consists of masks composed of odd numbers, named M pB The second group is composed of even-numbered masks, named M pR ; respectively in M pB and M pR Generate a three-step phase-shifted fringe sequence P i pB and P i pR , as shown below: Among them, (u p ,v p ) represents the projection pixel coordinates; A and B are both constants; F represents the fringe frequency; i represents the i-th fringe pattern in the three-step phase shift method, i = 1, 2, 3; The regional fringe patterns are projected alternately in the following order: {P1 pB , P1 pR , P2 pB , P2 pR , P3 pB , P3 pR }; In the weak light mask M q Generate a low-intensity three-step phase-shifted fringe sequence P i q , where parameters A and B are both strong photomask M p Reduce by half; The sequence P i pB and P i pR The stripes corresponding to the weak light mask area in the image are replaced by P i q , and obtain the updated adaptive intensity fringe sequence P i pB and P i pR ; The adaptive intensity fringe sequence P i pB and P i pR All are loaded into the projector and projected onto the surface of the object being measured.

7. The method for three-dimensional measurement by regional adaptive fringe projection of spectrum separation according to claim 6, characterized in that: Projection sequence P i pB When using a blue light source, the sequence P i pR Use red light source.

8. The method for three-dimensional measurement by regional adaptive fringe projection of spectrum separation according to claim 1, characterized in that: The sequentially collecting regional fringe images and performing phase demodulation and phase fusion include: The camera exposure time is set to be greater than or equal to the projection time of two adjacent regional fringe patterns, so that the camera can capture the sequence P at the same time during the acquisition of one image. i pB and P i pR two regional fringe images; The camera collects data at least once in sequence, and saves at least one spectrally separated phase-shift fringe image respectively; The collected spectrally separated fringe pattern is separated into RGB channels to obtain high-quality phase results without interference from multiple reflected light; The high-quality phase results are used to obtain complete phase results for three-dimensional reconstruction.

9. The method for three-dimensional measurement by regional adaptive fringe projection of spectrum separation according to claim 8, characterized in that: The collected spectral separation fringe image is separated into RGB channels, and the two images of the R channel and the B channel are taken to perform phase extraction independently as follows: Among them, Ф R and Φ B The phases obtained for the red and blue channels, I 1R -I 3R is the red channel stripe image, I 1B -I 3B This is the blue channel stripe image.

10. The spectral separation regional adaptive fringe projection three-dimensional measurement method according to claim 8, characterized in that: The three-dimensional reconstruction using the phase-height mapping model includes: The three-dimensional reconstruction is performed using the following mapping formula: Where H(x,y) represents the actual depth value at the coordinate (x,y), a(x,y), b(x,y), and c(x,y) are phase-height mapping parameters obtained by calibration; is the phase distribution of the object.

Images