Quick point cloud splicing method based on structured light three-dimensional reconstruction

Abstract

The invention discloses a point cloud rapid splicing method based on structured light three-dimensional reconstruction, and the method comprises the steps: obtaining and unfolding a wrapped phase, and the obtaining and unfolding of the wrapped phase comprise a four-step phase shift method and a three-frequency extrapolation principle; camera calibration, wherein the camera calibration comprises obtaining a coordinate mapping relation between camera image pixels and a three-dimensional space object; hand-eye calibration, wherein the hand-eye calibration comprises the step of obtaining a coordinate mapping relation between a coordinate system at the tail end of the mechanical arm and a three-dimensional space object; point cloud splicing, wherein the point cloud splicing comprises coarse point cloud splicing and fine point cloud splicing; according to the method provided by the invention, the surface information of the three-dimensional space object is measured through the structured light principle, and the surface information of the three-dimensional space object of a large scene can be quickly and effectively obtained through the point cloud rough splicing and fine splicing after hand-eye calibration.

Description

Technical Field

[0001] The present invention mainly relates to the field of point cloud processing, and in particular to a point cloud fast splicing method based on structured light three-dimensional reconstruction. Background Art

[0002] In recent years, with the development of industrial automation and intelligence, automation technology has also received widespread attention, especially vision-based automation technology. As one of the typical technologies in the field of three-dimensional visual imaging, structured light three-dimensional imaging technology has been widely used in measurement, industry, medical diagnosis and other fields. Structured light three-dimensional imaging technology can take into account both system flexibility and measurement accuracy, and is one of the mainstream technologies for optical three-dimensional surface imaging and measurement. When scanning the environment, the structured light scanning device often cannot measure the point cloud data of the environment at one time in the same coordinate system. The reason is that the size of the environment exceeds the measurement range of the scanning device, and the objects in the environment block each other. It is unlikely that the point cloud scanning device can scan the complete point cloud of the object at one angle. After obtaining multiple pieces of point cloud data, we need a technology to rotate and translate the multiple pieces of point cloud data to a unified coordinate system so that they can form a complete environmental point cloud data. This technology is called point cloud stitching. Summary of the invention

[0003] The purpose of the present invention is to provide a point cloud fast stitching method based on structured light 3D reconstruction, which measures the surface information of three-dimensional space objects through the principle of structured light, and through coarse and fine stitching of point clouds after hand-eye calibration, can quickly and effectively obtain the surface information of three-dimensional space objects in large scenes.

[0004] The method for rapid point cloud stitching based on structured light three-dimensional reconstruction of the present invention comprises the following steps:

[0005] Step 1, acquisition and expansion of the wrapped phase, the acquisition and expansion of the wrapped phase includes a four-step phase shift method and a three-frequency extrapolation principle, by projecting four sinusoidal fringes with uniform step lengths onto the surface of the object;

[0006] I 1 (x, y)=I′(x, y)+I″(x, y)cos(2πfx) (1)

[0007] I 2 (x, y)=I′(x, y)+I″(x, y)cos(2πfx+π / 2) (2)

[0008] I 3 (x, y)=I′(x, y)+I″(x, y)cos(2πfx+π) (3)

[0009] I 4(x, y)=I′(x, y)+I″(x, y)cos(2πfx+3π / 2) (4)

[0010] In formulas (1), (2), (3), and (4), I 1 (x, y), I 2 (x, y), I 3 (x, y), I 4 (x, y) is the fringe diagram I 1 ,I 2 ,I 3 ,I 4 The light intensity of , where I′(x, y) is the DC component of the image, I″(x, y) is the amplitude, and f is the number of cycles contained in the fringe pattern. The wrapping phase can be calculated using formulas (1), (2), (3), and (4):

[0011]

[0012] In formula (5), To solve the wrapped phase, since the inverse tangent function is used in the phase calculation, the phase value range is [-π, π]. In order to get the continuous phase distribution we want in the image, we need to further process and expand the phase to ensure the uniqueness of the phase value in the entire image.

[0013] Through the principle of three-frequency interpolation, two phase functions of different periods are superimposed and fused to obtain a phase function of different frequency, and finally a phase with a period of 1 within the image range is obtained. After system calibration, the point cloud of the object surface is obtained.

[0014] Step 2: camera calibration. The camera calibration includes obtaining the coordinate mapping relationship between camera image pixels and three-dimensional space objects. The coordinate mapping relationship includes an internal parameter matrix and an external parameter matrix. The three-dimensional coordinates of real-world objects are converted into the camera coordinate system through rotation and translation. According to the traditional pinhole imaging principle, the conversion relationship between the camera coordinate system and the image coordinate system can be obtained. The image coordinate system can be converted into a pixel coordinate system through a certain proportion of expansion and translation.

[0015] Step three, hand-eye calibration, the hand-eye calibration includes obtaining the coordinate mapping relationship between the robot arm end coordinate system and the three-dimensional space object, the transformation relationship from the robot arm flange coordinate system to the base coordinate system is directly obtained by the robot operating system, the conversion relationship between the camera coordinate system and the calibration plate coordinate system is obtained by the external parameters of the camera, the transformation relationship between the camera coordinate system and the robot arm flange coordinate system remains unchanged during the movement of the robot arm, the transformation relationship from the calibration plate coordinate system to the robot arm base coordinate system is unknown, and the calibration plate coordinate system and the robot arm base coordinate system are kept stationary during the calibration process.

[0016] In the actual operation, the camera is fixed to the end of the robotic arm, the calibration plate is placed in the camera field of view and kept fixed, the robotic arm is moved to shoot images of the calibration plate at different heights and angles, and the coordinates and posture of the robotic arm flange are saved at the same time. A total of 18 sets of images and postures need to be obtained. Among the 18 images, four images are taken in each of the four directions of front, back, left, and right. The remaining two images are taken as much as possible above the calibration plate. It should be ensured that each image has a complete calibration plate. By calculating the transformation matrix between the camera and the robotic arm flange, the calculation result can be expressed as Euler angles and translation vectors to obtain a 3×3 transformation matrix.

[0017] Step 4: rough stitching of point clouds. Rotate and translate the object surface point clouds at two different angles. Calculate the rotation and translation matrix between the camera coordinate system and the robot flange coordinate system, and obtain the rotation and translation matrix between the robot flange coordinate system and the end of the robot from the robot operating system. Rotate and translate the object surface point cloud twice and transform it to the robot base coordinate system. It can be expressed as:

[0018]

[0019] In formula (6), Represents the rotation and translation matrix between the robot flange coordinate system and the end of the robot, represents the rotation and translation matrix between the camera coordinate system and the robot flange coordinate system, (X c , Y c , Z c ) represents the point cloud coordinates of the object surface before rotation and translation, (X w , Y w , Z w ) represents the point cloud coordinates of the object surface after rotation and translation.

[0020] Step 5: precise point cloud stitching. Find the nearest point on the surface point cloud of the object according to the constraints. When the error function is minimized, calculate the rotation and translation matrix. The main steps include:

[0021] 1) Calculating the distance between the object surface point cloud and the matching point cloud, and minimizing the distance;

[0022] 2) Calculate the rotation matrix and translation matrix to minimize the mean square error between point clouds;

[0023] 3) Rotating and translating the object surface point cloud according to the rotation matrix and the translation matrix.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1) The present invention projects four sinusoidal fringe patterns on the surface of an object through a four-step phase shift method and a three-frequency extrapolation principle, calculates the wrapped phase and performs phase unwrapping. After the system is calibrated, the surface point cloud of the object can be obtained quickly and accurately.

[0026] 2) The present invention takes 18 images of the calibration plate in different positions and postures, and simultaneously records the transformation matrix from the robot flange coordinate system to the robot base coordinate system, and obtains the transformation matrix from the camera coordinate system to the robot base coordinate system by calculation.

[0027] 3) The present invention roughly splices the point cloud of the object surface in the robot base coordinate system, and then calculates the distance between the roughly spliced ​​point clouds. When the distance is minimum, the rotation matrix and the translation matrix are calculated, and the root mean square is minimized. Compared with the prior art, the present invention can quickly complete the rough splicing of point clouds, and calculate the transformation matrix when the root mean square is minimum, and can obtain an accurate rotation and translation matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method for rapid point cloud stitching based on structured light 3D reconstruction provided by an embodiment of the present invention.

[0029] Figure 2 This is a point cloud stitching image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] Reference Figure 1 The flowchart is a method for rapid point cloud stitching based on structured light 3D reconstruction according to the present invention, and the steps are as follows:

[0032] Step 1, acquisition and expansion of the wrapped phase, the acquisition and expansion of the wrapped phase includes a four-step phase shift method and a three-frequency extrapolation principle, by projecting four sinusoidal fringes with uniform step lengths onto the surface of the object;

[0033] I 1 (x, y)=I′(x, y)+I″(x, y)cos(2πfx) (1)

[0034] I 2 (x, y)=I′(x, y)+I″(x, y)cos(2πfx+π / 2) (2)

[0035] I 3 (x, y)=I′(x, y)+I″(x, y)cos(2πfx+π) (3)

[0036] I 4 (x, y)=I′(x, y)+I″(x, y)cos(2πfx+3π / 2) (4)

[0037] In formulas (1), (2), (3), and (4), I 1 (x, y), I 2 (x, y), I 3 (x, y), I 4 (x, y) is the fringe diagram I 1 ,I 2 ,I 3 ,I 4 The light intensity of , where I′(x, y) is the DC component of the image, I″(x, y) is the amplitude, and f is the number of cycles contained in the fringe pattern. The wrapping phase can be calculated using formulas (1), (2), (3), and (4):

[0038]

[0039] In formula (5), To solve the wrapped phase, since the inverse tangent function is used in the phase calculation, the phase value range is [-π, π]. In order to get the continuous phase distribution we want in the image, we need to further process and expand the phase to ensure the uniqueness of the phase value in the entire image.

[0040] Through the principle of three-frequency interpolation, two phase functions of different periods are superimposed and fused to obtain a phase function of different frequency, and finally a phase with a period of 1 within the image range is obtained. After system calibration, the point cloud of the object surface is obtained.

[0041] Step 2: camera calibration. The camera calibration includes obtaining the coordinate mapping relationship between camera image pixels and three-dimensional space objects. The coordinate mapping relationship includes an internal parameter matrix and an external parameter matrix. The three-dimensional coordinates of real-world objects are converted into the camera coordinate system through rotation and translation. According to the traditional pinhole imaging principle, the conversion relationship between the camera coordinate system and the image coordinate system can be obtained. The image coordinate system can be converted into a pixel coordinate system through a certain proportion of expansion and translation.

[0042] Step three, hand-eye calibration, the hand-eye calibration includes obtaining the coordinate mapping relationship between the robot arm end coordinate system and the three-dimensional space object, the transformation relationship from the robot arm flange coordinate system to the base coordinate system is directly obtained by the robot operating system, the conversion relationship between the camera coordinate system and the calibration plate coordinate system is obtained by the external parameters of the camera, the transformation relationship between the camera coordinate system and the robot arm flange coordinate system remains unchanged during the movement of the robot arm, the transformation relationship from the calibration plate coordinate system to the robot arm base coordinate system is unknown, and the calibration plate coordinate system and the robot arm base coordinate system are kept stationary during the calibration process.

[0043] In the actual operation, the camera is fixed to the end of the robotic arm, the calibration plate is placed in the camera field of view and kept fixed, the robotic arm is moved to shoot images of the calibration plate at different heights and angles, and the coordinates and posture of the robotic arm flange are saved at the same time. A total of 18 sets of images and postures need to be obtained. Among the 18 images, four images are taken in each of the four directions of front, back, left, and right. The remaining two images are taken as much as possible above the calibration plate. It should be ensured that each image has a complete calibration plate. By calculating the transformation matrix between the camera and the robotic arm flange, the calculation result can be expressed as Euler angles and translation vectors to obtain a 3×3 transformation matrix.

[0044] Step 4: rough stitching of point clouds. Rotate and translate the object surface point clouds at two different angles. Calculate the rotation and translation matrix between the camera coordinate system and the robot flange coordinate system, and obtain the rotation and translation matrix between the robot flange coordinate system and the end of the robot from the robot operating system. Rotate and translate the object surface point cloud twice and transform it to the robot base coordinate system. It can be expressed as:

[0045]

[0046] In formula (6), Represents the rotation and translation matrix between the robot flange coordinate system and the end of the robot, represents the rotation and translation matrix between the camera coordinate system and the robot flange coordinate system, (X c , Y c , Z c ) represents the point cloud coordinates of the object surface before rotation and translation, (X w , Y w , Z w ) represents the point cloud coordinates of the object surface after rotation and translation.

[0047] Step 5: precise point cloud stitching. Find the nearest point on the surface point cloud of the object according to the constraints. When the error function is minimized, calculate the rotation and translation matrix. The main steps include:

[0048] 1) Calculating the distance between the object surface point cloud and the matching point cloud, and minimizing the distance;

[0049] 2) Calculate the rotation matrix and translation matrix to minimize the mean square error between point clouds;

[0050] 3) Rotating and translating the object surface point cloud according to the rotation matrix and the translation matrix. Figure 2 This is the image after point cloud stitching.

Claims

1. A method for rapid point cloud stitching based on structured light 3D reconstruction, characterized in that: The steps include: Acquisition and expansion of wrapped phase, wherein the acquisition and expansion of wrapped phase include a four-step phase shift method and a triple-frequency extrapolation principle; Camera calibration, the camera calibration comprising obtaining a coordinate mapping relationship between camera image pixels and three-dimensional space objects; Hand-eye calibration, which includes obtaining a coordinate mapping relationship between the robot end coordinate system and the three-dimensional space object; Point cloud stitching, the point cloud stitching includes point cloud rough stitching and point cloud fine stitching.

2. The method for rapid point cloud stitching based on structured light 3D reconstruction according to claim 1, characterized in that: Acquisition and expansion of the wrapped phase, the acquisition and expansion of the wrapped phase includes a four-step phase shift method and a three-frequency extrapolation principle, the four-step phase shift method includes projecting four sinusoidal fringes of uniform step length onto the surface of the object: I1(x,y)=I′(x,y)+I″(x,y)cos(2πfx) (1) I2(x,y)=I′(x,y)+I"(x,y)cos(2πfx+π / 2) (2) I3(x,y)=I′(x,y)+I″(x,y)cos(2πfx+π) (3) I4(x,y)=I′(x,y)+I"(x,y)cos(2πfx+3π / 2) (4) In formulas (1), (2), (3), and (4), I1(x, y), I2(x, y), I3(x, y), and I4(x, y) are the light intensities of the fringe patterns I1, I2, I3, and I4, respectively, where I′(x, y) is the DC component of the image, I″(x, y) is the amplitude, and f is the number of cycles contained in the fringe pattern. The wrapping phase can be calculated using formulas (1), (2), (3), and (4): In formula (5), To solve for the parcel phase; The triple-frequency extrapolation principle includes superimposing and fusing two phase functions of different periods to obtain a phase function of different frequency, and finally obtaining a phase with a period of 1 within the image range, and obtaining a point cloud of the object surface after system calibration.

3. The method for rapid point cloud stitching based on structured light 3D reconstruction according to claim 1, characterized in that: Camera calibration, which includes obtaining the coordinate mapping relationship between camera image pixels and three-dimensional space objects. The coordinate mapping relationship includes an internal parameter matrix and an external parameter matrix. The three-dimensional coordinates of real-world objects are converted into the camera coordinate system through rotation and translation. According to the traditional pinhole imaging principle, the conversion relationship between the camera coordinate system and the image coordinate system can be obtained. The image coordinate system can be converted into a pixel coordinate system through a certain proportion of expansion and translation.

4. The method for rapid point cloud stitching based on structured light 3D reconstruction according to claim 1, characterized in that: Hand-eye calibration, which includes the coordinate mapping relationship between the end coordinate system of the robot arm and the coordinate mapping relationship of the three-dimensional space object. The camera is fixed to the end of the robot arm, the calibration plate is placed in the camera field of view and kept fixed, and the robot arm is moved to shoot the calibration plate images at different heights and angles. At the same time, the coordinates and posture of the robot arm flange are saved. A total of 18 sets of images and postures need to be obtained. Among the 18 images, four images are taken in each of the four directions of front, back, left, and right. The remaining two images are taken above the calibration plate as much as possible. It should be ensured that each image has a complete calibration plate. By calculating the transformation matrix between the camera and the robot flange, the calculation result can be expressed as Euler angles and translation vectors to obtain a 3×3 transformation matrix.

5. The method for rapid point cloud stitching based on structured light 3D reconstruction according to claim 1, characterized in that: Point cloud stitching, the point cloud stitching includes rough point cloud stitching and fine point cloud stitching, the rough point cloud stitching includes rotating and translating the object surface point cloud at two different angles, calculating the rotation and translation matrix between the camera coordinate system and the robot flange coordinate system, and obtaining the rotation and translation matrix between the robot flange coordinate system and the end of the robot from the robot operating system, rotating and translating the object surface point cloud twice and then converting it to the robot base coordinate system, which can be expressed as: In formula (6), Represents the rotation and translation matrix between the robot flange coordinate system and the end of the robot, represents the rotation and translation matrix between the camera coordinate system and the robot flange coordinate system, (X c , Y c , Z c ) represents the point cloud coordinates of the object surface before rotation and translation, (X w , Y w , Z w ) represents the point cloud coordinates of the object surface after rotation and translation.

6. The method for rapid point cloud stitching based on structured light 3D reconstruction according to claim 1, characterized in that: Point cloud stitching includes rough point cloud stitching and fine point cloud stitching. Fine point cloud stitching includes finding the nearest point on the surface point cloud of the object according to the constraint conditions, and calculating the rotation and translation matrix when the error function is minimized. The main steps include: 1) Calculating the distance between the object surface point cloud and the matching point cloud, and minimizing the distance; 2) Calculate the rotation matrix and translation matrix to minimize the mean square error between point clouds; 3) Rotating and translating the object surface point cloud according to the rotation matrix and the translation matrix.

Images