A method for automatic three-dimensional scanning measurement with robot path planning

Abstract

The present invention discloses a robot path planning-based automatic three-dimensional scanning and measuring method. First, a virtual simulation environment is built; then, the measurement pose and measurement path are set on the virtual simulation environment and the measurement path is evaluated; then, hand-eye calibration is completed in the actual measurement environment, and the entity robot carrying the structured light measuring instrument arrives at the set measurement poses in sequence to take pictures, obtaining point cloud images from multiple perspectives; afterwards, according to the hand-eye calibration result, the multi-perspective point clouds are unified to the robot base coordinate system through coordinate transformation to complete the rough stitching process of the point clouds, and finally, the missing point clouds are complemented at the missing shooting locations. The present invention effectively reduces the trial-and-error risk of multi-perspective measurement by performing robot path planning through a virtual simulation platform, completes the rough registration of the point clouds by converting the point clouds to the robot base coordinate system, provides good prior conditions for the subsequent fine registration process, reduces the burden of the subsequent registration task, improves the working efficiency of the structured light measuring instrument, and shortens the task cycle.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional shape measurement. More specifically, it relates to a robot path planning-based automatic three-dimensional scanning measurement method. Background Art

[0002] In recent years, with the rapid development of machine vision, structured light measurement technology has been widely applied to some high-precision measurement fields such as impact detection, cultural relic reconstruction, and medical scanning due to advantages such as high-precision and good-quality generated point clouds. Since the structured light measurement device is limited by the imaging range and shooting angle of the camera, it is impossible to obtain the complete three-dimensional shape information of the measured object at one time when performing actual three-dimensional shape measurement tasks. Generally, the solution is to move the structured light measurement device to the spatial poses corresponding to each perspective, obtain the point cloud information of the measured object from multiple perspectives, and then splice the multi-perspective point cloud information to make the point cloud information of the measured object complete. However, due to the large mass of high-precision structured light devices, the traditional manual moving method cannot accurately grasp the pose states of each acquisition perspective. In some environments with high load and high repeatability, robots can well replace traditional human labor, and pre-path planning can be performed on the robots to complete the measurement task.

[0003] In factories, the measurement path generation of traditional robots usually adopts the method of manual teaching, where operators select the path points of the robot measurement task one by one for the robot. This requires a high level of technical skills of the operators and familiarity with the on-site working environment. Since the accuracy of a single-perspective point cloud is greatly affected by the ranging of the structured light device, when facing a measurement device such as a structured light measuring instrument that has requirements for the shooting distance, the operator also needs to manually measure the distance between the measurement device and the measured object and judge whether it meets the shooting distance requirements. However, in some measurement occasions where it is difficult to obtain the distance between the measurement device and the measured object, such as in high-temperature environments, how to ensure that the distance between the measurement device and the measured object meets the requirements at each measurement pose has become a difficult problem. Therefore, it is necessary to adopt an effective measurement method to reduce the operation difficulty of the robot in the measurement task and accurately obtain the distance between the measurement device and the measured object, so as to obtain a higher-quality single-perspective point cloud. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a robot path planning-based automatic three-dimensional scanning measurement method to improve the working efficiency of the structured light measuring instrument, shorten the task cycle, provide a good initial value for the subsequent fine splicing process, and reduce the burden of the subsequent registration task.

[0005] To achieve the above invention purpose, the robot path planning-based automatic three-dimensional scanning measurement method of the present invention is characterized by including:

[0006] (1) Virtual simulation environment construction

[0007] 1.1) According to the actual measurement environment, build a virtual simulation environment on a computer equipped with an open-source robot system, import the description file of the robot into the virtual simulation environment to generate the corresponding robot. At the same time, in the virtual simulation environment, install the structured light measuring instrument and the depth camera on the fixed fixture of the robot's end flange;

[0008] 1.2) Simulate the object to be measured and place it directly in front of the robot and the structured light measuring instrument. Determine multiple measurement planes for the structured light measuring instrument to photograph the object to be measured and multiple measurement paths existing on each measurement plane. Among them, the i-th measurement path of the k-th measurement plane is denoted as k S i k = 1, 2,..., K, i = 1, 2,..., M k , where K is the number of measurement planes, and M k is the number of measurement paths on the k-th measurement plane. At the same time, each measurement path also contains multiple measurement points. The j-th measurement point, that is, the measurement pose, in the i-th path of the k-th measurement plane of the object to be measured is denoted as k P ij , j = 1, 2,..., N k , N k is the number of measurement poses on the measurement path of the k-th measurement plane;

[0009] The field of view of the structured light measuring instrument is a rectangle with a length of m centimeters and a width of n centimeters. The length of the circumscribed rectangle of the k-th measurement plane of the object to be measured is k a, and the width is k b. The measurement paths are longitudinal paths along the length direction. Then, the constraint needs to be satisfied: the number of measurement paths M k ≥ k a / m, and the number of measurement points N k ≥ k b / n. According to this constraint, the j-th measurement pose k P ij , j = 1, 2,..., N k of the i-th measurement path of the k-th measurement plane is obtained;

[0010] (2) Measurement path planning

[0011] 2.1) For the i-th measurement path of the k-th measurement plane, first, in the virtual simulation environment, drag the end of the robot to the measurement pose k P ij , where the measurement pose k P ij is (k Px ij,k Py ij,k Pz ij,k Rx ij,k Ry ij,k Rz ij ),( k Px ij,k Py ij, k Pz ij ), is the position coordinate of the robot end-effector,( k Rx ij,k Ry ij,k Rz ij ), is the attitude coordinate of the robot end-effector;

[0012] 2.2), At the measured pose of the robot k P ij , by using a depth camera to collect the image of the object to be measured, obtaining an RGB-D image and converting it into a grayscale image, then performing threshold segmentation to filter the background, and then extracting the contour information of the object to be measured to obtain the minimum bounding rectangle where the contour of the object to be measured is located After that, in the range of the minimum bounding rectangle corresponding to the RGB-D image, traverse the depth information d of all pixel points to find the shortest distance d between the object to be measured and the structured light measuring instrument min , and at the same time record the pixel point coordinates a(u, v) of the shortest distance, then judge the shortest distance d min Whether it meets the measurement range of the structured light measuring instrument: d min ∈[D - δ, D + δ], where D is the focal length during camera calibration of the structured light measuring instrument, and δ is the allowable error range of the structured light measuring instrument measurement; if it does not meet, go to step 2.3) to adjust the measured pose, if it meets, go to step 2.4);

[0013] 2.3), Convert the pixel point coordinates a(u, v) through coordinate transformation to obtain the corresponding spatial coordinates (x (u,v) , y (u,v) , z (u,v) ), and then combine with the position coordinate of the robot end-effector( k Px ij,k Py ij,k Pz ij ) to determine a spatial line:

[0014]

[0015] where x, y, z are the coordinates on the spatial line;

[0016] Then, from the position coordinate( k Px ij,k Py ij,kPz ij ), along this space straight line, a position coordinate is calculated Meet the measurement conditions:

[0017]

[0018] Obtain the position coordinates that meet the measurement conditions After that, combine with the original pose coordinates ( k Rx ij,k Ry ij,k Rz ij ) to form an updated measurement pose k P ij That is According to the measurement pose k P ij , the corresponding joint states of the robot are inversely solved by robot kinematics to achieve pose adjustment;

[0019] 2.4), Record the measurement pose k P ij , return to step 2.1), and perform the next measurement pose k P i(j+1) Setting until the measurement path k S i All measurement poses k P ij The shortest distance d min After the judgment is completed, then enter step (3). Among them, before setting the next measurement pose k P i(j+1) , if:

[0020] A. When the measurement pose k P ij+1 is set in the spatial pose where the robot's degrees of freedom degenerate and the inverse kinematics has no solution, that is, near the singular point, in order to ensure the smooth movement of the robot, a transition point needs to be set to avoid the singular point of the robot;

[0021] B. When the shape of the measured object is irregular and there are protrusions on the surface of the measured object, in order to obtain better measurement results and avoid collisions with the irregular measured object during the movement of the robot carrying the structured light measuring instrument, when setting the measurement pose k P i(j+1) , the actual measurement pose of the robot and the shape characteristics of the measured object need to be considered. A transition point k P ij and the measurement pose k P i(j+1) A transition point

[0022] Transition point Distance detection and adjustment are not required at this point, and the transition point is to artificially adjust the measurement path of the robot;

[0023] 2.5), for the measurement path k S i perform path evaluation

[0024] All measurement poses and all transition points constitute a complete measurement path k S i , for the measurement path k S i perform path evaluation, and the steps are as follows:

[0025] Step 2.5.1), execute the measurement path k S i in the virtual simulation environment, and the robot will move continuously from the measurement pose k P i1 to the measurement pose Record the process points by equidistant sampling during the movement process Set to form the actual movement path of the robot as

[0026] In the measurement path k S i , its starting position coordinates are ([[]] k Px i1,k Py i1,k Pz i1 ), and the ending position coordinates are Then the shortest distance between the ending position and the starting position is:

[0027]

[0028] In the actual movement path , its starting position coordinates are The ending position coordinates are Then the length of the actual movement of the robot is:

[0029]

[0030] Subtracting the two equations gives:

[0031] L = l 2 -l 1

[0032] Then the evaluation function f 1 (L) is expressed as: f 1 (L) = (δ1 (-L) / δ 1 , 0 ≤ L ≤ δ 1 , where δ 1 is the set maximum error threshold;

[0033] Step 2.5.2), find a pose in the actual motion path of the robot to the centroid coordinates (Px obj , Py obj , Pz obj ) of the measured object, and the distance l 3 is the smallest. Then the distance l 3 is expressed as:

[0034]

[0035] Then the evaluation function f 2 (l 3 ) is expressed as:

[0036] f 2 (l 3 ) = (l 3 - δ 2 ) / l 3

[0037] where δ 2 is the minimum distance to ensure that the structured light measuring instrument does not collide with the measured object;

[0038] 2.5.3), for the measurement path k S i the comprehensive evaluation function k F i is:

[0039] k F i = (0.5f 1 (L) + 0.5F(l 3 )) * 100

[0040] According to the comprehensive evaluation function k F i evaluate the set measurement path. When the comprehensive evaluation function k F i > g, it passes the evaluation and enters Step 2.7). Otherwise, go to Step 2.6), where g is a threshold defined by the measurement personnel according to different measurement scenarios, 0 < g < 100;

[0041] 2.6), traverse all poses in the measurement path k S i and find the pose closest to the pose ​k P is , then move the robot to the measurement pose k P is . Drag the end of the robot to manually increase the distance d between the current measurement pose of the robot and the object to be measured min , if the pose k P is is the measurement pose, the adjustment range still needs to meet the measurement conditions of the structured light measuring instrument: d min ∈[D - δ, D + δ]. If the pose k P is is a transition point, the distance d needs to be adjusted according to the setting conditions of the transition point min . Replace the pose k P is with the adjusted pose, so as to complete the correction of the measurement path k S i , and then enter step 2.7);

[0042] 2.7), Process all measurement paths of all measurement surfaces according to steps 2.1) to 2.6) to complete the simulation of measurement path planning. Convert the planned measurement path into a communication message that the robot can recognize and send it to the physical robot by the virtual simulation environment;

[0043] (3), Real environment measurement

[0044] 3.1), Perform hand - eye calibration to obtain the rotation - translation matrix of the camera coordinate system relative to the flange coordinate system at the end of the robot, that is, the hand - eye relationship matrix H cg ;

[0045] 3.2), After completing the hand - eye calibration, start measuring the object to be measured. After receiving the measurement path sent by the virtual simulation environment, the physical robot moves to the measurement pose k P ij in turn;

[0046] 3.3), When reaching the measurement pose k P ij , the physical robot sends the measurement pose k P ij to the structured light measuring instrument, and obtains the single - view point cloud k I ij of the object to be measured at the current measurement pose through the binocular vision camera, and uses the hand - eye relationship matrix H cg and the pose matrix from the flange coordinate system at the end of the robot to the base coordinate system of the robot obtained according to the measurement pose to convert the single - view point cloud k I ij to the base coordinate system of the robot;

[0047] 3.4), Repeat steps 3.2) and 3.3) to obtain the measured object point cloud I in the robot base coordinate system B That is, the rough stitching result of the measured object point cloud, and then observe the point cloud I B Check whether it is complete, that is, whether there is a phenomenon of missed shooting. If missed shooting occurs, make up the shooting according to the following steps:

[0048] 3.4.1), For the position of the missed shooting point cloud, determine two adjacent point clouds to the position of the missed shooting point cloud k I ij , k I i(j+1) , and then determine the measurement pose of the robot k P ij and k P i(j+1) , and finally determine the measurement pose of the physical robot at the missed shooting position

[0049]

[0050] 3.4.2), Operate the physical robot to reach the measurement pose Obtain the single-viewpoint cloud of the measured object at the current measurement pose, and convert the single-viewpoint cloud to the robot base coordinate system according to step 3.3) to complete the supplementation of the missed shooting point cloud.

[0051] The invention object of the present invention is achieved as follows:

[0052] The automatic three-dimensional scanning and measuring method for robot path planning of the present invention first builds a virtual simulation environment on a computer equipped with an open-source robot system (ROS) according to the actual measurement environment; then, on the premise of ensuring the integrity of three-dimensional shape measurement, the measurement pose and measurement path are set on the virtual simulation environment and the measurement path is evaluated; then, in the actual measurement environment, the camera calibration of the structured light measuring instrument and the hand-eye calibration between the structured light measuring instrument and the robot are completed, and the measurement path passed through the path evaluation is sent to the physical robot. The physical robot carries the structured light measuring instrument to reach the set measurement poses in turn for shooting to obtain multi-viewpoint cloud images; then, according to the hand-eye calibration result, the multi-viewpoint cloud is unified to the robot base coordinate system through coordinate transformation to complete the rough stitching process of the point cloud. Finally, it is judged whether the rough stitching image of the measured object point cloud in the robot base coordinate system is complete, and the missing point cloud is supplemented at the missing shooting positions until the three-dimensional shape measurement of the entire measured object is completed. The present invention effectively reduces the trial-and-error risk of multi-view measurement through robot path planning on a virtual simulation platform; by converting the point cloud to the robot base coordinate system to complete the rough registration of the point cloud, it provides good prior conditions for the subsequent fine registration process, reduces the burden of the subsequent registration task, improves the working efficiency of the structured light measuring instrument, and shortens the task cycle.

[0053] Specifically, the present invention has the following advantages and innovations:

[0054] (1) The present invention completes the construction of a virtual simulation scenario on ROS, and can conduct full simulation experiments in the simulation environment, reducing the trial-and-error risk when selecting measurement poses.

[0055] (2) By detecting the distance to judge whether the distance from the structured light measuring instrument to the measured object at the current pose meets the ranging requirements of the structured light measuring instrument, the measurement accuracy and shooting effect are effectively improved, and the automatic adjustment of the measurement pose is realized for the measurement poses with inappropriate ranging.

[0056] (3) According to a specific path evaluation function, the selection criteria for a more efficient and safer path are given, improving the overall measurement efficiency of the system and ensuring measurement safety.

[0057] (4) The point clouds of the measured object taken at different measurement poses are unified into the robot base coordinate system, realizing the rough registration of the point cloud and providing a good initial value for the subsequent fine registration process.

[0058] (5) Throughout the measurement process, by using a binocular structured light measuring instrument, the measured object is measured in a non-contact manner, without the need to stick marking points on the surface of the measured object, retaining more morphological information of the measured object while having the advantages of high precision and high speed. Description of the Drawings

[0059] Figure 1 is a flowchart of a specific implementation of the robot path planning-based automatic three-dimensional scanning measurement method of the present invention;

[0060] Figure 2 is a schematic diagram for determining the measurement path and measurement pose;

[0061] Figure 3 is a schematic diagram of the rough stitching coordinate system transformation;

[0062] Figure 4 is a schematic diagram of a specific example of the virtual simulation environment built;

[0063] Figure 5 is a schematic diagram of the measurement pose setting on the same measurement surface of the object to be measured and a single-viewpoint cloud map;

[0064] Figure 6 is a schematic diagram of the settings on different measurement surfaces of the object to be measured and the rough stitching result of a single measurement surface;

[0065] Figure 7 is the rough stitching result of the final point cloud. Specific implementation mode

[0066] The following describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be particularly noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be ignored here.

[0067] Figure 1 is a flowchart of a specific implementation of the robot path planning-based automatic three-dimensional scanning measurement method of the present invention.

[0068] In this embodiment, as Figure 1 shown, the robot path planning-based automatic three-dimensional scanning measurement method of the present invention includes the following steps:

[0069] Step S1: Building a virtual simulation environment

[0070] Step S1.1: Generating a robot and installing a structured light measuring instrument and a depth camera

[0071] According to the actual measurement environment, build a virtual simulation environment on a computer equipped with an open-source robot system, import the description file of the robot into the virtual simulation environment to generate the corresponding robot. At the same time, in the virtual simulation environment, install the structured light measuring instrument and the depth camera on the fixed fixture of the end flange of the robot.

[0072] Among them, the description file is a URDF (Universal Robot Description Format) description file, and the content it contains includes: links, joints, kinematic and dynamic parameters, visualization models, collision detection models, etc.

[0073] Step S1.2: Determine the measurement path

[0074] Figure 2 It is a schematic diagram for determining the measurement path and measurement pose. In this embodiment, as Figure 2 shown, in order to obtain the three-dimensional shape characteristics of the object to be measured more completely and accurately, and to ensure that there is an overlapping area between adjacent point clouds, the object to be measured is simulated and placed directly in front of the robot and the structured light measuring instrument, and multiple measurement planes for the structured light measuring instrument to photograph the object to be measured and multiple measurement paths existing on each measurement plane are determined. Among them, the i-th measurement path on the k-th measurement plane is denoted as k S i k = 1, 2,..., K, i = 1, 2,..., M k , where K is the number of measurement planes, and M k is the number of measurement paths on the k-th measurement plane. At the same time, each measurement path also contains multiple measurement points. The j-th measurement point, that is, the measurement pose, on the i-th path on the k-th measurement plane of the object to be measured is denoted as k P ij , j = 1, 2,..., N k , and N k is the number of measurement poses on the measurement path of the k-th measurement plane.

[0075] The field of view of the structured light measuring instrument is a rectangle with a length of m centimeters and a width of n centimeters. The length of the circumscribed rectangle of the k-th measurement plane of the object to be measured is k a, and the width is k b. The measurement paths are longitudinal paths along the length direction. In order to ensure that there is a common area for the point cloud information of adjacent two measurement paths and adjacent two measurement points, the constraint needs to be satisfied: the number of measurement paths M k ≥ k a / m, and the number of measurement points N k ≥ k b / n. According to this constraint, the j-th measurement pose k P ij on the i-th measurement path of the k-th measurement plane is obtained, where j = 1, 2,..., N k , as Figure 2 shown, so that there are overlapping areas both horizontally and longitudinally.

[0076] Step S2: Measurement path planning

[0077] Step S2.1: Drag the end of the robot to the measurement pose

[0078] For the i-th measurement path of the k-th measurement plane, first, in the virtual simulation environment, drag the end of the robot to the measurement pose k P ij , where the measurement pose k P ij is ( k Px ij,k Py ij,k Pz ij,k Rx ij,k Ry ij,k Rz ij ), ( k Px ij,k Py ij,k Pz ij ), where ( k Rx ij,k Ry ij,k Rz ij ) are the attitude coordinates of the end of the robot.

[0079] Step S2.2: Shortest distance detection and judgment

[0080] At the measurement pose k P ij of the robot, by using a depth camera to collect the image of the object to be measured, obtaining an RGB-D image and converting it into a grayscale image, then performing threshold segmentation to filter the background, and then extracting the contour information of the object to be measured to obtain the minimum circumscribed rectangle where the contour of the object to be measured is located After that, traverse the depth information d of all pixel points in the range of the minimum circumscribed rectangle corresponding to the RGB-D image, find the shortest distance d min between the object to be measured and the structured light measuring instrument, and at the same time record the pixel point coordinates a(u, v) of the shortest distance, and then judge whether the shortest distance d min meets the measurement range of the structured light measuring instrument: d min ∈[D - δ, D + δ], where D is the focal length when the structured light measuring instrument performs camera calibration, and δ is the allowable error range of the structured light measuring instrument measurement; if it does not meet, go to Step S2.3 to adjust the measurement pose, if it meets, go to Step S2.4.

[0081] Step S2.3: Pose adjustment

[0082] Convert the pixel point coordinates a(u, v) through coordinate system transformation to obtain the corresponding spatial coordinates (x (u,v) , y (u,v) , z (u,v)), and then combine with the position coordinates at the end of the robot ( k Px ij,k Py ij,k Pz ij ) to determine a straight line in space:

[0083]

[0084] where x, y, and z are the coordinates on the straight line in space;

[0085] Then, starting from the position coordinates ( k Px ij,k Py ij,k Pz ij ), along this straight line in space, calculate to obtain a position coordinate that satisfies the measurement condition:

[0086]

[0087] Obtain the position coordinate that satisfies the measurement condition After that, combine with the original pose coordinates ( k Rx ij,k Ry ij,k Rz ij ) to form an updated measurement pose k P ij That is According to the measurement pose k P ij , through the inverse kinematics of the robot, solve the corresponding joint states of the robot to achieve pose adjustment.

[0088] Step S2.4: Repeat steps S2.1 - S2.3 until the shortest distance judgment and pose adjustment of all measurement poses on the measurement path are completed

[0089] Record the measurement pose k P ij , return to step S2.1, and perform the next measurement pose k P i(j+1) setting, until the measurement path k S i All measurement poses k P ij The shortest distance d min is judged, and then, enter step S2.5, where, before setting the next measurement pose k P i(j+1) if:

[0090] A. The measurement pose k P i(j+1)When the robot is set at a spatial pose with degenerate degrees of freedom and no solution for inverse kinematics, i.e., near a singular point, a transition point needs to be set to ensure the smooth movement of the robot. to avoid the singular point of the robot;

[0091] B. When the shape of the object to be measured is irregular and there are protrusions on the surface of the object to be measured, in order to obtain better measurement results and avoid collision with the irregular object to be measured during the movement of the robot carrying the structured light measuring instrument, when setting the measurement pose k P i(j+1) it is necessary to consider the actual measurement pose of the robot and the shape characteristics of the object to be measured. A transition point is set between the measurement pose k P ij and the measurement pose k P i(j+1) A transition point is set between them

[0092] Transition point There is no need to perform distance detection and adjustment at the transition point. The transition point is to artificially adjust the measurement path of the robot.

[0093] Step S2.5: Evaluate the measurement path k S i All measurement poses

[0094] and all transition points constitute a complete measurement path k S i . In order to measure whether the measurement path k S i can be used as an executable path for the robot to complete the measurement task, the measurement path k S i is evaluated for its path from the perspectives of efficiency and safety. The steps are as follows:

[0095] Step S2.5.1: Execute the measurement path k S i in the virtual simulation environment. The robot will continuously move from the measurement pose k P i1 to the measurement pose Record the process points by equidistant sampling of the movement process Set The actual movement path of the robot is

[0096] In the measurement path k S i its starting position coordinates are ( k Px i1,k Pyi1,k Pz i1 ), the termination position coordinates are Then the shortest distance between the termination position and the starting position is:

[0097]

[0098] In the actual motion path Among them, the starting position coordinates are The termination position coordinates are Then the actual motion length of the robot is:

[0099]

[0100] Subtracting the two equations gives:

[0101] L = l 2 -l 1

[0102] Then the evaluation function f 1 (L) is expressed as: f 1 (L) = (δ 1 -L) / δ 1 , 0 ≤ L ≤ δ 1 , where δ 1 is the set maximum error threshold.

[0103] Step S2.5.2: Find a pose in the actual motion path of the robot to the centroid coordinates (Px obj , Py obj , Pz obj ) of the measured substance, and the distance l 3 is the smallest. Then the distance l 3 is expressed as:

[0104]

[0105] Then the evaluation function f 2 (l 3 ) is expressed as:

[0106] f 2 (l 3 ) = (l 3 -δ 2 ) / l 3

[0107] Among them, δ 2 is the minimum distance to ensure that the structured light measuring instrument does not collide with the measured object.

[0108] Step S2.5.3: For the measurement path k S iComprehensive evaluation function k F i is as follows:

[0109] k F i =(0.5f 1 (L)+0.5F(l 3 ))*100

[0110] According to the comprehensive evaluation function k F i evaluate the set measurement path. When the comprehensive evaluation function k F i > g, it passes the evaluation and enters step 2.7; otherwise, it proceeds to step S2.6, where g is a threshold defined by the measurement personnel according to different measurement scenarios, and 0 < g < 100.

[0111] Step S2.6: Correct the path

[0112] Traverse all poses in the measurement path k S i to find the pose nearest to the pose k P is . Then move the robot to the measurement pose k P is . Drag the end of the robot to manually increase the distance d min between the current robot measurement pose and the measured object. If the pose k P is is a measurement pose, the adjustment range still needs to meet the measurement conditions of the structured light measuring instrument: d min ∈[D - δ, D + δ]. If the pose k P is is a transition point, the distance d min needs to be adjusted according to the setting conditions of the transition point. The adjusted pose replaces the pose to complete the correction of the measurement path k S i , and then enter step S2.7.

[0113] Step S2.7: Process all measurement paths of all measurement surfaces according to steps S2.1 to S2.6 to complete the measurement path planning simulation. Convert the planned measurement path into a communication message that the robot can recognize and send it from the virtual simulation environment to the physical robot.

[0114] Step S3: Real - environment measurement

[0115] Step S3.1: Perform hand - eye calibration to obtain the rotation - translation matrix of the camera coordinate system relative to the robot end - flange coordinate system, i.e., the hand - eye relationship matrix Hcg 。

[0116] Figure 3 is a schematic diagram of the conversion of the rough splicing coordinate system. As Figure 3 shown, the positional relationship between the entity robot, the object to be measured, and the structured light scanner (camera), as well as the conversion between coordinate systems, are as Figure 3 shown. Among them, the camera coordinate system is represented by the subscript c, the flange coordinate system at the end of the robot is represented by the subscript g, and the base coordinate system of the robot is represented by the subscript B. Before the robot carries the scanner to complete the measurement task, hand-eye calibration needs to be carried out first to solve the relative position relationship matrix H between the structured light measuring instrument and the flange at the end of the robot cg , specifically:

[0117] Step S3.1.1: First, adjust the binocular structured light camera of the structured light scanner so that both the left and right cameras can capture the object to be measured, and complete camera calibration according to Zhang Zhengyou's calibration method.

[0118] Step S3.1.2: Manually operate the robot to take pictures of the calibration board from the q = 1, 2,..., r poses respectively, obtain the calibration board pictures in the corresponding poses, and record the pose coordinates of the flange at the end of the robot at the same time.

[0119] Step S3.1.3: Obtain the rotation vector R cq and translation vector T cq of the calibration board relative to the camera for each group of calibration board pictures according to Zhang's calibration method, and form a rotation-translation matrix from them, that is, obtain the external parameter matrix of the camera The pose matrix of the robot can be calculated according to the pose coordinates of the flange at the end of the robot

[0120] Step S3.1.4: Since the relationship between the robot base and the calibration board during any two (the pth and qth) pose transformations then there is:

[0121]

[0122] Among them, is the rotation-translation matrix from the flange at the end of the robot to the robot base coordinate system during the pth shooting, is the hand-eye relationship matrix during the pth shooting, is the rotation-translation matrix from the camera to the robot coordinate system during the pth shooting. has the same meaning as that during the pth shooting.

[0123] Since the relationship between the robot base and the calibration board remains unchanged during the two shootings, then there is the rotation-translation matrix H from the robot base to the calibration board rc :

[0124]

[0125] Since the pose relationship between the end of the robot flange and the camera remains unchanged in the two transformations, the hand-eye relationship matrix H exists. cg :

[0126]

[0127] Then the processed hand-eye calibration equation can be written as:

[0128] AX = XB

[0129] where X = H cg .

[0130] Step S3.1.5: Use the Tsai-Lenz algorithm to solve the hand-eye calibration equation, and then obtain the rotation and translation matrix of the camera coordinate system relative to the robot end flange coordinate system, that is, the hand-eye relationship matrix H. cg .

[0131] Step S3.2: After completing the hand-eye calibration, start measuring the object to be measured. After the physical robot receives the measurement path sent by the virtual simulation environment, it moves to the measurement pose k P ij in sequence.

[0132] Step S3.3: When reaching the measurement pose k P ij , the physical robot sends the measurement pose k P ij to the structured light measuring instrument, and obtains the single-viewpoint cloud of the object to be measured at the current measurement pose through the binocular vision camera k I ij , and uses the hand-eye relationship matrix H cg and the pose matrix from the robot end flange coordinate system to the robot base coordinate system obtained according to the measurement pose to transform the single-viewpoint cloud k I ij to the robot base coordinate system. Specifically:

[0133] Step S3.3.1: At the measurement pose k P ij , the left and right cameras of the structured light measuring instrument respectively take a depth image, and perform point cloud reconstruction according to the binocular camera principle to obtain the single-viewpoint cloud k P ij at the measurement pose k I ij , and according to the measurement pose k P ijObtain the pose matrix

[0134] Step S3.3.2: Convert the single-viewpoint point cloud k I ij to the robot base coordinate system to form a new point cloud The conversion method is as follows:

[0135]

[0136] Step S3.4: Repeat steps S3.2 and S3.3 to obtain the point cloud I of the object to be measured in the robot base coordinate system B That is, the rough stitching result of the point cloud of the object to be measured, the point cloud I in the robot base coordinate system B It can be expressed as:

[0137]

[0138] Then observe the point cloud I B to see if it is complete, that is, if there is a phenomenon of missed shooting. If missed shooting occurs, make up the shooting according to the following steps:

[0139] Step S3.4.1: For the position of the missed-shot point cloud, determine two adjacent point clouds to the position of the missed-shot point cloud k I ij 、 k I i(j+1) , and then determine the measurement poses of the robot k P ij and k P i(j+1) , and finally determine the measurement pose of the physical robot at the missed-shot position

[0140]

[0141] Step S3.4.2: Operate the physical robot to reach the measurement pose Obtain the single-viewpoint point cloud of the object to be measured at the current measurement pose, and convert the single-viewpoint point cloud to the robot's base coordinate system according to step S3.3 to complete the complement of the missed-shot point cloud.

[0142] Example

[0143] In this example, the object to be measured is a cuboid with dimensions of 600mm * 450mm * 200mm. First, build a 1:1 virtual simulation measurement environment in a computer equipped with the Ubuntu system and the ROS operating system according to the actual measurement scenario. The virtual simulation environment is as Figure 4As shown, the structured light measuring instrument is installed on the flange at the end of the six-degree-of-freedom robot, and the object to be measured is placed on the measuring platform directly in front of the robot. Secondly, the measurement path planning is carried out according to the path planning method described in step S2, and a surface of the object to be measured is selected for setting measurement points. Then other measurement surfaces are selected, and the measurement path planning is also carried out according to step S2. After all the measurement paths are set, the path information is sent to the physical six-degree-of-freedom robot.

[0144] In the actual measurement environment, the camera calibration and hand-eye calibration are completed according to steps S3.1 - S3.4, and the hand-eye matrix H is solved. cg Then, the six-degree-of-freedom robot carries the structured light measuring instrument to reach the set measurement points according to different paths for shooting. Figure 5 It is the single-viewpoint cloud of the object to be measured obtained by the six-degree-of-freedom robot carrying the structured light measuring instrument according to the planned measurement path under a single measurement surface. The single-viewpoint cloud on a single measurement surface is unified to the robot base coordinate system through coordinate transformation, and the result is as Figure 6 shown. It can be seen that the single-viewpoint clouds on the same measurement surface can be successfully stitched. Figure 7 It is the final rough point cloud stitching result. It can be seen that the point clouds on different measurement surfaces can basically restore the three-dimensional shape of the object to be measured after rough point cloud stitching.

[0145] Although the above describes the illustrative specific embodiments of the present invention for the convenience of those skilled in the art of this technology to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art of this technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

Claims

1. A robot path planning-based automatic three-dimensional scanning and measuring method, characterized by comprising: (1) Building a virtual simulation environment 1.1), According to the actual measurement environment, build a virtual simulation environment on a computer equipped with an open-source robot system, and import the robot's description file into the virtual simulation environment to generate the corresponding robot. At the same time, in the virtual simulation environment, install the structured light measuring instrument and the depth camera on the fixed fixture of the robot's end flange; 1.2), Simulate the object to be measured and place it directly in front of the robot and the structured light measuring instrument, and determine multiple measurement surfaces for the structured light measuring instrument to photograph the object to be measured and multiple measurement paths existing on each measurement surface; Among them, The $i$-th measurement path on the $k$-th measurement surface is denoted as k S i where $k = 1, 2, \cdots, K$ and $i = 1, 2, \cdots, M$ k , where $K$ is the number of measurement surfaces, and $M$ k is the number of measurement paths on the $k$-th measurement surface. At the same time, each measurement path also contains multiple measurement points. The $j$-th measurement point in the $i$-th path on the $k$-th measurement surface of the measured object, that is, the measurement pose, is denoted as k P ij , where $j = 1, 2, \cdots, N$ k , and $N$ k is the number of measurement poses on the measurement path of the $k$-th measurement surface; The field of view of the structured light measuring instrument is a rectangle with a length of m centimeters and a width of n centimeters. The circumscribed rectangle of the k-th measurement surface of the object to be measured has a length of k a and a width of k b. If the measurement path is a longitudinal path along the length direction, then the following constraints need to be satisfied: the number of measurement paths M on the same measurement surface k ≥ k a / m, and the number of measurement points N on the same measurement path k ≥ k b / n. According to this constraint, the j-th measurement pose of the i-th measurement path on the k-th measurement surface is obtained k P ij , where j = 1, 2,..., N k ; (2), Measurement path planning 2.1) For the i-th measurement path on the k-th measurement plane, first, in the virtual simulation environment, drag the robot end to the measurement pose k P ij , Among them, Measured pose k P ij is ( k Px ij,k Py ij,k Pz ij,k Rx ij,k Ry ij,k Rz ij ), ( k Px ij,k Py ij,k Pz ij ) is the position coordinates of the robot end-effector, ( k Rx ij,k Ry ij,k Rz ij ) is the attitude coordinates of the robot end-effector; 2.2) At the measured pose of the robot k P ij At the position of P, after collecting the image of the object to be measured using a depth camera, obtaining an RGB-D image and converting it into a grayscale image, performing threshold segmentation to filter the background, and then extracting the contour information of the object to be measured, the minimum circumscribed rectangle where the contour of the object to be measured is located is obtained After that, in the range of the minimum circumscribed rectangle corresponding to the RGB-D image, the depth information d of all pixel points is traversed to find the shortest distance d between the object to be measured and the structured light measuring instrument min , and at the same time, record the pixel point coordinates a(u, v) of the shortest distance, and then judge the shortest distance d min Whether it meets the measurement range of the structured light measuring instrument: d min ∈[D - δ, D + δ], where D is the focal length when the structured light measuring instrument performs camera calibration, and δ is the allowable error range of the structured light measuring instrument measurement; if it does not meet, go to step 2.3) to adjust the measurement pose, if it meets, go to step 2.4); 2.3) Convert the pixel point coordinates a(u, v) through coordinate system transformation to obtain the corresponding spatial coordinates (x (u,v) , y (u,v) , z (u,v) ), and then combine with the position coordinates of the robot end ([[]] k Px ij,k Py ij,k Pz ij ) to determine a spatial straight line: Among them, x, y, and z are the coordinates on the space straight line; Then, starting from the position coordinates ( k Px ij,k Py ij,k Pz ij ), along this straight line in space, a position coordinate is calculated to satisfy the measurement condition: Obtain the position coordinates that meet the measurement conditions After that, combine with the original pose coordinates ( k Rx ij, k Ry ij,k Rz ij ) to form an updated measured pose k P ij That is According to the measured pose k P ij , the corresponding joint states of the robot are solved by the inverse kinematics of the robot to achieve pose adjustment; 2.4), Record the measured pose k P ij , return to step 2.1) and perform the next measured pose k P i(j+1) setting until the measurement path k S i of all measured poses k P ij of the shortest distance d min is judged, and then step (3) is entered. Among them, before performing the next measured pose k P i(j+1) setting, if: A. Measuring pose k P i(j+1) When it is set at the spatial pose where the degrees of freedom of the robot degenerate and the inverse kinematics has no solution, that is, near the singular point, a transition point needs to be set to ensure the smooth movement of the robot to avoid the singular point of the robot; B. When the morphology of the object to be measured is irregular and there are protrusions on the surface of the object to be measured, in order to obtain better measurement results and avoid collision with the irregular object to be measured during the movement of the robot carrying the structured light measuring instrument, when setting the measurement pose k P i(j+1) it is necessary to consider the actual measurement pose of the robot and the morphological characteristics of the object to be measured. When setting the measurement pose k P ij and the measurement pose k P i(j+1) a transition point is set between them Transition point No distance detection and adjustment are required at the transition point to artificially adjust the measurement path of the robot; 2.5), evaluate the measurement path k S i for path evaluation All measured poses and all transition points constitute a complete measurement path k S i , for the measurement path k S i perform path evaluation, and the steps are as follows: Step 2.5.1), execute the measurement path in the virtual simulation environment k S i , the robot will continuously move from the measurement pose k P i1 to the measurement pose Perform equal-time sampling on the motion process to record the process points Set The actual motion path of the robot formed is In the measurement path k S i , the starting position coordinates are ( k Px i1,k Py i1,k Pz i1 ), and the ending position coordinates are Then the shortest distance between the ending position and the starting position is: In the actual motion path wherein, its starting position coordinates are and the ending position coordinates are then the actual motion length of the robot is: Subtracting the two equations gives: L＝l 2 -l 1 Then the evaluation function f 1 (L) is expressed as: f 1 (L) = (δ 1 - L) / δ 1 , 0 ≤ L ≤ δ 1 , where δ 1 is the set maximum error threshold; Step 2.5.2), find a pose in the actual movement path of the robot in which the distance l from the pose to the centroid coordinates (Px obj , Py obj , Pz obj ) of the measured object is the smallest, then the distance l 3 is expressed as: 3 ​ Then the evaluation function f 2 (l 3 ) is expressed as: f 2 (l 3 )=(l 3 -δ 2 ) / l 3 where δ 2 is the minimum distance to ensure that the structured light measuring instrument does not collide with the object to be measured; 2.5.3) For the measurement path k S i The comprehensive evaluation function of k F i is as follows: k F i =(0.5f 1 (L)+0.5f 2 (l 3 ))*100 According to the comprehensive evaluation function k F i evaluate the set measurement path. When the comprehensive evaluation function k F i > g, it passes the evaluation and proceeds to step 2.7); otherwise, it proceeds to step 2.6), where g is a threshold defined manually by the measurement personnel according to different measurement scenarios, and 0 < g < 100; 2.6) Traverse the measurement path k S i for all poses therein, and find the pose closest to the pose k P is . Then move the robot to the measurement pose k P is . Manually drag the end of the robot to increase the distance d between the current measurement pose of the robot and the object to be measured min . If the pose k P is is a measurement pose, the adjustment range still needs to meet the measurement conditions of the structured light measuring instrument: d min ∈[D - δ, D + δ]. If the pose k P is is a transition point, the distance d needs to be adjusted according to the setting conditions of the transition point min . The adjusted pose replaces the pose k P is , thus completing the correction of the measurement path k S i , and then proceed to step 2.7); 2.7), Process all the measurement paths of all the measurement surfaces according to steps 2.1) to 2.6) to complete the simulation of the measurement path planning, and convert the planned measurement path into a communication message that the robot can recognize and send it from the virtual simulation environment to the physical robot; (3), Real environment measurement 3.1), perform hand-eye calibration to obtain the rotation and translation matrix of the camera coordinate system relative to the robot end flange coordinate system, i.e., the hand-eye relationship matrix H cg ; 3.2) After completing the hand-eye calibration, start measuring the object to be measured. After the physical robot receives the measurement path sent by the virtual simulation environment, it moves to the measurement pose in sequence k P ij ; 3.3), when reaching the measurement pose k P ij At this point, the physical robot will send the measurement pose k P ij to the structured light measuring instrument, and obtain the single-viewpoint cloud of the object to be measured at the current measurement pose through the binocular vision camera k I ij , and use the hand-eye relationship matrix H cg , the pose matrix from the end flange coordinate system of the robot to the robot base coordinate system obtained according to the measurement pose to transform the single-viewpoint cloud k I ij to the robot base coordinate system; 3.4), Repeat steps 3.2) and 3.3) to obtain the point cloud I of the object to be measured in the robot base coordinate system B That is, the rough stitching result of the point cloud of the object to be measured, and then observe the point cloud I B Whether it is complete, that is, whether there is a phenomenon of missed shooting. If missed shooting occurs, make up the shooting according to the following steps: 3.4.1) For the missed point cloud location, determine the two point clouds adjacent to the missed point cloud location k I ij , k I i(j+1) , and then determine the robot's measurement posture k P ij and k P i(j+1) , and finally determine the measured pose of the physical robot at the missed position 3.4.2), Operate the entity robot to reach the measurement pose Obtain the single-viewpoint point cloud of the measured object at the current measurement pose, and according to step 3.3), convert the single-viewpoint point cloud to the base coordinate system of the robot to complete the complement of the missing point cloud.

Images