Color Three-Dimensional Reconstruction Method for Large Workpieces with Complex Shapes Based on Point Cloud Information and Hand-Eye Calibration

Abstract

The present invention discloses a method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information for hand-eye calibration. By using a structured light binocular vision system in combination with image fusion, it is possible to achieve rapid acquisition of high-precision point cloud data while ensuring the accuracy of three-dimensional reconstruction, taking advantage of the flexible configuration and high measurement efficiency of the binocular structured light system. On this basis, a color camera is combined to endow the point cloud with real color texture feature information, and a robotic arm auxiliary module is combined to efficiently measure the point cloud data from multiple perspectives of the large workpiece. In addition, taking advantage of the fact that binocular vision can directly obtain point cloud information, the coordinate information of the point cloud is used to complete hand-eye calibration. Compared with traditional hand-eye calibration based on calibration plates, it directly and fully utilizes the advantages of point cloud data, avoids multiple coordinate system transformation processes, and the calibration process is more efficient and the accuracy is guaranteed.

Description

[0001] Technical Field

[0002] The present invention belongs to the technical field of color three-dimensional reconstruction. More specifically, it relates to a method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information for hand-eye calibration. Background Art

[0003] As a high-precision flight device, a spacecraft undertakes important exploration and development tasks of celestial bodies and space. Its operating environment is complex, and the forms of security threats it suffers are diverse. For the measurement work of a spacecraft, high-precision three-dimensional shape measurement data and efficient measurement time are required simultaneously. At the same time, the spacecraft has a large measurement size, which further poses higher requirements for three-dimensional measurement.

[0004] Theodolite station layout measurement is one of the commonly used precision detection methods in the space station at present. A theodolite is a non-contact measurement system based on the principle of spatial intersection. It has a large measurement range and high precision. However, the measurement model requires the theodolite to be in a strictly leveled state. And it requires a complex calibration process. Therefore, the measurement cycle is long and the measurement cost is relatively high. At the same time, it is easily affected by external electromagnetic waves and light, and has high requirements for storage conditions.

[0005] A binocular structured light system projects an encoded structured light pattern onto the surface of an object to be measured by a projector and uses a binocular camera for shooting. It uses phase information for image matching and finally performs three-dimensional reconstruction to reconstruct the morphological information of the object in the binocular common area. It is a high-precision three-dimensional imaging technology.

[0006] However, both the theodolite and the binocular structured light system can only obtain the three-dimensional geometric information of the target. And color information, as important feature information, is of great significance for both human eye perception and feature extraction in machine vision.

[0007] In addition, the complete measurement of large-size targets of spacecraft is another challenge. The binocular structured light system is limited by the imaging range and viewing angle of the camera, and it is impossible to photograph a large-size workpiece (large workpiece), such as a spacecraft to be measured, completely at one time. A general strategy is to take multiple-viewpoint photographs to obtain multiple pieces of point cloud data, and then splice and align the multiple pieces of point cloud data to make the point cloud data of the object to be measured complete.

[0008] Multi-viewpoint photographing based on a robotic arm auxiliary module is an efficient point cloud data acquisition and splicing solution. Using the prior coordinate information provided by the robotic arm hardware, it can quickly and robustly calculate the rotation and translation matrices between multiple pieces of point cloud taken at each viewpoint to complete point cloud splicing. A key link among them is the hand-eye calibration link between the robotic arm and the camera. This link calculates the coordinate transformation matrix H from the position of the end of the robotic arm to the camera. gc, which is the basis for robotic arm-assisted splicing. However, the differences between the traditional hand-eye calibration process and the binocular calibration process result in a large gc computation error. SUMMARY OF THE INVENTION

[0009] The object of the present invention is to overcome the deficiencies of the prior art and provide a method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information hand-eye calibration. The method uses a robotic arm-assisted binocular structured light system to quickly obtain high-precision three-dimensional morphology data of the object to be measured, combines a color camera and image fusion technology to endow the point cloud with real color texture feature information, and directly performs hand-eye calibration using the robotic arm assistance and combining point cloud information. By making full use of the advantages of point cloud data, it avoids multiple coordinate system transformation processes, and the calibration process is more efficient and the accuracy is guaranteed.

[0010] To achieve the above object of the invention, a method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information hand-eye calibration is characterized by including:

[0011] (1), Using a robotic arm-assisted high-resolution binocular grayscale camera to perform three-dimensional morphology reconstruction on the large workpiece, i.e., the object to be measured, to obtain colorless three-dimensional point clouds of the object to be measured from each perspective, recorded as where obj represents the object to be measured, i represents the i-th shot, I represents the total number of shots. At the same time, record the 6 state parameters of the position of the robotic arm end during each shot, denoted as where is the position coordinate of the robotic arm end position, is the rotation coordinate of the robotic arm end position;

[0012] Use an optical RGB camera to take a low-resolution color image at the position of the robotic arm end, denoted as

[0013] (2), Perform a region-based interpolation algorithm based on the target orientation on each of the obtained low-resolution color images to unify the resolution of the low-resolution color images and the resolution of the grayscale images obtained by the high-resolution binocular grayscale camera in the binocular vision system

[0014] (2.1), Use to represent the left camera grayscale image obtained by the binocular vision system, where (x, y) represents the coordinate position of the pixel in the x-th row and y-th column of the left camera grayscale image, M h and N h represent the image size of the left camera grayscale image, that is, the total number of pixels in the left camera grayscale image is M h ×N h , simply denoted as

[0015] The registered color image of the low-resolution color image is represented as where (u l ', v l ) represents the coordinate position of the pixel at the u l 'th row and v l 'th column in the color image, M l and N l represent the image size of the color image, that is, the total number of pixels in the color image is M l ×N l , simply denoted as

[0016] (2.2) Use the edge detection algorithm to obtain the boundary point set ED of the object to be measured in the color image , and then perform image dilation on the boundary point set ED to expand the boundary to obtain the boundary point set ED exp :

[0017] ED exp = imdilate(ED)

[0018] Use the dilated boundary point set ED exp to divide the color image into the target region Zone obj , the background region Zone back and the boundary region Zone edge ;

[0019] (2.3) Perform coordinate transformation and mapping on the blank high-resolution color image to achieve the same resolution as the high-resolution grayscale image of the left camera :

[0020] First, create a blank high-resolution color image with the same resolution as the left camera grayscale image for the pixel values of the pixels to be determined , simply denoted as

[0021] Then perform the backward transformation from the coordinates (u , v h ) of the pixels in the high-resolution color image h to the coordinates (u, v) of the pixels in the registered low-resolution color image :

[0022]

[0023] (2.4) According to the color image obtained by the backward transformation The coordinates (u, v) of the middle pixel points and the point sets divided in step (2.2), for the high-resolution color image the pixel value of each pixel point in is assigned, that is, according to the high-resolution color image the coordinates (u h , v h ) of the pixel point mapped backward, determine the coordinates (u of each pixel point in the high-resolution color image h , v h ) pixel value Specifically;

[0024] (a), If the coordinates {(u, v)} ∈ Zone back , then the pixel point is located in the background area, and the interpolation quality requirement for it is the lowest, so its pixel value is the pixel value of the nearest neighbor point in the four-neighborhood where the coordinates (u h , v h ) are located in the mapped color image ;

[0025] (b), If the coordinates {(u, v)} ∈ Zone obj , then the pixel is located in the target area, and its pixel value is:

[0026]

[0027] Among them, (u′ l , v′ l ), (u′ l + 1, v′ l ), (u′ l , v′ l + 1), (u′ l + 1, v′ l + 1) represent the coordinates of the four neighboring pixel points of the coordinates (u, v) in the color image , represent the pixel values of the four neighboring pixel points, w 11 , w 12 , w 21 , w 22 are weight coefficients, which are determined by their positions and the relative distances from their respective nearest boundary region Zone edge pixel points, that is:

[0028]

[0029]

[0030]

[0031]

[0032] in,( 11 u′ ed , 11 v′ ed )、( 21 u′ ed , 21 v′ ed )、( 12 u′ ed , 12 v′ ed )、( 22 u′ ed , 22 v′ ed ) are the distance coordinates (u′ l ,v′ l )、(u′ l +1,v′ l )、(u′ l ,v′ l +1)、(u′ l +1,v′ l +1) The nearest boundary zone edge The coordinates of the pixel point, PP means to find the distance between two coordinates;

[0033] (c) If the coordinates {(u,v)}∈Zone edge , then the pixel is located in the boundary area, the pixel value Color image obtained by backward transformation The coordinates of the pixel point (u, v) are determined using a two-dimensional cubic spline interpolation algorithm;

[0034] (2.5) Correct the pixel values in the boundary area:

[0035] For the mapped coordinates {(u,v)} in the boundary area Zone edge , but the pixel value does not belong to the boundary point set ED Use weighted average pixel value for correction: If the mapped coordinates {(u,v)} are in the background area Zone back On one side, the pixel value is obtained according to step (2.4) (a), and then the weighted average of the pixel value obtained according to step (2.4) (c) is corrected. If the mapped coordinates {(u, v)} are in the target area Zone objOn one side, the pixel value is obtained according to step (2.4)(b), and then it is corrected by weighted averaging with the pixel value obtained according to step (2.4)(c).

[0036] (3) Use the image fusion algorithm to fuse each high-resolution color image after unifying the resolution and its corresponding high-resolution left camera grayscale image to obtain a fused image with color and texture information

[0037] (4) For each fused image Based on the binocular vision principle, map the 3-channel color information value at each coordinate in the fused image to the corresponding spatial coordinate position in the colorless three-dimensional point cloud to obtain an RGB three-dimensional point cloud with color information to achieve point cloud colorization;

[0038] (5) Use a standard sphere to calibrate the transformation matrix H gc between the end position of the robotic arm and the binocular vision system

[0039] (5.1) Keep the relative position between the standard sphere and the base of the robotic arm unchanged, and use the robotic arm to assist the high-resolution binocular grayscale camera to perform multiple three-dimensional shape reconstructions of the standard sphere to obtain the three-dimensional point clouds of the standard sphere from each perspective, recorded as where n represents the nth shot, N represents the total number of shots, and at the same time record the 6 state parameters of the end position of the robotic arm during each shot, and record them as where, is the position coordinate of the end position of the robotic arm, is the rotation coordinate of the end position of the robotic arm;

[0040] (5.2) Fit a sphere to each three-dimensional point cloud of the standard sphere to obtain the three-dimensional coordinates of the center of the sphere of each three-dimensional point cloud

[0041] (5.3) Use the 6 state parameters of the end position of the robotic arm during each shot to calculate the transformation matrix between the base coordinate system of the robotic arm and the end coordinate system of the robotic arm during each shot:

[0042]

[0043] where, R n = R Z_n * R Y_n * R X_n ,

[0044] (5.4) Based on the three-dimensional coordinates of the sphere center and the corresponding transformation matrix establish the objective function:

[0045]

[0046] Use the optimization algorithm to optimize the above minimization problem and solve for the optimal transformation matrix H gc to complete the calibration of the transformation matrix;

[0047] (6) Use the transformation matrix H between the calibrated end position of the robotic arm and the binocular vision system gc to splice the RGB three-dimensional point cloud with color information for splicing

[0048] (6.1) Based on the 6 state parameters of the end position of the robotic arm recorded when photographing the object to be measured calculate the transformation matrix

[0049]

[0050] where R i = R Z_i * R Y_i * R X_i ,

[0051] (6.2) For each pixel point in the RGB three-dimensional point cloud with color information perform rotation and translation:

[0052]

[0053] where is the m-th point in the i-th RGB three-dimensional point cloud and M is the number of points in the i-th RGB three-dimensional point cloud , is the colorized point cloud after rotation and translation, that is, the spliced color point cloud;

[0054] (7) Optimize the overlapping areas in the spliced color point cloud to obtain the final complete color three-dimensional reconstruction result of the object to be measured;

[0055] (7.1) For multiple RGB three-dimensional point clouds after splicing first regard them as an overall object regard the overall object PC allEach axis of the three-dimensional space where it is located is used to divide into eight sub-spaces. The octree structure is used to continuously divide the sub-spaces, and the smallest sub-space obtained is used as a voxel. The overlapping area between multiple three-dimensional point clouds is searched and located. A voxel with too many dense points, that is, more than one point, in a single voxel is defined as a voxel to be fused. Among them, the smallest sub-space is determined according to the resolution of the grayscale image and contains at least one point;

[0056] (7.2) For the l-th point existing in the voxel to be fused Voxel(k) represents the k-th voxel to be fused. According to its serial number, trace back to the corresponding RGB three-dimensional point cloud where it is located Taking the point as the center, find P nearest neighbor points, denoted as

[0057] Calculate the 3×3 covariance matrix

[0058]

[0059]

[0060] where x, y, z are the set the three-dimensional coordinate values corresponding to a point;

[0061] Taking the point and the P nearest neighbor points The covariance matrix of is used as the input of local PCA, and the smallest eigenvalue among the three non-negative eigenvalues of the covariance matrix is obtained and used as the significance value of the point in point cloud fusion;

[0062] (7.3) According to the calculated significance value, perform weighted averaging on all points in the voxel Voxel(k), and at the same time act on both color fusion and position fusion to obtain the fused point in the k-th voxel to be fused finally

[0063]

[0064] where L k is the total number of points contained in the k-th voxel Voxel(k), is the significance value corresponding to the l-th point in the k-th voxel Voxel(k). Delete all points in the voxel Voxel(k) and replace them with the fused point to complete point cloud fusion.

[0065] The object of the present invention is achieved as follows:

[0066] A method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information for hand-eye calibration uses a structured light binocular vision system combined with image fusion to achieve rapid acquisition of high-precision point cloud data while ensuring the accuracy of three-dimensional reconstruction, taking advantage of the flexible configuration and high measurement efficiency of the binocular structured light system. On this basis, a color camera is combined to endow the point cloud with real color texture feature information, and a robotic arm auxiliary module is combined to efficiently measure the point cloud data of the large workpiece from multiple perspectives. In addition, taking advantage of the fact that binocular vision can directly obtain point cloud information, the coordinate information of the point cloud is used to complete hand-eye calibration. Compared with traditional hand-eye calibration based on a calibration board, it directly and fully utilizes the advantages of point cloud data, avoids multiple coordinate transformation processes, and the calibration process is more efficient and the accuracy is guaranteed. Specifically, the present invention uses a robotic arm to assist the binocular structured light system to capture point clouds, obtaining multi-perspective point cloud data without color information. The resolution of the color camera image and the binocular left camera image is unified by using the method of regional interpolation. Thus, super-resolution is performed on the color camera image in a targeted manner. After the fusion of the color image and the binocular image, and the operation of coloring the point cloud, the original multi-perspective point cloud without color information is colored to obtain a colored multi-perspective point cloud. Then, a robotic arm is used to assist the binocular vision system to photograph a standard sphere, and the spherical point cloud data from multiple perspectives is obtained for hand-eye calibration of the robotic arm. An optimization equation is constructed using the center-of-sphere data of multiple spheres in the fitted multi-piece point clouds and the prior pose information of the robotic arm. The optimization algorithm is used to solve the equation directly to obtain the hand-eye calibration matrix H gc . The accuracy of hand-eye calibration is ensured by using binocular high-precision point cloud information. Finally, based on the end-effector pose information of the robotic arm at each perspective when photographing the target to be measured and the calculated hand-eye calibration matrix H gc , the colored multi-perspective point clouds are rotated and translated to complete the preliminary color point cloud stitching of the target to be measured. The overlapping area of the stitched point clouds is further optimized. After searching for the overlapping area of the point clouds through the octree structure, the local PCA information of each point in its corresponding original point cloud is used as the significance value. The significance of each point to be fused is used for weighted fusion, and finally the fusion of the position information and color information of the point cloud is completed.

[0067] Meanwhile, the method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information for hand-eye calibration of the present invention also has the following beneficial effects:

[0068] 1. The present invention takes advantage of the flexible configuration, high measurement efficiency, and high point cloud accuracy of the binocular structured light system to achieve rapid acquisition of high-precision point cloud data. At the same time, it combines the color texture information of the object collected by the color camera, and based on regional interpolation and image fusion technologies, real color texture information is given to the colorless point cloud data;

[0069] 2. Combine the robotic arm assistance module to achieve multi-view colorized point cloud stitching of large-sized measured objects. By leveraging the advantage of directly obtaining high-precision point cloud coordinate information from the binocular vision system, an optimization function is established to directly solve the hand-eye calibration matrix, which is more efficient than traditional hand-eye calibration and ensures accuracy.

[0070] 3. Address the problem of chaotic point clouds in the overlapping regions during the multi-view color point cloud stitching process of large workpieces. Use the point cloud fusion technology based on the local PCA algorithm to fuse the point cloud coordinates and color information in the overlapping regions, making the fused point cloud distribution more uniform and the color more harmonious. Description of the Drawings

[0071] Figure 1 is a flowchart of a specific implementation of the method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information for hand-eye calibration;

[0072] Figure 2 is a flowchart for unifying the resolutions of color images and binocular grayscale images based on the target-oriented sub-region interpolation algorithm;

[0073] Figure 3 is to calculate the transformation matrix H between the end position of the robotic arm and the binocular vision system based on point cloud information for hand-eye calibration gc algorithm flowchart;

[0074] Figure 4 is a result diagram of the original colorless point cloud data of the measured object from perspective 1 collected with the assistance of the robotic arm;

[0075] Figure 5 is a result diagram of the original colorless point cloud data of the measured object from perspective 2 collected with the assistance of the robotic arm;

[0076] Figure 6 is a result diagram of realizing point cloud colorization by mapping the color information to the original colorless point cloud of perspective 1 after fusing the color image and the binocular image with unified resolutions;

[0077] Figure 7 is a result diagram of realizing point cloud colorization by mapping the color information to the original colorless point cloud of perspective 2 after fusing the color image and the binocular image with unified resolutions;

[0078] Figure 8 is a scatter diagram of the optimization process and the decline process of the objective function value after establishing the optimization problem based on point cloud information for hand-eye calibration;

[0079] Figure 9 is based on the calibrated matrix H gc result diagram of point cloud stitching;

[0080] Figure 10 It is the result graph after fusing and optimizing the spliced point cloud. Detailed implementation manners

[0081] The following describes the detailed implementation manners of the present invention in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be particularly noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0082] Figure 1 It is a flowchart of a specific implementation manner of a method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information and hand-eye calibration of the present invention.

[0083] In this embodiment, as Figure 1 shown, the method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information and hand-eye calibration of the present invention includes the following steps:

[0084] Step S1: Use a robotic arm-assisted high-resolution binocular gray camera to perform three-dimensional shape reconstruction on the large workpiece, i.e., the object to be measured, to obtain colorless three-dimensional point clouds of the object to be measured from each perspective, recorded as where obj represents the object to be measured, i represents the i-th shot, I represents the total number of shots. At the same time, record 6 state parameters of the position of the robotic arm end during each shot, denoted as where is the position coordinate of the robotic arm end position, is the rotation coordinate of the robotic arm end position.

[0085] Use an optical RGB camera to take a low-resolution color image at the position of the robotic arm end, denoted as

[0086] Step S2: To ensure the effectiveness of color fusion, perform a target-oriented sub-region interpolation algorithm on each of the obtained low-resolution color images so as to unify the resolution of the low-resolution color images and the resolution of the gray images obtained by the high-resolution binocular gray camera in the binocular vision system.

[0087] The flowchart for unifying the resolution of the color image and the binocular gray image based on the target-oriented sub-region interpolation algorithm is as Figure 2 shown and includes the following steps:

[0088] Step S2.1: Use to represent the left camera gray image obtained by the binocular vision system, where (x, y) represents the coordinate position of the pixel in the x-th row and y-th column of the left camera gray image, M h and N hIndicates the image size of the left camera grayscale image, that is, the total number of pixels in the left camera grayscale image is M h ×N h , simply expressed as

[0089] The low-resolution color image After registration, the color image is expressed as where (u l ', v l ) represents the coordinate position of the pixel at the u l 'th row and v l 'th column in the color image, M l and N l Indicates the image size of the color image, that is, the total number of pixels in the color image is M l ×N l , simply expressed as

[0090] Step S2.2: Use the edge detection algorithm to obtain the boundary point set ED of the object to be measured in the color image , and then perform image dilation on the boundary point set ED to expand the boundary to obtain the boundary point set ED exp :

[0091] ED exp = imdilate(ED)

[0092] Use the dilated boundary point set ED exp To divide the color image Into the target area Zone obj , background area Zone back And boundary area Zone edge .

[0093] Step S2.3: Perform coordinate transformation and mapping on the blank high-resolution color image To achieve resolution unity with the high-resolution left camera grayscale image .

[0094] First, create a blank high-resolution color image with the same resolution as the left camera grayscale image To determine the pixel values of the pixels to be determined Simply expressed as

[0095] Then perform the backward transformation of the coordinates (u , v h ) of the pixels in the high-resolution color image h ) to the coordinates (u, v) of the pixels in the low-resolution registered color image :

[0096]

[0097] Step S2.4: Interpolate by region according to the mapping coordinates

[0098] The color image obtained according to the backward transformation For the coordinates (u, v) of the pixel points in it and the point set divided in Step S2.2, for the high-resolution color image Assign values to the pixel values of each pixel point That is, according to the high-resolution color image The coordinates (u h , v h ) of the pixel points after backward mapping, determine the high-resolution color image The coordinates (u h , v h ) of each pixel point Specifically:

[0099] (a), If the coordinates {(u, v)} ∈ Zone back , then this pixel point is located in the background area, and the interpolation quality requirement for it is the lowest, so its pixel value Is the pixel value of the nearest neighbor point in the four-neighborhood where the coordinates (u h , v h ) are located in the mapped color image ;

[0100] (b), If the coordinates {(u, v)} ∈ Zone obj , then this pixel is located in the target area, and its pixel value Is:

[0101]

[0102] Among them, (u′ l , v′ l ), (u′ l + 1, v′ l ), (u′ l , v′ l + 1), (u′ l + 1, v′ l ) represent the coordinates of the four neighboring pixel points of the coordinates (u, v) in the color image , Indicates the pixel values of the four neighboring pixel points, w 11 , w 12 , w 21 , w 22are weight coefficients, which are determined by their positions and the relative distances to the nearest boundary region Zone edge of the pixel points, that is:

[0103]

[0104]

[0105]

[0106]

[0107] wherein, ( 11 u′ ed , 11 v′ ed ), ( 21 u′ ed , 21 v′ ed ), ( 12 u′ ed , 12 v′ ed ), ( 22 u′ ed , 22 v′ ed ) are the coordinates of the nearest boundary region Zone l of the pixel points at the positions (u′ l , v′ l ), (u′ l + 1, v′ l ), (u′ l , v′ l + 1), (u′ l + 1, v′ edge + 1) respectively, and PP represents calculating the distance between two coordinates;

[0108] (c) If the coordinate {(u, v)} ∈ Zone edge , then the pixel is located in the boundary region, and the pixel value is determined by using the two-dimensional cubic spline interpolation algorithm according to the coordinates (u, v) of the pixel points in the color image obtained by the backward transformation;

[0109] Step S2.5: Correct the pixel values in the boundary region:

[0110] For the pixel values of the mapped coordinates {(u, v)} in the boundary region Zone edge , but not belonging to the boundary point set ED, correct them by using the weighted average pixel value: If the mapped coordinates {(u, v)} are in the background region Zoneback On one side, the pixel value is obtained according to step S2.4(a), and then it is corrected by weighted averaging with the pixel value obtained according to step S2.4(c). If the mapped coordinates {(u, v)} are in the target area Zone obj On one side, the pixel value is obtained according to step S2.4(b), and then it is corrected by weighted averaging with the pixel value obtained according to step S2.4(c);

[0111] Step S3: Use the image fusion algorithm to fuse each high-resolution color image after unifying the resolution and its corresponding high-resolution left camera grayscale image to perform image fusion and obtain a fused image with color and texture information Thereby endowing the binocular camera grayscale image with color and texture information.

[0112] Step S4: For each fused image Based on the binocular vision principle, map the 3-channel color information value at each coordinate in the fused image to the corresponding spatial coordinate position in the colorless three-dimensional point cloud to obtain an RGB three-dimensional point cloud with color information Realize point cloud colorization.

[0113] Step S5: Use a standard sphere to calibrate the transformation matrix H gc between the end position of the robotic arm and the binocular vision system. Calculate the transformation matrix H gc between the end position of the robotic arm and the binocular vision system based on the point cloud information. The algorithm flowchart is as Figure 3 shown and specifically includes the following steps:

[0114] Step S5.1: Keep the relative position between the standard sphere and the robotic arm base unchanged, and use the robotic arm to assist the high-resolution binocular grayscale camera to perform multiple three-dimensional shape reconstructions of the standard sphere to obtain the three-dimensional point clouds of the standard sphere from various perspectives, recorded as where n represents the nth shot, N represents the total number of shots, and at the same time record the 6 state parameters of the end position of the robotic arm during each shot and denote them as where, is the position coordinate of the end position of the robotic arm, is the rotation coordinate of the end position of the robotic arm.

[0115] Step S5.2: Fit a sphere to each three-dimensional point cloud of the standard sphere to obtain the three-dimensional coordinates of the center of the sphere of each three-dimensional point cloud

[0116] Step S5.3: Calculate the transformation matrix between the base coordinate system of the robotic arm and the end - effector coordinate system at each shooting using the six state parameters of the end - effector position at each shooting:

[0117]

[0118] where, R n = R Z_n * R Y_n * R X_n ,

[0119] Step S5.4: Based on the three - dimensional coordinates of the sphere center and the corresponding transformation matrix establish the objective function:

[0120]

[0121] Use an optimization algorithm to optimize the above minimization problem and solve for the optimal transformation matrix H gc , completing the calibration of the transformation matrix.

[0122] Step S6: Use the calibrated transformation matrix H gc between the end - effector position of the robotic arm and the binocular vision system to splice the RGB three - dimensional point cloud with color information;

[0123] Step S6.1: Calculate the transformation matrix based on the six state parameters of the end - effector position of the robotic arm recorded when shooting the object to be measured

[0124]

[0125] where, R = R Z * R Y * R X ,

[0126] Step S6.2: Rotate and translate each point in the colored point cloud :

[0127]

[0128] where, m = 1, …, M is the m - th point in this point cloud. is the colored point cloud after rotation and translation;

[0129] Step S7: Perform fusion optimization on the overlapping regions in the spliced color point cloud to obtain the final complete color three-dimensional reconstruction result of the object under test;

[0130] Step S7.1: For multiple RGB three-dimensional point clouds after splicing First, regard them as a whole object For the overall object PC all Divide each axis of the three-dimensional space where it is located into eight subspaces, continuously divide the subspaces using the octree structure, and take the smallest subspace as a voxel. Search and locate the overlapping regions among multiple three-dimensional point clouds. Define the voxel with too many dense points, that is, more than one point, in a single voxel as the voxel to be fused. Among them, the smallest subspace is determined according to the resolution of the grayscale image and contains at least one point;

[0131] Step S7.2: For the l-th point existing in the voxel to be fused Voxel(k) represents the k-th voxel to be fused. Trace back to the corresponding RGB three-dimensional point cloud according to its serial number Taking the point as the center, find P nearest neighbor points, denoted as

[0132] Calculate the 3×3 covariance matrix

[0133]

[0134]

[0135] where x, y, z are the three-dimensional coordinate values corresponding to a point;

[0136] Taking the point and the P nearest neighbor points as the input of local PCA, obtain the smallest eigenvalue among the three non-negative eigenvalues of the covariance matrix and use it as the significance value of the point in point cloud fusion;

[0137] Step S7.3: Perform weighted averaging on all the points in the voxel Voxel(k) according to the calculated significance value, and simultaneously apply it to both color fusion and position fusion to obtain the fused point in the final k-th voxel to be fused

[0138]

[0139] where, L k is the total number of points contained in the k-th voxel Voxel(k), is the significance value corresponding to the l-th point in the k-th voxel Voxel(k). Delete all points in the voxel Voxel(k) and replace them with the fused point to complete the point cloud fusion.

[0140] Example

[0141] In this embodiment, the object to be measured is an aircraft model. The manipulator is used to assist in photographing the point cloud data of the object to be measured from two perspectives, namely perspective 1 and perspective 2, and the corresponding color images 1 and 2 and binocular grayscale images 1 and 2 are obtained.

[0142] Based on the binocular structured light system, the point cloud data corresponding to the respective perspectives is collected, namely the perspective 1 point cloud and the perspective 2 point cloud, as Figure 4 and Figure 5 shown.

[0143] In this example, first, the resolution of the two captured color images of the object to be measured and their corresponding two binocular grayscale images in perspective 1 and perspective 2 are unified by using target-oriented sub-region interpolation. Then, the unified color image 1 and color image 2 are respectively fused with the corresponding binocular grayscale images 1 and 2 to obtain the fused images 1 and 2. After that, the color information in the fused images 1 and 2 is mapped into the perspective 1 point cloud and the perspective 2 point cloud. Finally, the colorized perspective 1 point cloud and the colorized perspective 2 point cloud are obtained. The results are as Figure 6 and Figure 7 shown.

[0144] The manipulator is used to assist in photographing the point cloud data of the standard sphere. Using the end-effector pose information during the photographing process and the center coordinate values of the sphere fitted by multiple standard spherical point clouds, an optimization objective function is established. And the transformation matrix H between the end position of the manipulator and the binocular vision system is calculated using an optimization algorithm gc . The optimization process and the iterative descent process of the objective function are as Figure 8 shown.

[0145] Based on the calibrated transformation matrix H gc the colorized perspective 1 point cloud and the colorized perspective 2 point cloud obtained previously are subjected to point cloud rotation and stitching. The preliminary stitched point cloud is obtained as Figure 9 shown.

[0146] The overlapping region in the preliminarily stitched point cloud is fused and optimized based on local PCA to obtain the final complete multi-perspective colored point cloud as Figure 10 shown. From Figure 9 and Figure 10 it can be seen that the incorrect color and incorrect point cloud information in the overlapping region of the fused and optimized colored point cloud are optimized, and the quality of the fused point cloud is improved.

[0147] Although the above description of the illustrative embodiments of the present invention is provided to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

Claims

1. A method for color three-dimensional reconstruction of large workpieces with complex shapes based on point cloud information for hand-eye calibration, characterized in that, Including: (1) Use a robotic arm to assist a high-resolution binocular grayscale camera to perform 3D shape reconstruction on a large workpiece, i.e., the object to be measured, and obtain colorless 3D point clouds of the object to be measured from each perspective, recorded as where obj represents the object to be measured, i represents the i-th shot, I represents the total number of shots. At the same time, record the 6 state parameters of the position of the end of the robotic arm during each shot, denoted as where is the position coordinate of the end of the robotic arm, is the rotation coordinate of the end of the robotic arm; Use an optical RGB camera to capture a low-resolution color image at the end position of the robotic arm, denoted as (2) For each obtained low-resolution color image Execute a target-oriented sub-region interpolation algorithm to unify the resolution of the low-resolution color image and the resolution of the grayscale image obtained by the high-resolution binocular grayscale camera in the binocular vision system (2.1), use to represent the grayscale image of the left camera obtained by the binocular vision system, where (x, y) represents the coordinate position of the pixel in the x-th row and y-th column of the grayscale image of the left camera, M h and N h represent the image size of the grayscale image of the left camera, that is, the total number of pixels in the grayscale image of the left camera is M h ×N h , simply expressed as Low-resolution color images The color image after registration is represented as Where (u l ',v l ') represents the uth l 'Line, v l 'The coordinate position of the column pixel, M l and N l Indicates the image size of the color image, that is, the total pixel value of the color image is M l ×N l , which can be simply expressed as (2.2) Use edge detection algorithm to obtain color image The boundary point set ED of the object to be measured is then expanded to obtain the boundary point set ED. exp : ED exp =imdilate(ED) Utilize the expanded boundary point set ED exp Partition the color image into regions of the target region Zone obj , the background region Zone back and the boundary region Zone edge ; (2.3) for blank high-resolution color images Perform coordinate transformation and mapping to achieve high-resolution left camera grayscale image The resolution is unified: First create a grayscale image of the left camera A blank high-resolution color image with the same resolution as the pixel values to be determined Simply expressed as Then the high-resolution color image The coordinates of the pixel point (u h ,v h ) to the low-resolution registered color image Backward transformation of the coordinates (u, v) of the pixel point in: (2.4) The color image obtained according to the backward transformation For the coordinate (u, v) of the pixel point in and the point set divided in step (2.2), for each pixel value in the high-resolution color image perform assignment, that is, according to the coordinate (u , v h ) of the pixel point mapped backward in the high-resolution color image, determine the coordinate (u h , v ) of each pixel point in the high-resolution color image h , v h ) pixel value Specifically; (a), if the coordinates \(\{(u, v)\}\in Zone back , then the pixel is located in the background area, and the lowest requirement for its interpolation quality is imposed. Its pixel value is the pixel value of the nearest neighbor point in the four-neighborhood where the coordinates \((u h , v h ) are located in the mapped color image ; (b) If the coordinates {(u,v)}∈Zone obj , then the pixel is located in the target area and its pixel value for: Among them, (u l ′, v l ′), (u l ′ + 1, v l ′), (u l ′, v l ′ + 1), (u l ′ + 1, v l ′ + 1) represent the coordinates of four neighboring pixel points of the coordinate (u, v) in the color image . indicates the pixel values of the four neighboring pixel points, w 11 , w 12 , w 21 , w 22 are weight coefficients, which are determined by their positions and the relative distances from their respective nearest boundary regions Zone edge pixel points, that is: in,( 11 u e ' d,11 v e ' d )、( 21 u e ' d,21 v e ' d )、( 12 u e ' d,12 v e ' d )、( 22 u e ' d,22 v e ' d ) are the distance coordinates (u l ′,v′l)、(u l ′+1,v l ′)、(u l ′,v l ′+1)、(u l ′+1,v l '+1) The nearest boundary zone edge The coordinates of the pixel point, PP means to find the distance between two coordinates; (c), if the coordinates \(\{(u, v)\}\in Zone edge , then the pixel is located in the boundary region, and the pixel value of the pixel point \((u, v)\) in the color image obtained according to the backward transformation is determined by using a two-dimensional cubic spline interpolation algorithm; (2.5) Correcting the pixel values of the boundary region: For the mapped coordinates {(u,v)} in the boundary area Zone edge , but the pixel value does not belong to the boundary point set ED Use weighted average pixel value for correction: If the mapped coordinates {(u,v)} are in the background area Zone back On one side, the pixel value is obtained according to step (2.4) (a), and then the weighted average of the pixel value obtained according to step (2.4) (c) is corrected. If the mapped coordinates {(u, v)} are in the target area Zone obj On one side, the pixel value is obtained according to step (2.4)(b), and then the weighted average of the pixel value obtained according to step (2.4)(c) is used for correction; (3) Use an image fusion algorithm to fuse each high-resolution color image after unified resolution and its corresponding high-resolution left camera grayscale image to perform image fusion and obtain a fused image F with color and texture information i obj ; (4) For each fused image F i obj , based on the binocular vision principle, map the 3-channel color information values at each coordinate in the fused image F i obj to the corresponding spatial coordinate positions in the colorless three-dimensional point cloud to obtain an RGB three-dimensional point cloud with color information to achieve point cloud colorization; (5) Calibrate the transformation matrix H between the position of the end of the robotic arm and the binocular vision system using a standard ball gc for calibration (5.1) Keep the relative position between the standard ball and the base of the robotic arm unchanged. Use the robotic arm to assist the high-resolution binocular grayscale camera to perform multiple three-dimensional shape reconstructions of the standard ball, obtain the three-dimensional point clouds of the standard ball from various perspectives, and record them as where n represents the nth shot, N represents the total number of shots. At the same time, record the 6 state parameters of the position of the robotic arm end during each shot, and denote them as Among them, is the position coordinate of the robotic arm end position, is the rotation coordinate of the robotic arm end position; (5.2), for each three-dimensional point cloud of the standard sphere Perform sphere fitting to obtain the 3D coordinates of the sphere center of each 3D point cloud (5.3) Calculating the transformation matrix between the base coordinate system of the robotic arm and the end coordinate system of the robotic arm during each shot using the 6 state parameters of the end position of the robotic arm during each shot: wherein, R n = R Z_n * R Y_n * R X_n , (5.4), Based on the three-dimensional coordinates of the center of the sphere and the corresponding transformation matrix to establish the objective function: Use the optimization algorithm to optimize the above minimization problem and solve the optimal transformation matrix H gc , complete the conversion matrix calibration; (6) Using the conversion matrix H between the calibrated end position of the manipulator and the binocular vision system gc RGB 3D point cloud with color information Splicing (6.1) Based on the six state parameters of the end position of the robotic arm recorded when photographing the object being measured Calculate the transformation matrix wherein, R i = R Z_i * R Y_i * R X_i , (6.2) RGB three-dimensional point cloud with color information Each pixel in is rotated and translated: Among them, is the m-th point in the i-th RGB three-dimensional point cloud , where M is the number of points in the i-th RGB three-dimensional point cloud , is the colorized point cloud after rotation and translation, that is, the spliced color point cloud; (7) Fusing and optimizing the overlapping regions in the stitched color point cloud to obtain the final complete color three-dimensional reconstruction result of the object under test; (7.1) For multiple RGB three-dimensional point clouds after splicing First, take them as a whole object Take the whole object PC all Use each axis of the three-dimensional space where it is located as a boundary to divide the subspace, obtaining eight subspaces. Continuously divide the subspace using the octree structure, and take the smallest subspace as a voxel. Search and locate the overlapping area between multiple three-dimensional point clouds. Define the voxel with too many dense points (i.e., more than one point) within a single voxel as a voxel to be fused. Among them, the smallest subspace is determined according to the resolution of the grayscale image and contains at least one point; (7.2) For the l-th point existing in the voxel to be fused Voxel(k) represents the k-th voxel to be fused, and according to its serial number, trace it back to the corresponding RGB three-dimensional point cloud Taking the point as the center, find P nearest neighbor points, denoted as Calculate the 3×3 covariance matrix where x, y, and z are sets the three-dimensional coordinate values corresponding to a point Take the point and the covariance matrix of the P nearest neighbor points as the input of local PCA, and obtain the smallest eigenvalue among the three non-negative eigenvalues of the covariance matrix and use it as the significance value of the point in point cloud fusion; (7.3) Weight the average of all points in voxel Voxel(k) according to the calculated significance value, and apply it to both color fusion and position fusion to obtain the fused point in the final k-th voxel to be fused. Among them, L k is the total number of points contained in the k-th voxel Voxel(k), is the significance value corresponding to the l-th point in the k-th voxel Voxel(k). Delete all points in the voxel Voxel(k) and replace them with the fusion point to complete the point cloud fusion.

Images