A method and device for three-dimensional reconstruction of workpieces for robot spraying production line

By constructing entity feature templates and calibrating laser cameras, stitching and segmenting point clouds, extracting workpiece features, and reconstructing the 3D model of the workpiece, the problem of missing point clouds caused by occlusion and reflection is solved, improving the modeling efficiency and model accuracy of the spraying process.

CN117974901BActive Publication Date: 2024-11-29BAOYING FANYANG SHIPPING ELECTRIC APPLIANCE FITTINGS MFG CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410173600.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-07
Publication Date
2024-11-29
Estimated Expiration
2044-02-07

AI Technical Summary

Technical Problem

During the robotic painting process, complex workpieces may suffer from point cloud defects due to issues such as occlusion and reflection, leading to reduced model reconstruction accuracy or even the inability to complete model reconstruction.

Method used

A solid feature-based modeling method is adopted. This method involves constructing a solid feature template, calibrating laser camera parameters, stitching together point clouds, removing noise, extracting workpiece feature skeleton lines and cross-sectional parameters, and reconstructing the workpiece's 3D model using Boolean operations.

Benefits of technology

It enables 3D reconstruction of workpieces even with missing point clouds, improves modeling efficiency, reduces computational costs for trajectory planning, and directly provides model parameters for path design.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117974901B_ABST
    Figure CN117974901B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for three-dimensional reconstruction of workpieces for a robot spraying production line, wherein the method comprises: S1, establishing a physical feature template of a workpiece to be sprayed based on a physical feature modeling method; S2, using a 3D line laser camera to collect images of the spraying production line to obtain a scene point cloud containing the workpiece; S3, removing background and noise from the scene point cloud, and segmenting the hook point cloud to obtain a workpiece point cloud; S4, extracting boundaries and performing straight line fitting on the workpiece point cloud, extracting skeleton lines of physical features and calculating cross-sectional parameters according to the spatial position information of the fitting straight line in the camera coordinate system; S5, using Open CASCADE to construct a three-dimensional model of the workpiece according to the extracted skeleton lines and cross-sectional parameters. The present invention can complete online three-dimensional reconstruction based on a single-viewpoint workpiece point cloud, and solve the problem that the model cannot be generated due to missing point clouds such as workpiece reflection or structural occlusion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an online three-dimensional reconstruction method and device for a workpiece for a robot spraying production line. Background Art

[0002] As a common surface processing technology, spraying is widely used in the fields of automobiles, ships, and aerospace. With the rapid development of automation technology, robotic spraying has shown advantages such as good coating thickness uniformity, high repeatability, wide applicability, and high efficiency. At the same time, it liberates workers from toxic, flammable, and explosive working environments. In the process of robotic automatic spraying, obtaining a three-dimensional model of the workpiece is an important prerequisite, which is a standard guide for spraying trajectory planning and posture estimation. For a variety of small batches of sprayed workpieces, three-dimensional models are rarely provided. Even if models are provided, the posture of the workpiece changes randomly on the spraying conveyor line, and real-time modeling is required to solve the deviation between the model and the real workpiece. Among them, point cloud-based model reconstruction has significant advantages in data accuracy, adaptability, and real-time performance, and is widely used in robotic spraying. However, for workpieces with complex structures, their structures will block each other, resulting in a large number of missing point clouds. Ambient light, workpiece reflection, and other problems will lead to a decrease in point cloud quality, making the point cloud of the sprayed surface incomplete, which in turn leads to a decrease in modeling accuracy or even inability to complete model reconstruction.

[0003] Therefore, a new technical solution is needed to solve the above problems. Summary of the invention

[0004] Purpose of the invention: In order to solve the above problems, the purpose of the present invention is to provide a three-dimensional reconstruction method for workpieces for a robotic spraying production line, aiming to solve the problem that the model cannot be generated due to the missing point cloud of the workpiece model in the robotic spraying production line.

[0005] Based on the same purpose, the present invention also provides a workpiece three-dimensional reconstruction device for a robot spraying production line.

[0006] Technical solution: The workpiece 3D reconstruction method for a robot spraying production line disclosed in the present invention can adopt the following technical solution:

[0007] A method for three-dimensional reconstruction of a workpiece for a robot spraying production line comprises the following steps:

[0008] S1. Constructing a solid feature template of the workpiece to be sprayed based on a solid feature modeling method, defining the skeleton lines and cross-sectional shapes required for constructing the solid feature template, and defining the spatial position sequence and connection relationship of each skeleton line;

[0009] S2. Calibrate the internal and external parameters and distortion coefficients of the laser camera to obtain the laser camera calibration result, calibrate the laser plane of image acquisition to obtain the laser plane calibration result, and obtain the geometric relationship between the laser projection plane and the camera imaging plane; use the grayscale centroid method to extract the center line of the laser stripe to obtain the two-dimensional pixel coordinates of the points on the center line; substitute the laser camera calibration result and the laser plane calibration result to calculate the corresponding spatial coordinates of the points on the center line of the laser stripe in the camera coordinate system; calibrate the direction vector of the movement direction of the spray conveyor line, and transform the position and posture of each center line point cloud through the displacement of the spray conveyor line movement within the laser stripe image acquisition interval, and obtain the scene point cloud containing the workpiece by stitching the line point cloud;

[0010] S3, removing the background and noise of the scene point cloud containing the workpiece, and segmenting the hook point cloud to obtain the workpiece point cloud;

[0011] S4, extracting the boundary and straight line fitting of the workpiece point cloud, projecting the fitting straight line to the camera coordinate system, obtaining the intersection of the fitting straight line and the coordinate axis, extracting the fitting straight line corresponding to the skeleton line of the entity feature according to the spatial position sequence and connection relationship of the skeleton line in step S1, and then extending the extracted fitting straight line with the adjacent fitting straight line to obtain the intersection point, and completing the skeleton line sequence point extraction; projecting the skeleton line sequence points of the entity feature template and the adjacent parallel fitting straight lines to the camera coordinate system, and obtaining the cross-sectional parameters of the entity feature by calculating the distance between the projection points of the coordinate axis;

[0012] S5. Reconstruct corresponding entity features according to the skeleton lines and cross-section parameters extracted in step S4, and fuse multiple entity features through Boolean operations to obtain a three-dimensional model of the workpiece.

[0013] Further, in step S2,

[0014] The correspondence between the image pixel points and the actual scene points calibrated using the internal and external parameters of the camera is formula 1:

[0015]

[0016] Where: u, v are the coordinates of the point on the image plane coordinate system; f x 、f y is the camera scale factor; u0 and v0 represent the camera principal point coordinates; (x w ,y w ,z w , 1) is the secondary coordinate value of the calibration plate coordinate system; M in 、M out are the camera internal and external parameter matrices respectively; R is the rotation component of the camera external parameter matrix; t is the translation component of the camera external parameter matrix;

[0017] Among them, the distortion model of the 3D line laser camera is formula 2:

[0018]

[0019] Where: (u,v) is the ideal pixel coordinate; (u p ,v p ) is the pixel coordinate after distortion; k1 and k2 are radial distortion parameters; k3 and k4 are tangential distortion parameters;

[0020] Calibrate the laser plane for image acquisition to obtain the geometric relationship between the laser projection plane and the camera imaging plane;

[0021] The equation of the laser plane in the camera coordinate system is formula 3:

[0022] A(x-x0)+B(y-y0)+C(z-z0)=1;

[0023] Substitute the calibration results of the camera and the light plane, and use Formula 1 and Formula 3 to calculate the corresponding spatial coordinates of the point on the center line of the laser stripe in the camera coordinate system:

[0024]

[0025] The direction vector of the spray conveyor line is calibrated, and the direction vector of the conveyor chain is calibrated as follows:

[0026]

[0027] Where: (x start ,y start ,z start ) is the coordinate of a corner point i in the calibration plate coordinate system when calibration starts; (x end ,y end ,z end ) is the coordinate of the same corner point after the calibration plate moves along the conveyor chain for a certain distance; cosθ xi 、cosθ yi 、cosθ zi is the projection of the i-th group of direction vectors in the camera coordinate system;

[0028] Select m groups of corner points for optimization, and the optimization objective function and optimization constraints satisfy the formula:

[0029]

[0030] Where: cosθ x 、cosθ y 、cosθ z To optimize the projection of the posterior direction vector in the camera coordinate system;

[0031] Using the calibration result of the conveyor chain motion direction vector, the pose of each line point cloud is transformed according to the following formula, and the line point clouds after pose transformation are spliced ​​to obtain the scene point cloud:

[0032]

[0033] Furthermore, in step S3, the RANSAC algorithm is used to fit the plane where the background point cloud is located to obtain the normal vector n of the plane where the background point cloud is located b ; Along n b Direction L b The distance exchange background point cloud plane's centroid is used to obtain the long-distance plane, whose point normal plane equation is:

[0034] A(x-x0)+B(y-y0)+C(z-z0)=0

[0035] Where (A, B, C) is the normal vector n b , (x0, y0, z0) is the center of mass after exchange;

[0036] The direction vector D of the upper boundary of the workpiece point cloud m The normal vector to the upper boundary of the workpiece point cloud is n m Cross product to obtain the normal vector n of the upper surface m :

[0037] n u =D m ×n m

[0038] The points on the upper boundary of the workpiece and the normal vector are used to construct the French hook splitting surface equation:

[0039] A′(x-x0′)+B′(y-y0′)+C′(z-z0′)=0

[0040] Where (A′, B′, C′) is the normal vector n of the upper surface u , (x0′, y0′, z0′) is any point on the upper boundary of the workpiece. Further, in step S4, the PCA algorithm is used to obtain the main plane of the workpiece point cloud, and the workpiece point cloud is projected onto the main plane to obtain the projected point cloud to enhance the local point cloud density; the Alpha Shapes algorithm is further used to extract the inner and outer boundaries of the projected point cloud, and then the RANSAC algorithm is used to perform straight line fitting on the boundary point cloud.

[0041] Furthermore, in step S5, the model construction using Open CASCADE is completed by creating and operating model entities and feature classes. First, straight line entities corresponding to solid features are created based on the feature cross-sectional shape and construction parameters, and then multiple straight line entities are closed into plane entities. Secondly, the plane entities are stretched and swept according to the extracted skeleton lines to complete the solid feature construction. Finally, the Union Boolean operation is used to merge multiple solid features to complete the reconstruction of the workpiece three-dimensional model.

[0042] Corresponding to the above method, the present invention also provides a technical solution for a workpiece three-dimensional reconstruction device for a robot spraying production line:

[0043] A workpiece three-dimensional reconstruction device for a robot spraying production line, comprising:

[0044] A solid feature template unit is used to construct a solid feature template of a workpiece to be sprayed based on a solid feature modeling method, define the skeleton lines and cross-sectional shapes required for constructing the solid feature template, and define the spatial position sequence and connection relationship of each skeleton line;

[0045] The scene point cloud generation unit is used to calibrate the internal and external parameters and distortion coefficients of the laser camera to obtain the laser camera calibration result, calibrate the laser plane of image acquisition to obtain the laser plane calibration result, and obtain the geometric relationship between the laser projection plane and the camera imaging plane; extract the center line of the laser stripe by the grayscale centroid method to obtain the two-dimensional pixel coordinates of the points on the center line; substitute the laser camera calibration result and the laser plane calibration result to calculate the corresponding spatial coordinates of the points on the center line of the laser stripe in the camera coordinate system; calibrate the direction vector of the movement direction of the spray conveyor line, transform the position and posture of each center line point cloud by the displacement of the spray conveyor line movement within the laser stripe image acquisition interval, and obtain the scene point cloud containing the workpiece by splicing the line point cloud;

[0046] The workpiece point cloud segmentation unit is used to remove the background and noise of the scene point cloud containing the workpiece, and segment the hook point cloud to obtain the workpiece point cloud;

[0047] The feature extraction unit is used to perform boundary extraction and straight line fitting on the workpiece point cloud, project the fitting straight line to the camera coordinate system, obtain the intersection of the fitting straight line and the coordinate axis, extract the fitting straight line corresponding to the entity feature skeleton line according to the spatial position sequence and connection relationship of the skeleton line in step S1, and then extend the extracted fitting straight line with the adjacent fitting straight line to obtain the intersection point, and complete the skeleton line sequence point extraction; project the skeleton line sequence points of the entity feature template and the adjacent parallel fitting straight lines to the camera coordinate system, and obtain the cross-sectional parameters of the entity feature by calculating the distance between the projection points on the coordinate axis;

[0048] The feature reconstruction and fusion unit is used to reconstruct the corresponding entity features according to the skeleton line and section parameters, and fuse multiple entity features through Boolean operations to obtain a three-dimensional model of the workpiece.

[0049] Beneficial effect: The present invention uses a point cloud stitching-workpiece segmentation-feature extraction-feature reconstruction strategy to complete online three-dimensional reconstruction of residual point clouds based on a small amount of prior knowledge, thereby solving the problem of the inability to generate a model due to missing point clouds such as workpiece reflection or structural occlusion.

[0050] As a further effect, the present invention proposes workpiece model reconstruction based on entity features. The model reconstruction parameters are the absolute position of the camera coordinate system. The established model does not need to be aligned to estimate the workpiece pose of the conveyor line. The surface point parameters of the model can be directly used for subsequent path point design, thereby improving modeling efficiency and reducing the computational cost of trajectory planning.

[0051] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flowchart of the steps of a method for online three-dimensional reconstruction of a workpiece for a robot spraying production line in an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of a solid feature template of a workpiece to be sprayed in an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of a sequence of entity feature positions and connection relationships in an embodiment of the present invention;

[0055] Figure 4 is a scene point cloud image obtained in an embodiment of the present invention;

[0056] Figure 5 Schematic diagram of removing background and noise from scene point cloud in an embodiment of the present invention;

[0057] Figure 6 Schematic diagram of workpiece point cloud segmentation in an embodiment of the present invention;

[0058] Figure 7 is a schematic diagram of boundary extraction and straight line fitting in an embodiment of the present invention;

[0059] Figure 8 is a schematic diagram of projecting a fitting straight line onto a camera coordinate system in an embodiment of the present invention;

[0060] Fig. 9 is a feature skeleton line graph extracted in an embodiment of the present invention;

[0061] Fig.10 is a schematic diagram of characteristic cross-sectional parameters of a solid in an embodiment of the present invention;

[0062] Fig.11 It is a three-dimensional reconstructed model diagram of the workpiece in the embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] The embodiment of the present invention discloses a method for online three-dimensional reconstruction of a workpiece for a robot spraying production line, such as Figure 1 As shown, the method comprises the following steps:

[0065] S1. Based on the solid feature modeling method, a solid feature template of the workpiece to be sprayed is established, and the skeleton line and cross-sectional shape required for solid feature construction are defined, as well as the relative position and connection relationship of the skeleton line of each solid feature.

[0066] Specifically, step S1 includes the following steps:

[0067] S1-1. Perform feature analysis on the workpiece to be sprayed, subdivide it into multiple entity feature combinations, establish a entity feature template for the workpiece to be sprayed, and construct the required skeleton line and cross-sectional shape given the entity features. Figure 2 As shown, taking an angle steel workpiece as an example: the workpiece to be sprayed is subdivided into a base entity, a contour entity and an intermediate entity, wherein the construction parameters of the base entity include a skeleton line L1 and a section side length E1, the construction parameters of the contour entity include a skeleton line L2, a section side length E2 and a section thickness D2, and the construction parameters of the intermediate entity include a skeleton line L3, a section side length E2 and a section thickness D2;

[0068] S1-2, projecting the entity feature combination of step S1-1 to the local coordinate system of the model, and obtaining the spatial position sequence and connection relationship of the skeleton lines of each entity feature template by extending the skeleton lines and intersecting the coordinate axis or adjacent skeleton lines. Figure 3 As shown, the base solid skeleton line L p1 , L p2 The spatial position sequence is y1 and y n ; The outline of the solid skeleton line is composed of straight line L O1 , straight line L O2 and the straight line L O3 Composition, straight line L O1 and the straight line L O3 With y p The axes intersect, and their spatial position sequences are y2 and yn-1 , straight line L O2 With x p The axes intersect, and their spatial position sequence is x1; the middle entity skeleton line L m With x p The axis intersects with the outline entity skeleton line, and its spatial position sequence is x3, and it is the same as x p When the number of axis intersections exceeds 4, there are intermediate entities.

[0069] S2. Use a 3D line laser camera to collect images of the spraying production line, extract the center line of the acquired laser stripe image to achieve point cloud generation and point cloud stitching, and obtain a scene point cloud containing the workpiece.

[0070] Specifically, step S2 includes the following steps:

[0071] S2-1, calibrate the internal and external parameters and distortion coefficients of the 3D line laser camera;

[0072] In step S2-1, the correspondence between the image pixel points and the actual scene points is calibrated using the camera's internal and external parameters

[0073] relation:

[0074]

[0075] Where: u, v are the coordinates of the point on the image plane coordinate system; f x 、f y is the camera scale factor; u0 and v0 represent the camera principal point coordinates; (x w ,y w ,z w , 1) is the secondary coordinate value of the calibration plate coordinate system; M in 、M out are the camera internal and external parameter matrices respectively; R is the rotation component of the camera external parameter matrix; t is the translation component of the camera external parameter matrix.

[0076] Distortion model of 3D line laser camera:

[0077]

[0078] Where: (u,v) is the ideal pixel coordinate; (u p ,v p ) are the pixel coordinates after distortion; k1 and k2 are radial distortion parameters; k3 and k4 are tangential distortion parameters.

[0079] S2-2, calibrating the laser plane for image acquisition to obtain the geometric relationship between the laser projection plane and the camera imaging plane;

[0080] The equation of the laser plane in the camera coordinate system is:

[0081] A(x-x0)+B(y-y0)+C(z-z0)=1 (3)

[0082] According to the principle of laser triangulation, the spatial coordinates of the measurement points on the laser stripe line satisfy both the camera imaging model and the laser plane equation. The camera is calibrated with external parameters, and multiple feature points are used to fit the coefficients of the light plane equation.

[0083] S2-3, extracting the center line of the laser stripe by using the grayscale centroid method to obtain the two-dimensional pixel coordinates of the points on the center line, and further calculating the spatial position of the center line of the laser stripe in the camera coordinate system according to the calibration results of steps S2-1 and S2-2, and generating a point cloud of the center line;

[0084] Substitute the calibration results of the camera and the light plane, combine equations (1) and (3), and calculate the corresponding spatial coordinates of the point on the center line of the laser stripe in the camera coordinate system:

[0085]

[0086] S2-4. Calibrate the direction vector of the movement direction of the spray conveyor line, further transform the position and posture of each center line point cloud through the displacement of the spray conveyor line movement within the laser stripe image acquisition interval, and obtain the scene point cloud containing the workpiece by splicing the line point cloud.

[0087] The calibrated conveyor chain direction vector is:

[0088]

[0089] Where: (x start ,y start ,z start ) is the coordinate of a corner point i in the calibration plate coordinate system when calibration starts; (x end ,y end ,z end ) is the coordinate of the same corner point after the calibration plate moves along the conveyor chain for a certain distance; cosθ xi 、cosθ yi 、cosθ zi is the projection of the i-th group of direction vectors in the camera coordinate system.

[0090] Since the calibration plate's posture will change in other directions when it is suspended, m groups of corner points are selected for optimization, and the optimization objective function and optimization constraints satisfy the following formula:

[0091]

[0092] Where: cosθ x 、cosθ y 、cosθz The projection of the direction vector in the camera coordinate system after optimization.

[0093] Using the calibration result of the conveyor chain motion direction vector, the pose of each line point cloud is transformed according to formula (7), and the line point clouds after pose transformation are spliced ​​to obtain the scene point cloud ( Figure 4 ).

[0094]

[0095] S3, removing the background and noise of the scenic spot cloud in step S2, and segmenting the hook point cloud to obtain the workpiece point cloud.

[0096] Specifically, step S3 includes the following steps:

[0097] S3-1. Use the RANSAC algorithm to fit the plane where the background point cloud is located and obtain its normal vector. Transform the centroid position along the normal vector direction by twice the distance between the spray conveyor line and the background plane to obtain a point on the long-distance segmentation surface, generate the point normal equation of the long-distance segmentation surface, and remove the long-distance noise including the background. Figure 5 As shown, the normal vector of the background point cloud plane is n b , along n b Direction L b The distance exchange background point cloud plane's centroid is used to obtain the long-distance plane, whose point normal plane equation is:

[0098] A(x-x0)+B(y-y0)+C(z-z0)=0 (8)

[0099] Where (A, B, C) is the normal vector n b , (x0,y0,z0) is the center of mass after exchange.

[0100] S3-2, cross-multiply the direction vector of the upper boundary of the workpiece point cloud with the normal line to obtain the normal vector of the upper surface, construct the hook segmentation surface according to the points on the upper boundary of the workpiece and the normal vector of the upper surface, and obtain the workpiece point cloud after segmentation. Figure 6 As shown, the direction vector of the upper boundary of the workpiece point cloud is D m , the normal vector of the upper boundary of the workpiece point cloud is n m , and the cross product of the two obtains the normal vector of the upper surface:

[0101] n u =D m ×n m (9) The points on the upper boundary of the workpiece and the normal vector are used to construct the French hook segmentation surface equation:

[0102] A′(x-x0′)+B′(y-y0′)+C′(z-z0′)=0 (10)

[0103] Where (A′, B′, C′) is the normal vector n of the upper surface u , (x0′, y0′, z0′) is any point on the upper boundary of the workpiece.

[0104] S4, perform boundary extraction and straight line fitting on the workpiece point cloud, map the relative position and connection relationship of each entity feature in the entity feature template in step S1 to the camera coordinate system, and extract the skeleton line of the entity feature and calculate the cross-sectional parameters according to the spatial position information of the fitting straight line in the camera coordinate system.

[0105] Specifically, step S4 includes the following steps:

[0106] S4-1. Use the PCA algorithm to obtain the main plane of the workpiece point cloud, project the workpiece point cloud onto the main plane to obtain the projected point cloud, and use it to enhance the local point cloud density; further use the Alpha Shapes algorithm to extract the inner and outer boundaries of the projected point cloud, and then use the RANSAC algorithm to perform straight line fitting on the boundary point cloud. Figure 7

[0107] As shown, the projection point cloud and the boundary point cloud are obtained in turn, and the boundary point cloud is fitted with a straight line and expressed in a point-wise manner, P i represents the point on line i, D i Represents the direction vector of line i, and the expression of the fitted line is:

[0108]

[0109] Where (m,n,p) is the direction vector D of the line i , (x0,y0,z0) is the point P on the line i .

[0110] S4-2, project the fitting line to the camera coordinate system to obtain the intersection of the fitting line and the coordinate axis, further arrange the intersections on the two coordinate axes in descending order according to the coordinate values, extract the fitting line corresponding to the entity feature skeleton line according to the spatial position sequence and connection relationship of the entity feature skeleton line in step S1-2, and then extend the extracted fitting line with the adjacent fitting line to obtain the intersection point, and complete the skeleton line extraction in the form of sequence points. Figure 8As shown, in the x-direction sequence, the first element ① corresponds to the fitting line 5, in the y-direction sequence, the first element ① and the last element ⑥ correspond to the fitting line 10 and the fitting line 9, and the second element ② and the second to last element ⑤ correspond to the fitting line 2 and the fitting line 1. According to the spatial position sequence and connection relationship of the skeleton line in step S1-2, the skeleton line of the base entity coincides with the fitting line 10 and the fitting line 9, and the skeleton line of the contour entity coincides with the fitting line 5, the fitting line 2 and the fitting line 1, and there is no intermediate entity. Further, the corresponding fitting line and the adjacent fitting line are extended to find the intersection point, and the skeleton line sequence point extraction is completed in the form of sequence points, such as Fig. 9 As shown in the figure, after the extracted fitting line and the adjacent fitting line are extended to find the intersection point, the skeleton line sequence points of the base entity are obtained as two groups P bstart and P bend , the skeleton line sequence point of the contour entity is P ostart , P omid1 , P omid2 and P oend .

[0111] S4-3, project the skeleton line sequence points of the entity feature template and the adjacent parallel fitting lines to the camera coordinate system, and obtain the cross-sectional parameters of the entity feature by calculating the distance between the projection points on the coordinate axis. Fig.10 As shown, the distance between the three sets of projection points of the contour entity skeleton line and the adjacent parallel fitting straight lines projected on the camera coordinates is e i The average value is used to obtain the cross-sectional parameters of the contour entity, e i The calculation formula is:

[0112]

[0113] Where (x1, y1) and (x2, y2) are the coordinates of the intersection of a set of skeleton lines and their parallel fitting lines with the coordinate axis.

[0114] The cross-sectional parameter E2 satisfies the following formula:

[0115]

[0116] Other cross-sectional parameters are given by artificial settings.

[0117] S5. Reconstruct corresponding entity features using Open CASCADE according to the skeleton lines and cross-section parameters extracted in step S4, and further fuse multiple entity features through Boolean operations to obtain a three-dimensional model of the workpiece.

[0118] In step S5, the model construction using Open CASCADE is completed by creating and operating model primitives and feature classes. First, the straight line primitives corresponding to the solid features are created based on the feature cross-sectional shape and construction parameters, and then the multi-segment straight line primitives are closed into plane primitives. Secondly, the plane primitives are stretched and swept according to the extracted skeleton line to complete the solid feature construction. Finally, the Union Boolean operation is used to merge multiple solid features to complete the reconstruction of the workpiece 3D model ( Fig.11 ).

[0119] In an embodiment of the present invention, the residual point cloud based on a small amount of prior knowledge is used to complete online three-dimensional reconstruction through the point cloud stitching-workpiece segmentation-feature extraction-feature reconstruction strategy, thereby solving the problem that the model cannot be generated due to the missing point cloud such as workpiece reflection or structure occlusion; workpiece model reconstruction based on entity features is proposed, and the model reconstruction parameters are the absolute position of the camera coordinate system. The established model does not need to be aligned to estimate the workpiece pose of the conveyor line, and the surface point parameters of the model can be directly used for the subsequent path point design, thereby improving the modeling efficiency while reducing the computational cost of trajectory planning.

[0120] In addition, there are many specific implementation methods and approaches of the present invention, and the above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method for workpieces in a robot spraying production line, characterized in that: The following steps are involved: S1. Constructing a solid feature template of the workpiece to be sprayed based on a solid feature modeling method, defining the skeleton lines and cross-sectional shapes required for constructing the solid feature template, and defining the spatial position sequence and connection relationship of each skeleton line; S2. Calibrate the internal and external parameters and distortion coefficients of the laser camera to obtain the laser camera calibration result, calibrate the laser plane of image acquisition to obtain the laser plane calibration result, and obtain the geometric relationship between the laser projection plane and the camera imaging plane; use the grayscale centroid method to extract the center line of the laser stripe to obtain the two-dimensional pixel coordinates of the points on the center line; substitute the laser camera calibration result and the laser plane calibration result to calculate the corresponding spatial coordinates of the points on the center line of the laser stripe in the camera coordinate system; calibrate the direction vector of the movement direction of the spray conveyor line, and transform the position and posture of each center line point cloud through the displacement of the spray conveyor line movement within the laser stripe image acquisition interval, and obtain the scene point cloud containing the workpiece by stitching the line point cloud; S3, removing the background and noise of the scene point cloud containing the workpiece, and segmenting the hook point cloud to obtain the workpiece point cloud; S4, extracting the boundary and straight line fitting of the workpiece point cloud, projecting the fitting straight line to the camera coordinate system, obtaining the intersection of the fitting straight line and the coordinate axis, extracting the fitting straight line corresponding to the skeleton line of the entity feature according to the spatial position sequence and connection relationship of the skeleton line in step S1, and then extending the extracted fitting straight line with the adjacent fitting straight line to obtain the intersection point, and completing the skeleton line sequence point extraction; projecting the skeleton line sequence points of the entity feature template and the adjacent parallel fitting straight lines to the camera coordinate system, and obtaining the cross-sectional parameters of the entity feature by calculating the distance between the projection points of the coordinate axis; S5. Reconstruct corresponding entity features according to the skeleton lines and cross-section parameters extracted in step S4, and fuse multiple entity features through Boolean operations to obtain a three-dimensional model of the workpiece.

2. The method for three-dimensional reconstruction of workpieces for a robot spraying production line according to claim 1, characterized in that: In step S2, The correspondence between the image pixel points and the actual scene points calibrated using the internal and external parameters of the camera is formula 1: Where: u, v are the coordinates of the point on the image plane coordinate system; f x 、f y is the camera scale factor; u0 and v0 represent the camera principal point coordinates; (x w ,y w ,z w , 1) is the homogeneous coordinate value of the calibration plate coordinate system; M in 、M out are the camera internal and external parameter matrices respectively; R is the rotation component of the camera external parameter matrix; t is the translation component of the camera external parameter matrix; Among them, the distortion model of the 3D line laser camera is formula 2: r 2 =u 2 +v 2 Where: (u,v) is the ideal pixel coordinate; (u p ,v p ) is the pixel coordinate after distortion; k1 and k2 are radial distortion parameters; k3 and k4 are tangential distortion parameters; Calibrate the laser plane for image acquisition to obtain the geometric relationship between the laser projection plane and the camera imaging plane; The equation of the laser plane in the camera coordinate system is formula 3: A(x-x0)+B(y-y0)+C(z-z0)=1; Substitute the calibration results of the camera and the light plane, and use Formula 1 and Formula 3 to calculate the corresponding spatial coordinates of the point on the center line of the laser stripe in the camera coordinate system: The direction vector of the spray conveyor line is calibrated, and the direction vector of the conveyor chain is calibrated as follows: Where: (x start ,y start ,z start ) is the coordinate of a corner point i in the calibration plate coordinate system when calibration starts; (x end ,y end ,z end ) is the coordinate of the same corner point after the calibration plate moves along the conveyor chain for a certain distance; cosθ xi 、cosθ yi 、cosθ zi is the projection of the i-th group of direction vectors in the camera coordinate system; Select m groups of corner points for optimization, and the optimization objective function and optimization constraints satisfy the formula: Where: cosθ x 、cosθ y 、cosθ z To optimize the projection of the posterior direction vector in the camera coordinate system; Using the calibration result of the conveyor chain motion direction vector, the pose of each line point cloud is transformed according to the following formula, and the line point clouds after pose transformation are spliced ​​to obtain the scene point cloud:

3. The method for three-dimensional reconstruction of workpieces for a robot spraying production line according to claim 1, characterized in that: In step S3, the RANSAC algorithm is used to fit the plane where the background point cloud is located to obtain the normal vector n of the plane where the background point cloud is located b ; Along n b Direction L b The distance exchange background point cloud plane's centroid is used to obtain the long-distance plane, whose point normal plane equation is: A(x-x0)+B(y-y0)+C(z-z0)=0 Where (A, B, C) is the normal vector n b , (x0, y0, z0) is the center of mass after exchange; The direction vector D of the upper boundary of the workpiece point cloud m The normal vector to the upper boundary of the workpiece point cloud is n m Cross product to obtain the normal vector n of the upper surface u : n u =D m ×n m The point on the upper boundary of the workpiece and the normal vector are used to construct the French hook splitting surface equation: A′(x-x0′)+B′(y-y0′)+C′(z-z0′)=0 Where (A′, B′, C′) is the normal vector n of the upper surface u , (x0′, y0′, z0′) is any point on the upper boundary of the workpiece.

4. The method for three-dimensional reconstruction of workpieces for a robot spraying production line according to claim 3, characterized in that: In step S4, the PCA algorithm is used to obtain the main plane of the workpiece point cloud, and the workpiece point cloud is projected onto the main plane to obtain the projected point cloud to enhance the local point cloud density; the Alpha Shapes algorithm is further used to extract the inner and outer boundaries of the projected point cloud, and then the RANSAC algorithm is used to perform straight line fitting on the boundary point cloud.

5. The method for three-dimensional reconstruction of workpieces for a robot spraying production line according to claim 4, characterized in that: In step S5, the model construction using Open CASCADE is completed by creating and operating model entities and feature classes. First, the straight line entities corresponding to the solid features are created based on the feature cross-sectional shape and construction parameters, and then multiple straight line entities are closed into plane entities. Secondly, the plane entities are stretched and swept according to the extracted skeleton lines to complete the solid feature construction. Finally, the Union Boolean operation is used to merge multiple solid features to complete the reconstruction of the workpiece 3D model.

6. A workpiece 3D reconstruction device for a robot spraying production line, characterized in that: include: A solid feature template unit is used to construct a solid feature template of a workpiece to be sprayed based on a solid feature modeling method, define the skeleton lines and cross-sectional shapes required for constructing the solid feature template, and define the spatial position sequence and connection relationship of each skeleton line; The scene point cloud generation unit is used to calibrate the internal and external parameters and distortion coefficients of the laser camera to obtain the laser camera calibration result, calibrate the laser plane of image acquisition to obtain the laser plane calibration result, and obtain the geometric relationship between the laser projection plane and the camera imaging plane; extract the center line of the laser stripe by the grayscale centroid method to obtain the two-dimensional pixel coordinates of the points on the center line; substitute the laser camera calibration result and the laser plane calibration result to calculate the corresponding spatial coordinates of the points on the center line of the laser stripe in the camera coordinate system; calibrate the direction vector of the movement direction of the spray conveyor line, transform the position and posture of each center line point cloud by the displacement of the spray conveyor line movement within the laser stripe image acquisition interval, and obtain the scene point cloud containing the workpiece by splicing the line point cloud; The workpiece point cloud segmentation unit is used to remove the background and noise of the scene point cloud containing the workpiece, and segment the hook point cloud to obtain the workpiece point cloud; The feature extraction unit is used to perform boundary extraction and straight line fitting on the workpiece point cloud, project the fitting straight line to the camera coordinate system, obtain the intersection of the fitting straight line and the coordinate axis, extract the fitting straight line corresponding to the entity feature skeleton line according to the spatial position sequence and connection relationship of the skeleton line in step S1, and then extend the extracted fitting straight line with the adjacent fitting straight line to obtain the intersection point, and complete the skeleton line sequence point extraction; project the skeleton line sequence points of the entity feature template and the adjacent parallel fitting straight lines to the camera coordinate system, and obtain the cross-sectional parameters of the entity feature by calculating the distance between the projection points on the coordinate axis; The feature reconstruction and fusion unit is used to reconstruct the corresponding entity features according to the skeleton line and section parameters, and fuse multiple entity features through Boolean operations to obtain a three-dimensional model of the workpiece.

7. The workpiece 3D reconstruction device for a robot spraying production line according to claim 6, characterized in that: In the scene point cloud generation unit, the correspondence between the image pixel points and the actual scene points is calibrated using the internal and external parameters of the camera as formula 1: Where: u, v are the coordinates of the point on the image plane coordinate system; f x 、f y is the camera scale factor; u0 and v0 represent the camera principal point coordinates; (x w ,y w ,z w , 1) is the homogeneous coordinate value of the calibration plate coordinate system; M in 、M out are the camera internal and external parameter matrices respectively; R is the rotation component of the camera external parameter matrix; t is the translation component of the camera external parameter matrix; Among them, the distortion model of the 3D line laser camera is formula 2: r 2 =u 2 +v 2 Where: (u,v) is the ideal pixel coordinate; (u p ,v p ) is the pixel coordinate after distortion; k1 and k2 are radial distortion parameters; k3 and k4 are tangential distortion parameters; Calibrate the laser plane for image acquisition to obtain the geometric relationship between the laser projection plane and the camera imaging plane; The equation of the laser plane in the camera coordinate system is formula 3: A(x-x0)+B(y-y0)+C(z-z0)=1; Substitute the calibration results of the camera and the light plane, and use Formula 1 and Formula 3 to calculate the corresponding spatial coordinates of the point on the center line of the laser stripe in the camera coordinate system: The direction vector of the spray conveyor line is calibrated, and the direction vector of the conveyor chain is calibrated as follows: Where: (x start ,y start ,z start ) is the coordinate of a corner point i in the calibration plate coordinate system when calibration starts; (x end ,y end ,z end ) is the coordinate of the same corner point after the calibration plate moves along the conveyor chain for a certain distance; cosθ xi 、cosθ yi 、cosθ zi is the projection of the i-th group of direction vectors in the camera coordinate system; Select m groups of corner points for optimization, and the optimization objective function and optimization constraints satisfy the formula: Where: cosθ x 、cosθ y 、cosθ z To optimize the projection of the posterior direction vector in the camera coordinate system; Using the calibration result of the conveyor chain motion direction vector, the pose of each line point cloud is transformed according to the following formula, and the line point clouds after pose transformation are spliced ​​to obtain the scene point cloud:

8. The workpiece 3D reconstruction device for a robot spraying production line according to claim 6 or 7, characterized in that: In the workpiece point cloud segmentation unit, the RANSAC algorithm is used to fit the plane where the background point cloud is located to obtain the normal vector n of the plane where the background point cloud is located b ; Along n b Direction L b The distance exchange background point cloud plane's centroid is used to obtain the long-distance plane, whose point normal plane equation is: A(x-x0)+B(y-y0)+C(z-z0)=0 Where (A, B, C) is the normal vector n b , (x0, y0, z0) is the center of mass after exchange; The direction vector D of the upper boundary of the workpiece point cloud m The normal vector to the upper boundary of the workpiece point cloud is n m Cross product to obtain the normal vector n of the upper surface m : n u =D m ×n m The point on the upper boundary of the workpiece and the normal vector are used to construct the French hook splitting surface equation: A′(x-x0′)+B′(y-y0′)+C′(z-z0′)=0 Where (A′, B′, C′) is the normal vector n of the upper surface u , (x0′, y0′, z0′) is any point on the upper boundary of the workpiece.

9. The workpiece 3D reconstruction device for a robot spraying production line according to claim 6 or 7, characterized in that: In the feature extraction unit, the PCA algorithm is used to obtain the main plane of the workpiece point cloud, and the workpiece point cloud is projected onto the main plane to obtain the projected point cloud to enhance the local point cloud density; the Alpha Shapes algorithm is further used to extract the inner and outer boundaries of the projected point cloud, and then the RANSAC algorithm is used to perform straight line fitting on the boundary point cloud.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

Citation Information

Patent Citations

  • Line laser scanning-based sand blasting robot automatic path generation method

    CN105354880A

  • Slice scanning processing method and system for manufacturing three-dimensional object

    CN106825570A