Online planning method for mixed-line product robot painting operation based on multi-depth camera

Abstract

This invention discloses an online planning method for robotic spraying operations on mixed-line production lines based on multi-depth cameras. The invention acquires multi-angle point cloud data of the product using multiple depth cameras and stitches these point clouds together into a complete point cloud using a turntable method. Based on a point cloud slicing algorithm, the point cloud contour lines are extracted to generate the end trajectory of the spraying robot's spray gun. Based on a 3D paint distribution model, the spraying thickness at each point on the product surface is accurately calculated, and the rationality of the spraying trajectory is verified, thereby generating the spraying operation instructions for the spraying robot. This invention completes the entire process from product point cloud acquisition, spraying effect evaluation, and spraying robot operation program generation, with an extremely short overall planning time. This invention can conveniently acquire product surface point cloud data, is suitable for mixed-line industrial production, and quickly and automatically generates the spraying robot's spraying operation trajectory based on the product's shape and spraying requirements, significantly improving the efficiency of spraying trajectory planning and meeting the real-time requirements of online spraying trajectory planning.

Description

Technical Field

[0001] This invention belongs to the field of automated spraying, and in particular, it is an online planning method for robot spraying operations of mixed-line products based on multi-depth cameras. Background Technology

[0002] Today, industrial products are becoming increasingly complex. To meet performance requirements, workpiece surfaces often have intricate shapes, and the quality of the surface coating has a significant impact on the overall performance of the product. For example, in manufactured products such as automobiles and ships, the quality of the surface coating is a crucial aspect of product performance indicators. A protective paint layer of a certain thickness and uniformity can prevent corrosion and deterioration of the product.

[0003] In large-scale industrial production, painting robots have replaced manual painting due to their advantages such as high operational flexibility, wide spraying range, high paint quality, and high spraying efficiency, thus avoiding the life hazards to workers caused by painting. Currently, the commonly used trajectory planning technology for painting robots in factories is the manual teaching method. Experienced painting workers guide the movement of the painting robot, record key spraying trajectory points through a teach pendant, and modify the trajectory based on the spraying effect. The robot can then reproduce and repeat the same spraying trajectory. However, the overall planning efficiency is low, and painting workers still have to endure the health hazards caused by painting.

[0004] Therefore, many factories have begun to adopt computer-aided spraying trajectory planning. However, since factories often adopt mixed-line production mode, which is beneficial to improve work efficiency, different types of products have very different shapes and spraying operation requirements. They cannot simply use the same spraying operation program for spraying. Therefore, it is necessary to collect shape data for different models of products and generate spraying trajectories online. How to quickly generate spraying robot spraying operation programs for different models of products and complete the spraying operation is a problem that is very worthy of research. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes an online planning method for robot spraying operations of mixed-line products based on multi-depth cameras. This method realizes the complete process from acquiring point clouds of large and complex product shapes to generating spraying trajectories, thereby completing online spraying operations.

[0006] The technical solution adopted in this invention is as follows:

[0007] I. An Online Planning Method for Robotic Spraying Operations of Mixed-Line Products Based on Multi-Depth Cameras

[0008] 1) Collect point cloud data of mixed-line product surfaces from multiple perspectives to form a complete product surface point cloud dataset;

[0009] 2) Extract the central contour polyline of the complete product surface point cloud data, and generate the end trajectory of the spray gun of the spraying robot based on the extracted contour polyline;

[0010] 3) Based on the three-dimensional spraying distribution model, verify the rationality of the current spray gun end trajectory by calculating the paint thickness. If it is not rational, repeat step 2) to update the spray gun end trajectory of the spraying robot until the final spray gun end trajectory is rational.

[0011] 4) Calculate the transformation relationship between the camera coordinate system and the spraying robot coordinate system, and generate the spraying operation instructions for the spraying robot based on the transformation relationship and the final spray gun end trajectory.

[0012] Specifically, 2) refers to:

[0013] 2.1) Determine the paint distribution model based on the spraying parameters, and then optimize and determine the spraying stroke spacing and spraying speed. Divide the complete product surface point cloud dataset into several point cloud slices based on the spraying stroke spacing.

[0014] 2.2) Generate the slice outline point set for each point cloud slice using a variable radius neighbor pair search method;

[0015] 2.3) After extracting the outer contour of the slice contour point set of the current point cloud slice, a polyline-enclosed contour is formed. Finally, the number of times each line segment endpoint in the polyline-enclosed contour connects with other line segments is counted. All endpoints that only connect once are extracted and then reordered from top to bottom according to the coordinate axis direction of the view sweep, thus forming a unidirectional continuous contour polyline.

[0016] 2.4) Simplify the unidirectional continuous contour polyline of the current point cloud slice to obtain a simplified polyline contour;

[0017] 2.5) The bisector of the outer angle of two adjacent polylines in the simplified polyline profile of the current point cloud slice is used as the spraying direction of the spray gun, and the spray gun end trajectory is generated in combination with the spraying height.

[0018] 2.6) Repeat steps 2.2)-2.5) to calculate the spray gun end trajectory of the remaining point cloud slices. Connect the spray trajectories of each slice end to end to obtain the complete spray trajectory.

[0019] In section 2.2), the generation of the contour point cloud for each point cloud slice specifically includes the following steps:

[0020] S1: Divide the current point cloud slice into two point cloud bands using a slice plane that is perpendicular to the slice direction of each point cloud slice and passes through the center of the current point cloud slice;

[0021] S2: Use a kd-tree to perform a nearest neighbor search on all points in the first point cloud by continuously increasing the search radius until the nearest point in the second point cloud is found. Then perform a reverse nearest point search on that nearest point. If two points are each other's nearest points across neighborhoods, they are determined to be a pair of intersection points.

[0022] S3: Repeat S2 until all intersection pairs in the two point cloud bands are found. Then find the intersection of the line connecting all intersection pairs with the slice plane and form the outline point cloud of the current point cloud slice on the slice plane.

[0023] In section 2.3), the outer contour of the slice contour point set is extracted by the alpha-shapes method. Specifically, points are taken sequentially on the vertical line passing through the center of the slice point cloud. Each point extends horizontally to form a horizontal cross-section. Only the outermost point of the slice contour point set on the horizontal cross-section is retained to form a polygonal contour.

[0024] Specifically, 2.4) refers to:

[0025] Based on the angle between two adjacent polylines, the polyline contour is clustered to obtain polyline clusters. Then, LSE line fitting is performed on each polyline cluster to obtain the polyline contour after initial fitting. Finally, based on the included angle method, small-sized polyline segments in the polyline contour after initial fitting are deleted to obtain the final simplified polyline contour.

[0026] In step 3), the angle between the normal vector of the sampling point on the product surface and the direction of the spray gun is used to determine whether occlusion has occurred. If the angle is greater than 90 degrees, the current trajectory of the spray gun end is unreasonable. The spray stroke spacing and spray speed are changed, and step 2 is repeated.

[0027] II. A computer device

[0028] The computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method.

[0029] III. A computer-readable storage medium

[0030] The computer-readable storage medium stores a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method.

[0031] The beneficial effects of this invention are as follows:

[0032] (1) This invention uses a multi-depth camera to collect point cloud data of the product surface from multiple angles to obtain a complete point cloud of the product surface. It adopts a point cloud-based spraying trajectory planning method, which greatly improves the convenience and efficiency of spraying operations.

[0033] (2) The present invention uses a spraying trajectory generation method based on point cloud slicing technology, which can generate trajectories by combining the contour features of large and complex product surfaces, reducing problems such as spraying path backtracking, repetition, and collision caused by small-sized features of product surfaces, while ensuring paint coverage and paint thickness, and reducing paint spraying time and paint usage.

[0034] (3) The present invention can accurately estimate the paint thickness at each point on the product surface through calculation, and verify the effectiveness of the spraying trajectory planning of the present invention through calculation.

[0035] (4) This invention provides a complete solution from product point cloud acquisition to spraying quality assessment, which is adapted to the fast-paced production rhythm in industry and the increasing complexity of industrial products. Attached Figure Description

[0036] Figure 1 This is a flowchart of the spraying trajectory planning in the embodiment;

[0037] Figure 2 This is a diagram showing the arrangement of the multi-angle depth cameras in the embodiment;

[0038] Figure 3 This is a schematic diagram of the overall spraying trajectory in the embodiment;

[0039] Figure 4 This is a schematic diagram illustrating the calculation of coating thickness in the embodiment;

[0040] Figure 5 This is a diagram showing the results of the spraying experiment in the embodiment. Detailed Implementation

[0041] To better understand the technical solution of this invention, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. The overall flowchart is as follows. Figure 1 As shown.

[0042] 1) Collect point cloud data of mixed-line product surfaces from multiple perspectives to form a complete product surface point cloud dataset;

[0043] In practice, firstly, three depth point cloud cameras are positioned on both the front and back sides of the product, with the product center as the center. These six depth point cloud cameras are then spaced apart on a circle with a radius of 2 meters, centered on the product center. The central camera faces the product directly, while the left and right cameras are positioned on the circle at a 60-degree angle. The multi-angle depth camera arrangement is as follows: Figure 2 As shown, it is connected to the computer via a data extension cable and a USB splitter;

[0044] Next, color frames and depth frames from each camera are acquired in sequence. After aligning the depth frames with the color frames, the depth value corresponding to each color pixel is obtained, thus obtaining point cloud data with color information.

[0045] Finally, by combining the relative positions of the cameras, the point cloud data acquired by each camera is transferred to the same camera coordinate system using the turntable method to obtain complete product surface point cloud data.

[0046] 2) Extract the central contour polyline of the complete product surface point cloud data based on the point cloud slicing algorithm, and generate the end trajectory of the spray gun of the spraying robot based on the extracted contour polyline.

[0047] 2) Specifically:

[0048] 2.1) Determine the paint distribution model q(x,y) based on the spraying parameters, and then optimize and determine the spraying stroke spacing and spraying speed. Divide the complete product surface point cloud dataset into several point cloud slices based on the spraying stroke spacing.

[0049] 2.2) Generate the slice outline point set for each point cloud slice using a variable radius neighbor pair search method;

[0050] In section 2.2), the generation of the contour point cloud for each point cloud slice specifically includes the following steps:

[0051] S1: Divide the current point cloud slice into two point cloud bands using a slice plane that is perpendicular to the slice direction of each point cloud slice and passes through the center of the current point cloud slice;

[0052] S2: Use a kd-tree to perform a nearest neighbor search on all points in the first point cloud by continuously increasing the search radius until the nearest point in the second point cloud is found. Then perform a reverse nearest point search on that nearest point. If two points are each other's nearest points across neighborhoods, they are determined to be a pair of intersection points.

[0053] S3: Repeat S2 until all intersection pairs are found in the two point cloud bands. Then, find the intersection points of the lines connecting all intersection pairs with the slicing plane and form the contour point cloud of the current point cloud slice on the slicing plane. The variable radius nearest neighbor search method proposed in this invention reduces the search range and significantly reduces the search time compared with the traditional traversal search.

[0054] 2.3) After extracting the outer contour of the slice contour point set of the current point cloud slice, a polyline bounding contour with a certain precision is formed. Finally, the number of times each line segment endpoint in the polyline bounding contour connects with other line segments is counted. All endpoints that only connect once are extracted and then reordered from top to bottom according to the coordinate axis direction of the view sweep, thus forming a unidirectional continuous contour polyline.

[0055] In section 2.3), the outer contour of the slice contour point set is extracted by the alpha-shapes method. Specifically, points are taken sequentially on the vertical line passing through the center of the slice point cloud. Each point extends horizontally to form a horizontal cross-section. Only the outermost point of the slice contour point set on this horizontal cross-section is retained, and the middle points are deleted, thus completing the removal of the inner contour line and forming a polyline-enclosed contour.

[0056] 2.4) Simplify the unidirectional continuous contour polyline of the current point cloud slice to obtain a simplified polyline contour;

[0057] 2.4) Specifically:

[0058] Based on the angle between two adjacent polylines, the polyline contour is clustered to obtain polyline clusters. Then, LSE line fitting is performed on each polyline cluster to obtain the polyline contour after initial fitting. Then, based on the included angle method, small-sized polyline segments in the polyline contour after initial fitting are deleted to obtain the final simplified polyline contour, thus retaining the features of large-sized polylines and the small-sized polyline segments being polyline segments smaller than the preset size.

[0059] 2.5) The bisector of the outer angle of two adjacent polylines in the simplified polyline contour of the current point cloud slice is used as the spraying direction of the spray gun. Combined with the spraying height, the end trajectory of the spray gun is generated, and the spray gun points from the end position point to the contour point.

[0060] 2.6) Repeat steps 2.2)-2.5) to calculate the spray gun end trajectory of the remaining point cloud slices. Connect the spray trajectories of each slice end to end to obtain the complete spray trajectory.

[0061] 3) Based on the spray gun tip position, spray gun posture, paint distribution model, and the curvature characteristics of the product surface, the paint distribution model of the product's three-dimensional surface is derived, such as... Figure 4 As shown. Based on the three-dimensional spraying distribution model, the rationality of the current spray gun end trajectory is verified by calculating the paint thickness. If it is not rational, repeat step 2) to update the spray gun end trajectory of the spraying robot until the final spray gun end trajectory is rational.

[0062] In step 3), the angle between the normal vector of the sampling point on the product surface and the direction of the spray gun is used to determine whether occlusion has occurred. If the angle is greater than 90 degrees, the spray trajectory cannot produce paint at that point, and the current end trajectory of the spray gun is unreasonable. Change the spray stroke spacing and spray speed, and repeat step 2). Otherwise, it is reasonable.

[0063] In practice, the paint thickness at each spray trajectory point is calculated according to the three-dimensional spray distribution model, and the formula is as follows:

[0064]

[0065] Where q(x,y,z) is the three-dimensional paint spraying distribution model, and q(x,y) is the paint spraying distribution model.

[0066] The final paint thickness at a given point is obtained by summing the paint thicknesses generated at all spraying trajectory points. The spraying effect is then evaluated after calculating the paint thickness at all points on the product surface.

[0067] In this embodiment, the paint distribution model was obtained through fitting:

[0068]

[0069] Therefore, the paint thickness at each spray trajectory point is:

[0070]

[0071] Where point s is any point on the freeform surface, and H is the standard height, i.e., the height during the flatbed spraying experiment. s θ refers to the vertical distance from the center of the spray gun to point s. s Let λ be the angle between the direction of the spray gun and the vertical direction. s This represents the angle between the normal vector at point s and the direction of the spray gun.

[0072] In practice, the spray gun height was determined to be 350mm. Based on the desired paint thickness of 50µm, the spray stroke spacing was optimized to 253.1276mm, and the spray gun moving speed was 720mm / s. The spray trajectory was generated using a point cloud slicing algorithm, as shown in the figure. Figure 3 As shown, the thicker white line segment represents the trajectory of the spray gun tip, and the thinner white line segment represents the posture of the spray gun tip. Pointing from the spray gun tip trajectory point to the product surface contour point, it can be seen that the spray trajectory is basically perpendicular to the product contour surface and covers the entire product surface. The calculated average paint thickness of the product is 57.5375um, which meets the paint thickness requirements.

[0073] 4) Calculate the transformation relationship between the camera coordinate system and the spraying robot coordinate system, and generate the spraying operation instructions for the spraying robot based on the transformation relationship and the final spray gun end trajectory.

[0074] 4) Specifically:

[0075] First, the relative positional relationship between the camera and the product hook on the production line is measured, and the position of the hook in the point cloud captured by the multi-depth camera is filtered and obtained. Then, the relative positional relationship between the painting robot and the product hook is measured. The coordinate system transformation relationship between the camera and the painting robot is calculated through the two sets of relative positional relationships. The final spray gun end trajectory of the painting robot is converted into the painting robot spraying operation trajectory in the coordinate system of the painting robot, and the painting robot spraying operation program is generated.

[0076] Tests showed that point cloud acquisition took 10 seconds, and the spraying robot trajectory generation took 12 seconds, for a total of 22 seconds, which is far lower than the production line cycle time of 90 seconds, meeting the requirements for online spraying operations.

[0077] In actual spraying operations, the painting time was 32 seconds, greatly reducing the spraying time. A total of 357 ml of paint and hardener was consumed, reducing paint consumption. The spraying result is as follows: Figure 5 As shown, the paint shop uses a mixed-air spraying method that combines air spraying and static pressure spraying. The paint utilization rate is about 40%, so the average paint thickness can be estimated to be 51.5 μm.

[0078] As can be seen from the image, the paint has been applied evenly to the engine surface. After drying, the average paint thickness on the product surface was measured to be 55.0 μm, which meets the paint quality requirements.

[0079] The basic principles and main features of the present invention have been described in detail above with reference to the accompanying drawings. Using this invention, product shape data can be obtained conveniently and quickly, and a spraying trajectory can be automatically generated based on the spraying process requirements, meeting the paint thickness requirements. The overall planning process is extremely time-efficient, satisfying the real-time requirements of online spraying trajectory planning. However, these descriptions should not be construed as limiting the scope of the present invention. The scope of protection of the present invention is defined by the appended claims, and any modifications made based on the claims of the present invention are within the scope of protection of the present invention.

Claims

1. A method for online planning of robotic spraying operations for mixed-line products based on multi-depth cameras, characterized in that, Includes the following steps: 1) Collect point cloud data of mixed-line product surfaces from multiple perspectives to form a complete product surface point cloud dataset; 2) Extract the central contour polyline of the complete product surface point cloud data, and generate the end trajectory of the spray gun of the spraying robot based on the extracted contour polyline; Specifically, 2) refers to: 2.1) Determine the paint distribution model based on the spraying parameters, and then optimize and determine the spraying stroke spacing and spraying speed. Divide the complete product surface point cloud dataset into several point cloud slices based on the spraying stroke spacing. 2.2) Generate the slice outline point set for each point cloud slice using a variable radius neighbor pair search method; 2.3) After extracting the outer contour of the slice contour point set of the current point cloud slice, a polyline-enclosed contour is formed. Finally, the number of times each line segment endpoint in the polyline-enclosed contour connects with other line segments is counted. All endpoints that only connect once are extracted and then reordered from top to bottom according to the coordinate axis direction of the view sweep, thus forming a unidirectional continuous contour polyline. 2.4) Simplify the unidirectional continuous contour polyline of the current point cloud slice to obtain a simplified polyline contour; Specifically, 2.4) refers to: Based on the angle between two adjacent polylines, the polyline contour is clustered to obtain polyline clusters. Then, LSE line fitting is performed on each polyline cluster to obtain the polyline contour after the first fitting. Then, based on the included angle method, small-sized polyline segments in the polyline contour after the first fitting are deleted to obtain the final simplified polyline contour. 2.5) The bisector of the outer angle of two adjacent polylines in the simplified polyline profile of the current point cloud slice is used as the spraying direction of the spray gun, and the spray gun end trajectory is generated in combination with the spraying height. 2.6) Repeat steps 2.2)-2.5) to calculate the spray gun end trajectory of the remaining point cloud slices. Connect the spray trajectories of each slice to obtain the complete spray trajectory. 3) Based on the three-dimensional spraying distribution model, verify the rationality of the current spray gun end trajectory by calculating the paint thickness. If it is not rational, repeat step 2) to update the spray gun end trajectory of the spraying robot until the final spray gun end trajectory is rational. 4) Calculate the transformation relationship between the camera coordinate system and the spraying robot coordinate system, and generate the spraying operation instructions for the spraying robot based on the transformation relationship and the final spray gun end trajectory.

2. The online planning method for robot spraying operations of mixed-line products based on multi-depth cameras according to claim 1, characterized in that, In section 2.2), the generation of the contour point cloud for each point cloud slice specifically includes the following steps: S1: Divide the current point cloud slice into two point cloud bands using a slice plane that is perpendicular to the slice direction of each point cloud slice and passes through the center of the current point cloud slice; S2: Use a kd-tree to perform a nearest neighbor search on all points in the first point cloud by continuously increasing the search radius until the nearest point in the second point cloud is found. Then perform a reverse nearest point search on that nearest point. If two points are each other's nearest points across neighborhoods, they are determined to be a pair of intersection points. S3: Repeat S2 until all intersection pairs in the two point cloud bands are found. Then find the intersection of the line connecting all intersection pairs with the slice plane and form the outline point cloud of the current point cloud slice on the slice plane.

3. The online planning method for robot spraying operations of mixed-line products based on multi-depth cameras according to claim 1, characterized in that, In section 2.3), the outer contour of the slice contour point set is extracted by the alpha-shapes method. Specifically, points are taken sequentially on the vertical line passing through the center of the slice point cloud. Each point extends horizontally to form a horizontal cross-section. Only the outermost point of the slice contour point set on the horizontal cross-section is retained to form a polygonal contour.

4. The online planning method for robot spraying operations of mixed-line products based on multi-depth cameras according to claim 1, characterized in that, In step 3), the angle between the normal vector of the sampling point on the product surface and the direction of the spray gun is used to determine whether occlusion has occurred. If the angle is greater than 90 degrees, the current trajectory of the spray gun end is unreasonable. The spray stroke spacing and spray speed are changed, and step 2 is repeated.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images