Real-time path planning method for automatic spraying process of large planar workpieces

Abstract

This invention discloses a real-time path planning method for an automated spraying process for large planar workpieces. The method includes the following steps: acquiring point cloud data of the workpiece to be sprayed based on structured light using a structured light camera; coarsely segmenting the planar point cloud according to the point cloud depth and filtering it; finely segmenting the planar point cloud to form an effective finely segmented plane; and creating the spraying trajectory of the spray gun based on the parameters of the structured light camera and the spray gun, combined with the effective finely segmented plane. This invention is based on the characteristics of the main spraying area of ​​the workpiece being planar with only a few ribs and beams. By segmenting all planar point clouds and using other unreconstructed point clouds and non-planar reconstructed point clouds as obstacle avoidance point clouds to generate the spraying trajectory, complex calculations are avoided and the generation time is short, thereby meeting the customer's production needs for automatic variety switching in a multi-variety, small-batch production mode.

Description

[Technical Field]

[0001] This invention belongs to the field of path planning, and in particular refers to a real-time path planning method for automatic spraying processes of large planar workpieces. [Background Technology]

[0002] The box girder painting production line requires real-time automated spraying path planning using vision equipment for large structural components to replace manual labor and improve safety, spraying quality, and efficiency. In actual production, some workpieces can reach dimensions of up to 33000×2000×3000mm, with planar surfaces and only a few ribs and beams as obstacles. The production line also operates on a high-volume, multi-variety basis with a fast cycle time. Real-time spraying path planning is required while avoiding obstacles such as ribs and beams to enable automatic product switching. Two existing methods exist: the first is an offline generation of preset spraying trajectories based on manual teaching / programming software; the second is an automated path planning scheme based on 3D reconstruction. The manual teaching method is inefficient, has poor trajectory accuracy, and can only handle simple curved surfaces, failing to meet online real-time requirements. While the software-based method is efficient, has high trajectory accuracy, and can handle complex curved surfaces, it requires a pre-existing CAD model, failing to meet the requirement of online real-time operation without CAD data. The second method offers high trajectory accuracy, can handle complex curved surfaces, and does not require a pre-existing CAD model, but involves massive computation and does not meet online real-time requirements. [Summary of the Invention]

[0003] The purpose of this invention is to provide a real-time path planning method for an automated spraying process for large planar workpieces, in order to solve the problems of inefficiency and poor trajectory accuracy of existing technologies based on manual teaching, or the huge computation and failure to meet online real-time requirements of automated path planning schemes based on three-dimensional reconstruction.

[0004] To achieve the above objectives, the real-time path planning method for automated spraying of large planar workpieces according to the present invention includes the following steps:

[0005] Use a structured light camera to acquire point cloud data of the workpiece to be coated based on structured light;

[0006] Based on the point cloud depth, the planar point cloud is coarsely segmented and then filtered.

[0007] Finely segment the planar point cloud to form an effective finely segmented plane;

[0008] Based on the parameters of the structured light camera and the spray gun, and combined with the effective fine segmentation plane, the spraying trajectory of the spray gun is created; wherein the coarse segmentation of the plane point cloud based on the point cloud depth and the subsequent filtering are achieved in the following way:

[0009] S201: Calculate the depth range of the point cloud;

[0010] S202: Calculate the number of segmentation layers;

[0011] S203: Divide each sub-point cloud layer to obtain multiple sub-point clouds;

[0012] S204: First select the first layer of sub-point cloud;

[0013] S205: Determine whether the proportion of the selected sub-point cloud exceeds the first threshold of the total point cloud. If not, proceed to step S209.

[0014] S206: If so, fit a plane based on the selected sub-point cloud;

[0015] S207: Determine whether the proportion of the number of valid fitted points exceeds the second threshold. If not, proceed to step S209.

[0016] S208: If so, record the valid fitted plane;

[0017] S209: Determine if all layers have been processed. If yes, end. If no, select the next layer and return to step S205;

[0018] The fitting plane based on the selected sub-point cloud in step S206 above is specifically implemented in the following way:

[0019] S300: Fits the initial plane based on all point groups;

[0020] S301: Calculate the projected distance of all points to the fitted initial plane;

[0021] S302: Filter valid point groups based on the standard that the projected distance is less than the average distance, and only retain the valid point cloud with a relatively small projected distance;

[0022] S303: Plane optimization based on effective point groups;

[0023] Furthermore, the precise segmentation of the planar point cloud and the formation of an effective precise segmentation plane are specifically achieved through the following methods;

[0024] S400: Select the coarse segmentation plane formed in the first coarse segmentation plane point cloud step;

[0025] S401: Calculate the average depth of the point cloud of the currently effective fitted plane;

[0026] S402: Re-segment the current layer's sub-point cloud based on average depth;

[0027] S403: Determine whether the proportion of the current layer's sub-point cloud exceeds the first threshold of the total point cloud; if not, proceed to step S407.

[0028] S404: If so, fit a plane based on the current layer sub-point cloud;

[0029] S405: Determine whether the percentage of valid points on the fitted plane exceeds the second threshold. If not, proceed to step S407.

[0030] S406: If so, record the valid fine segmentation plane;

[0031] S407: Determine if all coarse segmentation planes have been processed?

[0032] If so, the process ends. Otherwise, select the next coarse segmentation plane and return to step S401.

[0033] Finally, the creation of the spraying trajectory of the spray gun based on the parameters of the structured light camera and the spray gun, combined with the effective precision segmentation plane, is specifically achieved in the following way;

[0034] S500: Establish the spraying matrix, which is determined based on the far-field view of the structured light camera, camera resolution, horizontal spacing of the spray guns, and vertical spacing of the spray guns.

[0035] S501: Select the first column of the spraying matrix;

[0036] S502: Select the first row of the spraying matrix;

[0037] S503: Determine whether the current point belongs to the above-mentioned effective fine segmentation plane;

[0038] S504: If so, then set the position of the spray gun to {x,y,z-} 喷涂间隔 The spray gun attitude is the normal vector of the valid spraying plane to which the current point belongs;

[0039] S505: If not, then set the position of the spray gun to {x, y, 缺省避障深度 The spray gun's attitude is the default obstacle avoidance attitude;

[0040] S506: Add the current spray gun position and attitude to the spray trajectory;

[0041] S507: Determine whether all rows have been traversed; otherwise, select the next row and proceed to step S503.

[0042] S508: If so, determine whether all columns have been traversed; otherwise, select the next column and proceed to step S502.

[0043] Based on the above key features, the first threshold is 15% and the second threshold is 35%.

[0044] Compared with existing technologies, this invention is based on the characteristics of the main spraying area of ​​the workpiece being planar with only a few ribs and beams. By segmenting all planar point clouds and using other unreconstructed point clouds and non-planar reconstructed point clouds as obstacle avoidance point clouds to generate spraying trajectories, it avoids complex calculation processes such as surface reconstruction, Delaunay triangulation, spraying area segmentation, and intersecting triangulation surfaces of planar groups. This enables real-time spraying trajectory planning with obstacle avoidance function, thereby meeting the customer's production needs for automatic product switching in multi-variety, small-batch production modes. [Attached Image Description]

[0045] Figure 1 A schematic diagram illustrating the main process for implementing this invention.

[0046] Figure 2 A schematic diagram illustrating the steps for implementing the coarse segmentation of the planar point cloud in the main flowchart.

[0047] Figure 3 for Figure 2 A schematic diagram illustrating the specific process of fitting the midpoint cloud to the plane.

[0048] Figure 4 A schematic diagram illustrating the specific process of implementing the segmentation of the planar point cloud in the main flowchart.

[0049] Figure 5 A flowchart illustrating the process of creating a spray pattern.

[0050] Figure 6 This is a schematic diagram of the spraying area in one specific embodiment.

Detailed Implementation Methods

[0051] The overall concept of this invention is based on the fact that the workpiece is planar and has only a few obstacles such as ribs and beams. If the planar point cloud to be sprayed can be accurately segmented and the others can be classified as obstacles, the amount of calculation can be greatly reduced and the real-time planning can be improved.

[0052] Please see Figure 1 The diagram shown is a schematic representation of the main flow of implementing the present invention. The main flow of implementing the present invention includes the following steps:

[0053] Use a structured light camera to acquire point cloud data of the workpiece to be coated based on structured light;

[0054] Based on the point cloud depth, the planar point cloud is coarsely segmented and then filtered.

[0055] Finely segment the planar point cloud to form an effective finely segmented plane;

[0056] Based on the parameters of the structured light camera and the spray gun, and combined with the effective precision segmentation plane, the spraying trajectory of the spray gun is created.

[0057] In practice, point cloud data of the workpiece to be coated can be obtained through a structured light camera (such as a 3D industrial camera), and the point cloud data includes depth information.

[0058] Please see Figure 2 The diagram shown illustrates the specific steps involved in coarsely segmenting a planar point cloud. The coarse segmentation of the planar point cloud includes the following steps:

[0059] S201: Calculate the depth range of the point cloud;

[0060] S202: Calculate the number of segmentation layers; In specific implementation, the parameter setting interface (UI) provides a setting interface for the layer interval parameter. The process engineer sets the layer interval parameter according to the actual situation, that is, the thickness of each layer. Assuming the point cloud depth = {Zmin~Zmax}, and ΔZ = layer interval parameter, then the formula for calculating the number of segmentation layers in the coarse segmentation stage of the planar point cloud is: LayerNum_Coarse = (Zmax-Zmin) / ΔZ+1. For example, if the point cloud depth range is {1704.8116460000001, 3168.7312010000001} mm, and the value of ΔZ is: then LayerNum_Coarse = (Zmax-Zmin) / ΔZ+1 = 8 layers.

[0061] S203: Divide each sub-point cloud layer to obtain multiple sub-point clouds;

[0062] S204: First select the first layer of sub-point cloud;

[0063] S205: Determine whether the proportion of the selected sub-point cloud (or first-layer sub-point cloud when the first-layer sub-point cloud is selected) exceeds the first threshold of the total point cloud (e.g., 15%). If not, proceed to step S209. The proportion of the sub-point cloud can be set by the process personnel according to the actual situation on site. If the structured light camera has a resolution of 2048*1536, it has 3,145,728 pixels. Taking the example above, it is divided into 8 layers: Layer 0 has 82 pixels; Layer 1 has 164,832 pixels; Layer 2 has 17,836 pixels; Layer 3 has 936,171 pixels; Layer 4 has 73,087 pixels; Layer 5 has 22,379 pixels; Layer 6 has 83,169 pixels; and Layer 7 has 111,603 pixels. The successfully reconstructed point cloud contains 1,409,159 pixels = ∑ (number of pixels in each layer). If the selected sub-point cloud has fewer than 211,373 pixels (15% * 1,409,159), it is determined to be non-planar. Otherwise, it is a candidate plane (i.e., a potential plane). As can be seen from the example above, this step can exclude Layers 0, 1, 2, 6, and 7, leaving only Layers 3, 4, and 5.

[0064] S206: If so, then fit a plane based on the selected sub-point cloud. See the attached diagram for specific steps. Figure 3 As shown, details will be provided later;

[0065] S207: Determine if the percentage of valid fitted points exceeds the second threshold (e.g., 35%). If not, proceed to step S209. The second threshold (e.g., 35%) is set by the process engineers based on actual conditions. For example, with three layers, the number of points in the third layer is 936,171. The final number of valid points used for plane fitting (point clouds with projected distances less than the average distance) is 531,852, and the percentage is 531,852 / 936,171 = 56.811315863127568%.

[0066] S208: If so, record the valid fitted plane;

[0067] S209: Determine if all layers have been processed. If yes, end. If no, select the next layer and return to step S205.

[0068] Please see Figure 3 The diagram shown illustrates the specific process of fitting a point cloud to a plane, including the following steps:

[0069] S300: Fits the initial plane based on all point groups;

[0070] S301: Calculate the projected distance of all points to the fitted initial plane;

[0071] S302: Filter valid point groups based on the standard that the projected distance is less than the average distance, and only retain the valid point cloud with a relatively small projected distance;

[0072] S303: Optimize the plane based on effective point groups. In this step, the plane is no longer fitted based on the original point cloud, but rather based on effective point clouds with smaller projection distances. Because invalid point clouds are removed, the fitted plane parameters are more accurate.

[0073] Please see Figure 4 The diagram shown illustrates the detailed process of finely segmenting a planar point cloud, including the following steps:

[0074] S400: Select the coarse segmentation plane formed in the first coarse segmentation plane point cloud step. There may be two or more valid fitting planes in the coarse segmentation plane point cloud step.

[0075] S401: Calculate the average depth of the point cloud of the currently effective fitted plane;

[0076] S402: Re-segment the current layer sub-point cloud based on the average depth, where the layering interval parameter ΔZ can remain unchanged or become smaller in order to further subdivide the plane;

[0077] S403: Determine whether the proportion of the current layer sub-point cloud exceeds the first threshold of the total point cloud (e.g., 15%); if not, proceed to step S407.

[0078] S404: If so, fit a plane based on the current layer sub-point cloud;

[0079] S405: Determine whether the percentage of valid points on the fitted plane exceeds the second threshold (e.g., 35%). If not, proceed to step S407.

[0080] S406: If so, record the valid fine segmentation plane;

[0081] S407: Determine if all coarse segmentation planes have been processed?

[0082] If so, the process ends. Otherwise, select the next coarse segmentation plane and return to step S401.

[0083] By finely segmenting the planar point cloud, the surface to be sprayed on the workpiece can be determined more accurately. This surface may be one, or it may be two or more surfaces with height differences. Through the two processes of coarsely segmenting and finely segmenting the planar point cloud, the surface to be sprayed can be accurately determined.

[0084] Please see Figure 5 The diagram shown illustrates the process of creating a spray pattern, including the following steps:

[0085] S500: Establish the spraying matrix, which is determined based on the far-end field of view of the structured light camera, the camera resolution, the horizontal spacing of the spray guns, and the vertical spacing of the spray guns. For example, the far-end field of view of the structured light camera (Z=3000mm) is 3000×2400mm, the camera resolution is 2048*1536 pixels, so the camera resolution is 1.5mm / pixel. The horizontal spacing of the spray guns is 90mm, and the vertical spacing of the spray guns is 90mm. Therefore, the spacing between adjacent horizontal and vertical spraying points is 90 / 1.5=60 pixels. Thus, the spraying point matrix is ​​35 (=2048 / 60+1) columns * 26 (1536 / 60+1) rows, which is a 35*26 matrix. Therefore, when planning the spraying trajectory, it is only necessary to determine the points of the above spraying matrix.

[0086] S501: Select the first column of the spraying matrix;

[0087] S502: Select the first row of the spraying matrix;

[0088] S503: Determine whether the current point belongs to the valid spraying plane, that is, the valid fine division plane recorded in S406;

[0089] S504: If so, then set the position of the spray gun to {x,y,z-}喷涂间隔 The spray gun attitude is the normal vector of the valid spraying plane to which the current point belongs;

[0090] S505: If not, then set the position of the spray gun to {x, y, 缺省避障深度 The spray gun's attitude is the default obstacle avoidance attitude;

[0091] S506: Add the current spray gun position and attitude to the spray trajectory;

[0092] S507: Determine whether all rows have been traversed; otherwise, select the next row and proceed to step S503.

[0093] S508: If so, determine whether all columns have been traversed; otherwise, select the next column and proceed to step S502.

[0094] In the above way, the position and attitude of the spray gun can be added to the spraying trajectory to form the final spraying trajectory, which can then be output to the next station so that the spray gun can spray the workpiece according to this spraying trajectory.

[0095] Please refer to the details. Figure 6 The diagram shown is a schematic of the spraying area in a specific embodiment. The dots represent that the spraying point is an obstacle (i.e., a point in the obstacle avoidance point cloud, including unreconstructed point clouds and non-planar reconstructed point clouds). The spray gun should be in an obstacle avoidance position and attitude. The cross-shaped dots indicate that the spraying point (a point in the effective fine segmentation plane) is the point to be sprayed. The position of the spray gun is the position of this point, and the attitude of the spray gun is the direction of the plane normal vector.

[0096] Compared with existing technologies, this invention, based on the characteristics of the workpiece's main spraying area being planar with only a few ribs and beams, proposes a method for generating spraying trajectories by segmenting all planar point clouds and using the remaining unreconstructed point clouds and non-planar reconstructed point clouds as obstacle avoidance point clouds. Because it avoids complex computational processes such as surface reconstruction, Delaunay triangulation, spraying area segmentation, and intersecting triangulation surfaces of planar groups, it achieves real-time spraying trajectory planning with obstacle avoidance functionality (generating 3 million point clouds in 1 second, and trajectory generation in only 1.5 seconds), thus meeting the customer's production needs for automatic product switching in multi-variety, small-batch production modes.

[0097] It is understood that those skilled in the art can make equivalent substitutions or modifications to the technical solution and inventive concept of the present invention, and all such substitutions or modifications should fall within the protection scope of the appended claims.

Claims

1. A real-time path planning method for an automated spraying process for large planar workpieces, characterized in that... The method includes the following steps: Use a structured light camera to acquire point cloud data of the workpiece to be coated based on structured light; Based on the point cloud depth, the planar point cloud is coarsely segmented and then filtered. Finely segment the planar point cloud to form an effective finely segmented plane; Based on the parameters of the structured light camera and the spray gun, and combined with the effective precision segmentation plane, the spraying trajectory of the spray gun is created. The coarse segmentation and filtering of the planar point cloud based on the point cloud depth is achieved in the following way: S201: Calculate the depth range of the point cloud; S202: Calculate the number of segmentation layers; S203: Divide each sub-point cloud layer to obtain multiple sub-point clouds; S204: First select the first layer of sub-point cloud; S205: Determine whether the proportion of the selected sub-point cloud exceeds the first threshold of the total point cloud. If not, proceed to step S209. S206: If so, fit a plane based on the selected sub-point cloud; S207: Determine whether the proportion of the number of valid fitted points exceeds the second threshold. If not, proceed to step S209. S208: If so, record the valid fitted plane; S209: Determine if all layers have been processed? If yes, end; if no, select the next layer and return to step S205. The fitting plane based on the selected sub-point cloud in step S206 above is specifically implemented in the following way: S300: Fits the initial plane based on all point groups; S301: Calculate the projected distance of all points to the fitted initial plane; S302: Filter valid point groups based on the standard that the projected distance is less than the average distance, and only retain the valid point cloud with a relatively small projected distance; S303: Plane optimization based on effective point groups; Furthermore, the precise segmentation of the planar point cloud and the formation of an effective precise segmentation plane are specifically achieved through the following methods; S400: Select the coarse segmentation plane formed in the first coarse segmentation plane point cloud step; S401: Calculate the average depth of the point cloud of the currently effective fitted plane; S402: Re-segment the current layer's sub-point cloud based on average depth; S403: Determine whether the proportion of the current layer's sub-point cloud exceeds the first threshold of the total point cloud; if not, proceed to step S407. S404: If so, fit a plane based on the current layer sub-point cloud; S405: Determine whether the percentage of valid points on the fitted plane exceeds the second threshold. If not, proceed to step S407. S406: If so, record the valid fine segmentation plane; S407: Determine if all coarse segmentation planes have been processed? If so, the process ends; otherwise, select the next coarse segmentation plane and return to step S401. Finally, the creation of the spraying trajectory of the spray gun based on the parameters of the structured light camera and the spray gun, combined with the effective precision segmentation plane, is specifically achieved in the following way; S500: Establish the spraying matrix, which is determined based on the far-field view of the structured light camera, camera resolution, horizontal spacing of the spray guns, and vertical spacing of the spray guns. S501: Select the first column of the spraying matrix; S502: Select the first row of the spraying matrix; S503: Determine whether the current point belongs to the above-mentioned effective fine segmentation plane; S504: If so, then set the position of the spray gun to {x,y,z-} 喷涂间隔 The spray gun attitude is the normal vector of the valid spraying plane to which the current point belongs; S505: If not, then set the position of the spray gun to {x, y, 缺省避障深度 The spray gun's attitude is the default obstacle avoidance attitude; S506: Add the current spray gun position and attitude to the spray trajectory; S507: Determine whether all rows have been traversed; otherwise, select the next row and proceed to step S503. S508: If so, determine whether all columns have been traversed; otherwise, select the next column and proceed to step S502.

2. The real-time path planning method for automated spraying process of large planar workpieces as described in claim 1, characterized in that: The first threshold is 15%, and the second threshold is 35%.

Images