Point cloud three-dimensional measurement splicing method for intelligent spraying operation

Abstract

The invention discloses a point cloud three-dimensional measurement splicing method for intelligent spraying operation, and the method comprises the following steps: 1, employing two depth sensors to collect the original point cloud data of a plurality of calibration balls, extracting the spherical point cloud, color value and center coordinates of each calibration ball, carrying out the matching of homonymy points, and calibrating the two depth sensors; 2, collecting rough point clouds of a workpiece to be sprayed by using two depth sensors, splicing to obtain the spliced rough point clouds, and further obtaining an overall three-dimensional measurement path; 3, projection is carried out on the surface of the workpiece to be sprayed to form prompt information, and colored mark points are pasted according to the prompt information; 4, a mechanical arm is used for driving binocular structured light equipment to move according to the overall three-dimensional measurement path to measure the surface of the workpiece to be sprayed, and surface accurate point cloud is obtained; and splicing the surface accurate point clouds of the two camera visual angles by using the color information of each color mark point to obtain the spliced accurate point clouds of the surface of the workpiece to be sprayed.

Description

Technical Field

[0001] The present invention relates to the field of intelligent spraying point cloud data processing methods, and in particular to a point cloud three-dimensional measurement and splicing method for intelligent spraying operations. Background Art

[0002] In the field of intelligent spraying, faced with multi-variety, small-batch, customized, and refined spraying tasks, machine vision technology is used to obtain three-dimensional point cloud information of the surface of the workpiece to be sprayed. This point cloud is then combined with the spraying process to automatically generate the spraying trajectory, ultimately completing the intelligent spraying operation. Since a single-view point cloud often cannot reflect the complete surface information of the workpiece to be sprayed, it is necessary to use visual equipment to perform three-dimensional measurement of the sprayed workpiece from multiple perspectives, and then stitch the point clouds from multiple perspectives after measurement. Multi-view three-dimensional measurement path planning and multi-view point cloud stitching are technical difficulties in the intelligent spraying process.

[0003] The prior art patent application number "202310468797.7" proposes a self-calibration three-dimensional reconstruction device and method for intelligent spraying, which can perform three-dimensional measurement of workpieces at multiple positions and perspectives according to the shape and size of different types of workpieces, but these positions and perspectives need to be selected in advance according to the reconstruction requirements. The measurement trajectory is not automatically generated according to the workpiece to be sprayed. For different workpieces, manual reselection is required each time before the measurement begins. The efficiency and intelligence of three-dimensional measurement are low.

[0004] The prior art patent application number "202110621029.1" proposes a point cloud self-registration method based on calibration spheres, and proposes a method that can automatically match points with the same name to achieve point cloud registration with large angles and low overlap rates. The Euclidean distance between each pair of calibration spheres is used to form a feature descriptor to complete the automatic matching of points with the same name, but there must be a difference in the Euclidean distance between the centers of the two spheres, and there are strict requirements on the placement of the calibration spheres. The deployment of the system is not flexible enough.

[0005] The prior art patent application number "202410072024.1" proposes a method for precise stitching of aero-engine turbine blade point clouds. It creates feature descriptors for anti-tilt marker points under adjacent perspectives and matches homonymous points between adjacent views to achieve more accurate stitching of multi-surface point clouds. However, the anti-tilt marker points are essentially circular marker points, which are easily deformed by interference from the object surface, making visual detection more difficult. In addition, the Euclidean distance is used for matching homonymous points, which has strict requirements on the posting position. In addition, each circular marker point can only provide one point position information. In order to improve the matching accuracy, a large number of marker points need to be posted, and the pasting and removal process is very time-consuming and labor-intensive.

[0006] Therefore, in intelligent spraying operations, it is necessary to design a point cloud three-dimensional measurement and splicing method. Faced with workpieces to be sprayed of different shapes, sizes, and types, it is necessary to realize the generation of multi-perspective three-dimensional measurement trajectories and the high-precision splicing of multi-perspective point clouds after measurement, thereby improving the intelligence level of the spraying operation. Summary of the Invention

[0007] In order to overcome the above-mentioned problems of the prior art, the present invention provides a point cloud three-dimensional measurement and stitching method for intelligent spraying operations, so as to realize the three-dimensional measurement of a variety of small batches of workpieces to be sprayed, and the high-precision stitching of multi-view point clouds after measurement.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] The point cloud 3D measurement and splicing method for intelligent spraying operation includes the following steps:

[0010] Step 1: Use two depth sensors to collect the original point cloud data of multiple calibration balls of different colors placed at the position of the workpiece to be sprayed;

[0011] Based on the original point cloud data of each calibration sphere under each depth sensor's perspective, the spherical point cloud, color value, and sphere center coordinates of each calibration sphere under each depth sensor's perspective are extracted, and the spherical point clouds under the two depth sensor perspectives are matched with the same-name points.

[0012] Taking the coordinate system of one of the depth sensors as the reference, based on the spherical point cloud from the perspective of the two depth sensors after matching the same-name points, the relative position between the two depth sensors is solved, thereby obtaining the rotation and translation matrix T1 of the spherical calibration;

[0013] Step 2: Using the two depth sensors in step 1, collect a rough point cloud of the workpiece to be sprayed, which is placed at the workpiece placement position to be sprayed; based on the rotation and translation matrix T1 of the spherical calibration obtained in step 1, the rough point clouds of the workpiece to be sprayed collected by the two depth sensors are spliced to obtain a spliced rough point cloud;

[0014] Then, all surfaces of the stitched rough point cloud are obtained;

[0015] Next, based on the field of view of the movable binocular structured light device measuring the workpiece to be sprayed, a point cloud slicing algorithm is used to plan a three-dimensional measurement path of the binocular structured light device on each surface of the spliced rough point cloud; the binocular structured light device is mounted on a robotic arm, and the robotic arm drives the binocular structured light device to move;

[0016] Finally, the 3D measurement paths on each surface of the spliced rough point cloud are connected end to end to obtain the overall 3D measurement path of the workpiece to be sprayed;

[0017] Step 3: Using a robotic arm to drive the binocular structured light device to move along the overall three-dimensional measurement path obtained in step 2, and using a projector in the binocular structured light device to project prompt information on the surface of the workpiece to be sprayed to guide the pasting positions of each color mark point, and pasting each color mark point at the pasting position according to the prompt information;

[0018] Step 4: Using a robotic arm to drive the binocular structured light device to move along the overall three-dimensional measurement path obtained in step 2, and using the two cameras in the binocular structured light device to measure the surface of the workpiece to be sprayed, to obtain an accurate point cloud of the surface of the workpiece to be sprayed from the perspectives of the two cameras;

[0019] Using the color information of each colored marker point pasted in step 3 in the binocular structured light device, the precise surface point clouds of the two camera perspectives are spliced to obtain the precise point cloud of the workpiece surface to be sprayed, and the precise point cloud of the workpiece surface to be sprayed is converted into a point cloud under the robot arm base coordinate system.

[0020] In a further step 1, a random sampling consistency algorithm is used to extract the spherical point cloud, color value of the spherical point cloud, and sphere center coordinates of each calibration sphere under each depth sensor perspective based on the original point cloud data of each calibration sphere under each depth sensor perspective.

[0021] In a further step 1, the center coordinates of each calibration sphere under each depth sensor viewing angle are sorted in a set color order based on the color value of the spherical point cloud of each calibration sphere under the depth sensor viewing angle, thereby obtaining the center coordinates of each calibration sphere under each depth sensor viewing angle sorted in the color order, and the spherical point cloud corresponding to each center coordinate sorted in the color order;

[0022] Then, based on the spherical point clouds sorted in the color order under the two depth sensor perspectives, matching of points with the same name between the spherical point clouds under the two depth sensor perspectives is achieved.

[0023] In the further step 1, the SVD decomposition method is used to solve the relative pose between the two depth sensors based on the spherical point clouds under the perspectives of the two depth sensors after matching the same-name points.

[0024] In the further step 2, plane segmentation and region growing algorithm are used to obtain all surfaces of the spliced rough point cloud.

[0025] In a further step 2, based on the length and width of the field of view of the binocular structured light device, the slice width of the point cloud slicing algorithm and the distance between measurement points within the slice are determined, and then the point cloud slicing algorithm is used to plan each surface of the spliced rough point cloud to obtain the three-dimensional measurement path of the binocular structured light device.

[0026] In the further step 3, the color marker point pasting process is as follows:

[0027] Calibrate the eye-outside-hand coordinate system of the depth sensor as a reference, and solve the relative pose relationship between the depth sensor coordinate system as a reference and the robotic arm base coordinate system, thereby obtaining the eye-outside-hand rotation and translation matrix T2;

[0028] Based on the rotation and translation matrix T2 of the eye outside the hand, the overall three-dimensional measurement path obtained in step 2 is transformed into the overall three-dimensional measurement path in the robot arm base coordinate system;

[0029] The center of one of the cameras of the binocular structured light device is used as the reference coordinate system for the two-dimensional and three-dimensional imaging of the binocular structured light device, and the center of the structured light projector lens of the binocular structured light device is used as the tool center point TCP of the robot arm. The binocular structured light device is calibrated on the eye on the hand, and the relative position relationship between the coordinate system of the camera corresponding to the robot arm TCP in the binocular structured light device and the robot arm tool coordinate system is solved, thereby obtaining the eye on the hand rotation and translation matrix T4;

[0030] Based on the rotation and translation matrix T4 of the eye on the hand, the camera coordinate system used as the imaging reference in the binocular structured light device is transformed into the camera coordinate system under the robot arm tool coordinate system;

[0031] According to the overall three-dimensional measurement path transformed into the robot arm base coordinate system, the TCP of the robot arm is moved to the pasting position of each color mark point on the surface of the workpiece to be sprayed, and the projector of the binocular structured light device is used to project prompt information to the pasting position of each color mark point. Then, according to the prompt information of the pasting position of each color mark, the color mark points are pasted at the pasting position of each color mark point. Each color mark point contains the same multiple colors and has multiple corner points.

[0032] Furthermore, the process of step 4 is as follows:

[0033] The robot arm drives the binocular structured light device to move according to the overall three-dimensional measurement path obtained in step 2, and uses the two cameras in the binocular structured light device to measure the partial surface of the workpiece to be sprayed where each color mark point is located, thereby obtaining accurate point cloud data of the partial surface of the workpiece to be sprayed where each color mark point is located at each binocular structured light measurement point, and an image of the partial surface of the workpiece to be sprayed where each color mark point is located, wherein the description of the point cloud data is based on the camera coordinate system of the camera serving as the two-dimensional and three-dimensional imaging reference of the binocular structured light device, and the image is an image captured by the camera serving as the two-dimensional and three-dimensional imaging reference of the binocular structured light device;

[0034] From the texture mapping relationship, a one-to-one correspondence is obtained between the three-dimensional coordinate points in space and the two-dimensional coordinate points in the image under each binocular structured light measurement point;

[0035] For each partial surface where a color marker point is located, the robot arm TCP position when each partial surface where a color marker point is located is first photographed by the binocular structured light device is recorded respectively. The robot arm TCP position when each partial surface where a color marker point is located is first photographed serves as the reference for point cloud stitching of the precise point cloud of the partial surface where the corresponding marker point is located captured by the two cameras in the subsequent binocular structured light device. The rotation and translation transformation matrix T3 from the robot arm tool coordinate system to the robot arm base coordinate system corresponding to the partial surface where the corresponding color marker point is located is obtained from each point cloud stitching reference;

[0036] Obtain all corner point coordinate information of each color marker in the image at each binocular structured light measurement point;

[0037] Based on the characteristics that the color composition and the proportion of each color in the neighborhood of each corner point are different, multiple corner points of each color marker point at each binocular structured light measurement point are sorted according to the color composition and color proportion, and the corner point sorting information of each color marker point at two adjacent binocular structured light measurement points is obtained; based on the corner point sorting information of each color marker point, the color marker points with the same corner point sorting information at the two adjacent binocular structured light measurement points and the precise point cloud of the partial surface where the color marker points with the same corner point sorting information at the two adjacent binocular structured light measurement points are located are determined, and the precise point cloud of the partial surface where the color marker points with the same corner point sorting information at the two adjacent binocular structured light measurement points are located are matched with the same-named points, thereby achieving the matching of the same-named points of the precise point cloud of the partial surface where each color marker point at the two adjacent binocular structured light measurement points is located;

[0038] Based on the one-to-one correspondence between the three-dimensional coordinate points in space and the two-dimensional coordinate points in the image, the three-dimensional coordinates in space are obtained from the sequentially arranged image corner coordinates. SVD decomposition is used to solve the relative pose relationship of the point cloud data between two adjacent binocular structured light measurement points, and pixel-level precision point cloud stitching is completed to obtain the precise point cloud after stitching the partial surface where each color marker point of the workpiece to be sprayed is located.

[0039] Finally, the precise point cloud of the partial surface where each color mark point of the workpiece to be sprayed is located is spliced and converted into a point cloud under the robot arm base coordinate system.

[0040] Furthermore, the rotation and translation matrix T2 and the rotation and translation transformation matrix T3 of the eye outside the hand are used to convert the precise point cloud after splicing the partial surface where each color mark point of the workpiece to be sprayed is located into a point cloud under the robot arm base coordinate system.

[0041] Furthermore, the color marking points all have five colors: red, green, blue, purple and black.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. This invention uses a dual-depth sensor system to obtain a rough point cloud of the workpiece to be sprayed. After extracting each surface through plane segmentation and Euclidean clustering, it automatically plans a three-dimensional measurement path and guides the binocular structured light device to each planned measurement position to accurately acquire surface point cloud data. This can meet the needs of efficient and high-precision three-dimensional measurement of workpieces of different shapes, sizes, and types to be sprayed.

[0044] 2. The present invention uses calibration balls to calibrate the relative posture relationship between the dual depth sensors, and uses the color information of the calibration balls to complete the matching of homonymous points. There is no restriction on the positional relationship of the calibration balls. As long as the calibration balls appear in the public field of view of the dual depth sensors, the calibration balls can be placed arbitrarily, which improves the flexibility and efficiency of the deployment of the multi-depth sensor calibration system. The cost of the color information used is also relatively low, and the multi-depth sensor calibration system is optimized with almost zero hardware cost increment.

[0045] 3. The present invention designs colored marker points for point cloud stitching, which is suitable for planes, curved surfaces, etc. without significant features, and the stitching does not completely rely on the accuracy of the motion equipment. Stable corner point information is used as feature points to complete point cloud stitching, which reduces the difficulty of marker point detection and ensures the accuracy of point cloud stitching. The neighborhood color information of the corner points is used to complete the matching of points with the same name. There is no restriction on the positional relationship of the marker points. Each colored marker point provides multiple corner feature points. When the number of feature points used for matching is the same, the number of required colored marker points is less, which reduces the workload of pasting and removing marker points. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the overall steps of an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the overall structure of an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0048] Figure 3 This is a flowchart of sorting the centers of color-calibrated spheres for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0049] Figure 4 A schematic diagram of sphere center sorting for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0050] Figure 5 A schematic diagram of sphere center homonymous point matching for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0051] Figure 6 A schematic diagram of a rough point cloud of the surface of a workpiece to be sprayed and schematic diagrams of the point clouds of each segmented surface are provided for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0052] Figure 7 A schematic diagram of coordinate system transformation for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0053] Figure 8 A schematic diagram of a rough point cloud of the surface of a workpiece to be sprayed, which is an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0054] Figure 9 A schematic diagram of automatic scanning path planning for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0055] Figure 10 A schematic diagram of the structure of a binocular structured light system for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0056] Figure 11 This is a schematic diagram of the pasting position of structured light projection indicating marker points in an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0057] Figure 12 A schematic diagram of colored marker points of an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0058] Figure 13 A schematic diagram of the neighborhood information of color marker corner points of an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0059] Figure 14 This is a flowchart of sorting color marker points and corner points of an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0060] Figure 15 A schematic diagram of the sorting of colored marker corner points of an implementation method of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0061] Figure 16 This is a schematic diagram of matching color marker points with the same name in an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0062] Figure 17 This is a schematic diagram of color marker point cloud stitching for an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention.

[0063] Figure 18 A schematic diagram of an accurate point cloud of the surface of a workpiece to be sprayed, according to an implementation of a three-dimensional measurement and point cloud stitching method for intelligent spraying operations proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] To help those skilled in the art better understand the present invention, the following detailed description of the embodiments of the present invention is provided in conjunction with the accompanying drawings and examples. This will help those skilled in the art to fully understand and implement the present invention by applying technical means to solve technical problems and achieve corresponding technical effects. The embodiments of the present invention and the various features therein may be combined with each other as long as they do not conflict with each other, and the resulting technical solutions are all within the scope of protection of the present invention.

[0065] Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0066] It should be noted that the terms "include" and "have" in the specification, claims and drawings of the present invention and any variations thereof are intended to cover non-exclusive inclusions.

[0067] like Figure 1 As shown, this embodiment discloses a point cloud three-dimensional measurement and splicing method for intelligent spraying operation, comprising the following steps:

[0068] Step 1: Figure 2 As shown, a dual-depth sensor system 1 is constructed to collect raw point cloud data from three calibration spheres 2 of different colors. The calibration spheres 2 are used to calibrate the relative position relationship between the two depth sensors in the dual-depth sensor system 1. The depth sensor has a large measurement range but low measurement accuracy. It is suitable for obtaining a rough point cloud of the surface of the workpiece to be sprayed, providing visual guidance for subsequent precise three-dimensional measurement. The specific process is as follows:

[0069] (1.1) With the workpiece to be sprayed as the center, two depth sensors of model RealSenseD435i are arranged diagonally, marked as RealSense A11 and RealSense B12 respectively. The installation position, height and angle of the two depth sensors RealSense A11 and RealSense B12 are fixed. The two depth sensors RealSense A11 and RealSense B12 form a dual depth sensor system.

[0070] (1.2) Place three calibration balls 2 of different colors at the location where the workpiece to be sprayed is placed, namely a red calibration ball 21, a green calibration ball 22, and a blue calibration ball 23. As long as the three calibration balls can appear within the field of view of the two depth sensors, there are no other requirements for the position relationship of the three calibration balls.

[0071] (1.3) Use two depth sensors, RealSense A11 and RealSense B12, to collect the original point cloud data of the red calibration ball 21, the green calibration ball 22, and the blue calibration ball 23 placed at the position where the workpiece to be sprayed is placed.

[0072] The raw point cloud data collected by each depth sensor is preprocessed separately, including through-filtering and statistical filtering. Through-filtering is used to remove background points from the raw point cloud data, while statistical filtering is used to remove noise and outliers from the raw point cloud data.

[0073] (1.4) A random sampling consistency algorithm is used to extract the spherical point cloud, color value, and center coordinates of each calibration sphere under each depth sensor's perspective based on the original point cloud data of each calibration sphere under each depth sensor's perspective, and the spherical point clouds under the two depth sensor perspectives are matched with the same-name points.

[0074] Specifically, a random sampling consistency algorithm is used to extract the spherical point cloud of each calibration sphere based on the original point cloud data of each calibration sphere. Four non-coplanar points are randomly selected from the spherical point cloud of each calibration sphere as the starting internal point set. Based on these four points, the spherical model corresponding to each calibration sphere can be determined. The parameters include the sphere center C i (X i ,Y i ,Z i ) and radius R, as follows:

[0075] (x i -X i ) 2 +(y i -Y i ) 2 +(z i -Z i ) 2 =R 2 (1.1)

[0076] Where: X i is the horizontal coordinate of the center of the sphere, Y i is the vertical coordinate of the center of the sphere, Z i is the vertical coordinate of the center of the sphere, x i is the horizontal coordinate of a point on the sphere, y i is the ordinate of a point on the sphere, z i is the vertical coordinate of a point on the sphere.

[0077] Calculate the distance between the remaining points in the spherical point cloud of each calibration sphere and the spherical model. If the distance is less than the set threshold, the point is considered to be an internal point of the spherical model and added to the internal point set. Otherwise, the point is considered to be an external point and discarded. i (x i ,y i ,z i ) is a point in the spherical point cloud of each calibration ball, then the point p i (x i ,y i ,z i The distance d from the spherical model is calculated as follows:

[0078]

[0079] Traverse all points in the spherical point cloud of each calibration sphere. After completion, count the number of in-library points in the spherical model corresponding to each calibration sphere and record the spherical model and the number of in-library points. Repeat the random sampling consistency process until the set number of iterations is reached. Select the spherical model with the largest number of in-library points as the final spherical model for each calibration sphere. Use the final spherical model to obtain the center coordinates and radius of the corresponding calibration sphere, and extract the spherical point cloud data containing color information for the corresponding calibration sphere.

[0080] Since the center coordinates extracted from the spherical point cloud are in disorder, in order to solve the relative posture relationship, in this embodiment, the center coordinates of each calibration sphere under each depth sensor perspective are sorted in a set color order based on the color value of the spherical point cloud of each calibration sphere under the depth sensor perspective. Thus, the center coordinates of each calibration sphere sorted in the color order under each depth sensor perspective and the spherical point clouds corresponding to each center coordinate sorted in the color order are obtained, and the spherical point clouds with the same color sorting under the two depth sensor perspectives are matched with the same points, thereby completing the matching of the same points on the spherical point clouds under the two depth sensor perspectives.

[0081] Specific examples Figure 3 The process shown is based on the RGB average value R of each calibration sphere point cloud i ,G i ,B i Determine the spherical color of the calibration ball. If R i >G i And R i >B i , then the calibration ball is marked as "red", if G i >R i And G i >B i , then the calibration ball is marked as "green", if B i >R i And B i >G i , then the calibration ball is marked as "blue". When the color sequence is set to red, green, blue, such as Figure 4 As shown in the figure, the coordinates of the center of each calibration sphere from the perspective of the two depth sensors are sorted in red, green, and blue order, and the spherical point clouds corresponding to the coordinates of the center of each calibration sphere in this order are obtained. When matching homonymous points, the spherical point clouds of the red calibration sphere, the green calibration sphere, and the blue calibration sphere from the perspective of the two depth sensors are matched, thus completing the homonymous point matching of the spherical point clouds from the perspective of the two depth sensors.

[0082] (1.5) Taking the coordinate system of one of the depth sensors as the reference, this embodiment takes the depth sensor RealSense A11 coordinate system O1-X1Y1Z1 as the reference, such as Figure 5 As shown in the figure, the SVD decomposition method is used to solve the relative posture relationship between the two depth sensors based on the spherical point cloud under the perspective of the two depth sensors after matching the same-name points, thereby obtaining the rotation and translation matrix T1 of the spherical calibration.

[0083] Step 2: Use the dual depth sensor system 1 to collect point cloud information on the surface of the workpiece to be sprayed, placed at the workpiece placement location. Using the relative pose relationship between the two depth sensors, RealSense A11 and RealSense B12, and taking the coordinate system O1-X1Y1Z1 of one of the depth sensors, RealSense A11, as the reference, complete point cloud stitching to obtain a rough point cloud of the workpiece surface to be sprayed. Based on the rough point cloud information and the field of view of the binocular structured light device 4, the path for precise measurement by the binocular structured light 4 is automatically planned. The specific process is as follows:

[0084] (2.1) Using the two depth sensors RealSense A11 and RealSense B12 in step 1, a rough point cloud of the workpiece to be sprayed is collected at the workpiece placement position to be sprayed.

[0085] The collected rough point cloud is preprocessed, which includes straight-through filtering and statistical filtering. The straight-through filtering is used to remove the background point cloud in the rough point cloud, and the statistical filtering is used to remove the noise and outliers in the rough point cloud. The filtered point cloud P A like Figure 6 (a) shows the point cloud P from the perspective of the depth sensor RealSense B12. B like Figure 6 (b) shown.

[0086] (2.2), such as Figure 7 The coordinate system transformation relationship of the visual system is shown in FIG. 1 . Based on the rotation and translation matrix T1 of the ball calibration obtained in step 1, the rough point clouds of the workpiece to be sprayed collected by the two depth sensors are spliced to obtain the spliced rough point cloud. Figure 8 As shown in (a), the transformation relationship of the rough point cloud P1 of the workpiece surface to be sprayed in the RealSense A coordinate system O1-X1Y1Z1 of the depth sensor as the reference is as follows:

[0087] P1=T1P A +T1P B (1.3)

[0088] (2.3), such as Figure 8As shown in (b), plane segmentation and region growing algorithms are used to obtain all surfaces of the spliced rough point cloud. In this embodiment, a total of 5 surfaces are obtained.

[0089] like Figure 9 As shown in (a), based on the length and width of the field of view of the movable binocular structured light device 4 to be sprayed, the slice width of the point cloud slicing algorithm and the distance between the measurement points in the slice are determined, and then the point cloud slicing algorithm is used to plan each surface of the rough point cloud after splicing to obtain the three-dimensional measurement path of the binocular structured light device. Figure 9 As shown in (b), taking one of the planes and one curved surface as examples, the point cloud slicing algorithm is used on the surface to automatically plan the three-dimensional measurement path Q1 of the binocular structured light device.

[0090] In order to facilitate the subsequent calculation of the posture of the end of the robotic arm, the normal vector Save, the data format of the midpoint of the 3D measurement path Q1 is:

[0091] [x,y,z,nx,ny,nz](1.4)

[0092] Where: x, y, z are the horizontal and vertical coordinates of the path point, and nx, ny, nz are the normal vectors of the path point.

[0093] In this embodiment, the binocular structured light device 4 is installed on the robot arm 3, and the robot arm 3 drives the binocular structured light device 4 to move. Figure 10 As shown, the binocular structured light device 4 includes a left camera 41, a right camera 42, a left lens 43 corresponding to the left camera 41, a right lens 44 corresponding to the right camera 42, a projector 45, and an active light source 46. The binocular structured light device 4 is installed on the end flange of the robotic arm 3, and the robotic arm 3 is installed on the base.

[0094] (2.4) Repeat step (2.3) until the 3D measurement path planning of all surfaces of the spliced rough point cloud is completed. Connect the 3D measurement paths on each surface of the spliced rough point cloud end to end to obtain the overall 3D measurement path of the workpiece to be sprayed.

[0095] Step 3: Use the robotic arm 3 to move the binocular structured light device 4 along the overall three-dimensional measurement path obtained in step 2. Use the projector 45 in the binocular structured light device 4 to project prompt information onto the surface of the workpiece to be sprayed, indicating the location of each color marker. The colored markers are then affixed to each location according to the prompt information. The binocular structured light device has a small measurement range but high measurement accuracy, making it suitable for accurately measuring sprayed workpieces along the planned measurement path. The specific process is as follows:

[0096] (3.1) Calibrate the RealSense A11 depth sensor coordinate system as the reference with the eye outside the hand, and solve the relative position relationship between the depth sensor coordinate system O1-X1Y1Z1 as the reference and the robotic arm base coordinate system O3-X3Y3Z3, thereby obtaining the rotation and translation matrix T2 of the eye outside the hand.

[0097] (3.2), such as Figure 7 The visual system coordinate system transformation relationship shown in the figure is based on the rotation and translation matrix T2 of the eye outside the hand. The overall three-dimensional measurement path Q1 obtained in step 2 is transformed into the overall three-dimensional measurement path Q3 in the robot arm base coordinate system O3-X3Y3Z3, as shown in the following formula:

[0098] Q3=T2Q1(1.5)

[0099] (3.3) Take the center of one of the cameras of the binocular structured light device (the center of the left camera 41 in this embodiment) as the reference coordinate system for two-dimensional and three-dimensional imaging of the binocular structured light device, take the center of the structured light projector lens of the binocular structured light device as the tool center point TCP of the robot arm, calibrate the binocular structured light device on the eye, and solve the relative position relationship between the coordinate system O5-X5Y5Z5 of the left camera 41 of the binocular structured light device corresponding to the robot arm TCP and the robot arm tool coordinate system O4-X4Y4Z4, thereby obtaining the eye-on-hand rotation and translation matrix T4.

[0100] Based on the rotation and translation matrix T4 of the eye on the hand, the camera coordinate system used as the imaging reference in the binocular structured light device is transformed into the camera coordinate system under the robot arm tool coordinate system.

[0101] (3.4), such as Figure 11 As shown in (a), according to the overall three-dimensional measurement path transformed into the robot base coordinate system, the TCP of the robot is moved to the pasting position of each color mark point on the surface of the workpiece to be sprayed, and the projector of the binocular structured light device is made to project cross-line prompt information to the pasting position of each color mark point, and then the cross-line prompt information of each color mark pasting position is projected; Figure 11 As shown in (b), the colored marker points are pasted at the cross-line prompt information of the pasting position of each colored marker point.

[0102] Each color mark point contains the same multiple colors and has multiple corner points. In this embodiment, each color mark point is in the shape of a crosshair and has 12 corner points. Each color mark point has five colors: red, green, blue, purple, and black.

[0103] Step 4: Use the robotic arm 3 to drive the binocular structured light device 4 to move along the overall three-dimensional measurement path obtained in step 2, and use the two cameras 41 and 42 in the binocular structured light device 4 to measure the surface of the workpiece to be sprayed, obtaining an accurate point cloud of the surface of the workpiece to be sprayed from the perspectives of the two cameras 41 and 42. Then, using the color information of the various colored marker points pasted in step 3 in the binocular structured light device, the accurate surface point clouds from the two camera perspectives are spliced together to obtain an accurate spliced point cloud of the workpiece surface to be sprayed, and the accurate spliced point cloud of the workpiece surface to be sprayed is converted into a point cloud in the robotic arm base coordinate system. The specific process is as follows:

[0104] (4.1) The robot arm 3 drives the binocular structured light device 4 to move according to the overall three-dimensional measurement path obtained in step 2, and uses the two cameras 41 and 42 in the binocular structured light device to measure the partial surface of the workpiece to be sprayed where each color mark point is located, thereby obtaining accurate point cloud data of the partial surface of the workpiece to be sprayed where each color mark point is located at each binocular structured light measurement point, and a picture of the partial surface of the workpiece to be sprayed where each color mark point is located, wherein the description of the point cloud data is based on the left camera coordinate system, and the picture refers to the image obtained by the left camera.

[0105] From the texture mapping relationship, a one-to-one correspondence is obtained between the three-dimensional coordinate points in space and the two-dimensional coordinate points in the image under each binocular structured light measurement point.

[0106] For the partial surface where each color marker point is located, the robot arm TCP position when each color marker point is first photographed by the binocular structured light device is recorded respectively. The robot arm TCP position when each color marker point is first photographed is used as the reference for point cloud stitching of the precise point cloud of the partial surface where the corresponding marker point is located collected by the two cameras in the subsequent binocular structured light device. The rotation and translation transformation matrix T3 from the robot arm tool coordinate system O4-X4Y4Z4 to the robot arm base coordinate system O3-X3Y3Z3 corresponding to the partial surface where the corresponding color marker point is located is obtained from each point cloud stitching reference.

[0107] (4.2), such as Figure 12 As shown in the figure, each color marker has 12 corner points. Corner point detection is used to obtain the coordinate information of all corner points of each color marker in the image at each binocular structured light measurement point, and stable corner points are used as feature points for splicing the surface point clouds of each part.

[0108] (4.3), such as Figure 13As shown in the figure, the color markers include five colors: red, green, blue, purple, black and white. The RGB values of these five colors are quite different and easy to distinguish. The main structure of the color markers is four right-angled trapezoids. Red, green, blue and purple are the main structural colors, and black and white are the background colors. The color composition and proportion of each color in the neighborhood of each corner point are different.

[0109] Based on the characteristics that the color composition and proportion of each color in the neighborhood of each corner point are different, the multiple corner points of each color marker point under each binocular structured light measurement point are sorted according to the color composition and color proportion, and the corner point sorting information of each color marker point under two adjacent binocular structured light measurement points is obtained. According to the corner point sorting information of each color marker point, the color marker points with the same corner point sorting information under the two adjacent binocular structured light measurement points and the precise point cloud of the partial surface where the color marker points with the same corner point sorting information under the two adjacent binocular structured light measurement points are located are determined. Then, the precise point cloud of the partial surface where the color marker points with the same corner point sorting information under the two adjacent binocular structured light measurement points are located is matched with the same-named points, thereby achieving the matching of the same-named points of the precise point cloud of the partial surface where each color marker point under the two adjacent binocular structured light measurement points is located.

[0110] Taking red as an example, there are four neighborhoods containing red, of which two have two colors and two have four colors. Due to the different angles of the right trapezoid, the proportion of red in the neighborhood is different. In the neighborhood with two colors, one has a large proportion of red and the other has a small proportion. In the neighborhood with four colors, one has a large proportion of red and the other has a small proportion. The neighborhoods containing red are sorted as "two colors and a large proportion of red", "two colors and a small proportion of red", and "four colors and a large proportion of red". The discarded neighborhood will be classified into the sorting of other colors.

[0111] like Figure 14 As shown, extract the neighborhood image of the corner point and analyze the RGB value R of the pixel point i ,G i ,B i , for each set of values R in the existing color RGB value library j ,G j ,B j The following judgments are made:

[0112]

[0113] Where: C is the color tolerance, and two sets of RGB values with an error less than C are considered to be the same color.

[0114] If the operation results are greater than C, then R i , G i、B i Add the RGB value library of the existing color. If the calculation result is less than C, then the corresponding RGB value library of the existing color R j , G j 、B j The number of occurrences is incremented by 1. RGB values are quickly grouped using color tolerance. Each group records only one RGB value and the number of occurrences of the color representing that group. This avoids analyzing each pixel's RGB value to determine its color, improving the efficiency of neighborhood color analysis. Each representative RGB value is then analyzed to determine the colors within the neighborhood, reflecting the color composition. The number of repetitions of each color is counted to reflect the color ratio. Due to possible noise in the image acquisition, colors with low occurrences are ignored.

[0115] Sort the neighborhood by the color types and color occurrence counts it contains, and perform one-to-one correspondence between adjacent frame markers according to the sorting to complete the matching of points with the same name. The specific sorting order is as follows: Figure 15 shown.

[0116] (4.4), such as Figure 16 and 17 As shown, according to the one-to-one correspondence between the three-dimensional coordinate points in space and the two-dimensional coordinate points in the image, the three-dimensional coordinates in space arranged in sequence are obtained from the coordinates of the image corner points arranged in sequence. SVD decomposition is used to solve the relative pose relationship of the point cloud data between two adjacent binocular structured light measurement points, and the point cloud stitching with pixel-level accuracy is completed to obtain the precise point cloud after stitching of the partial surface where each color mark point of the workpiece to be sprayed is located.

[0117] (4.5) Repeat steps (4.1) to (4.4) until the three-dimensional measurement and splicing of the point clouds of all measurement path points are completed, and the accurate point cloud U of each surface of the workpiece to be sprayed is obtained. i .

[0118] like Figure 7 The coordinate system transformation relationship of the visual system shown in the figure uses the rotation and translation matrix T2 of the eye on the hand to perform posture transformation, transforming the precise point cloud after splicing the surface of the workpiece where the color marking points are located to the robot arm tool coordinate system O4-X4Y4Z4, and then using T3 to perform posture transformation, transforming the precise point cloud O4-X4Y4Z4 in the robot arm tool coordinate system to the robot arm base coordinate system O3-X3Y3Z3. The transformation relationship is as follows:

[0119]

[0120] Where: n is the number of surfaces of the workpiece to be sprayed, U is the precise point cloud of the entire workpiece to be sprayed in the robot base coordinate system O3-X3Y3Z3; T 3-iIt is the rotation and translation transformation matrix from the robot tool coordinate system to the robot base coordinate system obtained by the robot TCP position recorded by each surface when it is first photographed by the binocular structured light device.

[0121] like Figure 18 As shown, the final accurate point cloud U of the workpiece to be sprayed in the robot arm base coordinate system O3-X3Y3Z3 is obtained, which can be used for subsequent spraying path planning based on surface point cloud data.

[0122] The preferred embodiments of the present invention are described in detail above with reference to the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. Such combinations should also be regarded as the contents disclosed in this disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0123] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the scope of the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by those skilled in the art should fall within the scope of protection of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A point cloud three-dimensional measurement and splicing method for intelligent spraying operation, characterized in that: The following steps are involved: Step 1: Use two depth sensors to collect the original point cloud data of multiple calibration balls of different colors placed at the position of the workpiece to be sprayed; Based on the original point cloud data of each calibration sphere under each depth sensor's perspective, the spherical point cloud, color value, and sphere center coordinates of each calibration sphere under each depth sensor's perspective are extracted, and the spherical point clouds under the two depth sensor perspectives are matched with the same-name points. Taking the coordinate system of one of the depth sensors as the reference, based on the spherical point cloud from the perspective of the two depth sensors after matching the same-name points, the relative position between the two depth sensors is solved, thereby obtaining the rotation and translation matrix T1 of the spherical calibration; Step 2: Using the two depth sensors in step 1, collect a rough point cloud of the workpiece to be sprayed placed at the workpiece placement position to be sprayed; Based on the rotation and translation matrix T1 of the spherical calibration obtained in step 1, the rough point clouds of the workpiece to be sprayed collected by the two depth sensors are spliced to obtain a spliced rough point cloud; Then, all surfaces of the stitched rough point cloud are obtained; Next, based on the field of view of the movable binocular structured light device measuring the workpiece to be sprayed, a point cloud slicing algorithm is used to plan a three-dimensional measurement path of the binocular structured light device on each surface of the spliced rough point cloud; the binocular structured light device is mounted on a robotic arm, and the robotic arm drives the binocular structured light device to move; Finally, the 3D measurement paths on each surface of the spliced rough point cloud are connected end to end to obtain the overall 3D measurement path of the workpiece to be sprayed; Step 3: Using a robotic arm to drive the binocular structured light device to move along the overall three-dimensional measurement path obtained in step 2, and using a projector in the binocular structured light device to project prompt information on the surface of the workpiece to be sprayed to guide the pasting positions of each color mark point, and pasting each color mark point at the pasting position according to the prompt information; Step 4: Using a robotic arm to drive the binocular structured light device to move along the overall three-dimensional measurement path obtained in step 2, and using the two cameras in the binocular structured light device to measure the surface of the workpiece to be sprayed, to obtain an accurate point cloud of the surface of the workpiece to be sprayed from the perspectives of the two cameras; Using the color information of each colored marker point pasted in step 3 in the binocular structured light device, the precise surface point clouds of the two camera perspectives are spliced to obtain the precise point cloud of the workpiece surface to be sprayed, and the precise point cloud of the workpiece surface to be sprayed is converted into a point cloud under the robot arm base coordinate system.

2. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 1 is characterized in that: In step 1, a random sampling consistency algorithm is used to extract the spherical point cloud, color value of the spherical point cloud, and sphere center coordinates of each calibration sphere under each depth sensor's perspective based on the original point cloud data of each calibration sphere under each depth sensor's perspective.

3. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 1 is characterized in that: In step 1, the center coordinates of each calibration sphere under each depth sensor's viewing angle are sorted in a set color order based on the color value of the spherical point cloud of each calibration sphere under the depth sensor's viewing angle, thereby obtaining the center coordinates of each calibration sphere sorted in the color order under each depth sensor's viewing angle, and the spherical point cloud corresponding to each center coordinate sorted in the color order; Then, based on the spherical point clouds sorted in the color order under the two depth sensor perspectives, matching of points with the same name between the spherical point clouds under the two depth sensor perspectives is achieved.

4. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 1 is characterized in that: In step 1, the SVD decomposition method is used to solve the relative pose between the two depth sensors based on the spherical point cloud from the perspective of the two depth sensors after matching the same-name points.

5. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 1 is characterized in that: In step 2, plane segmentation and region growing algorithms are used to obtain all surfaces of the spliced rough point cloud.

6. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 1 is characterized in that: In step 2, based on the length and width of the field of view of the binocular structured light device, the slice width of the point cloud slicing algorithm and the distance between measurement points within the slice are determined, and then the point cloud slicing algorithm is used to plan each surface of the spliced rough point cloud to obtain the three-dimensional measurement path of the binocular structured light device.

7. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 1 is characterized in that: In step 3, the process of pasting the colored marker points is as follows: Calibrate the eye-outside-hand coordinate system of the depth sensor as a reference, and solve the relative pose relationship between the depth sensor coordinate system as a reference and the robotic arm base coordinate system, thereby obtaining the eye-outside-hand rotation and translation matrix T2; Based on the rotation and translation matrix T2 of the eye outside the hand, the overall three-dimensional measurement path obtained in step 2 is transformed into the overall three-dimensional measurement path in the robot arm base coordinate system; The center of one of the cameras of the binocular structured light device is used as the reference coordinate system for the two-dimensional and three-dimensional imaging of the binocular structured light device, and the center of the structured light projector lens of the binocular structured light device is used as the tool center point TCP of the robot arm. The binocular structured light device is calibrated on the eye on the hand, and the relative position relationship between the coordinate system of the camera corresponding to the robot arm TCP in the binocular structured light device and the robot arm tool coordinate system is solved, thereby obtaining the eye on the hand rotation and translation matrix T4; Based on the rotation and translation matrix T4 of the eye on the hand, the camera coordinate system used as the imaging reference in the binocular structured light device is transformed into the camera coordinate system under the robot arm tool coordinate system; According to the overall three-dimensional measurement path transformed into the robot arm base coordinate system, the TCP of the robot arm is moved to the pasting position of each color mark point on the surface of the workpiece to be sprayed, and the projector of the binocular structured light device is used to project prompt information to the pasting position of each color mark point. Then, according to the prompt information of the pasting position of each color mark, the color mark points are pasted at the pasting position of each color mark point. Each color mark point contains the same multiple colors and has multiple corner points.

8. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 7 is characterized in that: Step 4 process is as follows: The robot arm drives the binocular structured light device to move according to the overall three-dimensional measurement path obtained in step 2, and uses the two cameras in the binocular structured light device to measure the partial surface of the workpiece to be sprayed where each color mark point is located, thereby obtaining accurate point cloud data of the partial surface of the workpiece to be sprayed where each color mark point is located at each binocular structured light measurement point, and an image of the partial surface of the workpiece to be sprayed where each color mark point is located, wherein the description of the point cloud data is based on the camera coordinate system of the camera serving as the two-dimensional and three-dimensional imaging reference of the binocular structured light device, and the image is an image captured by the camera serving as the two-dimensional and three-dimensional imaging reference of the binocular structured light device; From the texture mapping relationship, a one-to-one correspondence is obtained between the three-dimensional coordinate points in space and the two-dimensional coordinate points in the image under each binocular structured light measurement point; For each partial surface where a color marker point is located, the robot arm TCP position when each partial surface where a color marker point is located is first photographed by the binocular structured light device is recorded respectively. The robot arm TCP position when each partial surface where a color marker point is located is first photographed serves as the reference for point cloud stitching of the precise point cloud of the partial surface where the corresponding marker point is located captured by the two cameras in the subsequent binocular structured light device. The rotation and translation transformation matrix T3 from the robot arm tool coordinate system to the robot arm base coordinate system corresponding to the partial surface where the corresponding color marker point is located is obtained from each point cloud stitching reference; Obtain all corner point coordinate information of each color marker in the image at each binocular structured light measurement point; Based on the characteristics that the color composition and the proportion of each color in the neighborhood of each corner point are different, multiple corner points of each color marker point at each binocular structured light measurement point are sorted according to the color composition and color proportion, and the corner point sorting information of each color marker point at two adjacent binocular structured light measurement points is obtained; based on the corner point sorting information of each color marker point, the color marker points with the same corner point sorting information at the two adjacent binocular structured light measurement points and the precise point cloud of the partial surface where the color marker points with the same corner point sorting information at the two adjacent binocular structured light measurement points are located are determined, and the precise point cloud of the partial surface where the color marker points with the same corner point sorting information at the two adjacent binocular structured light measurement points are located are matched with the same-named points, thereby achieving the matching of the same-named points of the precise point cloud of the partial surface where each color marker point at the two adjacent binocular structured light measurement points is located; Based on the one-to-one correspondence between the three-dimensional coordinate points in space and the two-dimensional coordinate points in the image, the three-dimensional coordinates in space are obtained from the sequentially arranged image corner coordinates. SVD decomposition is used to solve the relative pose relationship of the point cloud data between two adjacent binocular structured light measurement points, and pixel-level precision point cloud stitching is completed to obtain the precise point cloud after stitching the partial surface where each color marker point of the workpiece to be sprayed is located. Finally, the precise point cloud of the partial surface where each color mark point of the workpiece to be sprayed is located is spliced and converted into a point cloud under the robot arm base coordinate system.

9. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 8, characterized in that: Using the rotation and translation matrix T2 and the rotation and translation transformation matrix T3 of the eye outside the hand, the precise point cloud after splicing the partial surface where each color mark point of the workpiece to be sprayed is located is converted into a point cloud under the robot arm base coordinate system.

10. The point cloud three-dimensional measurement and splicing method for intelligent spraying operation according to claim 7 or 8, characterized in that: The color marking points all have five colors: red, green, blue, purple and black.

Images