Workpiece grabbing method and system

Abstract

The invention discloses a workpiece grabbing method and system. The method comprises the steps that point cloud data collected for a to-be-grabbed workpiece is obtained; segmenting the point cloud data to obtain a plurality of independent point cloud clusters corresponding to different workpieces; adopting an ISS algorithm to extract key points of the point cloud cluster and the workpiece template, and performing uniform sampling processing on the extracted key points; performing coarse registration on the point cloud cluster and the workpiece template by adopting an SAC-IA algorithm, and performing fine registration on the point cloud cluster and the workpiece template by adopting an ICP algorithm improved based on a KD tree algorithm to obtain a pose transformation relationship between the point cloud cluster and the workpiece template; according to the pose transformation relation, the pose of the workpiece template is transformed to determine the pose of the point cloud cluster; and a target point cloud cluster is determined, and the mechanical arm is controlled to grab the target workpiece according to the pose of the target point cloud cluster. According to the method, the workpiece can be accurately grabbed.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial automation, and more specifically, to a workpiece grasping method and system. Background Art

[0002] With the rapid development of industrial robots and machine vision industries, the degree of manual participation in industrial production has been greatly reduced, and the degree of automation of production lines has been improved. However, the working environment of workpiece grasping is complex and the working requirements are high, resulting in a low success rate and poor efficiency of automatic grasping of workpieces. The positioning accuracy is difficult to meet the requirements, which greatly increases the difficulty of robot grasping. Summary of the invention

[0003] A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0004] In view of the shortcomings of the prior art, a first aspect of an embodiment of the present invention provides a workpiece grasping method, comprising:

[0005] Acquire point cloud data collected for the workpiece to be grasped;

[0006] Segmenting the point cloud data to obtain a plurality of independent point cloud clusters, wherein different point cloud clusters correspond to different workpieces;

[0007] An internal morphological descriptor algorithm is used to extract key points of the point cloud cluster and key points of the workpiece template, and the extracted key points are uniformly sampled;

[0008] Based on the key points of the point cloud cluster and the key points of the workpiece template, a sampling consistency algorithm is used to roughly align the point cloud cluster with the workpiece template to obtain a first pose transformation relationship;

[0009] Based on the first pose transformation relationship, the point cloud cluster and the workpiece template are precisely aligned using a nearest neighbor iterative algorithm to obtain a second pose transformation relationship between the point cloud cluster and the workpiece template. During the precise alignment process, a KD tree algorithm is used to extract the nearest neighbor point pairs having a corresponding relationship between the point cloud cluster and the workpiece template;

[0010] transforming the pose of the workpiece template according to the second pose transformation relationship to determine the pose of the point cloud cluster;

[0011] A target point cloud cluster is determined from the plurality of point cloud clusters, and a robotic arm is controlled to grasp a target workpiece according to the position and posture of the target point cloud cluster, wherein the target workpiece corresponds to the target point cloud cluster.

[0012] In one embodiment, after controlling the robotic arm to grasp the target workpiece according to the position and posture of the target point cloud cluster, the method further includes:

[0013] Acquiring a two-dimensional image of the robotic arm and the target workpiece;

[0014] Determining the precise position and posture of the target workpiece according to the two-dimensional image;

[0015] The robot arm is adjusted according to the precise position and posture to place the target workpiece at a target position.

[0016] In one embodiment, segmenting the point cloud data to obtain a plurality of independent point cloud clusters includes:

[0017] In the point cloud data, edge points located in the contact area of ​​adjacent point cloud clusters are determined;

[0018] Eliminating the edge points, and clustering the point cloud data after eliminating the edge points to obtain a plurality of point cloud clusters; and,

[0019] The edge points are filled into the point cloud cluster.

[0020] In one embodiment, determining edge points located in contact areas of adjacent point cloud clusters includes:

[0021] For a first point cloud point in the point cloud data, calculating a mean value of normal angles of a plurality of point cloud points in a neighborhood of the first point cloud point;

[0022] If the average value of the normal angles is greater than a preset threshold, the first point cloud point is determined as an edge point.

[0023] In one embodiment, the step of using a nearest neighbor iterative algorithm to accurately align the point cloud cluster with the workpiece template based on the first posture transformation relationship includes:

[0024] The first posture transformation relationship is used as the initial posture of the nearest neighbor iteration algorithm to perform nearest neighbor iteration to obtain the second posture transformation relationship.

[0025] In one embodiment, the use of a sampling consistency algorithm to roughly align the point cloud cluster with the workpiece template to obtain a first pose transformation relationship includes:

[0026] Extracting key points to be registered in the workpiece template, and obtaining a fast point feature histogram feature descriptor of the key points to be registered;

[0027] Determining a target key point that matches the key point to be registered among the key points of the point cloud cluster according to a fast point feature histogram feature descriptor of the key point to be registered and a fast point feature histogram feature descriptor of the key point of the point cloud cluster;

[0028] The first posture transformation relationship is obtained according to the coordinates of the key points to be registered and the coordinates of the target key points.

[0029] In one embodiment, based on the first posture transformation relationship, the nearest neighbor iterative algorithm is used to accurately align the point cloud cluster with the workpiece template to obtain a second posture transformation relationship between the point cloud cluster and the workpiece template, including:

[0030] Performing a nearest neighbor search on the point cloud cluster and the workpiece template based on the KD tree algorithm to obtain a plurality of nearest neighbor point pairs having a corresponding relationship between the point cloud cluster and the workpiece template;

[0031] The first posture transformation relationship is iteratively corrected according to the distances between the multiple feature point pairs until the iteration error is less than a preset iteration error or the number of iterations reaches a preset number of iterations, so as to obtain the second posture transformation relationship.

[0032] In one embodiment, determining a target point cloud cluster from the plurality of point cloud clusters comprises:

[0033] The target point cloud cluster is determined according to the positions and postures of the plurality of point cloud clusters.

[0034] In one embodiment, determining a target point cloud cluster from the plurality of point cloud clusters comprises:

[0035] The target point cloud cluster is determined according to the number of point cloud points in the plurality of point cloud clusters.

[0036] In one embodiment, before segmenting the point cloud data, the method further includes preprocessing the point cloud data, wherein the preprocessing includes filtering out noise points and / or background points in the point cloud data.

[0037] In one embodiment, before segmenting the point cloud data, the method further includes:

[0038] Calibrate a point cloud data acquisition device to obtain a position and posture transformation relationship between a coordinate system of the point cloud data acquisition device and a world coordinate system, wherein the point cloud data acquisition device is used to acquire the point cloud data; and

[0039] The point cloud data acquisition device and the robotic arm are calibrated to obtain a posture transformation relationship between a coordinate system of the point cloud data acquisition device and a coordinate system of the robotic arm.

[0040] Another aspect of an embodiment of the present invention provides a workpiece grasping system, the workpiece grasping system comprising a point cloud data acquisition device, a mechanical arm and a control device, the control device connecting the point cloud data acquisition device and the mechanical arm;

[0041] The control device is used to execute the workpiece grasping method as described above, so as to control the robot arm to grasp the workpiece.

[0042] The workpiece grasping method and system proposed in the embodiments of the present invention can distinguish a single workpiece from multiple workpieces by dividing the point cloud data into single independent point cloud clusters, thereby facilitating the grasping of the workpieces one by one; by using an internal morphological descriptor algorithm to extract key points, using a sampling consistency algorithm to coarsely align the point cloud cluster with the workpiece template, and using a KD tree-improved nearest neighbor iterative algorithm to finely align the two, the accuracy and efficiency of the alignment can be improved, and the accurate position and posture of the point cloud cluster can be obtained according to the alignment results, so that the robotic arm can be controlled to accurately grasp the workpiece. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0044] Figure 1 is a schematic flow chart of a workpiece grasping method according to an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of point cloud data of a workpiece according to an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a workpiece identified from point cloud data according to an embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of a workpiece to be grasped according to an embodiment of the present invention;

[0048] Figure 5 FIG. 4 is a schematic block diagram of a workpiece grasping system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0050] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present application. However, it is apparent to those skilled in the art that the present application can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present application, some technical features well known in the art are not described.

[0051] It should be understood that the present application can be implemented in different forms and should not be interpreted as being limited to the embodiments set forth herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0052] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present application. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0053] In order to thoroughly understand the present application, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present application. The optional embodiments of the present application are described in detail as follows, but in addition to these detailed descriptions, the present application may also have other implementation methods.

[0054] The following describes the workpiece grasping method and system proposed in the embodiment of the present invention with reference to the accompanying drawings. Figure 1 , Figure 1 FIG. 1 is a schematic flow chart of a workpiece grasping method 100 according to an embodiment of the present invention. Figure 1 As shown, the workpiece grasping method 100 of the embodiment of the present invention includes the following steps:

[0055] In step S110, point cloud data collected for the workpiece to be grasped is obtained;

[0056] In step S120, the point cloud data is segmented to obtain a plurality of independent point cloud clusters, and different point cloud clusters correspond to different workpieces;

[0057] In step S120, an internal morphological descriptor algorithm is used to extract key points of the point cloud cluster and key points of the workpiece template, and the extracted key points are uniformly sampled;

[0058] In step S140, based on the key points of the point cloud cluster and the key points of the workpiece template, a sampling consistency algorithm is used to roughly align the point cloud cluster with the workpiece template to obtain a first pose transformation relationship;

[0059] In step S150, based on the first pose transformation relationship, the point cloud cluster and the workpiece template are precisely aligned using a nearest neighbor iterative algorithm to obtain a second pose transformation relationship between the point cloud cluster and the workpiece template. During the precise alignment process, a KD tree algorithm is used to extract the nearest neighbor point pairs having a corresponding relationship between the point cloud cluster and the workpiece template.

[0060] In step S160, the pose of the workpiece template is transformed according to the second pose transformation relationship to determine the pose of the point cloud cluster;

[0061] In step S170, a target point cloud cluster is determined from the plurality of point cloud clusters, and a robotic arm is controlled to grasp a target workpiece according to the position and posture of the target point cloud cluster, wherein the target workpiece corresponds to the target point cloud cluster.

[0062] The workpiece grasping method 100 of the embodiment of the present invention can distinguish a single workpiece from multiple stacked workpieces by dividing the point cloud data into single independent point cloud clusters, thereby facilitating the grasping of the workpieces one by one; by using an internal morphological descriptor algorithm to extract key points, using a sampling consistency algorithm to coarsely align the point cloud cluster with the workpiece template, and using a KD tree-improved nearest neighbor iterative algorithm to finely align the two, the accuracy and efficiency of the alignment can be improved, and the accurate position and posture of the point cloud cluster can be obtained according to the alignment result, so that the robotic arm can be controlled to accurately grasp the workpiece.

[0063] Exemplarily, before implementing the workpiece grasping method 100 of an embodiment of the present invention, it is first necessary to construct a workpiece grasping system. The workpiece grasping system includes at least a point cloud data acquisition device, a robotic arm, and a control device. Among them, the point cloud data acquisition device is used to collect point cloud data for the workpiece, and it can realize the collection of point cloud data based on stereo vision, structured light, line laser, etc. In one example, the point cloud data acquisition device can be a 3D camera such as a binocular structured light camera, a laser galvanometer stereo camera, etc. The control device is used to process the point cloud data, obtain the coordinates of the workpiece to be grasped, and send the coordinates to the robotic arm; the robotic arm is used to grasp the workpiece and move it to the target position.

[0064] In one embodiment, the workpiece grasping system also includes a 2D camera, which is arranged at the turntable station. After the robotic arm grasps the workpiece, it moves to the turntable station. The 2D camera captures the workpiece image and sends it to the control device. The control device completes the precise positioning of the workpiece, corrects the end posture of the robotic arm flange, and enables the robotic arm to accurately place the workpiece on the centering table positioning pin.

[0065] In addition, before executing step S110, the point cloud data acquisition device and the robotic arm need to be calibrated. The calibration of the point cloud data acquisition device is used to obtain the position and posture transformation relationship between the coordinate system of the point cloud data acquisition device and the world coordinate system, and the calibration of the point cloud data acquisition device and the robotic arm is used to obtain the position and posture transformation relationship between the coordinate system of the point cloud data acquisition device and the coordinate system of the robotic arm.

[0066] For example, taking a binocular structured light camera as an example, when calibrating a point cloud data acquisition device, the Zhang Zhengyou calibration method can be used to complete the calibration of camera-related parameters. The camera calibration principle is to obtain the internal and external parameters of the camera through rotation matrix constraints and optimization methods, and establish the transformation relationship between the world coordinate system and the camera coordinate system.

[0067] Specifically, a binocular structured light camera is used, and the mature calibration package in OpenCV is used to calibrate the camera. The relevant library functions are called to write a program for calibration. A 12×9 checkerboard calibration plate is used, the number of corner points is 11×8, and the side length of each square is 20mm. The calibration plate is placed under the camera, and the posture of the calibration plate is adjusted so that the left and right cameras simultaneously capture the image of the calibration plate. The left and right cameras capture a total of 24 calibration plate images. The calibration plate images taken by the left and right cameras are calibrated by single targets to obtain the parameters of each camera, and then the intrinsic calibration results of the two cameras are used to complete the binocular stereo vision calibration, thereby obtaining the rotation matrix and translation matrix from the left camera to the right camera.

[0068] When calibrating between the robot arm and the camera, the hand-eye calibration method with the eye outside the hand can be used, that is, the camera is fixed and does not move with the end of the robot arm, so as to obtain the transformation relationship between the camera coordinate system and the robot coordinate system.

[0069] Specifically, during calibration, a high-precision cone calibration block of a certain size is used as a hand-eye calibration target, and the cone calibration block is installed on the flange at the end of the robot through a connecting plate. During calibration, the robot is controlled to adjust the posture of the cone calibration block so that its cone vertex is vertically upward, the cone calibration block is moved to different positions in space and the robot coordinates are recorded. When the cone calibration block is moved, the position of the cone calibration block is distributed at different positions and heights within the field of view of the 3D camera. Next, the 3D camera is triggered by visual software to collect images at each cone calibration block position, and the image coordinates of the cone calibration block vertex are calculated and recorded using a host computer program written based on the three-dimensional software Halcon. Finally, the data is substituted into the robot to solve the hand-eye calibration matrix of the robot vision system.

[0070] In step S110, point cloud data collected for the workpiece to be grasped is obtained. Exemplarily, a material box containing the workpiece to be grasped is placed under a 3D camera, and the 3D camera is used to scan and obtain point cloud data of the workpiece. The workpiece to be grasped may be a stacked disordered workpiece. The 3D camera projects light with certain structural features onto the stacked workpieces through a near-infrared laser, and then a special infrared camera is used to collect the light. This light with a certain structure will reflect information with a structured light pattern due to the different depths of the workpiece. The change in this structure is converted into depth information through a computing unit to obtain point cloud data.

[0071] Optionally, the 3D camera can be a laser galvanometer stereo camera, which emits multiple laser lines to the galvanometer reflector through a multi-line laser and reflects them onto the workpiece to be measured. The laser lines scan the surface of the workpiece to obtain the RGB and depth information of the workpiece, and perform point cloud conversion on it to obtain three-dimensional point cloud data of the workpiece.

[0072] Since the collected point cloud data usually contains noise points and a large number of background points that are irrelevant to the target object (ie, workpiece), in order to obtain accurate point cloud data of the target workpiece, the original point cloud data needs to be preprocessed.

[0073] For example, based on the unique attributes of the point cloud data, such as coordinate information, color values, etc., a straight-through filtering algorithm can be used to set reasonable thresholds on the attributes of the point cloud points, retain the information of the point cloud points within the threshold, and delete the information of the point cloud points outside the threshold, thereby quickly removing a large number of invalid point clouds and narrowing the scope of searching for the target workpiece point cloud.

[0074] Optionally, a voxel network downsampling method of centroid points can be used. By building a 3D voxel network on the point cloud, the point cloud points closest to the centroid are found in the cell and retained, and other point cloud points in the cell are removed. This can effectively maintain the topological relationship between the point cloud data, maintain the surface features of the point cloud, and reduce the number of original point clouds.

[0075] Optionally, a radius filtering algorithm can also be used to set the filtering range with a certain point cloud point as the center. When the number of point cloud points in the filtering range is greater than a threshold, the point cloud point as the center is retained, otherwise the point cloud point is removed, thereby reducing the impact of outliers and noise on algorithm processing.

[0076] In the above preprocessing steps, the straight-through filtering method is used to remove a large number of invalid point clouds in the original point cloud data, narrowing the scope of searching for the target point cloud; the voxel filtering method is used to downsample on the basis of retaining the basic characteristics of the scene point cloud, reducing the density of the scene point cloud and the amount of calculation of the point cloud processing algorithm; the radius filtering method is used to eliminate the influence of outliers and noise on the algorithm processing. Figure 2 , which shows the pre-processed point cloud data.

[0077] Next, step S120 is executed to segment the point cloud data to obtain a plurality of independent point cloud clusters, where different point cloud clusters correspond to different workpieces. Figure 3 , which shows multiple artifacts identified from point cloud data.

[0078] Specifically, optimized Euclidean clustering can be used to segment the preprocessed point cloud data. When the workpieces are stacked and placed in disorder, there are a lot of occlusions and overlaps between the workpieces. Therefore, first, edge points located in the contact area of ​​adjacent point cloud clusters are determined in the point cloud data; then the edge points are removed, and the point cloud data after the edge points are removed is clustered to obtain multiple point cloud clusters; finally, the edge points are filled into the point cloud clusters.

[0079] Exemplarily, an edge point removal method based on normal angle can be used to separate the workpiece point cloud clusters that are adhered to each other in space, thereby completing the segmentation of point cloud clusters corresponding to different workpieces. For the first point cloud point in the point cloud data, calculate the mean of the normal angles of multiple point cloud points in its neighborhood; if the mean of the normal angles is greater than a preset threshold, the first point cloud point is determined as an edge point. Specifically, KD-Tree is used to establish the topological relationship of the point cloud data p, construct a workpiece point cloud cluster template, set the preset threshold δ of the mean normal angle in its neighborhood, and determine its neighborhood radius r, for any point cloud point p in the point cloud data p. i , calculate the mean normal angle δ of all point cloud points within the neighborhood radius of the point cloud point pi , if δ is satisfied pi>δ, then the point is defined as an edge point, otherwise it is a non-edge point. Finally, the edge points in the point cloud data are eliminated.

[0080] After removing the edge points, the remaining point cloud data is segmented based on Euclidean clustering. The essence of Euclidean clustering is to distinguish the distance of the neighborhood relationship of point cloud points, and use the distance between neighboring point cloud points as the judgment criterion to achieve clustering segmentation of point cloud. The main process of Euclidean clustering is: input point cloud data P, determine a query point P i , and set the threshold r, and use the KD-Tree nearest neighbor search method to find k P in the point cloud data i The neighbor point P j (j=1, 2, ..., k), and then calculate the Euclidean distance d between the neighboring point and the query point, and divide the points whose distance is less than the threshold into class M. When the number of points in class M no longer increases, the segmentation is completed, and the point cloud points in class M are a point cloud cluster. The calculation formula of the Euclidean distance d is:

[0081]

[0082] After Euclidean clustering, the point cloud data is clustered into multiple different point cloud clusters, each of which corresponds to an independent workpiece. However, each point cloud cluster has the problem of missing edges, so it is necessary to fill in the edge points that were previously removed into the point cloud cluster. For example, the edge points can be classified into different point cloud clusters by determining whether the distance from the edge point to its nearest neighbor in the point cloud cluster is less than a threshold. The edge filling algorithm steps include:

[0083] 1) Establish each point cloud cluster P in the point cloud data P cluster_j KD-Tree topological relationship;

[0084] 2) Set the point cloud cluster P cluster_j The threshold δ of the nearest neighbor distance;

[0085] 3) For any edge point, calculate the point to point cloud cluster Pc luster_j The Euclidean distance d to its nearest neighbor in the point cloud is defined as point cloud cluster P if d < δ. cluster_j The edge point is determined, otherwise the next edge point is determined;

[0086] 4) Repeat step 3 until all edge points are traversed and filled into the corresponding subset P cluster_j middle.

[0087] After filling the edge points, the segmentation of point cloud data of different workpieces is completed.

[0088] The point cloud segmentation method based on optimized Euclidean clustering in the embodiment of the present invention solves the problem that the traditional point cloud segmentation algorithm cannot effectively segment stacked and adhered workpiece targets, and can segment the preprocessed point cloud data into single independent point cloud clusters. On the premise of ensuring the accuracy of the segmentation results, the robustness of the segmentation is improved, and the accuracy of subsequent target workpiece recognition and posture judgment is also improved. However, it should be noted that in addition to the above segmentation method, other point cloud data segmentation algorithms can also be used to segment point cloud data. For example, deep learning network models such as PointNet model, PointNet++ model, etc. can be used to segment point cloud data.

[0089] After the point cloud data is divided into multiple point cloud clusters corresponding to different workpieces, the robot arm still cannot determine the specific coordinates of the workpiece to be grasped. In order to determine the accurate posture of the workpiece, the point cloud cluster can be registered with the workpiece template to obtain the posture change relationship between the point cloud cluster and the workpiece template, and the posture of the workpiece template can be transformed according to the posture transformation relationship obtained by registration to obtain the posture of the point cloud cluster. Among them, the posture of the workpiece template can be obtained in advance through calibration.

[0090] In the registration process, first, in step S130, the ISS (Internal Morphological Descriptor) algorithm is used to extract the key points of the workpiece template and the point cloud cluster, and the extracted key points are uniformly sampled. The ISS algorithm is a key point detection method based on the curvature and normal vector of the point cloud, which uses the inherent morphology of the point cloud point to determine whether the point cloud point is a key point. Uniform sampling processing is to uniformly select a part of the sample points from the key points to reduce the density of the key points and retain the key features.

[0091] Next, in step S140, based on the key points of the point cloud cluster and the key points of the workpiece template, the sample consistency (Sample Consensus Initial Alignment, SAC-IA) algorithm is used to coarsely align the point cloud cluster with the workpiece template to obtain the first pose transformation relationship. Coarse alignment can provide an initial pose transformation relationship with a large error for pose estimation, and also provide initial conditions for subsequent fine alignment.

[0092] Specifically, the key points to be registered in the workpiece template are first extracted, and the fast point feature histogram (FPFH) feature descriptors of the key points to be registered are obtained. For example, by setting a threshold d, n sampling points with a distance between each pair greater than d are selected from the workpiece template, i.e., the key points to be registered. When the FPFH algorithm is used to extract the FPFH feature descriptor, for each query point p in the point cloud cluster, s (For example, the key point to be registered mentioned above), calculates the distance between itself and the neighboring point p i The triplet parameters between are used to describe the query point p s and each of its neighboring points pi Normal deviation, generate feature histogram, and then use SPFH to p s The histogram of is weighted to calculate the FPFH feature description of each point cloud point in the point cloud cluster. The calculation formula is as follows:

[0093]

[0094] Among them, w i For p s to p i distance.

[0095] Next, according to the FPFH feature descriptors of the key points to be registered and the FPFH feature descriptors of the key points of the point cloud cluster, the target key points that match the key points to be registered are determined in the key points of the point cloud cluster. Specifically, the target key points can be selected based on the similarity of the FPFH features of the key points to be registered in the workpiece template and the key points in the point cloud cluster. After that, the three-dimensional rigid body transformation matrix between the workpiece template and the point cloud cluster is obtained according to the coordinates of the key points to be registered and the coordinates of the target key points, that is, the first pose transformation relationship. Furthermore, the quality of the registration can be judged according to the distance difference between the actual pose and the matched pose, and the error is calculated using the Huber penalty function. The above steps are repeated many times, and the rigid body transformation with the smallest error function value is selected as the optimal solution, which is the final rough registration result based on the SAC-IA algorithm.

[0096] Next, in step S150, based on the above-mentioned first pose transformation relationship, the point cloud cluster and the workpiece template are precisely aligned using the Iterative Closest Point (ICP) algorithm to obtain a second pose transformation relationship between the point cloud cluster and the workpiece template. Specifically, the first pose transformation relationship can be used as the initial pose of the ICP algorithm to perform the nearest neighbor iteration to obtain the second pose transformation relationship, thereby improving the iteration speed of the IPC algorithm.

[0097] The fine registration uses the ICP (Iterative Closest Point) nearest point iterative algorithm, which can accurately estimate the pose obtained by the rough registration, further reduce the pose estimation error, and improve the positioning accuracy. The ICP algorithm continuously selects corresponding points and calculates the corresponding rigid body transformation until the convergence meets the correct registration requirements. However, the ICP algorithm has a high error rate when searching for local optimal solutions, and it is easy to fall into the local optimal solution, and the original data structure is not conducive to processing point cloud data sets with large data volumes. In order to improve the limitations of the traditional ICP algorithm, the embodiment of the present invention optimizes its data structure and iteration efficiency so that it can meet the point cloud registration of various scenarios.

[0098] Specifically, under the premise of ensuring the registration accuracy, the KD tree (KD-Tree) is used to optimize the ICP algorithm. That is, in the process of precise registration, the KD tree algorithm is used to extract the nearest neighbor point pairs with corresponding relationships between the point cloud cluster and the workpiece template, thereby accelerating the global registration of the point cloud, optimizing the point cloud data structure, reducing the amount of algorithm iteration calculation, and more reasonably establishing the correspondence between the nearest neighbor points of the point cloud and planning the point cloud data structure. The central idea of ​​KD-Tree is to divide the entire space into several specific parts, and perform search and comparison operations in the specific space. The process of the ICP algorithm optimized by KD-Tree is as follows:

[0099] 1) Treat the registered point cloud cluster and artifact templates Construct KD-Tree to search for nearest neighbor points respectively, traverse P and Q, find multiple nearest neighbor point pairs with corresponding relationships between point cloud clusters and workpiece templates, and form corresponding registration point sets, set termination threshold τ and maximum number of iterations;

[0100] 2) Solve the rigid body transformation matrix of the registration point set to obtain the rotation matrix R and translation matrix T. The iterative error function formula is as follows:

[0101]

[0102] 3) Repeated iteration, that is, iteratively correct the first pose transformation relationship according to the distance between multiple feature point pairs. If the error of two iterations is less than the preset iteration error τ or the number of iterations reaches the maximum number of iterations N max , the iteration ends, and the last obtained posture transformation relationship is used as the second posture transformation relationship; if the termination condition is not met, the corresponding point set is updated and the above steps are executed again until the iteration termination condition is met.

[0103] After obtaining the second posture transformation relationship, step S160 can be executed to transform the posture of the workpiece template according to the second posture transformation relationship to obtain the posture of the point cloud cluster. Exemplarily, the posture obtained by the posture transformation step is the posture in the coordinate system of the point cloud data acquisition device; according to the posture transformation relationship between the coordinate system of the point cloud data acquisition device and the coordinate system of the robot arm, the posture of the workpiece in the coordinate system of the robot arm can be obtained.

[0104] The embodiment of the present invention uses the FPFH feature descriptor to establish corresponding point pairs for the key points of the workpiece template and the point cloud cluster, uses the SAC-IA coarse matching algorithm to obtain the initial rotation and translation matrix for pose estimation, and adopts the KD-Tree to optimize the ICP algorithm to improve the registration success rate, and reaches the termination condition with the least number of iterations, thereby improving the efficiency of precise registration, shortening the recognition and pose estimation time of the visual system, and indirectly reducing the application and maintenance costs of the visual system.

[0105] Next, step S170 is executed to determine a target point cloud cluster from the multiple point cloud clusters, and to control the robotic arm to grasp a target workpiece corresponding to the target point cloud cluster according to the position and posture of the target point cloud cluster.

[0106] In one embodiment, the poses of multiple point cloud clusters can be obtained based on the above-mentioned registration operation, and the target point cloud cluster can be determined according to the poses of the multiple point cloud clusters. Specifically, the pose of each point cloud cluster can be scored according to a preset scoring rule, and the workpiece corresponding to the point cloud cluster with the highest score can be grasped first. Scoring rules include, for example, the distance between the point cloud cluster and the robotic arm, whether the motion trajectory of the robotic arm collides with the material box, and the number of point cloud points in the point cloud cluster. Among them, since the number of point cloud points in the unobstructed point cloud cluster is larger, the target point cloud cluster can be determined according to the number of point cloud points in the multiple point cloud clusters, and the workpiece corresponding to the point cloud cluster with a larger number of point cloud points can be grasped first.

[0107] After determining the target workpiece, according to the position information of the target workpiece, the principle of the order of workpiece grabbing, and the result of the robot hand-eye calibration, the robot moves the gripper above the target workpiece and adjusts the end position. The robot successfully grabs the target workpiece and moves the robot to the transfer station. Specifically, the control device is connected to the robot control cabinet via Ethernet, and the Hilscher board is used to establish Profinet communication between the two. The robot control cabinet controls the end magnetic gripper to grab the target workpiece through I / O signals.

[0108] Furthermore, after the grasping is completed, the two-dimensional images of the robot arm and the target workpiece can be collected; the precise position and posture of the target workpiece can be determined based on the two-dimensional image; the robot arm can be adjusted based on the precise position and posture, and the target workpiece can be placed at the target position. The 3D visual processing described above can ensure that the workpiece is successfully grasped from the material box, and the 2D visual processing can further accurately locate the target workpiece, correct the end posture of the robot arm flange, and enable the robot arm to accurately place the target workpiece on the positioning pin of the centering table.

[0109] Specifically, Figure 4 Taking the workpiece of the hinge nut mounting plate on the side door as an example, the precise position and posture of the workpiece can be determined based on the two circular holes on the workpiece. For example, a 2000W camera and an 8mm lens are used to take pictures of the target workpiece on the gripper of the robotic arm, and the image distortion is removed by the undistort() function in OpenCV. The template matching method is used to select the two circular holes on the workpiece as feature points, and the feature point search area is divided and the feature point template is made. The feature point matching function in OpenCV is used to implement template matching of the feature point area to ensure that the feature point image is extracted.

[0110] Specifically, in order to reduce the impact of noise in the image, improve the signal-to-noise ratio, enhance image details, retain useful information in the image, and make image processing more accurate and stable, the median filtering algorithm can be used to filter the image and eliminate noise points in the image. The idea is to set the grayscale value of a pixel in the image to the median of the grayscale values ​​of the pixels in its neighborhood, and use the median filtering function cv2.mediaBlur() in OpenCV to smooth the image, thereby filtering to obtain a grayscale image.

[0111] Specifically, in order to further extract the positioning information in the image, the Canny operator edge detection algorithm is used to process the image and extract the contour information, which is implemented using the cv2.Canny() function in OpenCV, thereby obtaining a binary image that removes most of the irrelevant background information and accurately retains the edge of the circular hole of the feature point.

[0112] Specifically, the extracted circular contour is traversed, and an ellipse is fitted to the normal contour of length through the cv2.fitEllipse() function in OpenCV. According to the ratio of the major axis to the minor axis of the fitted ellipse, a reasonable threshold is set to filter the fitted ellipse, thereby completing the precise positioning of the feature points.

[0113] After completing the precise positioning of the target workpiece, the end position of the robot arm flange is corrected according to the positioning result, the robot arm motion trajectory is planned, the robot arm is moved, and the target workpiece is accurately placed on the positioning pin, thereby completing the placement task. Finally, the robot arm is controlled to return to the initial position to grab the next workpiece.

[0114] In summary, the workpiece grasping method 100 of the embodiment of the present invention can distinguish a single workpiece from multiple stacked workpieces by dividing the point cloud data into single independent point cloud clusters, thereby facilitating the grasping of the workpieces one by one; the point cloud cluster is aligned with the workpiece template using the SAC-IA algorithm and the ICP algorithm improved by the KD tree algorithm, and the position and posture of the point cloud cluster is obtained according to the alignment result, so as to control the robotic arm to accurately grasp the workpiece.

[0115] The embodiment of the present invention further provides a workpiece grasping system, which can be used to implement the workpiece grasping method 100 described above. Figure 5 , Figure 5 FIG. 5 is a schematic block diagram of a workpiece grasping system 500 according to an embodiment of the present invention.

[0116] like Figure 5As shown, the workpiece grasping system 500 includes a point cloud data acquisition device 510, a mechanical arm 520 and a control device 530, wherein the control device 530 is connected to the point cloud data acquisition device 510 and the mechanical arm 520; the control device 530 is used to execute the workpiece grasping method described above, so as to control the mechanical arm 520 to grasp the workpiece. Exemplarily, the control device 530 includes a memory, a processor and a computer program stored in the memory and running on the processor, and the workpiece grasping method 100 described above can be implemented when the processor executes the computer program.

[0117] In some embodiments, the workpiece grasping system 500 further includes an image acquisition device, which is connected to the controller 530. The image acquisition device is used to acquire a two-dimensional image of the robotic arm and the target workpiece after the robotic arm grasps the target workpiece; the control device 530 determines the precise position and posture of the target workpiece according to the two-dimensional image, and adjusts the robotic arm according to the precise position and posture, so as to place the target workpiece at the target position.

[0118] The workpiece grasping system according to the embodiment of the present invention is used to implement the workpiece grasping method 100 described above, and thus also has similar advantages.

[0119] The embodiment of the present invention further proposes a computer storage medium on which a computer program is stored. When the computer program is executed, the workpiece grasping method 100 of the embodiment of the present invention can be implemented. The computer storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0120] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0121] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0123] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0124] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0125] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0126] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0127] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some modules according to the embodiments of the present application. The application can also be implemented as a device program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0128] It should be noted that the above embodiments illustrate rather than limit the present application, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0129] The above is only a specific implementation method or an explanation of a specific implementation method of the present application, and the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A workpiece grasping method, characterized in that: The method comprises: Acquire point cloud data collected for the workpiece to be grasped; Segmenting the point cloud data to obtain a plurality of independent point cloud clusters, wherein different point cloud clusters correspond to different workpieces; An internal morphological descriptor algorithm is used to extract key points of the point cloud cluster and key points of the workpiece template, and the extracted key points are uniformly sampled; Based on the key points of the point cloud cluster and the key points of the workpiece template, a sampling consistency algorithm is used to roughly align the point cloud cluster with the workpiece template to obtain a first pose transformation relationship; Based on the first pose transformation relationship, the point cloud cluster and the workpiece template are precisely aligned using a nearest neighbor iterative algorithm to obtain a second pose transformation relationship between the point cloud cluster and the workpiece template. During the precise alignment process, a KD tree algorithm is used to extract the nearest neighbor point pairs having a corresponding relationship between the point cloud cluster and the workpiece template; transforming the pose of the workpiece template according to the second pose transformation relationship to determine the pose of the point cloud cluster; A target point cloud cluster is determined from the plurality of point cloud clusters, and a robotic arm is controlled to grasp a target workpiece according to the position and posture of the target point cloud cluster, wherein the target workpiece corresponds to the target point cloud cluster.

2. The workpiece grasping method according to claim 1, characterized in that: After controlling the robotic arm to grasp the target workpiece according to the position and posture of the target point cloud cluster, the method further includes: Acquiring a two-dimensional image of the robotic arm and the target workpiece; Determining the precise position and posture of the target workpiece according to the two-dimensional image; The robot arm is adjusted according to the precise position and posture to place the target workpiece at a target position.

3. The workpiece grasping method according to claim 1, characterized in that: The step of segmenting the point cloud data to obtain a plurality of independent point cloud clusters includes: In the point cloud data, edge points located in contact areas of adjacent point cloud clusters are determined; Eliminating the edge points, and clustering the point cloud data after eliminating the edge points to obtain a plurality of point cloud clusters; and, The edge points are filled into the point cloud cluster.

4. The workpiece grasping method according to claim 3, characterized in that: The step of determining edge points located in a contact area between adjacent point cloud clusters includes: For a first point cloud point in the point cloud data, calculating a mean value of normal angles of a plurality of point cloud points in a neighborhood of the first point cloud point; If the average value of the normal angles is greater than a preset threshold, the first point cloud point is determined as an edge point.

5. The workpiece grasping method according to claim 1, characterized in that: The step of using a nearest neighbor iterative algorithm to accurately align the point cloud cluster with the workpiece template based on the first posture transformation relationship includes: The first posture transformation relationship is used as the initial posture of the nearest neighbor iteration algorithm to perform nearest neighbor iteration to obtain the second posture transformation relationship.

6. The workpiece grasping method according to claim 1, characterized in that: The method of using a sampling consistency algorithm to roughly align the point cloud cluster with the workpiece template to obtain a first pose transformation relationship includes: Extracting key points to be registered in the workpiece template, and obtaining a fast point feature histogram feature descriptor of the key points to be registered; Determining a target key point that matches the key point to be registered among the key points of the point cloud cluster according to a fast point feature histogram feature descriptor of the key point to be registered and a fast point feature histogram feature descriptor of the key point of the point cloud cluster; The first posture transformation relationship is obtained according to the coordinates of the key points to be registered and the coordinates of the target key points.

7. The workpiece grasping method according to claim 1, characterized in that: Based on the first posture transformation relationship, the point cloud cluster and the workpiece template are precisely aligned using a nearest neighbor iterative algorithm to obtain a second posture transformation relationship between the point cloud cluster and the workpiece template, including: Performing a nearest neighbor search on the point cloud cluster and the workpiece template based on the KD tree algorithm to obtain a plurality of nearest neighbor point pairs having a corresponding relationship between the point cloud cluster and the workpiece template; The first posture transformation relationship is iteratively corrected according to the distances between the multiple feature point pairs until the iteration error is less than a preset iteration error or the number of iterations reaches a preset number of iterations, so as to obtain the second posture transformation relationship.

8. The workpiece grasping method according to claim 1, characterized in that: The step of determining a target point cloud cluster from among the plurality of point cloud clusters comprises: The target point cloud cluster is determined according to the positions and postures of the plurality of point cloud clusters.

9. The workpiece grasping method according to claim 1, characterized in that: The step of determining a target point cloud cluster from among the plurality of point cloud clusters comprises: The target point cloud cluster is determined according to the number of point cloud points in the plurality of point cloud clusters.

10. The workpiece grasping method according to claim 1, characterized in that: Before segmenting the point cloud data, the method further includes preprocessing the point cloud data, wherein the preprocessing includes filtering out noise points and / or background points in the point cloud data.

11. The workpiece grasping method according to claim 1, characterized in that: Before segmenting the point cloud data, the method further includes: Calibrate a point cloud data acquisition device to obtain a position and posture transformation relationship between a coordinate system of the point cloud data acquisition device and a world coordinate system, wherein the point cloud data acquisition device is used to acquire the point cloud data; and The point cloud data acquisition device and the robotic arm are calibrated to obtain a posture transformation relationship between a coordinate system of the point cloud data acquisition device and a coordinate system of the robotic arm.

12. A workpiece grasping system, characterized in that: The workpiece grasping system comprises a point cloud data acquisition device, a mechanical arm and a control device, wherein the control device is connected to the point cloud data acquisition device and the mechanical arm; The control device is used to execute the workpiece grasping method described in any one of claims 1 to 11 to control the robot arm to grasp the workpiece.

Images