A 6D depth pose estimation network system and a workpiece pose estimation method

Abstract

The present invention discloses a deep 6D pose estimation network system and a workpiece pose estimation method. The workpiece pose estimation method solves the problem of difficulty in obtaining large-volume point cloud datasets by generating a simulated dataset through a physical engine. The complete scene point cloud is directly used as input with pure geometric point cloud coordinates. The local and global features of the input point cloud can be extracted through the semantic and instance segmentation parts of the point cloud, thereby improving the network's ability to understand the scene. The pose estimation network with multi-layer feature fusion outputs accurate poses, which solves the problems of object stacking and self-occlusion to a certain extent, and is robust to various symmetrical objects. Experimental verification of datasets simulating real scenes shows that the method proposed in the present invention has obvious advantages in overall accuracy and stability, and has higher robustness.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and specifically provides a deep 6D pose estimation network system and a workpiece pose estimation method. Background Art

[0002] With the advancement of technology, robots are increasingly widely used in grasping, assembly, packaging, processing, logistics sorting, etc. Among them, grasping and assembly operations are the most common application scenarios of robots. A major challenge is the problem of accurate grasping of objects in complex environments. In traditional structured environments, although high accuracy and success rate can be achieved by manually finding teaching points in advance and programming them into programs, and then running them according to fixed programs, it is difficult to complete precise grasping and assembly operations in complex scenarios such as unstructured or semi-structured environments due to the lack of environmental perception and interaction.

[0003] A reliable robot grasping and assembly system needs to accurately obtain the position and posture of the target in the scene point cloud, that is, the 6D posture. Existing methods are mainly divided into category-level recognition and instance-level recognition. Category-level posture estimation means that the object recognition and posture estimation tasks can be completed even when objects of the same category have different shapes or when new objects of a certain category are added. On the contrary, instance-level recognition requires each category of objects to have a fixed shape, such as a CAD model.

[0004] Most industrial parts are rigid objects with standard production models and almost no intra-class changes, which reduces the difficulty of recognition. However, most parts in industrial scenes have weak textures and similar or identical colors. In this case, RGB color information becomes unreliable. Therefore, in order to improve the reliability of recognition, this paper starts with point clouds that only contain 3D geometric information, and mines the 6D pose of objects from the edge information and geometric relationships of objects. On the other hand, because the objects to be grasped are often in cluttered scenes, mutual occlusion and stacking between objects are inevitable, which still brings great difficulties to tasks such as robot grasping based on vision guidance. Summary of the invention

[0005] The purpose of the present invention is to solve the above-mentioned defects in the prior art, and a deep 6D pose estimation network system and a workpiece pose estimation method are designed; the network system combines a point cloud segmentation method and a point cloud pose estimation method to realize the pose estimation of multiple irregular workpieces in complex scenes.

[0006] The technical solution of the present invention to solve the technical problem is: design a deep 6D pose estimation network system, characterized in that the network system includes three submodules: a point cloud segmenter, a feature clustering sampler and a pose estimator:

[0007] The point cloud segmenter includes a feature extractor, a feature generator and a feature discriminator, wherein the feature extractor is composed of multiple SA layers, the feature generator is composed of multiple FP layers, and the feature discriminator is composed of two MLPs;

[0008] The multiple SA layers of the feature extractor perform multi-layer sampling on the input point cloud data to extract feature points. The input point cloud data is input to the last FP layer of the feature generator in addition to the first SA layer. The feature points extracted by the first SA layer are input to the second SA layer and the second to last FP layer of the feature generator. By analogy, the feature points extracted by each SA layer are input to the FP layer of the farthest unconnected SA layer in addition to the next SA layer. The feature points extracted by the last SA layer are input to the first FP layer. After the input N×3 point cloud data is processed by the feature extractor and feature generator, an N×128-dimensional feature vector is finally output.

[0009] The sizes of the two MLPs are [64, 128] and [64, 32], respectively. The output of the last FP layer of the feature generator is input into the two MLPs respectively to obtain the point-level high-dimensional instance features and semantic label predictions of the input point cloud data.

[0010] The feature clustering sampler is composed of a combination of a machine learning method based on density clustering and a sampling method. For the high-dimensional instance features generated by the point cloud segmenter, the feature clustering sampler clusters instance points with similar features to generate instances, and controls the number of points to a uniform value of M through the farthest point iterative downsampling method or the ball query upsampling method, and sends them to the next link in combination with the semantic label. Specifically, if the number of points of the generated instance is greater than M, the iterative farthest point sampling method is used; and when the number of points of the generated instance is less than M, the ball query method is used to divide the point cloud into N spherical areas around the center point, merge the points of the predicted instance with the input original point cloud, and search for the nearest original point within the query radius of each predicted point, and cyclically search until M points are found. For point clusters with too small a number, that is, points with less than 100 points, ball query and subsequent storage processing are not performed.

[0011] The posture estimator includes two posture feature extraction modules, which have the same structure. The M point instances and corresponding semantic labels output by the cluster sampler are input into the posture estimator, which processes them in two aspects: the first aspect is to directly transmit the M point instances and the corresponding semantic labels to the first posture feature extraction module, and obtain the predicted rotation vector of the point cloud instance with the semantic label through the first posture feature extraction module; the second aspect is to obtain the mean of the M point instances, and then subtract the mean of the M point instances from the M point instances and the corresponding semantic labels and transmit them to the second posture feature extraction module to obtain the predicted relative offset, and then add the relative offset to the mean of the M point instances to obtain the predicted translation vector of the point cloud instance with the semantic label.

[0012] The posture feature extraction module is based on the improved PointNet structure, and its working principle is as follows: first, the input point cloud instance is subjected to feature extraction through an MLP of size [64, 64, 128, 256], and then the multiple dimensional features extracted by each layer of the MLP are concatenated to obtain a 512-dimensional multi-level feature; then the 512-dimensional multi-level feature is input into a fully connected layer to obtain a 1024-dimensional multi-level feature, which is then input into the pooling layer to obtain the global feature; finally, the global feature is input into an MLP of size [512, 256, 3] to obtain the predicted rotation vector or translation vector of the semantically labeled point cloud instance.

[0013] Furthermore, the present invention provides a method for estimating the pose of a workpiece, characterized in that the method is based on the deep 6D pose estimation network system as described above, and comprises the following steps:

[0014] Step 1: Prepare point cloud data including instances and 6D pose labels:

[0015] According to the 3D model of the object to be identified, the physical engine is used to simulate the natural fall of the object from a random position at a fixed height, and the collision parameters with the ground and between objects are simulated. After the collision is stable, the point cloud is collected by the virtual camera, and point cloud data with point-level instance labels and accurate 6D poses are generated; finally, objects with severe occlusion in the data are eliminated, that is, the point cloud points of a class of objects are statistically sorted, and the 30% data with the least points are eliminated to obtain the point cloud data set of the network system; the point cloud data set of the network system is divided according to a certain ratio, and no less than 60% of it is divided into a training set, and the rest is a test set;

[0016] Step 2: Training of the deep 6D pose estimation network system

[0017] 2.1 Use the training set obtained in step 1 to train the point cloud segmenter and posture estimator of the deep 6D pose estimation network system respectively.

[0018] The point cloud data in the training set and the corresponding point-level semantic and instance segmentation labels are processed by the parameter-initialized point cloud segmenter to obtain point-level high-dimensional instance features and semantic label predictions, and then the loss functions generated by the semantic labels and instance features are calculated respectively, and the weighted sum of the two is obtained to obtain the loss function of the point cloud segmenter; according to the loss function value of the point cloud segmenter, back propagation is performed to complete a parameter update, and then the point cloud segmenter with updated parameters is used to process the point cloud data in the training set and the corresponding point-level semantic and instance segmentation labels again, and the loss function value of the point cloud segmenter corresponding to the current parameter value is used to continue back propagation to complete a parameter update, and the parameter update process is continuously repeated until the set number of cycles is reached to complete the parameter training of the point cloud segmenter.

[0019] The point cloud instances in the training set and the corresponding accurate 6D pose labels are input into the parameter-initialized pose estimator to obtain the predicted rotation vector and translation vector of the corresponding point cloud instance, and then the overall loss function of the pose estimator is calculated according to the rotation and translation loss function, and the overall loss function value of the pose estimator is back-propagated to achieve a parameter update of the pose estimator; then the point cloud instances in the training set and the corresponding accurate 6D pose labels are processed again by the pose estimator with updated parameters, and the overall loss function value of the pose estimator corresponding to the current parameter value is used to continue back-propagation to achieve a parameter update of the pose estimator, and the parameter update process is continuously repeated until the set number of cycles is reached to complete the parameter training of the pose estimator.

[0020] 2.2 Use the test set obtained in step 1 to test the deep 6D pose estimation network system

[0021] The point cloud data and the corresponding point-level semantic and instance segmentation labels in the test set in step 1 are input into the point cloud segmenter that has completed parameter training to obtain the point-level high-dimensional instance features and semantic labels of the predicted point cloud data. Then, the point-level high-dimensional instance features and semantic labels of the predicted point cloud data are input into the feature clustering sampler to obtain M point instances and corresponding semantic labels; finally, the M point instances and corresponding semantic labels output by the clustering sampler are input into the posture estimator that has completed parameter training to obtain the predicted rotation vector and translation vector of the corresponding point cloud instance.

[0022] The evaluation index of the object pose estimation error is the average point pair distance, which is as follows:

[0023]

[0024] Where M represents the point cloud set of the 3D model, m is the number of points, and x represents any point in the point cloud of the 3D model; 1 is any point in the point cloud of the object model, x 2 To transform the point cloud of the object model into the 6D pose predicted by the deep 6D pose estimation network system 1 The nearest point; R and t represent the real rotation matrix and translation vector respectively, and They represent the predicted rotation matrix and translation vector respectively; during evaluation, the predicted rotation vector is converted into a rotation matrix for evaluation.

[0025] The prediction quality evaluation of 6D pose estimation is expressed as the accuracy of the average point pair distance less than 10% of the diameter d of the corresponding model of each object; when the accuracy is greater than 90, the deep 6D pose estimation network system is an effective system.

[0026] Step 3: Workpiece pose estimation

[0027] The scene point clouds of multiple workpieces collected by the camera are input into the deep 6D pose estimation network system tested as valid in step 2. The scene point cloud is instance-segmented by the point cloud segmenter to obtain high-dimensional instance features of the entire scene point cloud. The features are sent to the feature clustering sampler to obtain the point cloud instances of each workpiece, which are then sent to the pose estimator to obtain high-precision 6D poses of each workpiece, which can be used in industrial sorting and grasping operations.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The method of the present invention applies the virtual data generation method to the 6D posture estimation problem of industrial grasping. It has the efficient detection effect of deep learning technology while alleviating the dependence of deep learning on large quantities of data. Under the premise of ensuring recognition accuracy and real-time performance, it does not require a large amount of labeled training data, reduces labor costs, and uses the instance segmentation method to separately segment the workpiece in advance for estimation. The trained model has better generalization ability for the recognition of various axisymmetric, centrally symmetric, and other irregular objects, and its performance is far superior to traditional machine vision and general posture estimation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the working principle of an embodiment of a deep 6D pose estimation network system of the present invention;

[0031] Figure 2 A schematic diagram of the working principle of a point cloud segmenter of a deep 6D pose estimation network system of the present invention;

[0032] Figure 3A schematic diagram of the working principle of a feature clustering sampler of a deep 6D pose estimation network system of the present invention;

[0033] Figure 4 A schematic diagram of the working principle of a posture estimator of a deep 6D posture estimation network system of the present invention;

[0034] Figure 5 This is a schematic diagram of the working principle of a posture feature extraction module of a posture estimator of a deep 6D posture estimation network system of the present invention. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The present invention provides a deep 6D pose estimation network system, which includes three submodules: a point cloud segmenter, a feature clustering sampler and a pose estimator:

[0037] The point cloud segmenter (see Figure 2 ) includes a feature extractor, a feature generator and a feature discriminator, wherein the feature extractor is composed of multiple SA (the abbreviation of SetAbstraction, a feature point sampling and feature extraction module, the same below) layers, the feature generator is composed of multiple FP (FeaturePropagation, a feature propagation module, which is a cross-layer connection to realize the generation of feature vectors corresponding to feature points, the same below) layers, and the feature discriminator is composed of two MLP (multi-layer perceptron, the same below);

[0038] The multiple SA layers of the feature extractor perform multi-layer sampling on the input point cloud data to extract feature points. The input point cloud data is input into the last FP layer of the feature generator in addition to the first SA layer. The feature points extracted by the first SA layer are input into the second SA layer and the second to last FP layer of the feature generator, and so on. The feature points extracted by each SA layer are input into the FP layer of the farthest unconnected SA layer in addition to the next SA layer. The feature points extracted by the last SA layer are input into the first FP layer.

[0039] The feature extractor and feature generator continuously sample feature points and extract high-dimensional features by jumping feature layers. The SA layer is used for point cloud sampling, point combination, and local feature extraction tasks at the combination points, while the FP layer is used to transfer multi-scale features and finally output the predicted features of each point. After the input N×3 point cloud data is processed by the feature extractor and feature generator, it finally outputs an N×128-dimensional feature vector.

[0040] The sizes of the two MLPs are [64, 128] and [64, 32], respectively. The output of the last FP layer of the feature generator is input into the two MLPs respectively to obtain the point-level high-dimensional instance features and semantic label predictions of the input point cloud data.

[0041] The loss function of the point cloud segmenter is the weighted sum of the loss functions of the two branches:

[0042] L=θ·L prediction +η·L embedding (1)

[0043] Where θ, η are equilibrium parameters, L prediction , L embedding Represent the loss functions of semantic label and instance feature generation, respectively, and the loss function L of the semantic label branch prediction is the cross entropy loss function, the loss function L of the instance feature generation branch embedding The definition is as follows:

[0044] L embedding =α·L inward +β·L outward +γ·L disperse (2)

[0045]

[0046]

[0047] Among them, α, β, and γ are weight parameters, which means that if there are C instances in the scene, N C is the cth one, μ C is the mean of the features of this instance; x i is the instance feature of the i-th point; L inward To calculate the loss of each instance point towards its own instance center point, the prediction of each instance point is close to its true center point; L outward To calculate the distance loss between the center points of each instance, increase the predicted distance between the center points of each instance; δ v and δ d For L inward and L outward The loss limit, L disperse is a pulling force that makes each point approach the origin to keep the activation bounded; during the training of the point cloud segmenter, the stochastic gradient descent method is used to make L inward , L outward , L disperse minimize.

[0048] The feature clustering sampler is composed of a combination of a machine learning method based on density clustering and a sampling method. For the high-dimensional instance features generated by the point cloud segmenter, the feature clustering sampler clusters instance points with similar features to generate instances, and controls the number of points to a uniform value of M (M=4096 in this patent, the same below) through the farthest point iterative downsampling method or the ball query upsampling method, and sends them to the next link in combination with the semantic label. Specifically, if the number of points of the generated instance is greater than M, the iterative farthest point sampling method is used. If the number of points of the generated instance is less than M, the ball query method is used to divide the point cloud into N spherical areas around the center point, merge the points of the predicted instance with the input original point cloud, and search for the nearest original point within the query radius of each predicted point, and cyclically search until M points are found. For point clusters that are too small, that is, points with less than 100 points, ball queries and subsequent storage processing are not performed. Its process structure is as follows Figure 3 As shown in the figure, first, for the high-dimensional features sent in, the MeanShift algorithm is used for feature clustering to generate point-by-point instance predictions. During the evaluation, although the same instance is predicted, the prediction numbers are inconsistent, resulting in a data association problem where the evaluation is incorrect. The KM algorithm is introduced, and its corresponding weight is set to the corresponding data variance of the point count statistical set of the predicted instance and the point count statistical set of the real instance. Its purpose is to make the two groups of points with closer points have higher connection weights, so that the predicted label of the same instance has the same value as the real label.

[0049] After clustering generates instances, since the number of points in each instance is different, they need to be sampled to a uniform value and sent to the subsequent network. Here, the farthest point sampling and ball query methods are used. If the number of points of the instance is greater than M, the FPS sampling method is used. Compared with other point cloud sampling methods, this method can better cover all points in the space and retain the characteristics of the source point cloud as much as possible, which is convenient for subsequent network analysis. When the number of points of the instance is less than M, the ball query method is used to divide the point cloud into N spherical areas around the center point, merge the points of the predicted instance with the input original point cloud, and search for the nearest original point within the query radius of each predicted point. The search is cyclic until M points are found, and the found original point set is sent as an instance for subsequent analysis.

[0050] The posture estimator includes two posture feature extraction modules, which have the same structure. The M point instances and corresponding semantic labels output by the cluster sampler are input into the posture estimator, which processes them in two aspects: the first aspect is to directly transmit the M point instances and the corresponding semantic labels to the first posture feature extraction module, and obtain the predicted rotation vector of the point cloud instance with the semantic label through the first posture feature extraction module; the second aspect is to obtain the mean of the M point instances, and then subtract the mean of the M point instances from the M point instances and the corresponding semantic labels and transmit them to the second posture feature extraction module to obtain the predicted relative offset (translation residual), and then add the relative offset to the mean of the M point instances to obtain the predicted translation vector of the point cloud instance with the semantic label.

[0051] The posture feature extraction module is based on the improved PointNet structure, and its working principle is as follows: first, the input point cloud instance is subjected to feature extraction through an MLP of size [64, 64, 128, 256], and then the multiple dimensional features extracted by each layer of the MLP are concatenated to obtain a 512-dimensional multi-level feature; then the 512-dimensional multi-level feature is input into a fully connected layer to obtain a 1024-dimensional multi-level feature, which is then input into the pooling layer to obtain the global feature; finally, the global feature is input into an MLP of size [512, 256, 3] to obtain the predicted rotation vector or translation vector of the semantically labeled point cloud instance.

[0052] When calculating the Loss, the network removes the mean of the input point cloud before calculating it, so the output translation is the translation residual, so the predicted translation vector The residual translation predicted by the network μ is obtained by averaging the input M×3 point cloud t Additive composition:

[0053]

[0054] Loss function for translation vector T is the real translation vector t and the predicted translation vector The second norm of the distance between:

[0055]

[0056] The loss function of the rotation vector r is to predict the rotation vector Convert the real rotation vector r to the corresponding rotation matrix After compositing with R, calculate the rotation difference between the two in three-dimensional space, that is, the geodesic distance:

[0057]

[0058] The overall loss function of the attitude estimator is obtained by weighting the loss functions of rotation and translation. Specifically, the translation loss function is scaled by a certain proportion and then added to the rotation loss function, as shown in formula (9):

[0059]

[0060] Wherein, α is a proportional constant, and its value range is 2.5 to 3.5. In this embodiment, the value is 3.

[0061] When training the posture estimator, the network parameters of the two posture feature extraction modules are adjusted so that the overall loss function of the posture estimator is minimized.

[0062] The main task of the posture estimator is to complete the 6D object posture estimation task of the input point cloud instance. Its purpose is to find the transformation relationship of the object to be estimated from the object coordinate system to the camera coordinate system, that is, the rotation and translation parameters, so as to perform subsequent processing, such as robotic arm grasping, etc., and the translation and rotation are relatively independent of each other. Among them, the translation parameter is generally represented by a 3×1 translation vector T, which represents the displacement along the three coordinate axes when the object is transformed, and the rotation specifies the rotation around the three coordinate axes. The present invention adopts the output method of the rotation vector in the network, and its specific structure is as follows Figure 4 As shown in the figure, the posture feature extraction module extracts features from the input point cloud through a multi-layer perceptron, and then concatenates features of multiple dimensions to obtain multi-level features of 512 dimensions, thereby enabling the network to learn and perceive multi-layer features of the object, and then increases the feature dimension to 1024 dimensions through a fully connected layer, and sends it to the pooling layer to obtain the global features, and then regresses the rotation or translation vector of the object through a multi-layer perceptron.

[0063] Furthermore, the present invention provides a method for estimating a workpiece pose, which is based on the deep 6D pose estimation network system as described above, and comprises the following steps:

[0064] Step 1: Prepare point cloud data including instances and 6D pose labels:

[0065] According to the 3D model of the object to be identified, the physical engine is used to simulate the natural fall of the object from a random position at a fixed height, and the collision parameters with the ground and between objects are simulated. After the collision is stable, the point cloud is collected by a virtual camera, and point cloud data with point-level instance labels and accurate 6D poses are generated. Finally, objects with severe occlusion in the data are eliminated, that is, the point cloud points of a class of objects are statistically sorted, and the 30% data with the least points are eliminated to obtain the point cloud data set of the network system; the point cloud data set of the network system is divided according to a certain ratio, and no less than 60% of it is divided into a training set, and the rest is a test set;

[0066] In this embodiment, the point cloud data set of the network system is obtained using the open source physics engine Blensor, and the specific process is as follows:

[0067] (1) Randomly import CAD mesh models of 4 to 7 repeatable parts, and randomly generate 6-DOF poses of each model above the preset grabbing plane to initialize the position of the parts. The plane to be photographed is a square area with a length and width of 256 mm, and collision baffles are set around it to prevent objects from falling from the plane and penetrating the mold, making it impossible to obtain accurate data.

[0068] (2) Add a physical engine to the model in the simulation scene, set the corresponding rigid body collision parameters and dynamic parameters, and retain the final state of the workpiece in free fall under the action of gravity to simulate the scattered and stacked placement of the workpiece on the plane.

[0069] (3) Add the sensor model, set the sensor resolution and focal length parameters, and add noise parameters to improve the similarity between the data set and the actual sensor data.

[0070] (4) Save the surface point cloud in the scene in the sensor coordinate system as a data set sample; save the 6-DOF posture information and point-level semantic information of each part as a data set label.

[0071] (5) For point clouds with multiple similar artifacts in each scene, the artifacts are separated based on semantic information, and the point cloud model is transformed to the actual position through the 6-DOF posture information. The KNN (K nearest neighbor) algorithm is used to obtain one or more points adjacent to each transformed point cloud model in the original scene and extract them as a single instance, thereby obtaining accurate instance labels for the entire scene point cloud.

[0072] (6) Labels of objects that are severely occluded or cannot be effectively presented in the scene point cloud due to collision or penetration are removed. The severe occlusion judgment threshold is 30% of the statistical mean of all generated object surface point clouds to ensure the reliability of recognition and grasping.

[0073] Step 2: Training of the deep 6D pose estimation network system

[0074] 2.1 Use the training set obtained in step 1 to train the point cloud segmenter and posture estimator of the deep 6D pose estimation network system respectively.

[0075] The point cloud data in the training set and the corresponding point-level semantic and instance segmentation labels are processed by the parameter-initialized point cloud segmenter to obtain point-level high-dimensional instance features and semantic label predictions, and then the loss functions generated by the semantic labels and instance features are calculated respectively, and the weighted sum of the two is obtained to obtain the loss function of the point cloud segmenter; according to the loss function value of the point cloud segmenter, back propagation is performed to complete a parameter update, and then the point cloud segmenter with updated parameters is used to process the point cloud data in the training set and the corresponding point-level semantic and instance segmentation labels again, and the loss function value of the point cloud segmenter corresponding to the current parameter value is used to continue back propagation to complete a parameter update, and the parameter update process is continuously repeated until the set number of cycles is reached to complete the parameter training of the point cloud segmenter.

[0076] The point cloud instances in the training set and the corresponding accurate 6D pose labels are input into the parameter-initialized pose estimator to obtain the predicted rotation vector and translation vector of the corresponding point cloud instance, and then the overall loss function of the pose estimator is calculated according to the rotation and translation loss function, and the overall loss function value of the pose estimator is back-propagated to achieve a parameter update of the pose estimator; then the point cloud instances in the training set and the corresponding accurate 6D pose labels are processed again by the pose estimator with updated parameters, and the overall loss function value of the pose estimator corresponding to the current parameter value is used to continue back-propagation to achieve a parameter update of the pose estimator, and the parameter update process is continuously repeated until the set number of cycles is reached to complete the parameter training of the pose estimator.

[0077] 2.2 Use the test set obtained in step 1 to test the deep 6D pose estimation network system

[0078] The point cloud data and corresponding point-level semantic and instance segmentation labels in the test set in step 1 are input into the point cloud segmenter that has completed parameter training to obtain the point-level high-dimensional instance features and semantic labels of the predicted point cloud data. Then, the point-level high-dimensional instance features and semantic labels of the predicted point cloud data are input into the feature clustering sampler to obtain M point instances and corresponding semantic labels; finally, the M point instances and corresponding semantic labels output by the clustering sampler are input into the posture estimator that has completed parameter training to obtain the predicted rotation vector and translation vector of the corresponding point cloud instance.

[0079] The commonly used evaluation index for the error in object pose estimation is the average point pair distance, which is the average Euclidean distance of the point pairs after the 3D model point cloud is transformed into the real pose and the predicted pose. Since the correspondence between the point pairs of the 3D model point cloud after the real pose transformation and the predicted pose transformation of the symmetrical object is uncertain, the calculation of the average point pair distance is also divided into AD for asymmetrical objects and AD-S for symmetrical objects to distinguish between asymmetrical objects, as shown below:

[0080]

[0081] Where M represents the point cloud set of the 3D model, m is the number of points, and x represents any point in the point cloud of the 3D model; 1 is any point in the point cloud of the object model, x 2 To transform the point cloud of the object model into the 6D pose predicted by the deep 6D pose estimation network system 1 The nearest point; R and t represent the real rotation matrix and translation vector respectively, and They represent the predicted rotation matrix and translation vector respectively; during evaluation, the predicted rotation vector is converted into a rotation matrix for evaluation.

[0082] The prediction quality evaluation of 6D pose estimation is usually expressed as the metric accuracy of the average point pair distance (AD or AD-S value) being less than 10% of the diameter d of the corresponding model of each object. That is, when the average point pair distance is less than 0.1 times the diameter d of the minimum circumscribed circle of the object model, the 6D pose estimation of the object is successful; when the accuracy is greater than 90, the deep 6D pose estimation network system is an effective system.

[0083] The prediction method of the deep 6D pose estimation network system (abbreviated as Ours+ICP) proposed in the present invention is compared with the other three methods, namely FPFH+RANSAC+ICP (abbreviated as FPFH+ICP), PPF+3D-Hough+ICP (abbreviated as PPF+ICP) and CloudPose+ICP. The first two are common recognition schemes in traditional registration methods based on artificial descriptors, and the third is based on pose estimation using deep learning methods directly on point cloud data. In order to comprehensively compare the performance of the network system proposed in the present invention, the four methods also differ in the form of input data. FPFH+RANSAC+ICP and CloudPose+ICP are recognized on point cloud blocks that have been segmented under real labels, and the semantics corresponding to the point cloud blocks are known, while PPF+3D-Hough+ICP and the network system proposed in the present invention both make pose estimates of multiple objects on the entire scene point cloud. The four prediction methods use the same data set.

[0084] Table 1 Accuracy of four prediction methods

[0085]

[0086] The accuracy of the four prediction methods is shown in Table 1. According to the prediction results, except for the slightly higher accuracy PPF+ICP of object D (asymmetric object) and object F (symmetric object) (all within 0.3), the accuracy of the remaining objects proposed by the network system of the present invention has obvious advantages (the maximum accuracy improvement is about 25, which is about 10 compared with the best of other methods, and the bold font is the type corresponding to the object). From the results, the prediction effect of the network system proposed by the present invention is more stable.

[0087] Step 3: Workpiece pose estimation

[0088] The scene point clouds of multiple workpieces collected by the camera are input into the deep 6D pose estimation network system tested as valid in step 2. The scene point cloud is instance-segmented by the point cloud segmenter to obtain high-dimensional instance features of the entire scene point cloud. The features are sent to the feature clustering sampler to obtain the point cloud instance of each workpiece, which is then sent to the pose estimator to obtain a high-precision 6D pose of each workpiece (composed of a three-dimensional rotation vector and a three-dimensional translation vector). This pose can be used in industrial sorting and grasping operations.

[0089] As an embodiment, when the deep 6D pose estimation network system is trained, the number of data sets used is configured as shown below.

[0090] Table 2 Basic training configuration table

[0091]

[0092] The computer configuration used in this embodiment is a Core i9-9820X ​​CPU, 32GB of memory, and a single NVIDIA GeForceRTX 2080ti graphics card. The test results show that the semantic segmentation accuracy of the point cloud segmentor for the training set is not less than 98%, and the instance segmentation accuracy is not less than 92%. After the point cloud segmentor outputs high-dimensional instance features, it is processed by the feature clustering sampler and then outputs the instance, which is then sent to the posture estimator. The results show that the network system has a recognition accuracy of not less than 96% for asymmetric objects and a recognition accuracy of not less than 95% for symmetric objects.

[0093] In this embodiment, a total of seven different objects were tested for 6D posture recognition, including axisymmetric objects, centrally symmetrical objects, and other irregular objects of different sizes. The method of the present invention can accurately identify the 6D posture of the object, and provides a low-cost solution for the application of deep learning methods in industry, and can be used in industrial sorting, grasping and assembly, and other links.

[0094] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0095] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A deep 6D pose estimation network system, characterized in that: The network system consists of three sub-modules: point cloud segmenter, feature clustering sampler and posture estimator: The point cloud segmenter includes a feature extractor, a feature generator and a feature discriminator, wherein the feature extractor is composed of multiple SA layers, the feature generator is composed of multiple FP layers, and the feature discriminator is composed of two MLPs; The multiple SA layers of the feature extractor perform multi-layer sampling on the input point cloud data to extract feature points. The input point cloud data is input to the last FP layer of the feature generator in addition to the first SA layer. The feature points extracted by the first SA layer are input to the second SA layer and the second to last FP layer of the feature generator. By analogy, the feature points extracted by each SA layer are input to the FP layer farthest from the unconnected SA layer in addition to the next SA layer. The feature points extracted by the last SA layer are input to the first FP layer. After the input N×3 point cloud data is processed by the feature extractor and the feature generator, an N×128-dimensional feature vector is finally output. The sizes of the two MLPs are [64, 128] and [64, 32] respectively, and the output of the last FP layer of the feature generator is input into the two MLPs respectively to obtain the point-level high-dimensional instance features and semantic label predictions of the input point cloud data respectively; The feature clustering sampler is composed of a combination of a machine learning method based on density clustering and a sampling method. For the high-dimensional instance features generated by the point cloud segmenter, the feature clustering sampler clusters instance points with similar features to generate instances, and controls the number of points to a uniform value of M through the farthest point iterative downsampling method or the ball query upsampling method, and sends them to the next link in combination with the semantic label; specifically, if the number of points of the generated instance is greater than M, the iterative farthest point sampling method is used; and when the number of points of the generated instance is less than M, the ball query method is used to divide the point cloud into N spherical areas around the center point, merge the points of the predicted instance with the input original point cloud, and search for the nearest original point within the query radius of each predicted point, and cyclically search until M points are found; for point clusters with too small a number, that is, points with less than 100 points, ball query and subsequent storage processing are not performed; The posture estimator comprises two posture feature extraction modules, which have the same structure. The M point instances and corresponding semantic labels output by the cluster sampler are input into the posture estimator, which processes them in two aspects: the first aspect is to directly transmit the M point instances and corresponding semantic labels to the first posture feature extraction module, and obtain the predicted rotation vector of the point cloud instance with semantic labels through the first posture feature extraction module; the second aspect is to obtain the mean of the M point instances, and then transmit the M point instances and corresponding semantic labels minus the mean of the M point instances to the second posture feature extraction module to obtain the predicted relative offset, and then add the relative offset to the mean of the M point instances to obtain the predicted translation vector of the point cloud instance with semantic labels; The posture feature extraction module is based on the improved PointNet structure, and its working principle is as follows: first, the input point cloud instance is subjected to feature extraction through an MLP of size [64, 64, 128, 256], and then the multiple dimensional features extracted by each layer of the MLP are concatenated to obtain a 512-dimensional multi-level feature; then the 512-dimensional multi-level feature is input into a fully connected layer to obtain a 1024-dimensional multi-level feature, which is then input into the pooling layer to obtain the global feature; finally, the global feature is input into an MLP of size [512, 256, 3] to obtain the predicted rotation vector or translation vector of the semantically labeled point cloud instance.

2. A deep 6D pose estimation network system according to claim 1, characterized in that: The loss function of the point cloud segmenter is the weighted sum of the loss functions of the two branches: L=θ·Lprediction +η·Lembedding (1) Where θ, η are equilibrium parameters, L prediction , L embedding Represent the loss functions of semantic label and instance feature generation, respectively, and the loss function L of the semantic label branch prediction is the cross entropy loss function, the loss function L of the instance feature generation branch embedding The definition is as follows: L embedding =α·L inward +β·L outward +γ·L disperse (2) Among them, α, β, and γ are weight parameters, which means that if there are C instances in the scene, N C is the cth one, μ C is the mean of the features of this instance; x i is the instance feature of the i-th point; L inward To calculate the loss of each instance point towards its own instance center point, the prediction of each instance point is close to its true center point; L outward To calculate the distance loss between the center points of each instance, increase the predicted distance between the center points of each instance; δ v and δ d For L inward and L outward The loss limit, L disperse is a pulling force that makes each point approach the origin to keep the activation bounded; during the training of the point cloud segmenter, the stochastic gradient descent method is used to make L inward , L outward , L disperse minimize.

3. A deep 6D pose estimation network system according to claim 1, characterized in that: The number of points in the feature clustering sampler is M=4096.

4. A deep 6D pose estimation network system according to claim 1, characterized in that: When the attitude estimator calculates the loss, the network removes the input point cloud mean before calculating, so the output translation is the translation residual, so the predicted translation vector The residual translation predicted by the network μ is obtained by averaging the input M×3 point cloud t Additive composition: Loss function for translation vector T is the real translation vector t and the predicted translation vector The second norm of the distance between: The loss function of the rotation vector r is to predict the rotation vector Convert the real rotation vector r to the corresponding rotation matrix After compositing with R, calculate the rotation difference between the two in three-dimensional space, that is, the geodesic distance: The overall loss function of the attitude estimator is obtained by weighting the loss functions of rotation and translation. Specifically, the translation loss function is scaled by a certain proportion and then added to the rotation loss function, as shown in formula (9): Wherein, α is a proportional constant, the value range is 2.5 to 3.5, and the value in this embodiment is 3; When training the posture estimator, the network parameters of the two posture feature extraction modules are adjusted so that the overall loss function of the posture estimator is minimized.

5. A method for estimating a workpiece posture, characterized in that: The method is based on the deep 6D pose estimation network system according to any one of claims 1 to 4, and comprises the following steps: Step 1: Prepare point cloud data including instances and 6D pose labels: According to the 3D model of the object to be identified, the physical engine is used to simulate the natural fall of the object from a random position at a fixed height, and the collision parameters with the ground and between objects are simulated. After the collision is stable, the point cloud is collected by the virtual camera, and point cloud data with point-level instance labels and accurate 6D poses are generated; finally, objects with severe occlusion in the data are eliminated, that is, the point cloud points of a class of objects are statistically sorted, and the 30% data with the least points are eliminated to obtain the point cloud data set of the network system; the point cloud data set of the network system is divided according to a certain ratio, and no less than 60% of it is divided into a training set, and the rest is a test set; Step 2: Training of the deep 6D pose estimation network system 2.1 Use the training set obtained in step 1 to train the point cloud segmenter and attitude estimator of the deep 6D pose estimation network system respectively; The point cloud data in the training set and the corresponding point-level semantic and instance segmentation labels are processed by the parameter-initialized point cloud segmenter to obtain point-level high-dimensional instance features and semantic label predictions, and then the loss functions generated by the semantic labels and instance features are calculated respectively, and the weighted sum of the two is obtained to obtain the loss function of the point cloud segmenter; according to the loss function value of the point cloud segmenter, back propagation is performed to complete a parameter update, and then the point cloud segmenter with updated parameters is used to process the point cloud data in the training set and the corresponding point-level semantic and instance segmentation labels again, and the loss function value of the point cloud segmenter corresponding to the current parameter value is used to continue back propagation to complete a parameter update, and the parameter update process is continuously repeated until the set number of cycles is reached to complete the parameter training of the point cloud segmenter; The point cloud instances in the training set and the corresponding accurate 6D pose labels are input into the parameter-initialized pose estimator to obtain the predicted rotation vector and translation vector of the corresponding point cloud instance, and then the overall loss function of the pose estimator is calculated according to the loss function of rotation and translation, and the overall loss function value of the pose estimator is back-propagated to achieve a parameter update of the pose estimator; then the pose estimator with updated parameters is used to process the point cloud instances in the training set and the corresponding accurate 6D pose labels again, and the overall loss function value of the pose estimator corresponding to the current parameter value is used to continue back-propagation to achieve a parameter update of the pose estimator, and the parameter update process is continuously repeated until the set number of cycles is reached to complete the parameter training of the pose estimator; 2.2 Use the test set obtained in step 1 to test the deep 6D pose estimation network system The point cloud data in the test set obtained in step 1 and the corresponding point-level semantic and instance segmentation labels are input into the point cloud segmenter that has completed parameter training to obtain the point-level high-dimensional instance features and semantic labels of the predicted point cloud data; then the point-level high-dimensional instance features and semantic labels of the predicted point cloud data are input into the feature clustering sampler to obtain M point instances and corresponding semantic labels; finally, the M point instances and corresponding semantic labels output by the clustering sampler are input into the posture estimator that has completed parameter training to obtain the predicted rotation vector and translation vector of the corresponding point cloud instance; The evaluation index of the object pose estimation error is the average point pair distance, which is as follows: Where M represents the point cloud set of the 3D model, m is the number of points, x represents any point in the point cloud of the 3D model; x1 is any point in the point cloud of the object model, x2 is the point in the scene that transforms the point cloud of the object model according to the 6D pose predicted by the deep 6D pose estimation network system to the point closest to x1; R and t represent the real rotation matrix and translation vector respectively. and They represent the predicted rotation matrix and translation vector respectively; during evaluation, the predicted rotation vector is converted into a rotation matrix for evaluation; The prediction quality evaluation of 6D pose estimation is expressed as the accuracy of the average point pair distance less than 10% of the diameter d of the corresponding model of each object; when the accuracy is greater than 90, the deep 6D pose estimation network system is an effective system; Step 3: Workpiece pose estimation The scene point clouds of multiple workpieces collected by the camera are input into the deep 6D pose estimation network system tested as valid in step 2. The scene point cloud is instance-segmented by the point cloud segmenter to obtain high-dimensional instance features of the entire scene point cloud. The features are sent to the feature clustering sampler to obtain the point cloud instances of each workpiece, which are then sent to the pose estimator to obtain high-precision 6D poses of each workpiece, which can be used in industrial sorting and grasping operations.

6. A workpiece pose estimation method according to claim 5, characterized in that: In step 1, the point cloud dataset of the network system is obtained using the open source physics engine Blensor. The specific process is as follows: (1) Randomly import CAD mesh models of 4 to 7 repeatable parts, and randomly generate 6-DOF poses of each model above the preset grabbing plane to initialize the position of the parts. The plane to be photographed is a square area with a length and width of 256 mm, and collision baffles are set around it; (2) Add a physical engine to the model in the simulation scene, set the corresponding rigid body collision parameters and dynamic parameters, and retain the final state of the workpiece in free fall under the action of gravity to simulate the scattered and stacked placement of the workpiece on the plane; (3) Add the sensor model, set the sensor resolution and focal length parameters, and add noise parameters to improve the similarity between the data set and the actual sensor data; (4) Save the surface point cloud in the scene in the sensor coordinate system as a data set sample; save the 6-DOF posture information and point-level semantic information of each part as a data set label; (5) For point clouds with multiple similar artifacts in each scene, the artifacts are separated according to semantic information, and the point cloud model is transformed to the actual position through the 6-DOF posture information. The K-nearest neighbor algorithm is used to obtain one or more points adjacent to each transformed point cloud model in the original scene and extract them as a single instance, thereby obtaining accurate instance labels for the entire scene point cloud; (6) Labels of objects that are severely occluded or cannot be effectively presented in the scene point cloud due to collision or penetration are removed; the severe occlusion judgment threshold is 30% of the statistical mean of all generated object surface point clouds to ensure the reliability of recognition and capture.

Images