Industrial pipeline workpiece point cloud completion and pose estimation method and system

Abstract

This invention discloses a method and system for point cloud completion and pose estimation of workpieces on industrial production lines. The method includes: image acquisition, point cloud preprocessing, intelligent completion, pose estimation, pose fine-tuning, and guided grasping. The system includes a 3D vision sensor unit: mounted above or to the side of the conveyor belt, used to acquire 3D point cloud data of workpieces on the production line. This sensor can be a laser profilometer, structured light camera, binocular stereo camera, or ToF camera. A computing and processing unit includes a high-performance graphics processing unit (GPU) and a central processing unit (CPU) for running the core algorithm of this invention. A robotic arm control unit receives the 6D pose of the workpiece output by the processing unit, generates a motion trajectory, and controls the robotic arm end effector for precise grasping.

Description

Technical Field

[0001] This invention belongs to the field of point cloud pose estimation technology, and relates to a method and system for point cloud completion and pose estimation of workpieces in industrial production lines. Background Technology

[0002] In modern intelligent manufacturing and industrial automation production lines, the automatic gripping, sorting, and assembly of workpieces are core processes. With the development of flexible manufacturing systems, the variety of workpieces on production lines is increasing. Furthermore, to reduce the complexity and cost of upstream material supply systems, workpieces are often placed randomly and haphazardly on conveyor belts in arbitrary postures (6 degrees of freedom, or 6D). How to enable industrial robots (robotic arms) to quickly and accurately perceive the position and posture of these workpieces and guide end effectors (such as grippers and suction cups) to complete precise gripping is crucial to the efficiency and reliability of the production line.

[0003] To achieve this goal, vision-based guidance technologies have become mainstream. Among these, acquiring 3D point cloud data of the workpiece using 3D sensors (such as laser profilometers, structured light cameras, and binocular stereo cameras) and then performing pose estimation is currently the most effective technical approach. However, in the specific scenario of an industrial production line, a fundamental technical challenge exists: the workpiece point cloud data collected by sensors with single or multiple fixed viewpoints is often partial and incomplete. This is mainly due to occlusion caused by the complex geometry of the workpiece (either occlusion by the workpiece itself or mutual occlusion between workpieces), limitations in the sensor's field of view, and point cloud gaps caused by the workpiece's surface material (such as highly reflective metals or black plastics).

[0004] Using partial point cloud data directly for pose estimation presents two serious problems: First, the lack of geometric features in the point cloud causes feature matching algorithms to fail, making it impossible to accurately establish a correspondence with the standard model. Second, even with certain iterative algorithms (such as Iterative Closest Point, ICP), insufficient information during initial alignment can easily lead to getting trapped in local optima, resulting in a significant decrease in pose estimation accuracy, or even complete failure. Therefore, how to intelligently and faithfully recover the complete shape of a workpiece from partially observed point clouds, and on this basis achieve high-precision 6D pose estimation, has become a cutting-edge technology urgently needing breakthroughs in the field of industrial vision.

[0005] Currently, for the problem of pose estimation of workpieces based on partial point clouds on industrial production lines, the industry mainly has the following similar implementation solutions: (1) Scheme based on traditional feature point matching and global optimization This is one of the most widely used methods in industry. Its core idea is as follows: First, from the partial point cloud of the workpiece (scene) and the workpiece's computer-aided design (CAD) model (a standard complete point cloud generated offline), manually designed local geometric feature descriptors, such as Point Feature Histogram (PFH), Fast Point Feature Histogram (FPFH), and View Feature Histogram (VFH), are extracted respectively. Then, these descriptors are used to match in the feature space to establish candidate corresponding point pairs between the scene point cloud and the model point cloud. Next, algorithms such as Random Sample Consensus (RANSAC) are used to iteratively solve for an optimal rigid transformation matrix (i.e., the initial pose) from a large number of corresponding point pairs containing outliers (mismatches). Finally, this initial pose is used as the initial value for the Iterative Closest Point (ICP) algorithm. By minimizing the point-to-point or point-to-plane distance between the scene point cloud and the model point cloud, the pose is refined and iteratively optimized to obtain the final 6D pose estimation result.

[0006] (2) Template matching-based scheme Such methods typically involve pre-establishing a large template library. Through offline rendering, the CAD model of the workpiece is discretized and sampled in 6D space at specific angles and positional steps, generating a massive number of 2D or 3D templates. During online execution, partial point clouds or corresponding depth maps acquired by sensors are compared one by one with all templates in the library. The comparison process usually employs methods such as Normalized Cross-Correlation (NCC), Histogram of Oriented Gradients (HOG), or Pair Feature Analysis (PPF) to calculate similarity, selecting the pose corresponding to the one or several templates with the highest similarity as the final estimation result. To improve efficiency, hash tables (such as those used in the PPF method) are often incorporated to index and quickly retrieve templates.

[0007] (3) End-to-end pose estimation scheme based on deep learning With the development of deep learning, many end-to-end network models have emerged in recent years that directly regress 6D pose from image or point cloud data. These methods are mainly divided into two categories: one is based on correspondence, such as PointNet++, which learns the correspondence between each point in the point cloud and points on the object's surface, and then solves the pose using the PnP (Perspective n-Point) algorithm or direct regression. The other is based on keypoints, such as PVNet, which predicts the projection positions of predefined 3D keypoints of the object onto a 2D image through the network, and then uses the PnP algorithm to solve the pose. Some methods also use fully convolutional networks to directly regress pose parameters (such as quaternions plus translation vectors).

[0008] Although the aforementioned existing technologies can achieve certain results under specific conditions, they still have the following significant drawbacks when dealing with workpieces that are randomly placed on industrial production lines, severely obstructed, complex in shape, and have diverse surface materials: 1. Disadvantages of existing technologies (1) Poor robustness to point cloud incompleteness: Traditional feature point matching schemes heavily rely on the completeness of the point cloud to extract stable and repeatable geometric features. When the point cloud is missing a large amount of structure due to occlusion, the calculation of feature descriptors becomes extremely unreliable, and a large number of incorrect matches are introduced during the matching process, making it difficult for the RANSAC algorithm to converge to the correct solution. Although template matching schemes cover part of the pose through template libraries, they are essentially still based on matching of partial geometric information. When the missing part happens to be the key feature for distinguishing similar poses, the matching results are very easy to cause confusion or ambiguity. Although end-to-end networks have a certain generalization ability, when the distribution of training data is inconsistent with the distribution of severe occlusion in the real scene, the accuracy and stability of their pose regression will decrease significantly.

[0009] (2) Strong prior assumptions about the geometric and physical properties of the workpiece: Most existing methods assume that the workpiece surface has diffuse reflection characteristics and can stably reflect light, or assume that the workpiece material is uniform and has no highlights. In industrial settings, many workpieces (such as machined metal parts and painted parts) have highly reflective surfaces, resulting in a large number of holes and noise in the point cloud collected by the 3D sensor. Traditional methods have almost no ability to handle this kind of point cloud missing caused by the material itself. In addition, for workpieces with similar shapes but subtle differences (such as gears of different models), traditional methods have difficulty making accurate distinctions, which can easily lead to grasping failures.

[0010] (3) The contradiction between computational efficiency and accuracy is difficult to reconcile: Although traditional ICP algorithms and their variants can achieve high-precision pose optimization, their iterative process involves a large amount of computation and is highly sensitive to the initial pose. In order to ensure real-time performance, it is often necessary to sacrifice some accuracy in the feature matching stage to gain speed, which makes it difficult for the final pose estimation result to simultaneously meet the requirements of high accuracy and real-time performance. Although template matching methods are fast online, their accuracy depends entirely on the discrete density of the template library. However, building a high-precision (e.g., angle step size < 1°) 6D template library requires exponentially increasing storage space and offline computation time, which is not feasible in engineering.

[0011] (4) Lack of deep understanding and intelligent reasoning regarding workpiece shape: The methods described above are essentially "searching" or "fitting" a known CAD model, lacking the ability to "understand" the essence of the workpiece shape. They cannot actively and intelligently "imagine" the geometric structure of the occluded part. In other words, they are "guessing" a transformation with incomplete data, rather than first "filling in" the incomplete shape through "mentalization" before performing pose estimation. This strategy of "estimating first and then filling in" or "estimating and filling in simultaneously" makes pose estimation fundamentally limited by the quality of the observed data. Summary of the Invention

[0012] To address the shortcomings of the existing technologies, the present invention aims to provide a method and system for point cloud completion and pose estimation of workpieces in industrial production lines. The present invention aims to solve the following technical problems: how to explicitly introduce edge geometric constraints in a single-stage instance segmentation framework to fundamentally eliminate edge adhesion caused by prototype combination; and how to establish a cross-scale feature alignment mechanism so that the generation of prototype masks can adapt to the scale changes of instances, thereby simultaneously improving the completeness of small object segmentation and the precision of large object segmentation.

[0013] The technical solution of this invention is: a method for point cloud completion and pose estimation of workpieces in industrial production lines, comprising the following steps: S10, Image Acquisition: When the workpiece on the conveyor belt moves to the area below the 3D vision sensor unit, the sensor is triggered to acquire a 3D point cloud image. S20, Point Cloud Preprocessing: The calculation and processing unit performs filtering, centering, and normalization on the original point cloud and extracts the normal vector of each point. S30 intelligent completion: The processed partial point cloud is fed into a pre-trained multi-scale local-global encoder to extract the global shape code. Subsequently, the geometry decoupling generator utilizes... First, generate the structural skeleton. Then, a complete point cloud is generated through the deformation field. ; S40, pose estimation, complete point cloud completion and the original point cloud The initial 6D pose is fed into a two-branch pose estimation network in parallel and outputs the initial 6D pose. ; S50, pose fine-tuning, utilizes a differentiable point-to-plane distance loss to perform a small number of iterations to optimize the initial pose, resulting in the final high-precision pose. ; S60, the guiding grasping, calculation and processing unit sends the final pose information along with the workpiece type information to the robotic arm control unit. The industrial robotic arm plans the grasping path based on this pose and completes the grasping operation.

[0014] Preferably, step S20 includes the following steps: S21, let the original partial point cloud collected by the sensor be... ,in The number of points, each point contains Coordinate information; S22, outliers are removed using statistical filtering and radius filtering to obtain a clean partial point cloud. ; S23, In order to maintain the original dimensions and orientation information of the workpiece during the completion process, ... After centering and normalizing, the coordinates of the center point of the input point cloud are calculated. : ; in, It is an input point cloud The point in the middle; S24, translate the point cloud to the origin, making its center located at... To obtain a centralized point cloud ; S25, Calculate the maximum radius of the point cloud. Scaling: ; Normalized point cloud This normalization operation helps the network learn scale-independent geometric structures and improves its generalization ability. S26, Input point cloud data is represented as .

[0015] Preferably, in step S20, in order to introduce more local geometric details, not only are the original three-dimensional coordinates used, but the normal vector of each point is also calculated. As an additional feature, the normal vector is calculated in the local neighborhood of the point cloud through principal component analysis. The feature vector of each point input into the network is... .

[0016] Preferably, the S30 intelligent completion employs a generative network based on an encoder-decoder structure, including a multi-scale local-global feature encoder and a geometric structure decoupling generator, from partial observations... The complete point cloud was recovered. ,in It represents the number of points after completion.

[0017] Preferably, step S30 specifically includes the following steps: S31, multi-scale local and global feature encoding, specifically adopts a hierarchical encoder, which consists of multiple feature extraction layers stacked together. Each layer adopts a set abstraction module, which downsamples the point cloud by sampling the farthest point and aggregates the features in the neighborhood of each sampling point. Let the first The input point set of the layer is ,in It's points. It is the feature dimension; the output point set Obtained through the following operations: For each output point Find its radius is within the spherical neighborhood Neighboring points ; Through a shared multilayer perceptron Encode local features and perform maximum pooling: ; in, Indicates feature splicing, It is the first Feature vectors of points; go through After layer encoding, global features with different receptive fields are obtained. To preserve fine local details, features from different layers of the encoder are... Store it so that it can be connected across layers during the decoding stage; Global features Subsequently, a point cloud convolutional network is used for further extraction to obtain a high-dimensional global shape encoding vector. This vector implicitly encodes the complete shape prior of the input workpiece, a process expressed as: ; in, A function representing the entire encoder network; S32, decoupled geometric structure generation, decomposes the completion task into two parallel subtasks: structural skeleton generation and geometric detail generation. First, from the global encoding A low-resolution structural skeleton point cloud was decoded from the data. ,in , Used to capture the overall posture, symmetry, and main contours of a workpiece; structural skeleton generator. It is a multilayer perceptron consisting of fully connected layers and upsampling layers: ; Also includes a geometry detail generator Instead of directly generating the final point, it generates a deformation field or residual flow, which acts on... Above, each skeleton point Mapped to a precise, complete point containing rich surface geometric details. : ; in, It is the displacement vector predicted by the geometry detail generator. Essentially, it is a continuous function that takes global features and skeleton point coordinates as input, implemented using implicit neural representation or point-based MLP; the final complete point cloud Through the Obtained by applying the deformation field: ; S33, completion loss function, specifically a multi-level loss function, including chamfer distance, seismograph distance, and a structural consistency loss for supervising skeleton generation. , chamfer distance Measuring Predicted Point Clouds Compared to a true complete point cloud Differences between them: ; Seismograph Distance Seeking the optimal match between two sets of points is more sensitive to capturing global shape differences. ; in It is a bijection, that is, a one-to-one mapping relationship from the predicted point set to the real point set, which guarantees that the matching between the two point sets is globally optimal and there is no duplicate matching; Structural consistency loss Global features extracted from the input point cloud by the forced encoder Global features extracted from a real complete point cloud Align in implicit space: ; The final total loss of the completed network is: ; in It is a weighting coefficient that balances the various losses.

[0018] Preferably, S40 specifically involves initial pose estimation based on a dual-branch network, designing a dual-branch network structure, and simultaneously completing the point cloud. and the original point cloud The network learns pose features, and its dual-branch structure includes: Complete point cloud branch: Input into a lightweight PointNet++ network to extract its global features. ,because The shape is complete, and this feature can provide global attitude information such as workpiece orientation and axis of symmetry. Partial point cloud branch: This refers to the process of dividing the original partial point cloud into smaller branches. Input into another network structure to extract its local features This feature focuses on capturing the positional relationship of the currently observed part relative to the whole, providing the anchor point information required for pose estimation; Subsequently, the features from the two branches are concatenated and fed into a pose regression head composed of fully connected layers to predict the 6D pose parameters of the workpiece. Rotation is represented by axis angles, and three-dimensional vectors are used. Representing the rotation axis and angle using a three-dimensional vector. Indicates translation; The pose regression head outputs a 6-dimensional vector. This process can be represented as: ; in It is a positional return network.

[0019] Preferably, S50 specifically includes: a lightweight geometric fine-tuning module, which utilizes point cloud completion. Point cloud of the standard CAD model of the workpiece Iterative optimization is performed on the geometric consistency between the points, using the differentiable point-to-plane distance as the loss, and rapid fine-tuning is performed once or several times through gradient descent; First, the predicted pose Convert to transformation matrix ,in It is determined by the axis angle The rotation matrix is ​​calculated using the Rodriguez formula; Then, complete the point cloud. Transform to the model coordinate system: ; The goal of fine-tuning is to minimize and The distance from a point to a plane between them, for Each point in ,exist Find the nearest point and its corresponding normal vector The distance from the point to the plane is The overall loss function is: ; in and These are the pose increment parameters to be optimized, with initial values ​​of unit rotation and zero translation. Since this loss function is relative to... and It is differentiable, and the pose is optimized through several gradient descent iterations to obtain the finely tuned pose. .

[0020] Preferably, step S50 further includes jointly training the completion network and the pose estimation network, with a total loss function. Includes compensation loss Initial pose regression loss And pose fine-tuning loss ; The initial regression loss of pose is adopted loss or Loss, pose directly predicted by the monitoring network Compared with the true pose The difference between them: ; in These are the weights used to balance the rotation and translation losses, and the pose fine-tuning loss. This is the sum of the final optimized point-to-plane distances in the aforementioned geometric fine-tuning module. The total loss for end-to-end training is: ; By minimizing The entire network can learn how to complete the point cloud to assist in pose estimation and finally output a high-precision 6D pose.

[0021] To achieve the above objectives, the present invention also provides a point cloud completion and pose estimation system for industrial assembly line workpieces, employing the aforementioned point cloud completion and pose estimation method for industrial assembly line workpieces, comprising the following sequentially connected components: 3D vision sensor unit: Installed directly above or to the side of the conveyor belt, used to collect 3D point cloud data of workpieces on the production line, including laser profilometer, structured light camera, binocular stereo camera or ToF camera; Computing and processing unit: Includes graphics processor and central processing unit, which performs point cloud intelligent completion and pose estimation algorithms; Robotic arm control unit: Receives the 6D pose of the workpiece from the calculation and processing unit, generates the motion trajectory, and controls the robotic arm end effector to perform precise grasping. Compared with the prior art, the present invention has at least the following beneficial effects: First, it overcomes the fundamental constraint of point cloud incompleteness on pose estimation. By introducing point cloud completion technology based on a depth generative model, incomplete and occluded input point clouds are intelligently restored into a complete and high-fidelity 3D shape. This "complete first, estimate later" strategy fundamentally changes the information input source for pose estimation, making it no longer dependent on incomplete observation data, but based on a "complete" and inferred workpiece model.

[0022] Second, it enhances adaptability and robustness to complex industrial scenarios. By constructing a joint learning framework, point cloud completion, shape reconstruction, and 6D pose estimation are integrated into an end-to-end trainable network. This network can be trained on massive amounts of simulation data (including simulations of various occlusions and reflective noise) to learn prior knowledge of the workpiece shape. Thus, even when faced with unknown occlusions and highly reflective materials in real-world scenarios, it can still robustly complete the shape and predict the pose.

[0023] Third, a balance between high accuracy and high efficiency is achieved. By designing a lightweight network structure and an efficient inference process, the total time for completion and pose estimation can meet the real-time requirements of industrial production lines (e.g., single-frame processing time less than 50ms). Simultaneously, since the completed point cloud provides complete geometric information, subsequent pose estimation can be simplified from traditional complex iterative optimization (such as ICP) to a one-time forward inference or a minimal number of iterations, thereby significantly improving efficiency while maintaining accuracy.

[0024] Fourth, it endows industrial robot systems with the capabilities of "intelligent perception" and "reasoning and prediction." The ultimate goal of this invention is to build an intelligent vision system that, like an experienced worker, can immediately "fill in" the complete shape of a workpiece and determine its placement posture even if it only sees a part of it, providing core technological support for achieving truly flexible, intelligent, and unmanned production. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the steps of the point cloud completion and pose estimation method for industrial assembly line workpieces according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the internal structure and working principle of the geometric decoupling generator in S30 of the point cloud completion and pose estimation method for industrial assembly line workpieces in an embodiment of the present invention. Figure 3 This is a schematic diagram of S40 of the point cloud completion and pose estimation method for industrial assembly line workpieces according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the system for point cloud completion and pose estimation of workpieces in industrial production lines according to an embodiment of the present invention. Figure 5 This is a comparison diagram of the point cloud completion and pose estimation method for industrial production line workpieces in the embodiment of the present invention and the existing technology method. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0028] See Figure 1 The image shows a method for point cloud completion and pose estimation of workpieces in industrial production lines according to an embodiment of the present invention, which includes the following steps: S10, Image Acquisition: When the workpiece on the conveyor belt moves to the area below the 3D vision sensor unit, the sensor is triggered to acquire a 3D point cloud image. S20, Point Cloud Preprocessing: The calculation and processing unit performs filtering, centering, and normalization on the original point cloud and extracts the normal vector of each point. S30 intelligent completion: The processed partial point cloud is fed into a pre-trained multi-scale local-global encoder to extract the global shape code. Subsequently, the geometry decoupling generator utilizes... First, generate the structural skeleton. Then, a complete point cloud is generated through the deformation field. ; S40, pose estimation, complete point cloud completion and the original point cloud The initial 6D pose is fed into a two-branch pose estimation network in parallel and outputs the initial 6D pose. ; S50, pose fine-tuning, utilizes a differentiable point-to-plane distance loss to perform a small number of iterations to optimize the initial pose, resulting in the final high-precision pose. ; S60, the guiding grasping, calculation and processing unit sends the final pose information along with the workpiece type information to the robotic arm control unit. The industrial robotic arm plans the grasping path based on this pose and completes the grasping operation.

[0029] S20 includes the following steps: S21, let the original partial point cloud collected by the sensor be... ,in The number of points, each point contains Coordinate information; S22, outliers are removed using statistical filtering and radius filtering to obtain a clean partial point cloud. ; S23, In order to maintain the original dimensions and orientation information of the workpiece during the completion process, ... After centering and normalizing, the coordinates of the center point of the input point cloud are calculated. : ; in, It is an input point cloud The point in the middle; S24, translate the point cloud to the origin, making its center located at... To obtain a centralized point cloud ; S25, Calculate the maximum radius of the point cloud. Scaling: ; Normalized point cloud This normalization operation helps the network learn scale-independent geometric structures and improves its generalization ability. S26, Input point cloud data is represented as .

[0030] In S20, to incorporate more local geometric details, not only are the original 3D coordinates used, but the normal vector for each point is also calculated. As an additional feature, the normal vector is calculated in the local neighborhood of the point cloud through principal component analysis. The feature vector of each point input into the network is... .

[0031] The S30 intelligent completion employs a generative network based on an encoder-decoder structure, including a multi-scale local-global feature encoder and a geometric structure decoupling generator, to complete partial observations. The complete point cloud was recovered. ,in It represents the number of points after completion.

[0032] S30 specific steps: S31, multi-scale local and global feature encoding, specifically adopts a hierarchical encoder, which consists of multiple feature extraction layers stacked together. Each layer adopts a set abstraction module, which downsamples the point cloud by sampling the farthest point and aggregates the features in the neighborhood of each sampling point. Let the first The input point set of the layer is ,in It's points. It is the feature dimension; the output point set Obtained through the following operations: For each output point Find its radius is within the spherical neighborhood Neighboring points ; Through a shared multilayer perceptron Encode local features and perform maximum pooling: ; in, Indicates feature splicing, It is the first Feature vectors of points; go through After layer encoding, global features with different receptive fields are obtained. To preserve fine local details, features from different layers of the encoder are... Store it so that it can be connected across layers during the decoding stage; Global features Subsequently, a point cloud convolutional network is used for further extraction to obtain a high-dimensional global shape encoding vector. This vector implicitly encodes the complete shape prior of the input workpiece, a process expressed as: ; in, A function representing the entire encoder network; S32, decoupled geometric structure generation, decomposes the completion task into two parallel subtasks: structural skeleton generation and geometric detail generation. First, from the global encoding A low-resolution structural skeleton point cloud was decoded from the data. ,in , Used to capture the overall posture, symmetry, and main contours of a workpiece; structural skeleton generator. It is a multilayer perceptron consisting of fully connected layers and upsampling layers: ; Also includes a geometry detail generator Instead of directly generating the final point, it generates a deformation field or residual flow, which acts on... Above, each skeleton point Mapped to a precise, complete point containing rich surface geometric details. : ; in, It is the displacement vector predicted by the geometry detail generator. Essentially, it is a continuous function that takes global features and skeleton point coordinates as input, implemented using implicit neural representation or point-based MLP; the final complete point cloud Through the Obtained by applying the deformation field: ; See Figure 2 This is a schematic diagram of the internal structure and working principle of the geometric decoupling generator.

[0033] S33, completion loss function, specifically a multi-level loss function, including chamfer distance, seismograph distance, and a structural consistency loss for supervising skeleton generation. , chamfer distance Measuring Predicted Point Clouds Compared to a true complete point cloud Differences between them: ; Seismograph Distance Seeking the optimal match between two sets of points is more sensitive to capturing global shape differences. ; in It is a bijection, that is, a one-to-one mapping relationship from the predicted point set to the real point set, which guarantees that the matching between the two point sets is globally optimal and there is no duplicate matching; Structural consistency loss Global features extracted from the input point cloud by the forced encoder Global features extracted from a real complete point cloud Align in implicit space: ; The final total loss of the completed network is: ; in It is a weighting coefficient that balances the various losses.

[0034] S40 specifically involves initial pose estimation based on a dual-branch network. A dual-branch network structure is designed, and this is used to simultaneously complete the point cloud. and the original point cloud Learning pose features, see Figure 3 The dual-branch network structure includes: Complete point cloud branch: Input into a lightweight PointNet++ network to extract its global features. ,because The shape is complete, and this feature can provide global attitude information such as workpiece orientation and axis of symmetry. Partial point cloud branch: This refers to the original partial point cloud... Input into another network structure to extract its local features This feature focuses on capturing the positional relationship of the currently observed part relative to the whole, providing the anchor point information required for pose estimation; Subsequently, the features from the two branches are concatenated and fed into a pose regression head composed of fully connected layers to predict the 6D pose parameters of the workpiece. Rotation is represented by axis angles, and three-dimensional vectors are used. Representing the rotation axis and angle using a three-dimensional vector. Indicates translation; The pose regression head outputs a 6-dimensional vector. This process can be represented as: ; in It is a positional return network.

[0035] The S50 specifically includes: a lightweight geometry fine-tuning module that utilizes point cloud completion. Point cloud of the standard CAD model of the workpiece Iterative optimization is performed on the geometric consistency between the points, using the differentiable point-to-plane distance as the loss, and rapid fine-tuning is performed once or several times through gradient descent; First, the predicted pose Convert to transformation matrix ,in It is determined by the axis angle The rotation matrix is ​​calculated using the Rodriguez formula; Then, complete the point cloud. Transform to the model coordinate system: ; The goal of fine-tuning is to minimize and The distance from a point to a plane between them, for Each point in ,exist Find the nearest point and its corresponding normal vector The distance from the point to the plane is The overall loss function is: ; in and These are the pose increment parameters to be optimized, with initial values ​​of unit rotation and zero translation. Since this loss function is relative to... and It is differentiable, and the pose is optimized through several gradient descent iterations to obtain the finely tuned pose. .

[0036] S50 also includes joint training of the completion network and pose estimation network, with a total loss function. Includes compensation loss Initial pose regression loss And pose fine-tuning loss ; The initial regression loss of pose is adopted loss or Loss, pose directly predicted by the monitoring network Compared with the true pose The difference between them: ; in These are the weights used to balance the rotation and translation losses, and the pose fine-tuning loss. This is the sum of the final optimized point-to-plane distances in the aforementioned geometric fine-tuning module. The total loss for end-to-end training is: ; By minimizing The entire network can learn how to complete the point cloud to assist in pose estimation and finally output a high-precision 6D pose.

[0037] See Figure 4 A point cloud completion and pose estimation system for industrial assembly line workpieces, employing the aforementioned point cloud completion and pose estimation method for industrial assembly line workpieces, includes the following sequentially connected components: 3D vision sensor unit 1: Installed directly above or to the side of conveyor belt 2, used to collect 3D point cloud data of workpiece 3 on the production line, including laser profilometer, structured light camera, binocular stereo camera or ToF camera; transmits the collected and processed 3D point cloud data 8 to computing and processing unit 4.

[0038] The computing and processing unit 4 includes a graphics processor and a central processing unit, which performs intelligent point cloud completion and pose estimation algorithms; and transmits 6D pose information 9 to the robotic arm control unit 5.

[0039] Robotic arm control unit 5: Receives the workpiece 6D pose output by the calculation and processing unit 4, generates a motion trajectory, and controls the gripping actuator 7 at the end of the industrial robotic arm 6 to perform precise gripping.

[0040] See Figure 5The diagram compares the completion performance and pose estimation error of the method of this invention with other existing methods (such as FPFH+RANSAC+ICP and direct pose regression based on PointNet++) in occluded scenarios. Experiments show that even in extreme occlusion conditions with a point cloud missing rate as high as 70%, the pose estimation success rate of this invention is still more than 45% higher than traditional feature matching methods (such as FPFH+ICP) and more than 30% higher than end-to-end networks directly based on partial point clouds (such as PointNet++). Under the same occlusion conditions, the completion result of this invention can accurately recover the complete shape of the workpiece, while traditional methods suffer from pose estimation divergence due to insufficient information. In the grasping experiment for highly reflective workpieces, the method of this invention maintained a grasping success rate of over 90%, while the success rate of the traditional structured light-based point cloud acquisition + ICP scheme dropped to below 40%.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for point cloud completion and pose estimation of workpieces in industrial production lines, characterized in that, Includes the following steps: S10, Image Acquisition: When the workpiece on the conveyor belt moves to the area below the 3D vision sensor unit, the sensor is triggered to acquire a 3D point cloud image. S20, Point Cloud Preprocessing: The calculation and processing unit performs filtering, centering, and normalization on the original point cloud and extracts the normal vector of each point. S30 intelligent completion: The processed partial point cloud is fed into a pre-trained multi-scale local-global encoder to extract the global shape code. ; Subsequently, the geometry decoupling generator utilizes First, generate the structural skeleton. Then, a complete point cloud is generated through the deformation field. ; S40, pose estimation, complete point cloud completion and the original point cloud The initial 6D pose is fed into a two-branch pose estimation network in parallel and outputs the initial 6D pose. ; S50, pose fine-tuning, utilizes a differentiable point-to-plane distance loss to perform a small number of iterations to optimize the initial pose, resulting in the final high-precision pose. ; S60, the guiding grasping, calculation and processing unit sends the final pose information along with the workpiece type information to the robotic arm control unit. The industrial robotic arm plans the grasping path based on this pose and completes the grasping operation.

2. The method for point cloud completion and pose estimation of workpieces in industrial production lines according to claim 1, characterized in that, S20 includes the following steps: S21, let the original partial point cloud collected by the sensor be... ,in The number of points, each point contains Coordinate information; S22, outliers are removed using statistical filtering and radius filtering to obtain a clean partial point cloud. ; S23, In order to maintain the original dimensions and orientation information of the workpiece during the completion process, ... After centering and normalizing, the coordinates of the center point of the input point cloud are calculated. : ； in, It is an input point cloud The point in the middle; S24, translate the point cloud to the origin, making its center located at... To obtain a centralized point cloud ; S25, Calculate the maximum radius of the point cloud. Scaling: ； Normalized point cloud This normalization operation helps the network learn scale-independent geometric structures and improves its generalization ability. S26, Input point cloud data is represented as .

3. The method for point cloud completion and pose estimation of workpieces in industrial production lines according to claim 2, characterized in that, In step S20, to incorporate more local geometric details, not only are the original three-dimensional coordinates used, but the normal vector of each point is also calculated. As an additional feature, the normal vector is calculated in the local neighborhood of the point cloud through principal component analysis. The feature vector of each point input into the network is... .

4. The method for point cloud completion and pose estimation of workpieces in industrial production lines according to claim 2, characterized in that, The S30 intelligent completion employs a generative network based on an encoder-decoder structure, including a multi-scale local-global feature encoder and a geometric structure decoupling generator, to complete partial observations. The complete point cloud was recovered. ,in It represents the number of points after completion.

5. The method for point cloud completion and pose estimation of workpieces in industrial production lines according to claim 3, characterized in that, The specific steps of S30 are as follows: S31, multi-scale local and global feature encoding, specifically adopts a hierarchical encoder, which consists of multiple feature extraction layers stacked together. Each layer adopts a set abstraction module, which downsamples the point cloud by sampling the farthest point and aggregates the features in the neighborhood of each sampling point. Let the first The input point set of the layer is ,in It's points. It is the feature dimension; Output point set Obtained through the following operations: For each output point Find its radius is within the spherical neighborhood Neighboring points ; Through a shared multilayer perceptron Encode local features and perform maximum pooling: ； in, Indicates feature splicing, It is the first Feature vectors of points; go through After layer encoding, global features with different receptive fields are obtained. To preserve fine local details, features from different layers of the encoder are... Store it so that it can be connected across layers during the decoding stage; Global features Subsequently, a point cloud convolutional network is used for further extraction to obtain a high-dimensional global shape encoding vector. This vector implicitly encodes the complete shape prior of the input workpiece, a process expressed as: ； in, A function representing the entire encoder network; S32, decoupled geometric structure generation, decomposes the completion task into two parallel subtasks: structural skeleton generation and geometric detail generation. First, from the global encoding A low-resolution structural skeleton point cloud was decoded from the data. ,in , Used to capture the overall posture, symmetry, and main contours of a workpiece; structural skeleton generator. It is a multilayer perceptron consisting of fully connected layers and upsampling layers: ； Also includes a geometry detail generator Instead of directly generating the final point, it generates a deformation field or residual flow, which acts on... Above, each skeleton point Mapped to a precise, complete point containing rich surface geometric details. : ； in, It is the displacement vector predicted by the geometry detail generator. Essentially, it is a continuous function that takes global features and skeleton point coordinates as input, implemented using implicit neural representation or point-based MLP; the final complete point cloud Through the Obtained by applying the deformation field: ； S33, completion loss function, specifically a multi-level loss function, including chamfer distance, seismograph distance, and a structural consistency loss for supervising skeleton generation. , chamfer distance Measuring Predicted Point Clouds Compared to a true complete point cloud Differences between them: ； Seismograph Distance Seeking the optimal match between two sets of points is more sensitive to capturing global shape differences. ； in It is a bijection, that is, a one-to-one mapping relationship from the predicted point set to the real point set, which guarantees that the matching between the two point sets is globally optimal and there is no duplicate matching; Structural consistency loss Global features extracted from the input point cloud by the forced encoder Global features extracted from a real complete point cloud Align in implicit space: ； The final total loss of the completed network is: ； in It is a weighting coefficient that balances the various losses.

6. The method for point cloud completion and pose estimation of workpieces in industrial production lines according to claim 5, characterized in that, Specifically, S40 involves initial pose estimation based on a dual-branch network, designing a dual-branch network structure, and simultaneously completing the point cloud. and the original point cloud The network learns pose features, and its dual-branch structure includes: Complete point cloud branch: Input into a lightweight PointNet++ network to extract its global features. ,because The shape is complete, and this feature can provide global attitude information such as workpiece orientation and axis of symmetry. Partial point cloud branch: This refers to the original partial point cloud... Input into another network structure to extract its local features This feature focuses on capturing the positional relationship of the currently observed part relative to the whole, providing the anchor point information required for pose estimation; Subsequently, the features from the two branches are concatenated and fed into a pose regression head composed of fully connected layers to predict the 6D pose parameters of the workpiece. Rotation is represented by axis angles, and three-dimensional vectors are used. Representing the rotation axis and angle using a three-dimensional vector. Indicates translation; The pose regression head outputs a 6-dimensional vector. This process can be represented as: ； in It is a positional return network.

7. The method for point cloud completion and pose estimation of workpieces in industrial production lines according to claim 6, characterized in that, Specifically, S50 includes: a lightweight geometric fine-tuning module that utilizes point cloud completion. Point cloud of the standard CAD model of the workpiece Iterative optimization is performed on the geometric consistency between the points, using the differentiable point-to-plane distance as the loss, and rapid fine-tuning is performed once or several times through gradient descent; First, the predicted pose Convert to transformation matrix ,in It is determined by the axis angle The rotation matrix is ​​calculated using the Rodriguez formula; Then, complete the point cloud. Transform to the model coordinate system: ； The goal of fine-tuning is to minimize and The distance from a point to a plane between them, for Each point in ,exist Find the nearest point and its corresponding normal vector The distance from the point to the plane is The overall loss function is: ； in and These are the pose increment parameters to be optimized, with initial values ​​of unit rotation and zero translation. Since this loss function is relative to... and It is differentiable, and the pose is optimized through several gradient descent iterations to obtain the finely tuned pose. .

8. The method for point cloud completion and pose estimation of workpieces in industrial production lines according to claim 7, characterized in that, S50 further includes jointly training the completion network and the pose estimation network, with a total loss function. Includes compensation loss Initial pose regression loss And pose fine-tuning loss ; The initial regression loss of pose is adopted loss or Loss, pose directly predicted by the monitoring network Compared with the true pose The difference between them: ； in These are the weights used to balance the rotation and translation losses, and the pose fine-tuning loss. This is the sum of the final optimized point-to-plane distances in the aforementioned geometric fine-tuning module. The total loss for end-to-end training is: ； By minimizing The entire network can learn how to complete the point cloud to assist in pose estimation and finally output a high-precision 6D pose.

9. A point cloud completion and pose estimation system for industrial assembly line workpieces, employing the point cloud completion and pose estimation method for industrial assembly line workpieces as described in any one of claims 1-8, characterized in that, Including those connected sequentially: 3D vision sensor unit: Installed directly above or to the side of the conveyor belt, used to collect 3D point cloud data of workpieces on the production line, including laser profilometer, structured light camera, binocular stereo camera or ToF camera; Computing and processing unit: Includes graphics processor and central processing unit, which performs point cloud intelligent completion and pose estimation algorithms; Robotic arm control unit: Receives the 6D pose of the workpiece from the calculation and processing unit, generates the motion trajectory, and controls the end effector of the robotic arm to perform precise grasping.

Images