Industrial part-oriented point cloud clustering segmentation method

Abstract

The invention discloses a point cloud clustering segmentation method for industrial parts. The method comprises the following steps: 1) data acquisition; according to the method, rapid clustering is achieved by determining the geometric center of each object to serve as a reference point of clustering, a redundant merging algorithm is not needed, the calculation time is saved, and errors can be better avoided when serious shielding and overlapping scenes are processed.

Description

Technical Field

[0001] The present invention relates to a point cloud segmentation technology in the field of computer vision, and in particular to a point cloud clustering segmentation method for industrial parts. Background Art

[0002] With the continuous improvement of industrial automation, robots are increasingly used in the manufacturing industry. When performing tasks, industrial robots often need to handle a large number of parts with complex shapes and sizes. These parts are usually piled up in a container, and the robot needs to pick out the target parts and process them. This process is called "bin-picking". In order to achieve accurate bin-picking, the robot must have efficient part recognition and positioning capabilities, which involves point cloud segmentation technology.

[0003] Point cloud segmentation technology refers to processing point cloud data obtained from 3D scanners or other sensors and dividing them into different parts for subsequent identification and processing. In industrial environments, parts are often stacked together and there are a lot of occlusions, which makes point cloud segmentation extremely challenging. Traditional point cloud segmentation methods often have difficulty achieving ideal results when dealing with such complex environments. Specifically, the main task of point cloud segmentation is to segment complex 3D point cloud data into meaningful parts according to predefined rules and algorithms, so as to achieve accurate identification and positioning of parts. In this process, how to effectively deal with occlusion, noise and the sparsity of point cloud data is the key to improving segmentation accuracy and efficiency.

[0004] Existing point cloud segmentation technologies mainly include methods based on geometric features and methods based on deep learning. Methods based on geometric features usually divide point clouds into different parts by analyzing the geometric shape, surface normal vector and other features of the point cloud. Such methods usually rely on the local features of each point in the point cloud, such as curvature, normal direction, etc., and divide the point cloud into different areas through these geometric features. However, this method often performs poorly when faced with complex occlusions and diverse part shapes, and it is difficult to achieve high-precision segmentation. Especially when the point cloud data volume is large and there is a lot of noise, methods based on geometric features are easily disturbed, resulting in unsatisfactory segmentation results.

[0005] In recent years, with the development of deep learning technology, point cloud segmentation methods based on deep learning have received widespread attention. Such methods can achieve good segmentation effects in complex scenes by constructing deep neural networks and using a large amount of labeled data for training. For example, models such as PointNet and DGCNN perform well on public datasets. However, these methods still have some shortcomings when dealing with point clouds of industrial parts with severe occlusion. Specifically, their segmentation accuracy still needs to be improved when faced with complex occlusions and diverse part shapes. In practical applications, there are many types of parts in industrial environments, with complex shapes and occlusions, which puts higher requirements on point cloud segmentation algorithms. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the current point cloud instance segmentation method performs poorly in the industrial parts point cloud segmentation scene, and proposes a point cloud clustering segmentation method for industrial parts. This method achieves fast clustering by determining the geometric center of each object as the reference point for clustering, without the need for redundant merging algorithms, greatly saving computing time, and avoiding errors when dealing with severe occlusion and overlapping scenes. The technical solution to achieve the purpose of the present invention is:

[0007] A point cloud clustering and segmentation method for industrial parts, comprising the following steps:

[0008] 1) Data acquisition: Convert the ground truth format of the acquired data set from a CSV file to a JSON file, that is, parse each line of data in the CSV file and convert the CSV file into a key-value pair structure that conforms to the JSON format. After the conversion is completed, save the JSON file;

[0009] Next, the depth image in the data set is converted into point cloud data. Point cloud data is a set of points in three-dimensional space, and each point contains three coordinates (x, y, z). When the depth image is converted into point cloud data, the corresponding three-dimensional coordinates can be calculated by the depth value of each point in the depth image;

[0010] Then the point cloud data is sampled and normalized. Data sampling can reduce the amount of point cloud data to improve processing speed and efficiency; normalization is to scale the coordinate values ​​of the point cloud data to a fixed range;

[0011] Finally, convert the processed point cloud data text file into an HDF5 format file. By saving the point cloud data as an HDF5 format file, the point cloud data can be easily loaded and used for model training.

[0012] 2) Build a feature extractor:

[0013] Adaptive graph convolution is used to transform the input point cloud M∈R n×f Process and obtain each point MK∈R n×f×k Then, two convolutional layers and a maximum pooling layer are used to eliminate the point cloud rotation anisotropy and low-level geometric structure features, respectively, to obtain MKL∈R n×64 , the obtained MKL is again passed through adaptive graph convolution, two convolution layers and a maximum pooling layer to obtain the intermediate layer geometric structure feature MKM∈R n×64 MKM again obtains the high-level geometric structure feature MKH∈R through adaptive graph convolution, two convolution layers and maximum pooling layers n×64 , MKL, MKM and MKH are combined, and the point cloud global feature MKG∈R is output through a single convolution layer and a maximum pooling layer 1024 Finally, MKL, MKM, MKM and MKG are combined to form a new point cloud geometric feature space, and the final point cloud encoding feature F is output through two fully connected layers. backbome ∈R n×256 ; The above is the construction process of the feature extractor;

[0014] 2-1) Adaptive Graph Convolution:

[0015] Since 3D sensors only collect point cloud information on the surface of objects, these point cloud information only contains the three-dimensional coordinates of the points but not the color and normal features. These point cloud information intuitively reflects the geometric shape of the object; therefore, when processing pure point cloud data, its local characteristics should be fully utilized. Adaptive graph convolution can better utilize the local characteristics of point clouds by capturing the local geometric structure of point clouds. Adaptive graph convolution first finds the neighbors of each point based on the k-nearest neighbor method. For each point i and its neighbor point j, the feature difference h′ is calculated. i =∑ j∈N(i) σ(h j -h i ||h i ), where N(i) represents the domain of point i, || represents the feature concatenation operation, and σ represents the sigmoid activation function; in this way, adaptive graph convolution can flexibly adjust the neighborhood relationship, capture more detailed geometric information, and improve the performance of the model in point cloud data processing;

[0016] 3) Training phase: After extracting the input point cloud features in the feature extractor, the center score branch and feature distance branch are used to predict the center score of each point in the point cloud and calculate the feature distance between each point in the point cloud;

[0017] 3-1) Center score branch:

[0018] The center score reflects the distance between a point and its instance center. The score of a point close to the center of an object is higher than that of a point on the boundary. In the center score branch, the extracted point cloud encoding feature F backbone ∈R n×256 After two layers of MLP, we get N×1 point-by-point features, and then use the center score loss function Optimize the center score of each point, where s center(i) represents the true score of each point, s^ center(i) represents the predicted score;

[0019] 3-2) Feature distance branch:

[0020] In the feature space, points belonging to the same instance should be close, while points of different instances should be far away. In order to make the features of points of the same instance similar, the extracted point cloud encoding features F backbone ∈R n×256 After one layer of MLP, we get N×128 point-by-point embedding features, and then use the feature distance loss function It forces the network to optimize the similarity matrix during training, so that the similarity between points in the same group increases and the similarity between points in different groups decreases, where w (i,j) represents the weight matrix obtained by element-by-element multiplication, k (i,j) Represents the loss based on point-to-point relationships;

[0021] Finally, the weighted sum of the center score loss and the feature distance loss is the final total loss, and the formula is L total =(1-α)·L CS +α·L EF ; By assigning appropriate weight coefficients to different types of losses, the model's focus on each task objective can be effectively adjusted to prevent certain sub-losses from excessively affecting the total loss, thereby achieving better training results and convergence speed;

[0022] 4) Inference clustering stage: Points above a fixed threshold are considered candidate center points. Non-maximum suppression is performed on all candidate center points to find the center of each instance. The point with the highest center score is selected as the first candidate point. At the same time, other points within a fixed radius of r centered on the candidate point and with a feature distance greater than the fixed threshold are removed. The feature distance comprehensively considers the attribute information of the point, so that the selected center point can better reflect the actual feature distribution of the points in the cluster, rather than just the proximity in geometric position, effectively reducing the overlap between clusters and fuzzy boundaries, and improving the accuracy of segmentation. This removal process is repeated until there are no more points. At this time, the cluster center point of each instance is obtained. Then, the instance to which each point belongs is determined by calculating the feature distance between each point and the cluster center point, and the clustering result is obtained, and the segmentation is finally completed.

[0023] Compared with the existing methods, this technical solution has the following three characteristics:

[0024] (1) Use adaptive graph convolution in the feature extractor construction stage: Adaptive graph convolution enhances the expressive power of point cloud features by introducing learnable weights. These weights can dynamically adjust the influence of each point in the neighborhood, allowing the model to adapt more flexibly to changes in local geometric structures. Such a design can capture more detailed geometric features and improve the ability to recognize complex structures. At the same time, adaptive graph convolution is highly robust to noise and outliers, and effectively reduces the impact of noise and outliers through learning. This makes the model more stable and accurate when processing noisy point cloud data that may be encountered in practical applications. In short, adaptive graph convolution performs well in tasks such as point cloud classification and segmentation, not only improving performance, but also enhancing computational efficiency and generalization capabilities.

[0025] (2) Introducing weight coefficients during the training phase to balance the contribution of different sub-losses to the total loss: Its advantage is that it can more accurately control the importance of each sub-task in the training process. By assigning appropriate weight coefficients to different types of losses, the model's attention to each task objective can be effectively adjusted to prevent certain sub-losses from excessively affecting the total loss, thereby achieving better training results and convergence speed. This method is particularly suitable for multi-task learning or complex models with multiple loss functions. It can improve the comprehensive performance of the model on different tasks and make it more balanced and optimized when processing data with diverse objectives. In general, the introduction of weight coefficients can improve the flexibility and effectiveness of the training process, allowing the model to perform more stably and accurately in practical applications.

[0026] (3) Selecting center points based on feature distance in the inference clustering stage: The advantage of selecting center points based on feature distance is that it can more accurately capture the local geometric and semantic features of point cloud data, ensuring that the center points are more representative. Feature distance comprehensively considers the attribute information of the points, so that the selected center points can better reflect the actual feature distribution of the points in the cluster, rather than just the proximity in geometric position. This method can effectively reduce the overlap and fuzzy boundaries between clusters and improve the accuracy of segmentation. In addition, selecting center points based on feature distance can reduce the interference of noise and outliers, and enhance the robustness of the model when processing complex and diverse point cloud data. Overall, this method optimizes the quality of point cloud segmentation and makes the model perform better and more stable in practical applications.

[0027] This technical solution method realizes fast clustering by determining the geometric center of each object as the reference point for clustering, without the need for redundant merging algorithms, greatly saving calculation time, and avoiding errors when dealing with severe occlusion and overlapping scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a method flow chart of an embodiment;

[0029] Figure 2 and Figure 3 A schematic diagram of an original point cloud of an embodiment;

[0030] Figure 4 and Figure 5 A schematic diagram of the segmentation result of an embodiment, where each color represents an instance. DETAILED DESCRIPTION

[0031] The content of the present invention is further described below in conjunction with the drawings and embodiments, but the present invention is not limited thereto.

[0032] Example:

[0033] Reference Figure 1 , a point cloud clustering and segmentation method for industrial parts, comprising the following steps:

[0034] 1) Data acquisition: Convert the ground truth format of the acquired dataset from a CSV file into a JSON file, that is, parse each row of data in the CSV file and convert the CSV file into a key-value pair structure that conforms to the JSON format. After the conversion is completed, save the JSON file; the dataset in this example comes from the "Fraunhofer IPA Bin-Picking dataset";

[0035] Next, the depth image in the data set is converted into point cloud data. Point cloud data is a set of points in three-dimensional space, and each point contains three coordinates (x, y, z). When the depth image is converted into point cloud data, the corresponding three-dimensional coordinates can be calculated by the depth value of each point in the depth image;

[0036] Then the point cloud data is sampled and normalized. Data sampling can reduce the amount of point cloud data to improve processing speed and efficiency. Normalization is to scale the coordinate values ​​of the point cloud data to a fixed range, usually [0, 1] or [-1, 1].

[0037] Finally, the processed point cloud data text file is converted into an HDF5 format file to obtain high-quality data for model training. The HDF5 format is an efficient format for storing and managing large-scale data, and is particularly suitable for training data for machine learning models. By saving the point cloud data as an HDF5 format file, the point cloud data can be easily loaded and used for model training.

[0038] 2) Constructing feature extractors: DGCNN effectively captures and utilizes local geometric structures in point clouds through dynamic composition and EdgeConv operations, with excellent performance and strong robustness and scalability; Figure 2 and Figure 3 As shown in the figure, however, there are limitations when using EdgeConv to extract point cloud features. In each layer, EdgeConv builds a graph structure based on a fixed k-nearest neighbor, which means that all points in the same layer are calculated using the same number of neighbors, and the importance between the center point of the graph and the other k points is the same, which limits the model's ability to capture complex local geometric structures. To address this problem, this example designs an adaptive graph convolution to replace EdgeConv.

[0039] Adaptive graph convolution is used to transform the input point cloud M∈R n×f Process and obtain each point MK∈R n×f×k Then, two convolutional layers and a maximum pooling layer are used to eliminate the point cloud rotation anisotropy and low-level geometric structure features, respectively, to obtain MKL∈R n×64 , the obtained MKL is again passed through adaptive graph convolution, two convolution layers and a maximum pooling layer to obtain the intermediate layer geometric structure feature MKM∈R n×64 MKM again obtains the high-level geometric structure feature MKH∈R through adaptive graph convolution, two convolution layers and maximum pooling layers n×64 , MKL, MKM and MKH are combined, and the point cloud global feature MKG∈R is output through a single convolution layer and a maximum pooling layer 1024 Finally, MKL, MKM, MKH and MKG are combined to form a new point cloud geometric feature space, and the final point cloud encoding feature F is output through two fully connected layers. backbone ∈R n×256 ; The above is the construction process of the feature extractor;

[0040] 2-1) Adaptive Graph Convolution:

[0041] Since 3D sensors only collect point cloud information on the surface of objects, these point cloud information only contains the three-dimensional coordinates of the points but not the color and normal features. These point cloud information intuitively reflects the geometric shape of the object; therefore, when processing pure point cloud data, its local characteristics should be fully utilized. Adaptive graph convolution can better utilize the local characteristics of point clouds by capturing the local geometric structure of point clouds. Adaptive graph convolution first finds the neighbors of each point based on the k-nearest neighbor method. For each point i and its neighbor point j, the feature difference h′ is calculated. i =∑ j∈N(i) σ(h j -h i ||h i), where N(i) represents the domain of point i, || represents the feature concatenation operation, and σ represents the sigmoid activation function;

[0042] 3) Training phase: After extracting the input point cloud features in the feature extractor, the center score branch and feature distance branch are used to predict the center score of each point in the point cloud and calculate the feature distance between each point in the point cloud;

[0043] 3-1) Center score branch:

[0044] The center score reflects the distance between a point and its instance center. The score of a point close to the center of an object is higher than that of a point on the boundary. In the center score branch, the extracted point cloud encoding feature F backbone ∈R n×256 After two layers of MLP, we get N×1 point-by-point features, and then use the center score loss function Optimize the center score of each point, where s center(i) represents the true score of each point, s^ center(i) represents the predicted score;

[0045] 3-2) Feature distance branch:

[0046] In the feature space, points belonging to the same instance should be close, while points of different instances should be far away. In order to make the features of points of the same instance similar, the extracted point cloud encoding features F backbone ∈R n×256 After one layer of MLP, we get N×128 point-by-point embedding features, and then use the feature distance loss function It forces the network to optimize the similarity matrix during training, so that the similarity between points in the same group increases and the similarity between points in different groups decreases, where w (i,j) represents the weight matrix obtained by element-by-element multiplication, k (i,j) Represents the loss based on point-to-point relationships;

[0047] Finally, the weighted sum of the center score loss and the feature distance loss is the final total loss, and the formula is L total =(1-α)·L CS +α·L EF ;

[0048] 4) Inference clustering stage: Points above a fixed threshold are considered candidate center points. Non-maximum suppression is performed on all candidate center points to find the center of each instance. The point with the highest center score is first selected as the first candidate point. At the same time, other points within a fixed radius r around the candidate point and with a feature distance greater than the fixed threshold are removed. This removal process is repeated until there are no more points. At this time, the cluster center point of each instance is obtained. Then, the instance to which each point belongs is determined by calculating the feature distance between each point and the cluster center point, and the clustering result is obtained, and the segmentation is finally completed.

[0049] Figure 4 and Figure 5 The schematic diagram of the industrial parts point cloud segmentation effect is shown. The method in this example uses adaptive graph convolution and feature distance-based inference clustering method to allow the network to learn more in-depth and accurate point cloud features and implement a fast and accurate clustering method, with excellent segmentation effect.

Claims

1. A point cloud clustering and segmentation method for industrial parts. It is characterized in that The steps include: 1) Data acquisition: Convert the ground truth format of the acquired data set from a CSV file to a JSON file, that is, parse each line of data in the CSV file and convert the CSV file into a key-value pair structure that conforms to the JSON format. After the conversion is completed, save the JSON file; Next, the depth image in the data set is converted into point cloud data. Point cloud data is a set of points in three-dimensional space, and each point contains three coordinates (x, y, z). When the depth image is converted into point cloud data, the corresponding three-dimensional coordinates can be calculated by the depth value of each point in the depth image; Then the point cloud data is sampled and normalized. Data sampling can reduce the amount of point cloud data to improve processing speed and efficiency; normalization is to scale the coordinate values ​​of the point cloud data to a fixed range; Finally, convert the processed point cloud data text file into an HDF5 format file. By saving the point cloud data as an HDF5 format file, the point cloud data can be easily loaded and used for model training. 2) Construct feature extractor: Adaptive graph convolution is used to input point cloud M∈R n×f Process and obtain each point MK∈R n×f×k Then, two convolutional layers and a maximum pooling layer are used to eliminate the point cloud rotation anisotropy and low-level geometric structure features, respectively, to obtain MKL∈R n×64 , the obtained MKL is again passed through adaptive graph convolution, two convolution layers and a maximum pooling layer to obtain the intermediate layer geometric structure feature MKM∈R n×64 MKM again obtains the high-level geometric structure feature MKH∈R through adaptive graph convolution, two convolution layers and maximum pooling layers n×64 , MKL, MKM and MKH are combined, and the point cloud global feature MKG∈R is output through a single convolution layer and a maximum pooling layer 1024 Finally, MKL, MKH, MKH and MKG are combined to form a new point cloud geometric feature space, and the final point cloud encoding feature F is output through two fully connected layers. backbone ∈R n×256 ; The above is the construction process of the feature extractor; 2-1) Adaptive Graph Convolution: Since 3D sensors only collect point cloud information on the surface of objects, these point cloud information only contains the three-dimensional coordinates of the points but not the color and normal features. These point cloud information intuitively reflects the geometric shape of the object; therefore, when processing pure point cloud data, its local characteristics should be fully utilized. Adaptive graph convolution can better utilize the local characteristics of point clouds by capturing the local geometric structure of point clouds. Adaptive graph convolution first finds the neighbors of each point based on the k-nearest neighbor method. For each point i and its neighbor point j, the feature difference h′ is calculated. i =∑ j∈N(i) σ(h j -h i ||h i ), where N(i) represents the domain of point i, || represents the feature concatenation operation, and σ represents the sigmoid activation function; 3) Training phase: After extracting the input point cloud features in the feature extractor, the center score branch and feature distance branch are used to predict the center score of each point in the point cloud and calculate the feature distance between each point in the point cloud; 3-1) Center score branch: The center score reflects the distance between a point and its instance center. The score of a point close to the center of an object is higher than that of a point on the boundary. In the center score branch, the extracted point cloud encoding feature F backbone ∈R n×256 After two layers of MLP, we get N×1 point-by-point features, and then use the center score loss function Optimize the center score of each point, where s center(i) represents the true score of each point, s^ center(i) represents the predicted score; 3-2) Feature distance branch: In the feature space, points belonging to the same instance should be close, while points of different instances should be far away. In order to make the features of points of the same instance similar, the extracted point cloud encoding features F backbone ∈R n×256 After one layer of MLP, we get N×128 point-by-point embedding features, and then use the feature distance loss function It forces the network to optimize the similarity matrix during training, so that the similarity between points in the same group increases and the similarity between points in different groups decreases, where w (i,j) represents the weight matrix obtained by element-by-element multiplication, k (i,j) Represents the loss based on point-to-point relationships; Finally, the weighted sum of the center score loss and the feature distance loss is the final total loss, and the formula is L total =(1-α)·L CS +α·L EF ; 4) Inference clustering stage: Points above a fixed threshold are considered candidate center points. Non-maximum suppression is performed on all candidate center points to find the center of each instance. The point with the highest center score is first selected as the first candidate point. At the same time, other points within a fixed radius r around the candidate point and with a feature distance greater than the fixed threshold are removed. This removal process is repeated until there are no more points. At this time, the cluster center point of each instance is obtained. Then, the instance to which each point belongs is determined by calculating the feature distance between each point and the cluster center point, and the clustering result is obtained, and the segmentation is finally completed.

Images