An industrial scattered part instance segmentation method based on an improved DGCNN network

Abstract

The application provides an industrial scattered part instance segmentation method based on an improved DGCNN network, comprising the following steps: obtaining a point cloud scene of an industrial scattered part accumulation, removing outliers and platform planes in the scene through a preprocessing means, generating a large amount of industrial scene part point cloud data based on a virtual dataset generation algorithm for instance segmentation, carrying out deep learning training by integrating a real industrial scene point cloud graph into the virtual dataset, training an instance segmentation model by using an improved DGCNN network FPCC-Net, and realizing a three-dimensional industrial scene instance segmentation task of an industrial robot in a scattered and stacked industrial part and under a partial occlusion condition, significantly meeting the three-dimensional intelligent perception and autonomous operation requirements of the industrial robot in a complex part scene, and effectively and accurately segmenting a target from a background.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial robotics, and in particular to a method for segmenting scattered industrial parts instances based on an improved DGCNN network. Background Technology

[0002] In scenarios involving unordered robotic sorting, many objects of the same category are stacked together. Traditional 2D imaging technology cannot meet the precise positioning requirements of robots during the grasping process. 3D vision sensors are gradually becoming a research focus, and their applications are expanding into fields such as autonomous driving and smart homes. Currently, research on 3D instance segmentation tasks is still in its early stages. Different methods, including feature-based clustering and bounding box-based deep learning methods, all face challenges in improving accuracy and speed.

[0003] How to fully utilize the 3D point cloud scene information acquired by the 3D vision system to robustly complete the 3D industrial scene instance segmentation task in real time when industrial parts are scattered and partially occluded is an urgent technical problem to be solved in this field. Summary of the Invention

[0004] To address at least some of the aforementioned problems, this invention provides a method for segmenting industrial scattered parts instances based on an improved DGCNN network. To achieve the above objectives, the following technical solution is provided:

[0005] A method for segmenting scattered industrial parts based on an improved DGCNN network includes the following steps:

[0006] Step 1: Acquire M point cloud images of a real industrial scene with 0-N parts using a 3D vision system;

[0007] Step 2: Use a statistical outlier removal algorithm to remove common outliers from the real point cloud image, and use them as the training set and test set respectively.

[0008] Step 3: Use an improved virtual dataset creation algorithm to generate a large amount of point cloud data that meets the requirements for training the instance segmentation network;

[0009] Step 4: Import the CAD model of the corresponding part, generate a collision model according to the V-HACD algorithm, and generate the corresponding URDF file and uniformly sampled point cloud file;

[0010] Step 5: Set the dataset generation parameters: maximum number of parts N per loop, data collection period K, maximum part size, and box size;

[0011] Step 6: Start Pybullet drop items, create a Pybullet virtual scene with boxes, drop N parts in sequence, record the part pose, RGB image, and depth map, clear the scene, and then repeat N+1 times until the maximum limit is reached.

[0012] Step 7: Reconstruct the complete scene using Open3d. Reconstruct the complete scene based on the .ply point cloud file generated in Step 2 and the pose from Step 4. Add a model point cloud size transformation function to adjust the input point cloud to the actual size of the workpiece to meet the actual grasping requirements.

[0013] Step 8: Calculate center scores based on the object model at each point, and generate a center score and segmentation index;

[0014] Step 9: Reconstruct the scene point cloud data from Step 5 based on the camera focal length filter, and save the filtered data to an .H5 file;

[0015] Step 10: After performing voxel downsampling and random sampling on the dataset created in Steps 3-9, the original 3D point cloud in the dataset is transformed to a new coordinate system and input into the improved DGCNN feature extractor to extract point-by-point features.

[0016] Step 11: The point features extracted in Step 10 are fed into the embedding feature branch and the center score branch respectively to generate embedding features of size N×128 and predicted center scores of size N×1;

[0017] Step 12: Perform non-maximum suppression on the points with center scores to obtain the center of each instance, and then generate predicted instances based on the distance from the features to the center points to obtain the classification results.

[0018] As a preferred embodiment, the specific process of outlier removal using statistical methods in step 2 is as follows:

[0019] Step 2.1: Set the number of points k to be calculated in the neighborhood and the standard deviation weighting coefficient β representing the threshold;

[0020] Step 2.2: For point p with index i in the point cloud i Calculate the average distance from it to the k nearest points, denoted as d. i ;

[0021] Step 2.3: Calculate the average distance of all points, and then calculate the mean and standard deviation of the distribution of d, denoted as μ and ε, respectively;

[0022] Step 2.4: Calculate the filtering distance threshold ds = μ + β·ε;

[0023] Step 2.5: Points pi whose average distance di is greater than the filtering distance threshold ds are considered outliers and removed.

[0024] As a preferred option, the specific steps of the improved virtual dataset creation algorithm in steps 3-9, which generates a large amount of point cloud data that meets the training requirements of the instance segmentation model, are as follows:

[0025] Since the point cloud feature extraction algorithm DGCNN used in this paper is insensitive to point cloud density, and the density of the workpiece point cloud map varies greatly in actual industrial scenarios and simulated industrial scenarios, and the point cloud density is uneven, a collision function is added in step a and the model is uniformly sampled to enable the network to be trained in a wider density range, thereby improving its adaptability to point clouds of different densities.

[0026] Step a: Import the CAD model of the workpiece. First, calculate the vertex normals of the model, uniformly sample from the model, and generate the model.ply file. Then, use the V-HACD (Volumetric Hierarchical Approximate ConvexDecomposition) algorithm to generate collisions. Set the parameter alpha (alp ha) to control the accuracy of the model; a smaller alpha value will cause leaf nodes to be preserved. Set the resolution parameter resolution, which determines the level of detail of the generated collision mesh, thus generating a more accurate collision model. Finally, write the basic information, inertia, visual, and collision attributes of the workpiece into the urdf file.

[0027] Step b: Set the physics engine parameters, ground model, and other simulation environment settings, as well as the box size and object drop range. Set the number of loops M. In each loop, place the box and randomly place 0-N objects every 0.25 seconds. Use a virtual camera to capture the object's viewpoint image in each loop, including RGB images, depth images, and segmentation images. Save the generated images to the specified folder, and save the object pose information (xyz coordinates and quaternions) and pose matrix (4x4 matrix) to a text file.

[0028] Step c: Read the pose information from step b, generate point cloud model instances into the scene through matrix transformation, and process the generated scene with labels (instance labels and center scores). Save the point cloud (xyz) and label of each instance to the corresponding scene file. The file format is that each line represents a point, including coordinates (x, y, z), instance label, and center score.

[0029] Step d: First, define a scene cropping function (scene_crop) to crop the scene point cloud from step c, ensuring it is within the camera's field of view; define a point cloud normalization function (samples_normalize d) to normalize the sampled point cloud data, mapping the data to the range [0,1]; then define a save_h5 function to save the normalized point cloud data as an H5 file, containing the point cloud's coordinates, center score, and label information; finally, iterate through each cycle and each object, reading the original point cloud data, performing cropping, sampling, and normalization, and saving the cropped point cloud file (scene_cropped) and the H5 file.

[0030] Step e: First, voxel downsampling and label processing are performed on the N real industrial scene point cloud images collected in step 1 through step c. The real industrial scene point cloud (xyz), labels and center scores are saved to the scene (scene_cam) file. Then, the scene (scene_cam) is cropped, sampled and normalized through step d to obtain the cropped point cloud file and the H5 file for training.

[0031] Preferably, in step 10, a hybrid sampling method of voxel downsampling and random sampling is used to sample the 3D point cloud in the dataset. The specific steps for feeding the sampled point cloud into the feature extraction network after coordinate system transformation are as follows:

[0032] Step 10.1: Perform voxel downsampling on the point cloud data, with a voxel size of 0.005, and then define a random sampling function to randomly sample the point cloud data after voxel downsampling.

[0033] Step 10.2: The specific process of 3D point cloud coordinate system transformation is as follows:

[0034] First, the coordinates p of the original 3D point cloud (N×3) are... i =(x i ,y i ,z i (i = 1, 2, ..., N) can be transformed to the new coordinate system using the following formula:

[0035]

[0036] The transformed point cloud (N×6) has each point composed of... The six-dimensional vector (6D) and the normalized position n of the entire scene x ,n y ,n z The transformed point cloud (N×6) is then fed into the feature extraction network.

[0037] Step 10.3: The specific process of extracting point-by-point features using the improved DGCNN algorithm in step 10 is as follows:

[0038] When extracting features from point clouds of workpieces in industrial scenarios, the DGCNN network utilizes local geometry to construct a local neighborhood graph and applies edge convolution (EdgeConv) operations on the edges connecting adjacent point pairs to leverage the local geometry.

[0039] (1) This invention employs the segmentation network of DGCNN and improves upon it. Through multi-layer feature connections, new features are learned from previous features using graph convolution in the current layer. First, the spatially transformed point features are input into EdgeConv1 and simultaneously connected to EdgeConv2 and EdgeConv3 modules to extract edge features. The output of net1 is input into EdgeConv2 to extract edge features, thus achieving multi-layer feature connections. Then, the feature B×N×1×192 obtained by fusing net1, net2, and net3 is expanded to B×N×1×1024 for each point after mlp{1024} and max pooling. The features of net1, net2, and net3 are then concatenated with the expanded feature B×N×1×1024 along the feature channels to output B×N×1×1216, completing global feature expansion and concatenation. The last two layers of the segmentation network are then removed.

[0040] (2) Spatial transformation module: The point cloud input is aligned with the typical space by applying an estimated 3×3 matrix. This 3×3 matrix is ​​estimated using a tensor that connects the coordinate differences between each point and k neighboring points.

[0041] (3) EdgeConv is an innovative convolutional operation designed to efficiently extract local geometric features from point cloud data. Unlike traditional convolutional operations that operate on regular grid structures, EdgeConv constructs a dynamic k-nearest neighbor (k-NN) graph at each layer, enabling it to capture complex local geometric relationships. This operation significantly enhances the ability of neural networks to process unstructured data (such as point clouds) by dynamically updating the adjacency graph at each layer based on the learned feature space.

[0042] Given input point cloud features X∈R N×F Vertex set:

[0043]

[0044] Where N is the number of points and F is the feature dimension.

[0045] Given a directed graph G = (υ, ξ) representing a local point cloud structure, with vertices and edges υ = {1, 2, ..., n} and ζ ∈ υ × υ. Construct a graph in R with vertices X.F The k-nearest neighbor (k-nn) graph G is used, and the k-nn graph of the points in the graph is recalculated in each layer.

[0046] d) EdgeConv operation

[0047] For each point i, its feature vector is x. i The eigenvectors x of its neighboring points j EdgeConv calculates the edge features between point i and its neighbor point j:

[0048] e ij =h θ (x i ,x j -x i )

[0049] =LeakyReLU(α) n ·(x j -x i )+β n ·x i )

[0050] Among them, h θ It is a learnable function (typically a multilayer perceptron) used to combine the features of a point itself with the feature differences relative to its neighbors. It explicitly combines the global shape structure captured by the coordinates of the point center xi with the local neighborhood information captured by xj-xi.

[0051] e) Feature aggregation operation

[0052] By analyzing the edge features e between point i and its neighbor point j ij Aggregation (such as max pooling or average pooling) is performed to obtain a new feature representation x' of the point. i :

[0053] x′ i =max j∈Ν(i) e ij

[0054] Where N(i) is the set of the k nearest points to each point i, containing j1, j2, ... j n .

[0055] Preferably, in step 11, the point-state features extracted in step 10 are fed into the embedding feature branch and the center score branch respectively, and the features of each point in the feature embedding space are encoded respectively. The specific process of inferring the center score of each point is as follows:

[0056] Step 11.1: In the center score branch, the point orientation features generated by the feature extractor in step 10 are activated by the sigmoid function after passing through two MLPs. Then, a predicted center score of size N×1 is obtained.

[0057] Step 11.2: In the embedding feature branch, the 256-dimensional features extracted in step 10 are processed by MLP to generate an embedding feature of size N×128. The embedding feature branch introduces the feature distance matrix, the effective distance matrix, and the attention mechanism to train the instance segmentation model, and calculates the embedding feature loss and the center score loss to supervise the training process.

[0058] (1) Feature distance matrix FDM

[0059] In the feature embedding space, points belonging to the same instance should be close, while points from different instances should be separate. To make the features of points within the same instance similar, the following feature distance matrix D is introduced. F ∈R N×N .

[0060]

[0061] (2) Effective distance matrix VDM

[0062] Effective distance matrix D V ∈R N×N It is a binary matrix where each element is either 0 or 1. The effective distance matrix is ​​used to determine whether two points within a certain Euclidean distance belong to the same instance. If the Euclidean distance between two points exceeds the maximum distance d... max If the two points are twice the same, then they do not belong to the same instance.

[0063]

[0064] The above formula indicates whether the Euclidean distance between points is within a reasonable range.

[0065] (3) Attention Scoring Mechanism (ASM)

[0066] In the FPCC method, we introduce an attention score matrix to improve the performance of point cloud clustering and segmentation. Specifically, the score matrix S is constructed as the sum of scores for each pair of points, and its symmetry and reasonable range are ensured through symmetry and pruning operations. Finally, the attention score matrix is ​​used to adjust the similarity matrix loss, giving more weight to more important point pairs in the loss calculation, thereby improving the accuracy of clustering and segmentation. The specific process is as follows:

[0067] Construct a score mask matrix: S represents the score matrix with shape [B,N,1], where B is the batch size and N is the number of points, and expand it to shape [B,N,N].

[0068] S i,j =s i +s j

[0069] S exp anded =Re peat(S i,j (dim=2, num=N)

[0070] Where s i It is the score of point i, s j It is the score for point j.

[0071] Symmetrically transform the score mask matrix and restrict it to the range [0,1]:

[0072]

[0073] S i,j =min(max(S) i,j ,0),1)

[0074] Applying the score mask matrix: Applying the score mask to the similarity matrix loss.

[0075] L simmat =L simmat ⊙S

[0076] Where ⊙ represents element-wise multiplication.

[0077] Calculate the average loss: the average loss of the similarity matrix.

[0078]

[0079] The attention mechanism described above weights the similarity between feature points, making the model pay more attention to important feature points, thereby enhancing the model's performance and improving overall performance.

[0080] (4) Center score

[0081] The center score reflects the distance between a point and its corresponding center; points closer to the center of the object receive a higher score than points on the boundary. i The center score is calculated as follows:

[0082]

[0083] Where β is a positive constant, and c i These are the coordinates of the geometric center of the instance.

[0084] (5) Embedded feature loss

[0085] In instance segmentation scenarios, point pairs (i,j) can only have two relationships: 1) belonging to the same instance; 2) belonging to different instances. Therefore, we define the embedding feature loss L... EF :

[0086]

[0087] Where, ω (i,j) Through D V and D A The element-wise multiplication yields W∈R N×N One of the elements.

[0088] ω (i,j) =d V(i,j) s A(i,j)

[0089] K (i,j) The loss is based on point-to-point relationships and is defined as:

[0090]

[0091] Where ∈1 and ∈2 are constants and satisfy 0 < ∈1 < ∈2.

[0092] (6) Center score loss

[0093] Due to the robustness of the L1 loss function, the smooth L1 loss is used as the center score branch loss function, and L is defined as follows: CS as follows:

[0094]

[0095] in, Indicates the predicted center score

[0096]

[0097] Preferably, in step 12, non-maximum suppression is applied to the points with center scores to obtain the center of each instance. Then, predicted instances are generated based on the distance from the features to the center points. The specific steps to obtain the classification results are as follows:

[0098] Step 12.1: After predicting the center scores, we use the NMS algorithm to find the geometric center of each object and use these points as reference points in the clustering process.

[0099] Non-maximum suppression (NMS) is applied to all points with center scores to find the center of each instance. Center scores above θ are considered center scores. thPoints with a center score of 0.6 are considered candidate points for the center. The point with the highest center score is selected as the first candidate point for the center, and points with a radius of γd centered on the candidate point are removed. max Other points in the sphere, where d max It is the maximum distance from the geometric center to the farthest point of the object, and γ is the selection factor. Repeat this process until no more points remain.

[0100] Step 12.2: Generate predicted instances based on the distance from the features to the center point. The specific steps to obtain the classification results are as follows:

[0101] After processing by the Non-Maximum Suppression (NMS) algorithm, the feature distance between the center point and other points is calculated:

[0102]

[0103] in, This represents the feature of the k-th center point selected in the NMS algorithm. Let p represent the feature of the i-th remaining point. All points except the center point are clustered with the center point whose feature distance is closest. Find point p in the feature embedding space. i The nearest center point c k Calculate p in 3D space i and c k The Euclidean distance d(p) between them i ,c k If d(p) i ,c k ) greater than d max If pi is considered noise, it will not be assigned an instance label.

[0104] The beneficial effects of this invention are as follows:

[0105] 1. High-quality datasets are provided for the training and testing phases of instance segmentation models through preprocessing, data augmentation, and other methods.

[0106] 2. In view of the problem of lack of actual workpiece datasets and the time-consuming manual annotation, this invention proposes an automatic generation method for workpiece instance segmentation datasets that does not require manual annotation. It can automatically generate various workpieces based on CAD models, saving a lot of manpower.

[0107] 3. This invention proposes a hybrid sampling method of voxel downsampling and random sampling before point cloud data is fed into the model, which effectively reduces the density of point cloud, reduces the amount of data, accelerates the subsequent processing speed, and improves the stability and accuracy of processing.

[0108] 4. This invention uses the FPCC-Net instance segmentation network as a foundation and makes improvements in several aspects to address the shape characteristics of the research object DB9. First, considering that DB9 contains a large number of vertices and faces, the model may exhibit unreasonable jitter, jumping, or jamming during simulation. A collision model is added to generate a simplified and accurate convex hull model, ensuring that the generated dataset accurately reflects the physical characteristics of the object, thereby improving the quality of the dataset and the training effect of the model. Then, this invention learns new features from previous features through multi-layer feature links. By improving the DGCNN network layer structure, the complexity of the model is reduced, and the computational efficiency is significantly improved. Attached Figure Description

[0109] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0110] Figure 1 This is a flowchart of an industrial scattered parts instance segmentation method based on an improved DGCNN network according to the present invention;

[0111] Figure 2 These are comparison images of point clouds before and after preprocessing. Figure 2 (a) A point cloud image of a real industrial scene taken with an industrial camera. Figure 2 (b) is the point cloud after outlier removal based on a statistical strategy;

[0112] Figure 3 The network structure graph is segmented for instances, where, Figure 3 (a) Improved DGCNN segmentation network structure Figure 3 (b) is a diagram of the FPCC-Net instance segmentation network structure based on the improved DGCNN feature extractor;

[0113] Figure 4 Visualize the instance segmentation results, including the real-world RGB image, 3D point cloud image, center score point cloud image, and segmentation result point cloud image;

[0114] Figure 5 This section compares the results of different algorithms on different datasets, including the segmentation results of four algorithms on DB9 and circular hole iron sheets. Detailed Implementation

[0115] The present invention will now be further described in conjunction with the accompanying drawings and embodiments.

[0116] The scattered parts scene point cloud described in this invention was obtained by scanning the environment of the Gocator3110 LED structured light 3D sensor, and contains the 3D coordinate information of points in the industrial scattered parts scene point cloud. As an application example, this invention provides test results for a self-made dataset with DB9 serial connection port as the instance segmentation target. The dataset contains 8000 scattered parts scenes, including 500 real industrial scene point cloud images (400 as the training set and 100 as the validation set) and 7500 synthetic industrial scene point cloud images (5000 as the training set and 2500 as the validation set). Each scene point cloud image in the training set contains 1-30 parts, and each scene point cloud image in the test set contains 10-30 parts.

[0117] like Figure 1 The diagram shows a flowchart of a method for segmenting industrial scattered parts instances based on an improved DGCNN network according to the present invention, which includes the following steps:

[0118] Step 1: Acquire M point cloud images of a real industrial scene with 0-N parts using a 3D vision system;

[0119] Step 2: Use a statistical outlier removal algorithm to remove common outliers from the real point cloud image, and use them as the training set and test set respectively.

[0120] The specific steps of the elimination method are as follows:

[0121] 2.1: Set the number of points to be calculated in the knn neighborhood search method to k = 20, and the standard deviation weighting coefficient of the threshold to β = 0.1;

[0122] 2.2: For the i-th point p i The average distance d to its k nearest points i ;

[0123] 2.3: Calculate the average distance p of all points i Calculate the mean and standard deviation of the distribution of d, denoted as μ and ε, respectively;

[0124] 2.4 Calculate the filter threshold distance d s =μ+β·ε;

[0125] 2.5 The average distance d i Greater than the filter distance threshold d s Points that are considered outliers are removed.

[0126] Step 3: Use an improved virtual dataset creation algorithm to generate a large amount of point cloud data that meets the requirements for training the instance segmentation network;

[0127] Step 4: Import the CAD model of the corresponding part, generate a collision model according to the V-HACD algorithm, and generate the corresponding URDF file and uniform sampling point cloud file;

[0128] Step 5: Set the dataset generation parameters: maximum number of parts N per cycle, data collection period K, maximum part size, and box size;

[0129] Step 6: Start Pybullet drop items, create a Pybullet virtual scene with boxes, drop N parts in sequence, record the part pose, RGB image, and depth map, clear the scene, and then repeat N+1 times until the maximum limit is reached.

[0130] Step 7: Reconstruct the complete scene using Open3d. Reconstruct the complete scene based on the .ply point cloud file generated in Step 2 and the pose from Step 4. Add a model point cloud size transformation function to adjust the input point cloud to the actual size of the workpiece to meet the actual grasping requirements.

[0131] Step 8: Calculate center scores based on the object model at each point, and generate a center score and segmentation index;

[0132] Step 9: Reconstruct the scene point cloud data from Step 5 based on the camera focal length filter, and save the filtered data to an .H5 file;

[0133] Furthermore, the specific steps of the improved virtual dataset creation algorithm in steps 3-9, which generates a large amount of point cloud data that meets the training requirements of the instance segmentation model, are as follows:

[0134] Since the point cloud feature extraction algorithm DGCNN used in this invention is insensitive to point cloud density, and the density of the workpiece point cloud map varies greatly in actual industrial scenarios and simulated industrial scenarios, and the point cloud density is not uniform, a collision function is added in step a and the model is uniformly sampled so that the network can be trained in a wider density range, thereby improving its adaptability to point clouds of different densities.

[0135] Step a: Import the CAD model of the workpiece. First, calculate the vertex normals of the model. Uniformly sample 100,000 points from the model to generate the model.ply file. Then, use the V-HACD (Volumetric Hierarchical Approximate Convex Decomposition) algorithm to generate collisions. Set the parameter alpha, which controls the model's accuracy, to 0.04. A smaller alpha value will cause leaf nodes to be preserved. Set the resolution parameter resolution, which determines the level of detail of the generated collision mesh, to 50,000 to generate a more accurate collision model. Finally, write the workpiece's basic information, inertia, visual, and collision attributes into the urdf file.

[0136] Step b: Set the physics engine parameters, ground model, and other simulation environment settings, as well as the box size and object drop range. Set the number of loops M. In each loop, place the box and randomly place 0-N objects every 0.25 seconds. Use a virtual camera to capture the object's viewpoint image in each loop, including RGB images, depth images, and segmentation images. Save the generated images to the specified folder, and save the object pose information (xyz coordinates and quaternions) and pose matrix (4x4 matrix) to a text file.

[0137] Step c: Read the pose information from step b, generate point cloud model instances into the scene through matrix transformation, and process the generated scene with labels (instance labels and center scores). Save the point cloud (xyz) and label of each instance to the corresponding scene file. The file format is that each line represents a point, including coordinates (x, y, z), instance label, and center score.

[0138] Step d: First, define a scene cropping function (scene_crop) to crop the scene point cloud from step c, ensuring it is within the camera's field of view; define a point cloud normalization function (samples_normalize d) to normalize the sampled point cloud data, mapping the data to the range [0,1]; then define a save_h5 function to save the normalized point cloud data as an H5 file, containing the point cloud's coordinates, center score, and label information.

[0139] Step e: First, perform voxel downsampling, random sampling, and label processing on the N real industrial scene point cloud images collected in step 1 through step c. Save the real industrial scene point cloud (xyz), labels, and center scores to the scene (scene_cam) file. Then, perform cropping, sampling, and normalization processing on the scene (scene_cam) through step d to obtain the cropped point cloud file and the H5 file used for training.

[0140] Furthermore, in step 10, the 3D point cloud in the dataset undergoes a hybrid sampling process of voxel downsampling and random sampling. The sampled point cloud is then transformed in coordinate system before being fed into the feature extraction network. The specific steps are as follows:

[0141] Step 10.1: Perform voxel downsampling on the point cloud data, with a voxel size of 0.005, and then define a random sampling function to randomly sample the point cloud data after voxel downsampling.

[0142] Step 10.2: The specific process of 3D point cloud coordinate system transformation is as follows:

[0143] First, the coordinates p of the original 3D point cloud (N×3) are... i =(x i ,y i ,z i (i = 1, 2, ..., N) can be transformed to the new coordinate system using the following formula:

[0144]

[0145] The transformed point cloud (N×6) has each point composed of... The six-dimensional vector (6D) and the normalized position n of the entire scene x ,n y ,n z The transformed point cloud (N×6) is then fed into the feature extraction network.

[0146] Step 10.3: The specific process of extracting point-by-point features using the improved multi-layer feature fusion network in step 10 is as follows:

[0147] First, the coordinate-transformed features are respectively combined with the second and third edge convolutional modules. The output features of net1 are added to the edge convolutional outputs (local descriptors) of net2 and net3. Through multi-layer feature fusion, new features are learned from previous features using the graph convolution of the current layer. Then, the fused features B×N×1×192 of net1, net2, and net3 are expanded to B×N×1×1024 for each point after mlp{1024} and max pooling. The features of net1, net2, and net3 are then concatenated with the expanded features B×N×1×1024 along the feature channels to output B×N×1×1216, completing global feature expansion and concatenation. Finally, the last two layers of the segmentation network are removed.

[0148] The specific process of feeding the point features extracted in step 8 into the embedding feature branch and the center score branch in step 11 is as follows:

[0149] First, the point-state features of each point are extracted using the multi-layer feature fusion network proposed in this invention and fed into two branches. These branches encode the features of each point in the feature embedding space, and the center score branch infers the center score of each point. In the embedding feature branch, the 256-dimensional features extracted in step 10 are processed by MLP to generate embedding features of size N×128. The embedding feature branch introduces a feature distance matrix, an effective distance matrix, and an attention mechanism to train the instance segmentation model, and calculates the embedding feature loss and center score loss to supervise the training process. In the center score branch, the point orientation features extracted in step 10 are activated by the sigmoid function after passing through two MLPs. Then, a predicted center score of size N×1 is obtained.

[0150] Step 12: Perform non-maximum suppression on the points with center scores to obtain the center of each instance. Then, generate predicted instances based on the distance from the features to the center points to obtain the classification results. The specific steps are as follows:

[0151] The NMS algorithm is used to find the geometric center of each object, and these points are used as reference points in the clustering process. Center scores higher than θ are considered geometrically positive. th Points with a score of 0.6 are candidate points. The point with the highest score is selected as the first candidate center point, and points with a radius of γd centered on these candidate points are removed. max Other points within the sphere. Calculate the characteristic distances between the center point and other points:

[0152]

[0153] in, This represents the feature of the k-th center point selected in the NMS algorithm. Let p represent the feature of the i-th remaining point. All points except the center point are clustered with the center point whose feature distance is closest. Find point p in the feature embedding space. i The nearest center point c k Calculate p in 3D space i and c k The Euclidean distance d(p) between them i ,c k If d(p) i ,c k ) greater than d max If pi is considered noise, it will not be assigned an instance label.

[0154] Experimental setup

[0155] The instance segmentation network in this invention is implemented in the PyTorch framework and trained using the Adam optimizer with an initial learning rate of 0.0001, a batch size of 2, and a momentum of 0.9. All training and testing were performed on an Nvidia GTX3060 GPU and an Intel Core i7-12700 with 16GB of RAM. During the training phase, the DB9, toroidal screw, and iron sheet were used... max The values ​​were 0.12, 0.08, and 0.08, respectively. The network was trained for 30 epochs.

[0156] Multi-layer feature fusion is set up, with the mixed sampling points A and B as the improved points, and the original FPCC-Net network as the baseline. The method combination to be verified in the ablation experiments is as follows:

[0157] Table 1: Instance segmentation results of industrial objects

[0158]

[0159]

[0160] Table 1 shows the precision and recall metrics for instance segmentation of gear shafts, circular hole iron sheets, and DB9, measured as precision (%) and recall (%), with an IoU threshold of 0.5. Using the improved DGCNN as the backbone network, FPCC-Net achieved the best results and was competitive with other networks. Ablation experiments show that, in terms of precision, the improved FPCC network after multi-layer feature fusion is superior to the original FPCC network. On the gear shaft dataset used by the original algorithm, the improved network achieved a 9.99% improvement in precision compared to the original network.

[0161] Table 2 Comparison of Calculation Speed

[0162]

[0163] Table 2 reports the average computation time for each scene. Each scene in the DB9 dataset contains 25 to 30 objects, while each scene in the iron sheet dataset contains 55 to 60 objects. The table shows that multi-layer feature fusion is the most computationally intensive improvement method. Hybrid sampling can reduce inference time to some extent.

[0164] Figure 4 The results of instance segmentation network in this invention on instance segmentation and center score prediction for three types of industrial objects are shown. Points closer to the center have higher scores than edge points. The instance segmentation network of this invention can clearly distinguish different instances even when the workpiece is heavily covered. For objects such as DB9, round hole iron sheet, and gear shaft, whose geometric center is on the part, our clustering algorithm is suitable for objects whose center is on the part, and the reference points used in clustering will not be offset from the object center. In real-world scenarios, many objects of the same category are stacked together, and the 3D point cloud uses... The values ​​are then input into the instance segmentation network of this invention. The results of instance segmentation are displayed in the last column.

[0165] Figure 5 The segmentation results of 3D-BoNet, PointGroup, FPCC, and the instance segmentation network of this invention are displayed. The objects in each scene are identical, and different predicted instances are displayed in different colors. In the segmentation of iron sheets and DB9 parts, 3D-BoNet performed the worst among the four algorithms, frequently misidentifying the same object as different objects when there was stacking and overlap. PointGroup performed poorly in iron sheet segmentation. Compared to the original FPCC network, the improved FPCC network achieved more accurate segmentation, with different instances clearly corresponding to different colors.

[0166] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method for segmenting industrial scattered parts instances based on an improved DGCNN network, characterized in that, The method includes the following steps: Step 1: Acquire M point cloud images of a real industrial scene with 0-N parts using a 3D vision system; Step 2: Use a statistical outlier removal algorithm to remove common outliers from the real point cloud image, and use them as the training set and test set respectively. Step 3: Use an improved virtual dataset creation algorithm to generate a large amount of point cloud data that meets the requirements for training the instance segmentation network; Step 4: Import the CAD model of the corresponding part, generate a collision model according to the V-HACD algorithm, and generate the corresponding URDF file and uniform sampling point cloud file; Step 5: Set the dataset generation parameters: maximum number of parts N per cycle, data collection period K, maximum part size, and box size; Step 6: Start Pybullet drop items, create a Pybullet virtual scene with boxes, drop N parts in sequence, record the part pose, RGB image, and depth map, clear the scene, and then repeat N+1 times until the maximum limit is reached. Step 7: Reconstruct the complete scene using Open3d. Reconstruct the complete scene based on the .ply point cloud file generated in Step 2 and the pose in Step 4; add a model point cloud size transformation function to adjust the input point cloud to the actual size of the workpiece to meet the actual grasping requirements. Step 8: Calculate center scores based on the object model at each point, and generate a center score and segmentation index; Step 9: Reconstruct the scene point cloud data from Step 5 based on the camera focal length filter, and save the filtered data to an .H5 file; Step 10: After performing voxel downsampling and random sampling on the dataset created in Steps 3-9, the original 3D point cloud in the dataset is transformed to a new coordinate system and input into the improved DGCNN feature extractor to extract point-by-point features; the improved DGCNN algorithm includes multi-layer feature fusion. Step 11: The point features extracted in Step 10 are fed into the embedding feature branch and the center score branch respectively to generate embedding features of size N×128 and predicted center scores of size N×1; Step 12: Perform non-maximum suppression on the points with center scores to obtain the center of each instance, and then generate predicted instances based on the distance from the features to the center points to obtain the classification results; The specific process of outlier removal using statistical methods in step 2 is as follows: Step 2.1: Set the number of points k to be calculated in the neighborhood and the standard deviation weighting coefficient β representing the threshold; Step 2.2: For point p with index i in the point cloud i Calculate the average distance from it to the k nearest points, denoted as d. i ; Step 2.3: Calculate the average distance of all points, and then calculate the mean and standard deviation of the distribution of d, denoted as μ and ε, respectively; Step 2.4: Calculate the filtering distance threshold d s =μ+β·ε; Step 2.5: Calculate the average distance d i Greater than the filter distance threshold d s point p i These are considered outliers and are removed.

2. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 1, characterized in that, By adding a collision function and uniformly sampling the model, the network can be trained over a wider density range, thereby improving its adaptability to point clouds of different densities.

3. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 1, characterized in that, In step 10, the 3D point cloud in the dataset undergoes a hybrid sampling process of voxel downsampling and random sampling. The sampled point cloud is then transformed in coordinate system before being fed into the feature extraction network. The specific steps are as follows: Step 10.1: Perform voxel downsampling on the point cloud data, with a voxel size of 0.005, and then define a random sampling function to randomly sample the point cloud data after voxel downsampling; Step 10.2: The specific process of 3D point cloud coordinate system transformation is as follows: First, the original 3D point cloud coordinates p i =(x i ,y i ,z i (i = 1, 2, ..., N) can be transformed to the new coordinate system using the following formula: ； Each point in the converted point cloud is composed of... The six-dimensional vector and the normalized position n of the entire scene x ,n y ,n z The converted point cloud is then fed into the feature extraction network; Step 10.3: The specific process of extracting point-by-point features using the improved DGCNN algorithm in step 10 is as follows: (1) The segmentation network of DGCNN is adopted and the network is improved. Through multi-layer feature connection, new features are learned from previous features using graph convolution of the current layer. First, the spatially transformed point features are input into EdgeConv1 and connected with EdgeConv2 and EdgeConv3 modules to extract edge features. The output of net1 is input into EdgeConv2 to extract edge features to realize multi-layer feature connection. Then, the feature B×N×1×192 obtained by fusing net1, net2 and net3 is extended to each point B×N×1×1024 after mlp{1024} and max pooling. The features of net1, net2 and net3 and the extended feature B×N×1×1024 are concatenated along the feature channel to output B×N×1×1216 to complete global feature expansion and concatenation. The last two layers of the segmentation network are removed. (2) Spatial transformation module: The point cloud input is aligned with the typical space by applying an estimated 3×3 matrix. This 3×3 matrix is ​​estimated using a tensor that connects the coordinate differences between each point and k neighboring points. (3) EdgeConv convolution operation is designed to effectively extract local geometric features from point cloud data. Unlike traditional convolution operations that run on regular grid structures, EdgeConv builds a dynamic k-nearest neighbor (k-NN) graph at each layer, enabling it to capture complex local geometric relationships. This operation significantly enhances the ability of neural networks to process unstructured data by dynamically updating the adjacency graph at each layer based on the learned feature space.

4. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 3, characterized in that, The EdgeConv convolution operation is as follows: a) Construction of dynamic K-NN graph Given input point cloud features X∈R N×F Vertex set: ； Where N is the number of points, and F is the feature dimension; Given a directed graph G = (υ, ξ) representing a local point cloud structure, with vertices and edges υ = {1, 2, ..., n} and ζ ∈ υ × υ; construct a graph in R with X as the vertex. F In the k-nearest neighbor graph G, the k-nn graph of points in the graph of each layer is recalculated; b) EdgeConv operation For each point i, its feature vector is x. i The eigenvectors x of its neighboring points j EdgeConv calculates the edge features between point i and its neighbor point j: e ij =h θ (x i ,x j -x i ) =LeakyReLU(α n (x j -x i )+β n ·x i ) Among them, h θ It is a learnable function (usually a multilayer perceptron) used to combine the features of a point itself with the feature differences relative to its neighbors; it explicitly combines the global shape structure captured by the coordinates of the point center xi with the local neighborhood information captured by xj-xi. c) Feature aggregation operation By analyzing the edge features e between point i and its neighbor point j ij Aggregation (such as max pooling or average pooling) is performed to obtain a new feature representation x' of the point. i : x′ i =max j∈Ν(i) e ij Where N(i) is the set of the k nearest points to each point i, containing j1, j2, ... j n .

5. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 1, characterized in that, In step 11, the point-state features extracted in step 10 are fed into the embedding feature branch and the center score branch, respectively, to encode the features of each point in the feature embedding space and infer the center score of each point. The specific process is as follows: Step 11.1: In the center score branch, the point orientation features generated by the feature extractor in step 10 are activated by the sigmoid function after passing through two MLPs; then, a predicted center score of size N×1 is obtained; Step 11.2: In the embedding feature branch, the 256-dimensional features extracted in step 10 are processed by MLP to generate an embedding feature of size N×128.

6. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 5, characterized in that, The embedded feature branch introduces a feature distance matrix, an effective distance matrix, and an attention mechanism to train the instance segmentation model, and calculates the embedded feature loss and center score loss to supervise the training process.

7. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 6, characterized in that, Effective distance matrix D V ∈R N×N It is a binary matrix where each element is either 0 or 1. The effective distance matrix is ​​used to determine whether two points within a certain Euclidean distance belong to the same instance. If the Euclidean distance between two points exceeds the maximum distance d... max If the value is twice that of the previous instance, then these two points do not belong to the same instance. ； The above formula indicates whether the Euclidean distance between points is within a reasonable range.

8. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 6, characterized in that, p i The center score is calculated as follows: ； Where β is a positive constant, and c i These are the coordinates of the geometric center of the instance. Define the embedding feature loss L EF : ； Where, ω (i,j) Through D V and D A The element-wise multiplication yields W∈R N×N One of the elements; oh (i,j) =d V(i,j) s A(i,j) K (i,j) The loss is based on point-to-point relationships and is defined as: ； Among them, ∈1 and ∈2 are constants and satisfy 0 < ∈1 < ∈2. Due to the robustness of the L1 loss function, the smooth L1 loss is used as the center score branch loss function, and L is defined as follows: CS as follows: ； in, Indicates the predicted center score 。 9. The method for segmenting industrial scattered parts instances based on an improved DGCNN network according to claim 1, characterized in that, In step 12, NMS is performed on the points with center scores to obtain the center of each instance. Then, predicted instances are generated based on the distance from the features to the center points. The specific steps to obtain the classification results are as follows: Step 12.1: After predicting the center scores, we use the NMS algorithm to find the geometric center of each object and use these points as reference points in the clustering process; NMS is performed on all points with center scores to find the center of each instance; points with center scores above a center score threshold are considered candidate centers; the point with the highest center score is selected as the first candidate center, and points with a radius of γd centered on the candidate center are removed. max Other points in the sphere, where d max It is the maximum distance from the geometric center to the farthest point of the object, and γ is the selection factor; repeat this process until no more points remain; Step 12.2: Generate predicted instances based on the distance from the features to the center point. The specific steps to obtain the classification results are as follows: After processing by the NMS algorithm, the feature distance between the center point and other points is calculated: ； in, This represents the feature of the k-th center point selected in the NMS algorithm. Represent the feature of the i-th point among the remaining points. All points except the center point are clustered with the center point that has the closest feature distance. Find point p in the feature embedding space. i The nearest center point c k Calculate p in three-dimensional space i and c k The Euclidean distance d(p) between them i ,c k If d(p) i ,c k ) greater than d max If pi is considered noise, it will not be assigned an instance label.

Images