A point cloud segmentation method for stacked parts

Abstract

The present invention provides a point cloud segmentation method for stacked parts, comprising the following steps: S1, constructing raw point cloud data; preprocessing the raw point cloud data to obtain point cloud data of the stacked parts to be segmented; S2, inputting the coordinate information of the point cloud data of the stacked parts to be segmented into a feature extraction backbone network to obtain point cloud features; S3, inputting the point-by-point features into a semantic score prediction branch and a geometric offset prediction branch to generate candidate blocks; S4, traversing all candidate blocks, generating instance kernels based on edge detection-attentive sampling results, splicing instance-level features with the sampling results, and dynamically convolving them with the instance kernels to generate binary instance masks; S5, sorting the instance mask point coordinates by instance mask score, removing duplicate results using a NMS algorithm, fusing the instance mask point coordinates with semantic labels, and outputting the final instance segmentation result. The present invention can achieve stable and efficient recognition and segmentation of parts in complex scenes, and can improve recognition and segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision processing, and more particularly to a point cloud segmentation method for stacked parts. Background Art

[0002] With the continuous development of industrial and robotic technologies, automated grasping and assembly based on machine vision are playing an increasingly important role in manufacturing, handling, palletizing, and intelligent logistics. In the past, industrial robots used teach-and-play programming or offline programming for grasping parts on assembly lines, following pre-set trajectories. The placement and angle of the parts to be grasped required strict settings, resulting in low intelligence and flexibility. Enabling machines to perceive and understand complex real-world scenarios has long been a research focus in advanced manufacturing and intelligent production, leading to a surge in research in areas such as point cloud recognition, pose estimation, point cloud segmentation, point cloud registration, and 3D reconstruction.

[0003] Point cloud instance segmentation is an advanced computer vision task that combines target detection and semantic segmentation. It requires not only assigning a category label to each point, but also distinguishing different instances belonging to the same category. A typical application scenario is to segment a single target to be grasped from the entire workpiece to provide target data for subsequent sorting. However, in practice, both semantic distinction and instance separation present significant challenges. Disordered parts are mixed in type, difficult to distinguish by metal color, and parts of different models and specifications have similar appearances. Stacked parts are prone to adhesion, occlusion, and nesting in posture. Point cloud instance segmentation is one of the technical difficulties in machine vision operations.

[0004] Existing instance segmentation methods mainly include proposal-based, clustering-based, and dynamic convolution-based implementation methods. Among them, the proposal-based method is a top-down approach that first generates a 3D bounding box and then segments the foreground area within each bounding box to form an instance. Its quality is heavily dependent on the bounding box prediction results, and its performance is not stable enough. The clustering-based method generates semantics, offsets, or other potential features point by point from the bottom up, and clusters points with similar features to form instances. The limitation of this method is that it is very sensitive to the judgment threshold of semantics and offset predictions. In the actual practice of workpiece stacking, it is easy to go to the two extremes of adjacent workpieces being unable to be separated or parts being over-segmented when occluded. The dynamic convolution-based method generates a convolution kernel and convolves it with point features to encode and decode the instance mask. The quality of the point features and the selection of the instance kernel largely determine the quality of the mask. The convolution and iteration overhead is high in complex environments. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies in the prior art and to provide a point cloud segmentation method for stacked parts. This point cloud segmentation method can achieve stable and efficient identification and segmentation of parts in complex scenes. At the same time, this method can improve the accuracy of part recognition and segmentation in complex working environments where parts are mixed and occluded from each other.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solution: a point cloud segmentation method for stacked parts, characterized in that it includes the following steps:

[0007] S1. Collect the depth information of the entire scene of stacked parts and construct the original point cloud data; separate, downsample, crop, and filter the original point cloud data to obtain the point cloud data of the stacked parts to be segmented;

[0008] S2. Input the coordinate information of the point cloud data of the stacked parts to be segmented into the feature extraction backbone network to obtain point cloud features one by one;

[0009] S3: Input the point-by-point features into the semantic score prediction branch and the geometric offset prediction branch. Soft-group the point cloud features based on the semantic score. Then, the geometric offset prediction branch moves each point cloud feature to the vicinity of its instance center according to its offset to generate a candidate block. Two point-level loss functions supervise the semantic prediction branch and the geometric offset branch respectively.

[0010] S4. Traverse all candidate blocks and generate instance kernels based on the sampling results of edge detection attention. The MLP structure is combined with a lightweight geometric radiation module to obtain instance-level features. The sampling results are spliced ​​with instance-level features and dynamically convolved with the instance kernel to generate a binary instance mask. The instance mask confidence L is calculated. mask To measure the credibility of the mask representation instance currently output;

[0011] S5, refine the instance boundary through the point aggregation layer and predict the semantic score L at the instance mask level for supervision class and the accuracy score L score The instance mask point coordinates are obtained by sorting the instance mask scores and removing duplicate results through the NMS algorithm. The instance mask point coordinates are fused with the semantic labels to output the final instance segmentation result.

[0012] Step S1 includes the following steps:

[0013] Step S11: collecting depth information of the entire scene of stacked parts and constructing raw point cloud data; the depth information of the scene includes background information; filtering out point cloud data of non-interested areas through straight-through filtering to separate the raw point cloud data;

[0014] Step S12: Downsampling uses uniform voxel filtering, replacing the points of the separated point cloud data under the entire voxel with the centroid coordinates of the grid, thereby maintaining the shape details of the point cloud data while reducing the data density;

[0015] Step S13: Select a random sampling consistency algorithm to fit the plane information and frame information in the background information, and remove the platform information and frame information from the point cloud data by combining the normal vectors through multiple iterations;

[0016] Step S14: Statistical filtering is used to analyze and remove outliers that are far away from the point cloud data of the part body, isolated, and sparse, to obtain the point cloud data of the stacked parts to be segmented.

[0017] Step S2 includes the following steps:

[0018] Step S21: uniformly mapping the point cloud data of the stacked parts to be segmented onto a grid for voxelization and inputting the data into a feature extraction network;

[0019] Step S22: applying reverse interpolation to upsample the output backbone features, and performing voxelized inverse mapping to obtain point cloud features one by one;

[0020] Step S23: Use the fully connected network to concatenate the point cloud features containing multi-scale and rich context information along the channel dimension to obtain the final encoded point cloud feature F p ; F p ∈R N×D , where N is the number of point cloud points and D is the dimension of the output channel.

[0021] Step S3 includes the following steps:

[0022] Step S31: Input the final encoded point cloud features into the constructed semantic prediction branch to obtain the semantic score S sem , and soft group the point cloud features;

[0023] Step S32: Input the final encoded point cloud features into the constructed geometric offset prediction branch to obtain the predicted geometric centroid and the offset of each point cloud feature to the center of its instance. According to the offset of each point cloud feature, the point is moved to the vicinity of its instance center to generate a candidate block.

[0024] Step S33: Construct a loss function for supervised semantic prediction and a loss function for supervised geometric offset prediction, and supervise the semantic prediction branch and the geometric offset branch respectively.

[0025] In step S31, the final encoded point cloud features are input into the constructed semantic prediction branch to obtain the semantic score S sem , and soft grouping of point cloud features means:

[0026] The semantic prediction branch consists of a two-layer MLP structure, a Norm layer, and a ReLU layer. The final encoded point cloud features are normalized and activated by the Norm layer and the ReLU layer in turn, and finally a semantic score S of size N×C is obtained. sem ;

[0027] Set two probability thresholds τ group and τ back , for the semantic score S sem The semantic probability P sem ={P1, ..., P N}∈(0, 1) N×C Traverse N points, first compare the maximum probability in the category with the threshold τ back Comparison, less than the threshold τ back If , the point is considered to be a background point and removed directly; the probability of all C categories and the threshold τ group Compare with the threshold τ group The comparison results are used to perform semantic group classification; the result of soft grouping is C semantic groups to be refined.

[0028] In step S32, the final encoded point cloud features are input into the constructed geometric offset prediction branch to obtain the predicted geometric centroid and the offset of each point cloud feature to the center of its instance. The point is moved to the vicinity of its instance center according to the offset of each point cloud feature. Generating candidate blocks means:

[0029] The geometric offset prediction branch consists of a two-layer MLP structure, a Norm layer, and a ReLU layer;

[0030] Offset relative to the center of mass Make a prediction:

[0031]

[0032] Represents the offset of a point i in the point cloud to the geometric center of the instance to which it belongs; by converting the original coordinates of point i p i (x i ,y i , z i ) and offset Add together to get the coordinates of the point in the offset space And generate candidate blocks; where the coordinates of the points in the offset space are for:

[0033]

[0034] Step S4 includes the following steps:

[0035] Step S41: traverse all candidate blocks and define the feature correlation measure h(p i, p ij ), represents the candidate block B i midpoint p i and its jth adjacent point p in its area ij The correlation degree between them is calculated by directly using the normalized correlation graph as the attention graph, and then calculating the standard deviation σ for each normalized correlation graph. i ; By sampling the standard deviation σ within the candidate block i The points above the threshold are regarded as the points that best represent the instance, and the instance kernel W is predicted by a single-layer MLP structure. i ;

[0036] Step S42: Adding a geometric radiation module to the MLP structure, the MLP structure is combined with the geometric affine module to obtain instance-level features from the sampling results;

[0037] Step S43: The relative position of each point in the sampling result and the instance-level features are spliced ​​together, and the generated instance convolution kernel is used for dynamic convolution, and a binary instance mask is generated after the sigmoid function is applied.

[0038] Step S44: Calculate the binary cross entropy loss mask confidence L mask To measure the credibility of the current output mask representation instance.

[0039] In step S41, all candidate blocks are traversed and the feature correlation measure h(p i , p ij ), represents the candidate block B i midpoint p i and its jth adjacent point p in its area ij The correlation degree between them is calculated by directly using the normalized correlation graph as the attention graph, and then calculating the standard deviation σ for each normalized correlation graph. i ; By sampling the standard deviation σ within the candidate block i Points above the threshold are regarded as the points that best represent the instance, and the instance kernel W is predicted through a single-layer MLP structure. i means:

[0040] Traverse all candidate blocks and define the feature correlation measure h(p i , p ij ), represents the candidate block B i midpoint p i and its jth adjacent point p in its area ij The degree of correlation between them, vector m i is the normalized correlation map between the center point and its neighboring points, defined as:

[0041] m i =softmax[h(p i , p ij)];

[0042] Use the normalized correlation map as the attention map:

[0043] h(p i , p ij )=Q(p i ) T K(p ij -p i );

[0044] Where Q and K represent the linear layers applied to the query input and key input respectively; the center point p i The feature of the neighbor point and the center point is used as the query input, and the feature difference p ij -p i As key input; count the feature dimensions Used as a scaling factor; the final normalized correlation map m i The definition is as follows:

[0045]

[0046] Calculate m i The standard deviation σ i ; By sampling σ in the candidate block i Above the threshold τ edge The point is taken as the point that best represents the instance; the instance kernel W is predicted by a single-layer MLP structure. i .

[0047] In step S42, a geometric radiation module is added to the MLP structure. The MLP structure combined with the geometric affine module obtains instance-level features from the sampling results.

[0048] Set with candidate block B i The original features of the sampling results The residual connection of It is candidate block B i The total number of points, It is candidate block B i The sampling result S i The total number of points in D is the feature dimension;

[0049] The sampling result S i Extracting instance-level features The operations of the MLP structure are expressed as follows:

[0050]

[0051] First, the k-nearest algorithm is used to select adjacent points for the sampling result S i Divide into local regions, k is 24, and the number of groups is rounded down; point j is any point in the local region group l, which is a D+3 dimensional feature vector; using the constructed geometric affine module, point j is transformed to obtain (f l，j )∈R K×(D+3) , the affine transformation is defined as:

[0052]

[0053] where α∈R D and β∈R D is a learnable variable, ⊙ is the dot product; ∈ takes a smaller value 1e -5 , σ is a scalar that describes the feature deviation between all local regions and channels:

[0054]

[0055] Input prediction network φ pre The shared weights are learned from the local area, and then after a maximum pooling, they are input into the network φ for extracting deep aggregation features. fea By stacking the feature extraction network of the local area four times, the receptive field is gradually expanded, and a complete point cloud geometric information model is established to obtain instance-level features. Feature aggregation network φ fea and the local prediction network φ pre They are all implemented by multi-layer perceptrons combined with residual connections, expressed as MLP(x)+x. The input of MLP(x) passes through a fully connected layer and is nonlinearly connected with the residual. The connected result is normalized and activated with the ReLU function.

[0056] Step S5 includes the following steps:

[0057] Step S51: performing spherical searches with different radii on the generated binary instance mask through two point aggregation layers to aggregate boundary points;

[0058] Step S52: Predict the mask-level semantic score L for supervision class and the accuracy score L score ; Based on the generated binary instance mask, the boundary points are dynamically added and deleted to achieve refinement;

[0059] Step S53: Filter out the mask results whose points in the instance do not meet the threshold, then obtain the total mask score by the square root of the product of the semantic category score and the accuracy score, and sort them from high to low. Use the NMS algorithm to select the most representative masks as the instance mask point coordinates, and remove redundant detection results;

[0060] Step S54: The mask under the candidate block directly inherits the semantic label with the highest score of the parent block. The semantic label is fused with the instance mask point coordinates to output the final instance segmentation result.

[0061] The proposed method uses dynamic convolution based on edge sampling, combining the advantages of top-down and bottom-up point cloud processing strategies to complement each other. First, blocks are divided based on semantic scores and relative centroid offsets, and soft grouping is used to correct semantic judgment errors. Edge-attention-based sampling is introduced to generate more discriminative instance kernels and improve instance feature extraction. Two point aggregation layers post-process candidate masks, iteratively adjusting the cutoff points between instances to calculate scores for supervision and improve instance segmentation accuracy.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] 1. The point cloud segmentation method for stacked parts of the present invention can achieve stable and efficient identification and segmentation of parts in complex scenes; at the same time, this method can improve the accuracy of part recognition and segmentation in complex working environments where parts are mixed and occluded from each other.

[0064] 2. Compared with proposal-based and clustering-based instance segmentation methods, this method combines the complementary advantages of top-down and bottom-up point cloud processing strategies. It avoids the problems of unstable bounding box prediction results and difficult to define semantic and offset thresholds for clustering, providing a more stable and efficient segmentation method for real-world industrial scenarios with inter-occluded, adhered parts, and mixed types. It also incorporates semantic and offset-based preliminary block segmentation and point aggregation-based mask refinement before and after mask generation, respectively, improving recognition and segmentation accuracy in complex scenarios.

[0065] 3. In the present invention, a transformer module is added to the feature extraction network, and the position information of each point is encoded into the average value of the direction vector through a position embedding layer, which improves the limitations of the limited receptive field and representation field of 3D sparse convolution. At the same time, enriching the geometric information is beneficial for subsequent instance encoding.

[0066] 4. This invention combines point-by-point semantic score and geometric offset prediction with a soft grouping strategy to preliminarily define the range for subsequent mask generation, reducing convolution and computational overhead and ensuring real-time detection requirements in industrial environments. Furthermore, this soft grouping strategy allows a point to exist in multiple semantic groups simultaneously, avoiding semantic conflicts caused by clustering and correcting potential semantic prediction errors, offering significant advantages in occluded scenes.

[0067] 5. The dynamic convolutional network designed in this paper generalizes pixel features from 2D images to 3D point clouds, introducing edge-attention-based sampling instead of farthest point sampling to obtain more discriminative instance kernels. It also improves the feature extraction method used in dynamic convolution methods. Compared with previous methods that use independent small deep networks, this paper uses an MLP structure combined with a lightweight geometric affine module, reducing training costs and improving stability.

[0068] 6. The present invention uses two point aggregation layers to post-process the candidate masks, dynamically adds and deletes boundary points based on the candidate blocks to achieve refinement, and iteratively predicts the semantic scores and confidence levels at the mask level for supervision, which greatly improves the accuracy of recognition and segmentation in complex working environments where parts are mixed and occluded. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Schematic diagram of the overall network framework of the point cloud segmentation method for stacked parts of the present invention;

[0070] Figure 2 This is a network flow chart of the attention-based edge sampling algorithm of the present invention;

[0071] Figures 3(a)-3(c) Schematic diagram of the attention-based edge sampling algorithm of the present invention;

[0072] Figure 4 Schematic diagram of the MLP structure with the geometric affine module added to the present invention; DETAILED DESCRIPTION

[0073] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0074] Example 1

[0075] like Figures 1 to 4 As shown, the point cloud segmentation method for stacked parts of the present invention includes the following steps:

[0076] S1. Collect the depth information of the entire scene of stacked parts and construct the original point cloud data; separate, downsample, crop, and filter the original point cloud data to obtain the point cloud data of the stacked parts to be segmented;

[0077] S2. Input the coordinate information of the point cloud data of the stacked parts to be segmented into the feature extraction backbone network to obtain point cloud features one by one;

[0078] S3: Input the point-by-point features into the semantic score prediction branch and the geometric offset prediction branch. Soft-group the point cloud features based on the semantic score. Then, the geometric offset prediction branch moves each point cloud feature to the vicinity of its instance center according to its offset to generate a candidate block. Two point-level loss functions supervise the semantic prediction branch and the geometric offset branch respectively.

[0079] S4. Traverse all candidate blocks and generate instance kernels based on the sampling results of edge detection attention. The MLP structure is combined with a lightweight geometric radiation module to obtain instance-level features. The sampling results are spliced ​​with instance-level features and dynamically convolved with the instance kernel to generate a binary instance mask. The instance mask confidence L is calculated. mask To measure the credibility of the mask representation instance currently output;

[0080] S5, refine the instance boundary through the point aggregation layer and predict the semantic score L at the instance mask level for supervision class and the accuracy score L score The instance mask point coordinates are obtained by sorting the instance mask scores and removing duplicate results through the NMS algorithm. The instance mask point coordinates are fused with the semantic labels to output the final instance segmentation result.

[0081] Specifically, step S1 includes the following steps:

[0082] Step S11: The depth camera indiscriminately collects depth information of the entire scene to construct raw point cloud data; the depth information of the scene includes background information; the point cloud data of non-interested areas is filtered out by straight-through filtering to separate the raw point cloud data.

[0083] Step S12: Downsampling uses uniform voxel filtering, and the centroid coordinates of the grid are used to replace the points of the separated point cloud data under the entire voxel, thereby maintaining the shape details of the point cloud data while reducing the data density.

[0084] Step S13: Considering that the experimental platform and frame are always fixed during the grasping process, the RANSAC random sampling consistency algorithm is selected to fit the plane information and frame information in the background information, and a random subset in the data set is extracted to fit the plane model. Repeated sampling is used to enhance the robustness to noise. Multiple iterations are combined with the normal vector to remove the platform information and frame information from the point cloud data, so that the points belonging to the background platform can be removed from the scene point cloud.

[0085] Step S14: Statistical filtering is used to analyze and remove outliers that are far away from the point cloud data of the part body, isolated, and sparse, to obtain the point cloud data of the stacked parts to be segmented.

[0086] Common statistical filtering methods include setting a threshold based on the number of points in the neighborhood and setting a threshold based on the average distance between a point and its neighbors. However, both require a neighborhood search for each point. The present invention directly calculates a hash value containing point quantity features based on the previous voxel grid with the help of a grid structure. It only needs to sort the hash values ​​and then query the number of points. If the number of point clouds in the surrounding grids is less than the set threshold τ0, the point is considered an outlier. Assume that the coordinates of point i in the point cloud are (x i ,y i , z i ), the center coordinate of the grid (x ig ,y ig , z ig ) can be obtained by the voxel index V corresponding to the coordinates of point i i The total number of points in the grid is N g , then the hash function of the grid is defined as:

[0087]

[0088] Step S2 of the present invention comprises the following steps:

[0089] Step S21: The point cloud data of the stacked parts to be segmented is uniformly mapped onto a grid for voxelization and then fed into the feature extraction network. Unlike preprocessing with grid downsampling, the purpose of voxelization here is to convert the disordered and sparse point cloud space into an ordered and dense voxel space, thereby enabling traditional convolution to be used for feature extraction in this voxel space.

[0090] The preferred feature extraction network uses a three-layer convolutional U-Net network, which uses submanifold sparse convolution to process voxel features to prevent the loss of feature sparsity as the number of convolution layers increases. Considering that 3D sparse convolution is additionally limited by its limited receptive field and representation capacity, a lightweight transformer is added. Through a position embedding layer, the position information of each point is encoded as the average of the direction vector to enhance long-range interaction at the top of each encoder layer.

[0091] Step S22: Apply reverse interpolation to the output backbone features to achieve upsampling, and perform voxelized inverse mapping to obtain point cloud features one by one.

[0092] The reverse interpolation operation is to obtain the k voxel centers closest to a point p in the point cloud data, and calculate the center index V according to the center index V. p ∈R k The index gets the corresponding k voxel features f∈R k , according to the weighted sum of distance, we get the feature f of point p p :

[0093]

[0094] in Weight ω i It is inversely related to the distance between the voxel center index and the point p, which means that the farther the voxel center is from the point, the smaller its contribution to the point feature.

[0095] Step S23: Use the fully connected network to concatenate the point cloud features containing multi-scale and rich context information along the channel dimension to obtain the final encoded point cloud feature F p ; F p ∈R N×D , where N is the number of point cloud points and D is the dimension of the output channel.

[0096] Step S3 of the present invention comprises the following steps:

[0097] Step S31: Input the final encoded point cloud features into the constructed semantic prediction branch to obtain the semantic score S sem , and soft grouping is performed on the point cloud features. The "soft" aspect of the present invention is that it allows points to be associated with multiple semantic categories, so as to prevent the incorrect prediction of semantic categories from directly affecting the subsequent instance segmentation.

[0098] The semantic prediction branch consists of a two-layer MLP structure, a Norm layer, and a ReLU layer. The final encoded point cloud features are normalized and activated by the Norm layer and the ReLU layer in turn, and finally a semantic score S of size N×C is obtained. sem The semantic scores are converted into a probability distribution indicating that the point belongs to a specific category through a softmax function. Grouping is then performed directly without mapping the scores to one-hot semantic labels, avoiding unnecessary conversion overhead. The original category information of the point is also preserved, allowing the number of semantic groups each point can enter to be determined by the complexity of its features.

[0099] Specifically set two probability thresholds τ group and τ back , for the semantic score S sem The semantic probability P sem ={P1, ..., P N}∈(0, 1) N×C Traverse N points, first compare the maximum probability in the category with the threshold τ back Comparison, less than the threshold τ back If , the point is considered to be a background point and removed directly; the probability of all C categories and the threshold τ group Compare with the threshold τ group The comparison results are used to perform semantic group classification; the result of soft grouping is C semantic groups to be refined.

[0100] Step S32: Input the final encoded point cloud features into the constructed geometric offset prediction branch to obtain the predicted geometric centroid and the offset of each point cloud feature to the center of the instance to which it belongs. Move the point to the vicinity of its instance center according to the offset of each point cloud feature to generate a candidate block.

[0101] The geometric offset prediction branch consists of a two-layer MLP structure, a Norm layer, and a ReLU layer;

[0102] Offset relative to the center of mass Make a prediction:

[0103]

[0104] Represents the offset of a point i in the point cloud to the geometric center of the instance to which it belongs; by converting the original coordinates of point i p i (x i ,y i , z i ) and offset Add together to get the coordinates of the point in the offset space And generate candidate blocks; where the coordinates of the points in the offset space are for:

[0105]

[0106] Compared to the original point space, the points in the offset space are more tightly clustered around their respective instance centers, which is particularly effective when separating multiple parts adjacent to each other.

[0107] Step S33: Construct a loss function for supervised semantic prediction and a loss function for supervised geometric offset prediction, and supervise the semantic prediction branch and the geometric offset branch respectively.

[0108] When grouping points, the semantic scores are converted to their respective probabilities, but when supervised, they are uniformly converted to the labels corresponding to the highest semantic scores: T pre ={T1, ..., T N}∈(-1, 1) N×1 The semantic prediction branch is supervised by the cross entropy loss:

[0109]

[0110] Where N is the number of points in the point cloud, C is the total semantic category, and T i gt and T i pre are the true semantic label of point i and the label corresponding to the highest predicted semantic score.

[0111] The L1 regression loss is used to supervise the offset branch and is defined as follows:

[0112]

[0113] in is an indicator function, representing the set of points belonging to the foreground among the N points in the entire point cloud scene. i gt and They represent the actual offset and predicted offset vector of point i relative to the true center, respectively.

[0114] Step S4 of the present invention comprises the following steps:

[0115] Step S41: traverse all candidate blocks and define the feature correlation measure h(p i , p ij ), represents the candidate block B i midpoint p i and its jth adjacent point p in its area ij The correlation degree between them is calculated by directly using the normalized correlation graph as the attention graph, and then calculating the standard deviation σ for each normalized correlation graph. i ; By sampling the standard deviation σ within the candidate block i The points above the threshold are regarded as the points that best represent the instance, and the instance kernel W is predicted by a single-layer MLP structure. i .

[0116] Specifically: traverse all candidate blocks and use the attention-based edge sampling algorithm to sample points within the blocks. The flow chart and algorithm structure diagram are as follows: Figure 2 , as shown in Figures 3(a), 3(b), and 3(c). Edge-attention-based sampling of points within a block uses the Canny edge detection algorithm. This algorithm compares each pixel with its neighboring pixels within each fixed-size patchset, calculating the intensity gradient and assuming that edge pixels have a larger intensity gradient. This generalizes the pixel features in a 2D image to 3D points. If there is a large correlation difference between points, the standard deviation σ of the point intensity will also be high.

[0117] Define the feature correlation measure h(p i , p ij ), represents the candidate block B i midpoint p i and its jth adjacent point p in its area ij The degree of correlation between them, vector m i is the normalized correlation map between the center point and its neighboring points, defined as:

[0118] m i =softmax[h(p i , P ij )];

[0119] The basic idea of ​​the attention mechanism is to assign a weight representing the importance of each element in the input sequence. Here, we pay special attention to the correlation measurement between point features in each region, so we directly use the normalized correlation map as the attention map:

[0120] h(p i , p ij )=Q(p i ) T K(p ij -p i );

[0121] Where Q and K represent the linear layers applied to the query input and key input respectively; the center point p i The feature of the neighbor point and the center point is used as the query input, and the feature difference p ij -p i As key input; count the feature dimensions Used as a scaling factor; the final normalized correlation map m i The definition is as follows:

[0122]

[0123] Calculate m i The standard deviation σ i ; By sampling σ in the candidate block i Above the threshold τ edge The point is taken as the point that best represents the instance; the instance kernel W is predicted by a single-layer MLP structure. i Since the sampling results of edge attention are quite different from the data distribution of the original point cloud, in order to ensure that the instance-level features extracted subsequently are more comprehensive, a cross-attention-based upsampling layer is designed to supplement the information. The key input is the features of the previous layer, and the query input is the features of the original point through the residual connection.

[0124] Step S42: Add a geometric radiation module to the MLP structure. The MLP structure is combined with the geometric affine module to obtain instance-level features from the sampling results.

[0125] Set with candidate block B i The original features of the sampling results The residual connection of It is candidate block B i The total number of points, It is candidate block B i The sampling result S i The total number of points in D is the feature dimension;

[0126] The point cloud analysis network is obtained by stacking the MLP structure combined with the geometric affine module four times, as shown in Figure 4 The MLP structure is shown in the schematic diagram. i Extracting instance-level features The operations of the MLP structure are expressed as follows:

[0127]

[0128] First, the k-nearest algorithm is used to select adjacent points for the sampling result S i Divide into local regions, k is 24, and the number of groups is rounded down; point j is any point in the local region group l, which is a D+3 dimensional feature vector; using the constructed geometric affine module, point j is transformed to obtain (f l，j )∈R K×(D+3) , the affine transformation is defined as:

[0129]

[0130] where α∈R D and β∈R D is a learnable variable, ⊙ is the dot product; ∈ takes a smaller value 1e -5 , σ is a scalar that describes the feature deviation between all local regions and channels:

[0131]

[0132] Input prediction network φ pre The shared weights are learned from the local area, and then after a maximum pooling, they are input into the network φ for extracting deep aggregation features. fea By stacking the feature extraction network of the local area four times, the receptive field is gradually expanded, and a complete point cloud geometric information model is established to obtain instance-level features. Feature aggregation network φ fea and the local prediction network φ pre Both are implemented using a multi-layer perceptron combined with residual connections, expressed as MLP(x)+x. The input of MLP(x) passes through a fully connected layer and then undergoes a nonlinear connection with the residual. The result of the connection is normalized and activated with the ReLU function. Compared to previous methods that construct network geometry to capture local characteristics through convolution, graph, or attention mechanisms, the MLP-based residual design possesses local connections and weight sharing properties, which prevents the disordered arrangement of points in the point cloud from affecting the network's learning of local structure. Residual connections efficiently help the network capture and represent subtle features of the data, simplifying the network structure and facilitating scalability. They allow for stacking multiple layers to obtain deep feature representations without exploding or vanishing gradients.

[0133] Step S43: The relative position of each point in the sampling result and the instance-level features are spliced ​​together, and the generated instance convolution kernel is used for dynamic convolution, and a binary instance mask is generated through the sigmoid function.

[0134] Step S44: Calculate the binary cross entropy loss mask confidence L mask To measure the credibility of the current output mask representation instance.

[0135] For each output instance mask, calculate the intersection over union (IoU) between it and all the real instance sets, and select the real instance with the highest IoU greater than or equal to 0.5 as the target instance of the predicted instance. The points in the instance mask set that overlap with the target instance set are correct predictions, otherwise they are incorrect predictions. Mask prediction loss L mask The design is binary cross entropy loss, which is defined as follows:

[0136]

[0137] where N ins is the total number of output mask sets, is the indicator function, which represents the set of mask instances whose intersection-over-union ratio with the target real instance is greater than 50, where there are N mask instances under instance mask i. i Points. j is the predicted mask value of point j in the mask, is the true mask value of point j.

[0138] Step S5 of the present invention comprises the following steps:

[0139] Step S51: performing spherical searches with different radii on the generated binary instance mask through two point aggregation layers to aggregate boundary points.

[0140] Preferably, similar to the idea in poinmet++, the purpose of iteratively applying two point aggregation layers with different radii is to diverge the points searched from the previous layer in the next layer, achieving an effect similar to stacking multiple convolutional layers in 2D images to increase the receptive field.

[0141] Step S52: Predict the mask-level semantic score L for supervision class and the accuracy score L score ; Based on the generated binary instance mask, boundary points are dynamically added and deleted to achieve refinement.

[0142] Preferably, the accuracy score L score It is the binary cross entropy loss, which is defined as follows:

[0143]

[0144] where N relable The total number of instances in the reliable mask prediction whose boundary adjustment results meet the threshold, score i is the accuracy score of the current instance mask i, score gt is the current instance mask i and the target instance set truth gt The corresponding real instance score with the highest IoU in .

[0145] The soft semantic grouping used in this invention allows a point to have multiple semantic labels simultaneously, meaning that a point can exist in semantic groups with different semantic labels. When the points within a group are segmented into instances, semantic prediction is re-performed on all aggregated points, using the group's semantic label as supervision. Points that do not belong to the group are removed from the group, and the next level of aggregation is iterated. This corrects any misclassifications that may have occurred in the initial semantic prediction, while also dynamically adjusting the points included in the instance mask.

[0146] Semantic score L class It is the CE cross entropy loss, which is defined as follows:

[0147]

[0148] Where K is the total number of masks, N mask is the number of points contained in the current mask point cloud, C gt is the true semantic label of the current group, are the predicted semantic labels of any point i in the point cloud.

[0149] Step S53: Filter out the mask results whose points in the instance do not meet the threshold, then obtain the total mask score by the square root of the product of the semantic category score and the accuracy score, and sort them from high to low. Use the NMS algorithm to screen out the most representative masks as the instance mask point coordinates, and remove redundant detection results to prevent the same object from being detected by multiple bounding boxes.

[0150] Step S54: The mask under the candidate block directly inherits the semantic label with the highest score of the parent block. The semantic label is fused with the instance mask point coordinates to output the final instance segmentation result.

[0151] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A point cloud segmentation method for stacked parts, characterized by: The following steps are involved: S1, collect the depth information of the entire scene of stacked parts and construct the original point cloud data; Separate, downsample, crop, and filter the original point cloud data to obtain the point cloud data of the stacked parts to be segmented; S2. Input the coordinate information of the point cloud data of the stacked parts to be segmented into the feature extraction backbone network to obtain point cloud features one by one; S3, input the point-by-point features into the semantic score prediction branch and the geometric offset prediction branch; Each point cloud feature is soft-grouped based on its semantic score. Then, the geometric offset prediction branch moves each point cloud feature to the vicinity of its instance center according to its offset to generate candidate blocks. Two point-level loss functions supervise the semantic prediction branch and the geometric offset branch respectively. S4. Traverse all candidate blocks and generate instance kernels based on the sampling results of edge detection attention. The MLP structure is combined with a lightweight geometric affine module to obtain instance-level features. The sampling results are spliced ​​with instance-level features and dynamically convolved with the instance kernel to generate a binary instance mask. The instance mask confidence L is calculated. mask To measure the credibility of the mask representation instance currently output; S5, refine the instance boundary through the point aggregation layer and predict the semantic score L at the instance mask level for supervision class and the accuracy score L score The instance mask point coordinates are obtained by sorting the instance mask scores and removing duplicate results through the NMS algorithm. The instance mask point coordinates are fused with the semantic labels to output the final instance segmentation result.

2. The point cloud segmentation method for stacked parts according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: collecting depth information of the entire scene of stacked parts and constructing raw point cloud data; the depth information of the scene includes background information; filtering out point cloud data of non-interested areas through straight-through filtering to separate the raw point cloud data; Step S12: Downsampling uses uniform voxel filtering, replacing the points of the separated point cloud data under the entire voxel with the centroid coordinates of the grid, thereby maintaining the shape details of the point cloud data while reducing the data density; Step S13: Select a random sampling consistency algorithm to fit the plane information and frame information in the background information, and remove the platform information and frame information from the point cloud data by combining the normal vectors through multiple iterations; Step S14: Statistical filtering is used to analyze and remove outliers that are far away from the point cloud data of the part body, isolated, and sparse, to obtain the point cloud data of the stacked parts to be segmented.

3. The point cloud segmentation method for stacked parts according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: evenly mapping the point cloud data of the stacked parts to be segmented onto a grid for voxelization and inputting the data into a feature extraction network; Step S22: applying reverse interpolation to upsample the output backbone features, and performing voxelized inverse mapping to obtain point cloud features one by one; Step S23: Use the fully connected network to concatenate the point cloud features containing multi-scale and rich context information along the channel dimension to obtain the final encoded point cloud feature F p ; F p ∈R N×D , where N is the number of point cloud points and D is the dimension of the output channel.

4. The point cloud segmentation method for stacked parts according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: Input the final encoded point cloud features into the constructed semantic prediction branch to obtain the semantic score S sem , and soft group the point cloud features; Step S32: Input the final encoded point cloud features into the constructed geometric offset prediction branch to obtain the predicted geometric centroid and the offset of each point cloud feature to the center of its instance. According to the offset of each point cloud feature, the point is moved to the vicinity of its instance center to generate a candidate block. Step S33: Construct a loss function for supervised semantic prediction and a loss function for supervised geometric offset prediction, and supervise the semantic prediction branch and the geometric offset branch respectively.

5. The point cloud segmentation method for stacked parts according to claim 4, characterized in that: In step S31, the final encoded point cloud features are input into the constructed semantic prediction branch to obtain the semantic score S sem , and soft grouping of point cloud features means: The semantic prediction branch consists of a two-layer MLP structure, a Norm layer, and a ReLU layer. The final encoded point cloud features are normalized and activated by the Norm layer and the ReLU layer in turn, and finally a semantic score S of size N×C is obtained. sem ; Set two probability thresholds τ group and τ back , for the semantic score S sem The semantic probability P sem ={P1,...,P N }∈(0,1) N×C Traverse N points, first compare the maximum probability in the category with the threshold τ back Comparison, less than the threshold τ back If , the point is considered to be a background point and removed directly; the probability of all C categories and the threshold τ group Compare with the threshold τ group The comparison results are used to perform semantic group classification; the result of soft grouping is C semantic groups to be refined.

6. The point cloud segmentation method for stacked parts according to claim 4, characterized in that: In step S32, the final encoded point cloud features are input into the constructed geometric offset prediction branch to obtain the predicted geometric centroid and the offset of each point cloud feature to the center of its instance. The point is moved to the vicinity of its instance center according to the offset of each point cloud feature. Generating candidate blocks means: The geometric offset prediction branch consists of a two-layer MLP structure, a Norm layer, and a ReLU layer; Offset relative to the center of mass Make a prediction: Represents the offset of a point i in the point cloud to the geometric center of the instance to which it belongs; by converting the original coordinates of point i p i (x i ,y i ,z i ) and offset Add together to get the coordinates of the point in the offset space And generate candidate blocks; Among them, the coordinates of the point in the offset space for:

7. The point cloud segmentation method for stacked parts according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: traverse all candidate blocks and define the feature correlation measure h(p i ,p ij ), represents the midpoint p of the candidate block B i and its jth adjacent point p in its area ij The correlation degree between them is calculated by directly using the normalized correlation graph as the attention graph, and then calculating the standard deviation σ for each normalized correlation graph. i ; By sampling the points with standard deviation σ higher than the threshold in the candidate block as the points that best represent the instance, and predicting the instance kernel W through a single-layer MLP structure i ; Step S42: Adding a geometric affine module to the MLP structure, the MLP structure combined with the geometric affine module obtains instance-level features from the sampling results; Step S43: The relative position of each point in the sampling result and the instance-level features are spliced ​​together, and the generated instance convolution kernel is used for dynamic convolution, and a binary instance mask is generated after the sigmoid function is applied. Step S44: Calculate the binary cross entropy loss mask confidence L mask To measure the credibility of the current output mask representation instance.

8. The point cloud segmentation method for stacked parts according to claim 7, characterized in that: In step S41, all candidate blocks are traversed and the feature correlation measure h(p i ,p ij ), represents the candidate block B i midpoint p i and its jth adjacent point p in its area ij The correlation degree between them is calculated by directly using the normalized correlation graph as the attention graph, and then calculating the standard deviation σ for each normalized correlation graph. i ; By sampling the standard deviation σ within the candidate block i Points above the threshold are regarded as the points that best represent the instance, and the instance kernel W is predicted through a single-layer MLP structure. i means: Traverse all candidate blocks and define the feature correlation measure h(p i ,p ij ), represents the candidate block B i midpoint p i and its jth adjacent point p in its area ij The degree of correlation between them, vector m i is the normalized correlation map between the center point and its neighboring points, defined as: m i =softmax[h(p i ,p ij )]; Use the normalized correlation map as the attention map: h(p i ,p ij )=Q(p i ) T K(p ij -p i ); Among them, Q and K represent the linear layers applied to the query input and key input respectively; the feature of the center point p is used as the query input, and the feature difference p between the neighbor point and the center point is used as the query input. ij -p i As key input; count the feature dimensions Used as a scaling factor; the final normalized correlation map m i The definition is as follows: Calculate m i The standard deviation σ i ; By sampling σ in the candidate block i Above the threshold τ edge The point is taken as the point that best represents the instance; the instance kernel W is predicted by a single-layer MLP structure. i .

9. The point cloud segmentation method for stacked parts according to claim 7, characterized in that: In step S42, a geometric affine module is added to the MLP structure. The MLP structure combined with the geometric affine module obtains instance-level features from the sampling results, which means: Set with candidate block B i The original features of the sampling results The residual connection of It is candidate block B i The total number of points, It is candidate block B i The sampling result S i The total number of points in D is the feature dimension; The sampling result S i Extracting instance-level features The operation of the MLP structure is expressed as follows: First, the k-nearest algorithm is used to select adjacent points for the sampling result S i Divide into local regions, k is 24, and the number of groups is rounded down; point j is any point in the local region group l, which is a D+3 dimensional feature vector; using the constructed geometric affine module, point j is transformed to obtain (f l,j )∈R K×(D+3) , the affine transformation is defined as: where α∈R D and β∈R D is a learnable variable, ⊙ is the dot product; ∈ takes a smaller value 1e -5 , σ is a scalar that describes the feature deviation between all local regions and channels: Input prediction network φ pre The shared weights are learned from the local area, and then after a maximum pooling, they are input into the network φ for extracting deep aggregation features. fea By stacking the feature extraction network of the local area four times, the receptive field is gradually expanded, and a complete point cloud geometric information model is established to obtain instance-level features. Feature aggregation network φ fea and the local prediction network φ pre They are all implemented by multi-layer perceptrons combined with residual connections, expressed as MLP(x)+x. The input of MLP(x) passes through a fully connected layer and is nonlinearly connected with the residual. The connected result is normalized and activated with the ReLU function.

10. The point cloud segmentation method for stacked parts according to claim 1, characterized in that: Step S5 includes the following steps: Step S51: performing spherical searches with different radii on the generated binary instance mask through two point aggregation layers to aggregate boundary points; Step S52: Predict the mask-level semantic score L for supervision class and the accuracy score L score ; Based on the generated binary instance mask, the boundary points are dynamically added and deleted to achieve refinement; Step S53: Filter out the mask results whose points in the instance do not meet the threshold, then obtain the total mask score by the square root of the product of the semantic category score and the accuracy score, and sort them from high to low. Use the NMS algorithm to select the most representative masks as the instance mask point coordinates, and remove redundant detection results; Step S54: The mask under the candidate block directly inherits the semantic label with the highest score of the parent block. The semantic label is fused with the instance mask point coordinates to output the final instance segmentation result.

Images