An instance segmentation method based on point cloud completion and merging

Abstract

This invention relates to the field of data processing technology, specifically to an instance segmentation method based on point cloud completion and merging. The method includes inputting point cloud information containing location and color information into a feature extraction backbone network for point-level feature extraction to obtain point cloud features. These features are then passed to an offset prediction branch and a semantic prediction branch, respectively. The semantic prediction branch is used to perform semantic prediction on the point cloud features to obtain a point cloud semantic score. The offset prediction branch is used to perform center offset prediction on the point cloud features to obtain a center offset vector between the point cloud and the center point of its respective instance. Point clustering is performed based on the semantic score and the center offset vector to obtain a preliminary instance segmentation result. The preliminary instance segmentation result is then merged again using point cloud features and point cloud completion to obtain a corrected instance prediction result. Finally, the corrected instance prediction result is adjusted by optimizing the network to obtain the final instance segmentation result, thus solving the problem of low accuracy in existing point cloud instance segmentation methods.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an instance segmentation method based on point cloud completion and merging. Background Technology

[0002] In indoor point cloud instance segmentation tasks, considering practical applications, low-cost portable acquisition devices equipped with LiDAR, such as iPad Pro, are generally used to collect point clouds in indoor scenes. However, due to current limitations in point cloud acquisition technology, the point clouds collected by low-cost point cloud acquisition devices inevitably contain missing points, resulting in low accuracy of point cloud instance segmentation. Summary of the Invention

[0003] The purpose of this invention is to provide an instance segmentation method based on point cloud completion and merging, which aims to solve the problem of low accuracy in existing point cloud instance segmentation.

[0004] To achieve the above objectives, this invention provides an instance segmentation method based on point cloud completion and merging, comprising the following steps:

[0005] Point cloud information containing location and color information is input into the feature extraction backbone network for point-level feature extraction to obtain point cloud features. The point cloud features are then passed into the offset prediction branch and the semantic prediction branch, respectively.

[0006] The semantic prediction branch is used to perform semantic prediction on the point cloud features to obtain a point cloud semantic score.

[0007] The offset prediction branch is used to perform center offset prediction on the point cloud features to obtain the center offset vector between the point cloud and the center point of its instance.

[0008] Point clustering is performed based on the semantic score and the center offset vector to obtain preliminary instance segmentation results;

[0009] The point cloud features are combined and point cloud completion is used to merge the preliminary instance segmentation results to obtain the corrected instance prediction results.

[0010] The final instance segmentation result is obtained by adjusting the corrected instance prediction result through network optimization.

[0011] The backbone network includes a voxelization module and a U-Net-type feature extraction network.

[0012] The step of inputting point cloud information with location and color information into the feature extraction backbone network for point-level feature extraction to obtain point cloud features includes:

[0013] The voxelization module converts the input point cloud information into multiple voxels and extracts the voxel features of each voxel to obtain the output feature map.

[0014] The output feature map is input into the U-Net-type feature extraction network, which outputs the processing result of the voxel features. The processing result is then mapped back to the point cloud to obtain the point cloud features.

[0015] The step of performing point clustering based on the semantic score and the center offset vector to obtain preliminary instance segmentation results includes:

[0016] The point cloud coordinates are added to the center offset vector to pull the point cloud in the original space toward the center point of the instance.

[0017] Based on the semantic score, cluster all point clouds in the offset space that satisfy the condition that the distance is less than the cluster radius and the semantic score is greater than the semantic score threshold to obtain preliminary instance segmentation results.

[0018] The step of combining the point cloud features and using point cloud completion to merge the preliminary instance segmentation results to obtain the corrected instance prediction results includes:

[0019] The point cloud features of size N×K are voxelized according to cluster sets to obtain preliminary voxel features of size M×K; the preliminary voxel features are input into the U-Net-type feature extraction network to obtain final voxel features of size M×K containing global and local features; the final voxel features are then transformed into N×K features through global average pooling. ins ×K cluster set-level semantic features; based on the cluster set-level semantic features, an MLP is used to predict a group of size N. ins ×N class The semantic score corresponding to each cluster set is taken as the maximum value to obtain the semantic label of the cluster set;

[0020] With size N i The point cloud features extracted by the backbone network from ×K and N i The point cloud coordinates of size N are concatenated by multiplying by 3. i The cluster set features of ×(K+3), where N i The number of point clouds in cluster set i is given, and the features of the cluster set are input into the encoder of the completion and merging module to obtain the point cloud features of the cluster set.

[0021] The center offset vector is transformed into an instance offset vector, and the instance offset vector and the cluster set point cloud features are input into the decoder of the completion and merging module to obtain the sparse point cloud of the complete instance.

[0022] Using the sparse point cloud of the complete instance, the bulldozer distance between sparse point clouds is used as the similarity score between sparse point clouds; combined with the semantic labels of the cluster set, the original cluster sets with the same semantic labels and the similarity scores between the completed sparse point clouds are merged with the merging threshold to obtain the corrected instance prediction result.

[0023] The step of inputting the cluster set features into the encoder of the completion and merging module to obtain the cluster set point cloud features includes:

[0024] The encoder will have a size of N i The cluster set features of ×(K+3) are transformed into N through a two-layer MLP. i Features of ×256;

[0025] N i A feature input of 256 × 106 is used to obtain a feature of 512 Ni × 12.

[0026] Using two layers of MLP to N i The feature of ×512 is transformed into N i ×(N com The characteristics of ×3×3), where N com This is a hyperparameter representing the number of sparse point clouds generated by point cloud completion for each cluster set;

[0027] N i ×(N com The features of (×3×3) are normalized and activated through Norm and ReLU layers, resulting in an encoded size of N. i ×(N com The clustered point cloud features of (×3×3) clusters.

[0028] The step of inputting the instance offset vector and the cluster set point cloud features into the decoder of the completion and merging module to obtain the sparse point cloud of the complete instance includes:

[0029] The decoder will have a size of Ni×(N) com The cluster set features of (×3×3) are transformed into a set of size 1×(N) through max pooling. com Features of (×3×3);

[0030] For a size of 1×(N) com The features of (×3×3) are transformed to obtain a shape of size N. com Features of ×(3×3);

[0031] Using two layers of MLP to process a size of N com The feature transformation of ×(3×3) is N com The sparse point cloud after completion (×3);

[0032] Copy the instance offset vector of the 1×3 instance set N times. com 1, stacked together, yields a size of N. com ×3 instance offset;

[0033] The instance offset is added to the sparse point cloud to obtain the sparse point cloud of the complete instance.

[0034] The step of combining the point cloud features and using point cloud completion to merge the preliminary instance segmentation results to obtain the corrected instance prediction results includes:

[0035] The point cloud features of size N×K are voxelized according to cluster sets to obtain preliminary voxel features of size M×K; the preliminary voxel features are input into the U-Net-type feature extraction network to obtain final voxel features of size M×K containing global and local features; the final voxel features are then transformed into N×K features through global average pooling. ins ×K cluster set-level semantic features; based on the cluster set-level semantic features, an MLP is used to predict a group of size N. ins ×N class The semantic score corresponding to each cluster set is taken as the maximum value to obtain the semantic label of the cluster set;

[0036] The initial voxel features of size M×K and the voxel coordinates of size M×3 corresponding to each voxel are input into the encoder of the lightweight completion and merging module to obtain the clustered set voxel features.

[0037] The center offset vector is transformed into an instance offset vector, and the cluster set voxel features and the instance offset vector are input into the decoder of the lightweight completion and merging module to obtain the sparse point cloud of the complete instance.

[0038] Using the sparse point cloud of the complete instance, the bulldozer distance between sparse point clouds is used as the similarity score between sparse point clouds; combined with the semantic labels of the cluster set, the original cluster sets with the same semantic labels and the similarity scores between the completed sparse point clouds are merged with the merging threshold to obtain the corrected instance prediction result.

[0039] Specifically, the process of inputting the initial voxel features of size M×K and the voxel coordinates of size M×3 corresponding to each voxel into the encoder of the lightweight completion and merging module to obtain clustered voxel features includes:

[0040] The encoder passes the initial voxel features of size M×K through a two-layer U-Net network to obtain global and local features;

[0041] The global and local features are concatenated with voxel coordinates of size M×3, and then the feature shape is transformed into an M×256 feature through a two-layer MLP.

[0042] The M×256 feature is sequentially transformed into M×512, M×1024, and M×(N) shapes through three two-layer MLPs. com The characteristics of ×3×3), where N com This is a hyperparameter representing the number of sparse point clouds generated by point cloud completion for each cluster set;

[0043] Using Norm and ReLU layers to convert M×(N) com Normalize and activate the features of (×3×3) to obtain an encoded size of M×(N). com The cluster features of the ×3×3) cluster set.

[0044] The step of inputting the cluster set voxel features and the instance offset vector into the decoder of the lightweight completion and merging module to obtain the sparse point cloud of the complete instance includes:

[0045] The decoder uses a two-layer MLP to convert M×(N) com The voxel characteristics of ×3×3) are converted to M×(N) com Features of ×3);

[0046] Global average pooling will M×(N) com The voxel characteristics of ×3) are transformed into N ins ×(N com ×3) cluster-level point cloud features, N ins The number of cluster sets;

[0047] The size of all cluster sets is N ins The set of ×3 instance offset vectors is copied N times respectively. ins This time, we get a size of N. ins ×(N com ×3) instance offset;

[0048] With size N ins ×(N com The instance offset and size are N ×3). ins ×(N com The sparse point cloud of the complete instance is obtained by adding the cluster set-level point cloud features of (×3) clusters.

[0049] This invention discloses an instance segmentation method based on point cloud completion and merging. The method involves inputting point cloud information containing location and color information into a feature extraction backbone network for point-level feature extraction, obtaining point cloud features. These features are then fed into an offset prediction branch and a semantic prediction branch. The semantic prediction branch is used to perform semantic prediction on the point cloud features, yielding a point cloud semantic score. The offset prediction branch is used to predict the center offset of the point cloud features, obtaining a center offset vector between the point cloud and the center point of its respective instance. Point clustering is performed based on the semantic score and the center offset vector to obtain a preliminary instance segmentation result. The preliminary instance segmentation result is then merged again using the point cloud features and point cloud completion to obtain a corrected instance prediction result. Finally, the corrected instance prediction result is adjusted by optimizing the network to obtain the final instance segmentation result. This method improves segmentation accuracy and solves the problem of low accuracy in existing point cloud instance segmentation methods. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a diagram illustrating the completion and merging of modules.

[0052] Figure 2 This is a schematic diagram of semantic prediction for cluster sets.

[0053] Figure 3 This is a schematic diagram of the encoder for the completion and merging module.

[0054] Figure 4 This is a schematic diagram of the completion and merging module decoder.

[0055] Figure 5 This is a diagram illustrating the merging of clustered sets.

[0056] Figure 6 This is a schematic diagram of a lightweight completion merging module encoder.

[0057] Figure 7 This is a schematic diagram of the lightweight completion merging module decoder.

[0058] Figure 8 This is a schematic diagram of a point cloud instance segmentation network based on point cloud completion and merging.

[0059] Figure 9This is a schematic diagram of the center offset vector. (a) Semantic prediction in the original space; (b) with the center offset vector added.

[0060] Figure 10 This is a schematic diagram of the scene visualization results of CAIS on the ScanNet V2 validation set.

[0061] Figure 11 This is a diagram illustrating the comparison and visualization results of the completed merge module.

[0062] Figure 12 This is a flowchart of an instance segmentation method based on point cloud completion and merging provided by the present invention. Detailed Implementation

[0063] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0064] Please see Figures 1 to 12 Example 1:

[0065] This invention provides an instance segmentation method based on point cloud completion and merging, comprising the following steps:

[0066] S1 inputs the point cloud information with location and color information into the feature extraction backbone network to perform point-level feature extraction, obtains point cloud features, and then passes the point cloud features into the offset prediction branch and the semantic prediction branch respectively.

[0067] The backbone network includes a voxelization module and a U-Net-type feature extraction network.

[0068] The U-Net-type feature extraction network comprises seven layers. The left side is used for downsampling, and the right side is used for upsampling. The first layer on the left side is a three-layer stacked convolutional layer (conv, stride 1). The second and third layers have the same network structure, consisting of a convolutional layer with a stride of 2 (conv-s2) and two convolutional layers with a stride of 1. Layers 2 through 7 perform downsampling until the output feature map reaches the desired size. The right side includes six upsampling layers with the same network structure. Each upsampling layer consists of a deconvolutional layer dconv-s2 with a stride of 2, a two-dimensional convolutional layer conv-s2, and a convolutional layer conv. The input of each upsampling layer includes the output of the previous upsampling layer and the output of the same layer on the left. The output of the last upsampling layer is the processed result of M×K dimensional (K is a preset value representing the feature dimension) voxel features, which is then back-mapped to the point cloud to obtain N×K dimensional point cloud features.

[0069] The step of inputting point cloud information containing location and color information into the feature extraction backbone network for point-level feature extraction to obtain point cloud features includes:

[0070] The S11 voxelization module converts the input point cloud information into multiple voxels and extracts the voxel features of each voxel to obtain the output feature map.

[0071] S12 inputs the output feature map into the U-Net-type feature extraction network, outputs the processing result of the voxel feature, and maps the processing result back to the point cloud to obtain the point cloud feature.

[0072] S2 uses the semantic prediction branch to perform semantic prediction on the point cloud features to obtain a point cloud semantic score;

[0073] Specifically, the semantic prediction loss adopts cross-entropy loss, which is used to supervise the training of the semantic prediction branch, as shown in Equation (4-5).

[0074]

[0075] Where N is the number of point clouds. and These are the predicted semantic label and the true semantic label for any point i in the point cloud, respectively.

[0076] S3 uses the offset prediction branch to perform center offset prediction on the point cloud features to obtain the center offset vector between the point cloud and the center point of its instance;

[0077] Specifically, the center offset loss is used to supervise the training of the offset prediction branch, as shown in Equation (4-6).

[0078]

[0079] Where I is the indicator function, P fg This represents the set of point clouds belonging to the foreground in the entire scene. and Let L represent the predicted center offset vector and the true center offset vector at point i, respectively. Center offset loss L off Background points in the scene (such as walls and floors) are ignored; only foreground points are included in the calculation.

[0080] S4 performs point clustering based on the semantic score and the center offset vector to obtain preliminary instance segmentation results;

[0081] The specific method is as follows:

[0082] S41 adds the point cloud coordinates to the center offset vector, thereby pulling the point cloud in the original space toward the instance center point;

[0083] Specifically, the point cloud coordinates of size N×3 are added to the center offset vector of size N×3 obtained by the offset prediction branch, thereby pulling the point cloud in the original space toward the center point of its instance.

[0084] S42 clusters all point clouds in the offset space that satisfy the condition that the distance is less than the clustering radius and the semantic score is greater than the semantic score threshold based on the semantic score, and obtains preliminary instance segmentation results.

[0085] Specifically, for all clusters in the offset space that satisfy the condition that the distance is less than the cluster radius r and the semantic score is greater than the semantic score threshold T... ins The point cloud is clustered. By clustering the points, a cluster set can be obtained, which is the preliminary point cloud instance segmentation result.

[0086] S5 combines the point cloud features and uses point cloud completion to merge the preliminary instance segmentation results again to obtain the corrected instance prediction results;

[0087] The specific method is as follows:

[0088] S51 converts the point cloud features of size N×K into voxels according to the cluster sets to obtain preliminary voxel features of size M×K; inputs the preliminary voxel features into the U-Net-type feature extraction network to obtain final voxel features of size M×K containing global and local features; and transforms the final voxel features into N×K voxel features through global average pooling. ins ×K cluster set-level semantic features; based on the cluster set-level semantic features, an MLP is used to predict a group of size N. ins ×N class The semantic score corresponding to each cluster set is taken as the maximum value to obtain the semantic label of the cluster set;

[0089] S52 will have a size of N i The point cloud features extracted by the backbone network from ×K and N i The point cloud coordinates of size N are concatenated by multiplying by 3. i The cluster set features of ×(K+3), where N i The number of point clouds in cluster set i is given, and the features of the cluster set are input into the encoder of the completion and merging module to obtain the point cloud features of the cluster set.

[0090] Specifically, the step of inputting the cluster set features into the encoder of the completion and merging module to obtain the cluster set point cloud features includes:

[0091] The encoder described in S521 has a size of N. i The cluster set features of ×(K+3) are transformed into N through a two-layer MLP. i Features of ×256;

[0092] S522 will N i A feature input of 256 × 106 is used to obtain a feature of 512 Ni × 12.

[0093] S523 utilizes a two-layer MLP to integrate N i The feature of ×512 is transformed into N i ×(N com The characteristics of ×3×3), where N com This is a hyperparameter representing the number of sparse point clouds generated by point cloud completion for each cluster set;

[0094] S524 will N i ×(N com The features of (×3×3) are normalized and activated through Norm and ReLU layers, resulting in an encoded size of N. i ×(N com The clustered point cloud features of (×3×3) clusters.

[0095] S53 transforms the center offset vector into an instance offset vector, and inputs the instance offset vector and the cluster set point cloud features into the decoder of the completion and merging module to obtain the sparse point cloud of the complete instance;

[0096] Specifically, the instance offset vector of cluster set i The calculation method is shown in formula (4-1).

[0097]

[0098] Where N i Let P be the number of point clouds in cluster set i. i Let Δx represent the point cloud in cluster set i. j Let be the center offset vector of point j.

[0099] The step of inputting the instance offset vector and the clustered point cloud features into the decoder of the completion and merging module to obtain the sparse point cloud of the complete instance includes:

[0100] The decoder described in S531 will have a size of N. i ×(N com The cluster set features of (×3×3) are transformed into a set of size 1×(N) through max pooling. com Features of (×3×3);

[0101] S532 pairs of sizes 1×(N) com The features of (×3×3) are transformed to obtain a shape of size N. com Features of ×(3×3);

[0102] S533 uses two layers of MLP to process a data of size N.com The feature transformation of ×(3×3) is N com The sparse point cloud after completion (×3);

[0103] S534 copies the instance offset vector of a 1×3 instance set to N. com 1, stacked together, yields a size of N. com ×3 instance offset;

[0104] S535 adds the instance offset to the sparse point cloud to obtain the sparse point cloud of the complete instance.

[0105] The completion prediction loss is used to supervise the training of the completion merging module, as shown in Equation (4-7).

[0106]

[0107] Where, N ins To predict the total number of instances in the set, EMD represents bulldozer distance, P i This represents all points in the prediction instance set i. This represents the point cloud of all real instances corresponding to the predicted instance set i.

[0108] S54 utilizes the bulldozer distance (EMD) between sparse point clouds in the sparse point cloud of the complete instance, and uses the bulldozer distance as the similarity score between sparse point clouds; combined with the semantic labels of the cluster set, the original cluster sets with the same semantic labels and the similarity scores between the completed sparse point clouds are merged with the merging threshold to obtain the corrected instance prediction result, as shown in Algorithm 4-1.

[0109] EMD can solve this problem well. EMD requires that the point cloud sets involved in the calculation have the same number of points, and then uses linear programming to find a one-to-one mapping relationship between the two point cloud sets so that the sum of the average shortest distances is minimized, as shown in Equation (4-4).

[0110]

[0111] Where P i and P j Let p represent two point cloud sets respectively. i For P i Point p in j Let N be a point in Pj. i This represents the number of point clouds in the point cloud set Pi. To find the point cloud set P that minimizes the average shortest distance i Point Cloud Collection P j A one-to-one mapping.

[0112]

[0113]

[0114] S6 adjusts the corrected instance prediction results by optimizing the network to obtain the final instance segmentation result.

[0115] Specifically, the N×K input features are first further extracted using a two-layer U-Net-type feature extraction network to obtain N×K features. During feature extraction, voxelization is performed according to the cluster categories to facilitate the extraction of instance information. Then, the N×K features are used to predict an N×1 mask through a two-layer MLP network. This mask is then used to filter out instance predictions with lower confidence levels, resulting in N×K features. fg ×K filtered point cloud features, where N fg This represents the number of point clouds after filtering. The size is N. fg The point cloud features of size ×K are back-mapped to an instance cluster set through average pooling, resulting in a set of size N. ins ×K cluster set features. Finally, a two-layer MLP is used to analyze N cluster sets. ins Predict N from the cluster set features of ×K ins The confidence score of the cluster set is calculated as ×1. Clusters with confidence scores less than the confidence threshold T are considered. score After instance prediction filtering, the final point cloud instance segmentation result is obtained.

[0116] Mask prediction loss L mask The mask score is used to supervise the training of the optimization network, as shown in Equation (4-8).

[0117]

[0118]

[0119] Where I is the indicator function, N ins To predict the total number of instances in the set, iou i To predict the intersection-union ratio (IU) of instance set i and the corresponding target instance set, N i To predict the number of points in instance set i, m j To provide a prediction mask for point j in the prediction instance set i, To predict the true mask value of point j in instance set i.

[0120] Confidence loss L score The training of the confidence score is supervised, as shown in equation (4-9).

[0121]

[0122] Where N insTo predict the total number of instances in the set, iou i To predict the intersection-union ratio (IU) of instance set i and the corresponding target instance set, the score... i , to predict the confidence score for instance set i.

[0123] The overall training of the network is achieved through multi-task loss L. Total Supervision, L Total The definition is shown in equation (4-10).

[0124] L Total =W sem ·L sem +W off ·L off +W com ·L com +W mask ·L mask +

[0125] W score ·L score (4-10)

[0126] Among them, L sem L is the semantic prediction loss defined in equation (4-5). off L is the offset prediction loss defined in equation (4-6). com L is the completion prediction loss defined in equation (4-7). mask L is the mask prediction loss defined in equation (4-8). score W represents the confidence loss as defined in equation (4-9). off W sem W com W mask and W score These are the weights corresponding to each loss, used to balance the descent gradient of each loss during network training, so that the network model reaches the overall optimum.

[0127] Example 2:

[0128] This invention provides an instance segmentation method based on point cloud completion and merging, comprising the following steps:

[0129] S1 inputs the point cloud information with location and color information into the feature extraction backbone network to perform point-level feature extraction, obtains point cloud features, and then passes the point cloud features into the offset prediction branch and the semantic prediction branch respectively.

[0130] The backbone network includes a voxelization module and a U-Net-type feature extraction network.

[0131] The U-Net-type feature extraction network comprises seven layers. The left side is used for downsampling, and the right side is used for upsampling. The first layer on the left side is a three-layer stacked convolutional layer (conv, stride 1). The second and third layers have the same network structure, consisting of a convolutional layer with a stride of 2 (conv-s2) and two convolutional layers with a stride of 1. Layers 2 through 7 perform downsampling until the output feature map reaches the desired size. The right side includes six upsampling layers with the same network structure. Each upsampling layer consists of a deconvolutional layer dconv-s2 with a stride of 2, a two-dimensional convolutional layer conv-s2, and a convolutional layer conv. The input of each upsampling layer includes the output of the previous upsampling layer and the output of the same layer on the left. The output of the last upsampling layer is the processed result of M×K dimensional (K is a preset value representing the feature dimension) voxel features, which is then back-mapped to the point cloud to obtain N×K dimensional point cloud features.

[0132] The step of inputting point cloud information containing location and color information into the feature extraction backbone network for point-level feature extraction to obtain point cloud features includes:

[0133] The S11 voxelization module converts the input point cloud information into multiple voxels and extracts the voxel features of each voxel to obtain the output feature map.

[0134] S12 inputs the output feature map into the U-Net-type feature extraction network, outputs the processing result of the voxel feature, and maps the processing result back to the point cloud to obtain the point cloud feature.

[0135] S2 uses the semantic prediction branch to perform semantic prediction on the point cloud features to obtain a point cloud semantic score;

[0136] S3 uses the offset prediction branch to perform center offset prediction on the point cloud features to obtain the center offset vector between the point cloud and the center point of its instance;

[0137] S4 performs point clustering based on the semantic score and the center offset vector to obtain preliminary instance segmentation results;

[0138] The specific method is as follows:

[0139] S41 adds the point cloud coordinates to the center offset vector, thereby pulling the point cloud in the original space toward the instance center point;

[0140] Specifically, the point cloud coordinates of size N×3 are added to the center offset vector of size N×3 obtained by the offset prediction branch, thereby pulling the point cloud in the original space toward the center point of its instance.

[0141] S42 clusters all point clouds in the offset space that satisfy the condition that the distance is less than the clustering radius and the semantic score is greater than the semantic score threshold based on the semantic score, and obtains preliminary instance segmentation results.

[0142] Specifically, for all clusters in the offset space that satisfy the condition that the distance is less than the cluster radius r and the semantic score is greater than the semantic score threshold T... ins The point cloud is clustered. By clustering the points, a cluster set can be obtained, which is the preliminary point cloud instance segmentation result.

[0143] S5 combines the point cloud features and uses point cloud completion to merge the preliminary instance segmentation results again to obtain the corrected instance prediction results;

[0144] The specific method is as follows:

[0145] S51 converts the point cloud features of size N×K into voxels according to the cluster sets to obtain preliminary voxel features of size M×K; inputs the preliminary voxel features into the U-Net-type feature extraction network to obtain final voxel features of size M×K containing global and local features; and transforms the final voxel features into N×K voxel features through global average pooling. ins ×K cluster set-level semantic features; based on the cluster set-level semantic features, an MLP is used to predict a group of size N. ins ×N class The semantic score corresponding to each cluster set is taken as the maximum value to obtain the semantic label of the cluster set;

[0146] S52 inputs the preliminary voxel features of size M×K and the voxel coordinates of size M×3 corresponding to each voxel into the encoder of the lightweight completion and merging module to obtain the clustered voxel features.

[0147] The specific method is as follows:

[0148] The encoder described in S521 passes the initial voxel features of size M×K through a two-layer U-Net network to obtain global and local features;

[0149] S522 concatenates the global and local features with voxel coordinates of size M×3, and then transforms the feature shape into an M×256 feature through a two-layer MLP.

[0150] S523 sequentially transforms the M×256 feature into M×512, M×1024, and M×(N) features through three double-layer MLPs. com The characteristics of ×3×3), where N com This is a hyperparameter representing the number of sparse point clouds generated by point cloud completion for each cluster set;

[0151] S524 utilizes Norm and ReLU layers to convert M×(N) com Normalize and activate the features of (×3×3) to obtain an encoded size of M×(N). com The cluster features of the ×3×3) cluster set.

[0152] S53 transforms the center offset vector into an instance offset vector, and inputs the cluster set voxel features and the instance offset vector into the decoder of the lightweight completion and merging module to obtain the sparse point cloud of the complete instance.

[0153] Specifically, the step of inputting the cluster set voxel features and the instance offset vector into the decoder of the lightweight completion and merging module to obtain the sparse point cloud of the complete instance includes:

[0154] The decoder described in S531 uses a dual-layer MLP to convert M×(N) com The voxel characteristics of ×3×3) are converted to M×(N) com Features of ×3);

[0155] S532 global average pooling will M×(N) com The voxel characteristics of ×3) are transformed into N ins ×(N com ×3) cluster-level point cloud features, N ins The number of cluster sets;

[0156] S533 assigns all cluster sets to a size of N. ins The set of ×3 instance offset vectors is copied N times respectively. ins This time, we get a size of N. ins ×(N com ×3) instance offset;

[0157] S534 will have a size of N ins ×(N com The instance offset and size are N ×3). ins ×(N com The sparse point cloud of the complete instance is obtained by adding the cluster set-level point cloud features of (×3) clusters.

[0158] S54 utilizes the bulldozer distance (EMD) between sparse point clouds in the sparse point cloud of the complete instance, and uses the bulldozer distance as the similarity score between sparse point clouds; combined with the semantic labels of the cluster set, the original cluster sets with the same semantic labels and whose similarity scores between the completed sparse point clouds are less than the merging threshold are merged to obtain the corrected instance prediction result.

[0159] S6 adjusts the corrected instance prediction results by optimizing the network to obtain the final instance segmentation result.

[0160] Specifically, the N×K input features are first further extracted using a two-layer U-Net-type feature extraction network to obtain N×K features. During feature extraction, voxelization is performed according to the cluster categories to facilitate the extraction of instance information. Then, the N×K features are used to predict an N×1 mask through a two-layer MLP network. This mask is then used to filter out instance predictions with lower confidence levels, resulting in N×K features. fg ×K filtered point cloud features, where N fg This represents the number of point clouds after filtering. The size is N. fg The point cloud features of size ×K are back-mapped to an instance cluster set through average pooling, resulting in a set of size N. ins ×K cluster set features. Finally, a two-layer MLP is used to analyze N cluster sets. ins Predict N from the cluster set features of ×K ins The confidence score of the cluster set is calculated as ×1. Clusters with confidence scores less than the confidence threshold T are considered. score After instance prediction filtering, the final point cloud instance segmentation result is obtained.

[0161] To verify the effectiveness of the network model, this paper conducts detailed and comprehensive experiments on the indoor scene point cloud dataset ScanNetV2. In addition, experiments are also conducted on the large-scene indoor scene dataset S3DIS to verify the effectiveness of CAIS in handling large scenes. This is based on a point cloud completion and merging network for 3D instance segmentation (CAIS).

[0162] The S3DIS dataset contains 6 regions, a total of 11 scenes, and 271 individual rooms. The indoor scenes in the S3DIS dataset are larger and contain more point clouds than those in the ScanNetV2 dataset. Therefore, experiments on the S3DIS dataset are more effective in illustrating the performance of network models in large-scale scenes.

[0163] The main evaluation metrics used for point cloud instance segmentation in the S3DIS dataset are accuracy, recall, AP, and AP50. Predictions with an IoU greater than 50% are considered correct samples. A predicted positive sample that is actually positive is denoted as TP (True Positive), a predicted positive sample that is actually negative is denoted as FP (False Positive), a predicted negative sample that is actually negative is denoted as TN (True Negative), and a predicted negative sample that is actually positive is denoted as FN (False Negative).

[0164] The accuracy rate is calculated as shown in equation (4-11).

[0165]

[0166] The recall rate is calculated as shown in equation (4-12).

[0167]

[0168] The experiments conducted on the S3DIS dataset in this paper follow general convention, setting Area5 of the six regions as the test set and the remaining five regions as the training set.

[0169] The experimental environment for this experiment is shown in Table 4-1.

[0170] Table 4-1 CAIS Experimental Environment

[0171]

[0172] Table 4-2 shows some parameter settings for CAIS on the ScanNetV2 and S3DIS training sets.

[0173] Table 4-2 Experimental parameter settings for CAIS on the training set

[0174]

[0175] Table 4-3 shows some of the hyperparameter settings for CAIS and the Lightweight Completion Aggregation for 3D Instance Segmentation (LCAIS) network based on point cloud completion merging, which includes a lightweight completion merging module.

[0176] Table 4-3 Some Hyperparameter Settings for CAIS and LCAIS

[0177]

[0178] The weight settings for each loss in the multi-task loss defined by equation (4-10) are shown in Table 4-4.

[0179] Table 4-4 shows the loss weight settings for CAIS.

[0180]

[0181] Unless otherwise specified, the CAIS models used in the experiments in this section are all network models with the complete completion and merging modules described in Section 4.2.1, and the experimental configurations for CAIS and LCAIS are the same.

[0182] Quantitative analysis

[0183] The CAIS network model was trained on the S3DIS training set (Area1, Area2, Area3, Area4, and Area6) according to the settings, resulting in a CAIS network model with trained parameters. The trained CAIS network model was then run on the S3DIS test set (Area5) to obtain the instance segmentation prediction results. The test results are shown in Table 4-5.

[0184] Table 4-5 Comparison of Instance Segmentation Results of CAIS on the S3DIS Test Set

[0185]

[0186] Table 4-5 presents the experimental results of point cloud instance segmentation on the S3DIS test set, and also shows the experimental results of other advanced point cloud instance segmentation algorithms on S3DIS. CAIS performs exceptionally well, achieving accuracies of 52.7%, 67.3%, 30.1%, and 75.4% on the four evaluation metrics: AP, AP50, mPrec50, and mRec50, respectively. The comparison reveals that CAIS achieves the highest accuracy in AP, AP50, and mRec50, with only a lower accuracy in mPrec50. In AP, CAIS's 52.7% is 1.1% higher than SoftGroup's 51.6%; in AP50, CAIS's 67.3% is 1.2% higher than SoftGroup's 66.1%; and in mRec50, CAIS's 75.4% is 11.2% higher than SoftGroup's 66.6%. Although CAIS performs exceptionally well on most metrics, it falls below average on mPrec50. This paper argues that this is primarily because CAIS tends to predict additional non-instance point clouds (including walls, floors, etc., that do not participate in instance segmentation) as instances during instance prediction.

[0187] The CAIS network was trained on the ScanNetV2 training set according to the settings, resulting in a CAIS network model with trained parameters. The trained CAIS network model was then run on the ScanNetV2 test set to obtain instance segmentation prediction results, which were uploaded to the ScanNetV2 official website for testing. The test results are shown in Table 4-6.

[0188] Table 4-6 Experimental Results of CAIS Instance Segmentation on ScanNetV2 Test Set

[0189]

[0190]

[0191] Table 4-6 presents the experimental results of point cloud instance segmentation on the ScanNetV2 test set, and also shows the experimental results of other advanced point cloud instance segmentation algorithms on ScanNetV2. CAIS performs exceptionally well, achieving accuracies of 46.5%, 70.7%, and 82.0% on the three evaluation metrics: AP, AP50, and AP25, respectively. The comparison reveals that CAIS achieves the highest accuracy across all three metrics. In AP, CAIS's 46.5% is 0.3% higher than the second-highest, SSEC's 46.2%; in AP50, CAIS's 70.7% is 0.7% higher than SSEC's 70.0%; and in AP25, CAIS's 82.0% is 1.7% higher than HAIS's 80.3%.

[0192] Specifically, the ScanNetV2 test set contains 18 different categories of instances. The segmentation and classification accuracy test results for each type of object instance in the ScanNetV2 test set are shown in Tables 4-7, 4-8, and 4-9.

[0193] Table 4-7 shows the AP accuracy of CAIS across various object categories, and also lists the classification AP accuracy of other state-of-the-art algorithms for comparison. It can be seen that CAIS achieved the highest AP accuracy in 7 out of the 18 instance categories on the ScanNetV2 test set. Specifically, it achieved an AP accuracy of 50.2% in the Bookshelf category, 3.1% higher than the second-highest MaskGroup (47.1%); 60.5% in the Curtain category, 2.8% higher than the second-highest PE category (57.7%); 32.3% in the Door category, 1.5% higher than the second-highest HAIS category (30.8%); and 45.1% in the Otherfurniture category. The AP accuracy was 2.1% higher than the second-highest SSEC (43.0%); in the Table category, it reached 50.6% AP accuracy, 3.9% higher than the second-highest SSEC (46.7%); in the Toilet category, it reached 90.0% AP accuracy, 1.1% higher than the second-highest HAIS (88.9%); and in the Windows category, it reached 36.6% AP accuracy, 0.4% higher than the second-highest MaskGroup (36.2%).

[0194] Table 4-7 shows the classification comparison results of CAIS on the ScanNet V2 test set (AP).

[0195]

[0196]

[0197] Table 4-8 shows the AP50 accuracy of CAIS across various object categories, and also lists the classification AP50 accuracy of other state-of-the-art algorithms for comparison. It can be seen that CAIS achieved the highest AP50 accuracy in 7 out of 18 instance categories on the ScanNetV2 test set. Specifically, it achieved 100.0% AP50 accuracy in the Bathtub category, tying for first place with some state-of-the-art algorithms; 92.4% AP50 accuracy in the Bookshelf category, 3.1% higher than the second-highest, Dyco3D's 89.3%; 86.2% AP50 accuracy in the Curtain category, 3.8% higher than the second-highest, SSEC's 82.4%; and achieved [missing data - likely a missing section] in the Otherfurniture category. With an AP50 accuracy of 62.4%, it surpasses the second-highest HAIS (59.6%) by 2.8%; in the shower curtain category, it achieves 100.0% AP50 accuracy, tying for first place with some advanced algorithms; in the table category, it achieves 76.8% AP50 accuracy, surpassing the second-highest SSEC (68.9%) by 7.9%; and in the toilet category, it achieves 100.0% AP50 accuracy, tying for first place with some advanced algorithms.

[0198] Table 4-8 shows the classification comparison results of CAIS on the ScanNetV2 test set (AP50).

[0199]

[0200]

[0201] Table 4-9 shows the AP25 accuracy of CAIS across various object categories, and also lists the classification AP25 accuracy of other state-of-the-art algorithms for comparison. It can be seen that CAIS achieves the highest AP25 accuracy on 10 out of 18 instance categories on the ScanNetV2 test set. Among them, the Bathtub category achieved 100.0% AP25 accuracy, tying for first place with other advanced algorithms; the Bookshelf category achieved 92.4% AP25 accuracy, 3.1% higher than Dyco3D's 89.3%; the Cabinet category achieved 82.6% AP25 accuracy, 4.5% higher than GICN's 78.1%; the Curtain category achieved 89.9% AP25 accuracy, 1.7% higher than HAIS's 88.2%; and the Door category achieved 67.3% AP25 accuracy, 3.2% higher than Dyco3D's 64.1%. In the Other Furniture category, it achieved an AP25 accuracy of 73.1%, 3.0% higher than the second-highest, PointGroup and PE, at 70.1%. In the Shower Curtain category, it achieved 100.0% AP25 accuracy, tying for first place with some advanced algorithms. In the Table category, it achieved 87.8% AP25 accuracy, 11.1% higher than the second-highest, HAIS, at 76.7%. In the Toilet category, it achieved 100.0% AP25 accuracy, tying for first place with some advanced algorithms. In the Windows category, it achieved 78.0% AP25 accuracy, 5.4% higher than the second-highest, HAIS, at 72.6%.

[0202]

[0203]

[0204] This paper conducts experiments on the accuracy of CAIS with configured completion merging module and LCAIS with configured lightweight completion merging module on the ScanNetV2 validation set. The results are shown in Table 4-10.

[0205] The lightweight completion merging module is less accurate than the original completion merging module, but as shown in Table 4-10, the difference is relatively small. On AP, LCAIS is 0.2% lower than CAIS; on AP50, LCAIS is 0.1% lower than CAIS; and on AP25, LCAIS is on par with CAIS.

[0206] Regarding efficiency, this paper conducted experiments on the inference time of CAIS with a configuration completion merging module and LCAIS with a configuration lightweight completion merging module on the ScanNetV2 validation set. The test results of the network model and other advanced algorithms are shown in Table 4-11.

[0207] Table 4-10 Comparison of Instance Segmentation Results between CAIS and LCAIS on the ScanNetV2 Validation Set

[0208]

[0209] Table 4-11 Comparison of inference time of CAIS on the ScanNetV2 validation set.

[0210]

[0211]

[0212] Table 4-11 shows that CAIS takes 160 seconds on the ScanNetV2 validation set containing 312 scenes on an RTX3090, with an average time of approximately 221 milliseconds per scene. LCAIS takes 144 seconds on the ScanNetV2 validation set, with an average time of approximately 462 milliseconds per scene. Both CAIS and LCAIS maintain good efficiency while achieving high accuracy, especially the lightweight completion merging module, which reduces the single-scene inference time of LCAIS by 51 milliseconds compared to CAIS.

[0213] Regarding resource usage, this paper conducted a memory usage test on the ScanNetV2 validation set, and the experimental results are shown in Table 4-12.

[0214] Table 4-12 Comparison of GPU memory usage between CAIS and LCAIS on the ScanNetV2 validation set.

[0215]

[0216] It can be observed that, with the support of the lightweight completion merging module, under the same conditions, LCAIS's memory usage during training decreased from 18905MB to 3733MB, and its memory usage during testing decreased from 6757MB to 6341MB. The lightweight completion merging module can effectively reduce the hardware requirements of the running device.

[0217] The experimental results above show that CAIS achieves higher accuracy than SSEC. While CAIS is not as efficient as SSEC, its efficiency can be improved to some extent with the help of LCAIS's lightweight merging module.

[0218] ablation experiment

[0219] CAIS innovatively proposes a completion and merging module. To further illustrate the actual effect of the completion and merging module, this paper conducts a series of ablation experiments to demonstrate the actual contribution of the completion and merging module in CAIS to the final point cloud instance segmentation effect from an objective numerical perspective.

[0220] Table 4-13 Results of modular ablation experiments of CAIS on the ScanNetV2 validation set

[0221]

[0222]

[0223] The results of the module ablation experiments are shown in Table 4-13. The addition of completion merging improved the AP of the network model from 42.6% to 46.0%, AP50 from 66.5% to 67.5%, and AP25 from 79.0% to 79.1%. The data in Table 4-13 fully demonstrates the effectiveness and rationality of the completion merging module in CAIS. The completion merging module uses a threshold segmentation method for merging. This paper conducted ablation experiments on the merging threshold Tcom, and the experimental results are shown in Table 4-14.

[0224] Table 4-14 Ablation experimental results of the merging threshold in CAIS on the ScanNet V2 validation set.

[0225]

[0226] As can be seen from Table 4-14, CAIS achieved the best results when Tcom=30, with the network model reaching 46.0% on AP, 67.5% on AP50, and 79.1% on AP25. Therefore, Tcom was ultimately set to 30.

[0227] Qualitative assessment

[0228] This section visualizes the various stages of CAIS in indoor point cloud scenarios, providing an intuitive understanding of CAIS.

[0229] It was shown, such as Figure 10 As shown, Figure 10From left to right, the visualizations are: input point cloud visualization, semantic segmentation ground truth label visualization, semantic segmentation predicted label visualization, instance segmentation ground truth label visualization, and instance segmentation predicted label visualization. In the semantic segmentation visualization, different colors are used to identify different semantic labels. In the instance segmentation visualization, different colors are only used to distinguish neighboring point clouds of different instances; that is, point clouds of the same color that are far apart do not belong to the same instance. It can be intuitively observed that CAIS's point cloud instance segmentation effect is excellent. This section also conducted experiments on CAIS in indoor point cloud scenes with and without the completion and merging module, and demonstrated the corresponding visualization effects, such as... Figure 11 As shown. Figure 11 The visualization results of some scenarios in the comparative experiment of the completion merging module are shown. From left to right, they are the visualization effects without the completion merging module, the visualization effects with the completion merging module, and the visualization effects of instance segmentation and real labels. The better results obtained by the network model with the help of the completion merging module are highlighted in red circles in the figure, which intuitively demonstrates the superiority of CAIS.

[0230] The above-disclosed embodiments are merely preferred embodiments of the point cloud completion and merging method for instance segmentation according to the present invention. They should not be construed as limiting the scope of the present invention. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. An instance segmentation method based on point cloud completion and merging, characterized in that, Includes the following steps: Point cloud information containing location and color information is input into the feature extraction backbone network for point-level feature extraction to obtain point cloud features. The point cloud features are then passed into the offset prediction branch and the semantic prediction branch, respectively. The semantic prediction branch is used to perform semantic prediction on the point cloud features to obtain a point cloud semantic score. The offset prediction branch is used to perform center offset prediction on the point cloud features to obtain the center offset vector between the point cloud and the center point of its instance. Point clustering is performed based on the semantic score and the center offset vector to obtain preliminary instance segmentation results; The point cloud features are combined and point cloud completion is used to merge the preliminary instance segmentation results to obtain the corrected instance prediction results. These include: The point cloud features of size N×K are voxelized according to cluster sets to obtain preliminary voxel features of size M×K. These preliminary voxel features are then input into a U-Net-type feature extraction network to obtain final voxel features of size M×K, which include both global and local features. Finally, global average pooling is used to transform the final voxel features into N... ins ×K cluster set-level semantic features; based on the cluster set-level semantic features, an MLP is used to predict a group of size N. ins ×N class The semantic score corresponding to each cluster set is taken as the maximum value to obtain the semantic label of the cluster set; With size N i The point cloud features extracted by the backbone network from ×K and N i The point cloud coordinates of size N are concatenated by multiplying by 3. i The cluster set features of ×(K+3), where N i The number of point clouds in cluster set i is given, and the features of the cluster set are input into the encoder of the completion and merging module to obtain the point cloud features of the cluster set. The center offset vector is transformed into an instance offset vector, and the instance offset vector and the cluster set point cloud features are input into the decoder of the completion and merging module to obtain the sparse point cloud of the complete instance. Using the sparse point cloud of the complete instance, the bulldozer distance between sparse point clouds is used as the similarity score between sparse point clouds; combined with the semantic labels of the cluster set, the original cluster sets with the same semantic labels and the similarity scores between the completed sparse point clouds are merged with the merging threshold to obtain the corrected instance prediction result. The final instance segmentation result is obtained by adjusting the corrected instance prediction result through network optimization.

2. The instance segmentation method based on point cloud completion and merging as described in claim 1, characterized in that, The backbone network includes a voxelization module and a U-Net-type feature extraction network.

3. The instance segmentation method based on point cloud completion and merging as described in claim 2, characterized in that, The step of inputting point cloud information containing location and color information into the feature extraction backbone network for point-level feature extraction to obtain point cloud features includes: The voxelization module converts the input point cloud information into multiple voxels and extracts the voxel features of each voxel to obtain the output feature map. The output feature map is input into the U-Net-type feature extraction network, which outputs the processing result of the voxel features. The processing result is then mapped back to the point cloud to obtain the point cloud features.

4. The instance segmentation method based on point cloud completion and merging as described in claim 3, characterized in that, The preliminary instance segmentation result obtained by performing point clustering based on the semantic score and the center offset vector includes: The point cloud coordinates are added to the center offset vector to pull the point cloud in the original space toward the center point of the instance. Based on the semantic score, cluster all point clouds in the offset space that satisfy the condition that the distance is less than the cluster radius and the semantic score is greater than the semantic score threshold to obtain preliminary instance segmentation results.

5. The instance segmentation method based on point cloud completion and merging as described in claim 1, characterized in that, The step of inputting the cluster set features into the encoder of the completion and merging module to obtain the cluster set point cloud features includes: The encoder will have a size of N i The cluster set features of ×(K+3) are transformed into N through a two-layer MLP. i Features of ×256; N i A feature input of 256 × 106 is used to obtain a feature of 512 Ni × 12. Using two layers of MLP to N i The feature of ×512 is transformed into N i ×(N com The characteristics of ×3×3), where N com This is a hyperparameter representing the number of sparse point clouds generated by point cloud completion for each cluster set; N i ×(N com The features of (×3×3) are normalized and activated through Norm and ReLU layers, resulting in an encoded size of N. i ×(N com The clustered point cloud features of (×3×3) clusters.

6. The instance segmentation method based on point cloud completion and merging as described in claim 5, characterized in that, The step of inputting the instance offset vector and the clustered point cloud features into the decoder of the completion and merging module to obtain the sparse point cloud of the complete instance includes: The decoder will have a size of Ni×(N) com The cluster set features of (×3×3) are transformed into a set of size 1×(N) through max pooling. com Features of (×3×3); For a size of 1×(N) com The features of (×3×3) are transformed to obtain a shape of size N. com Features of ×(3×3); Using two layers of MLP to process a size of N com The feature transformation of ×(3×3) is N com The sparse point cloud after completion (×3); Copy the instance offset vector of the 1×3 instance set N times. com 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 9, 1, 2, 1, 2, 3 ... com ×3 instance offset; The instance offset is added to the sparse point cloud to obtain the sparse point cloud of the complete instance.

Images