A 3D instance segmentation method and system based on dense and sparse convolution fusion

Abstract

The present invention discloses a three-dimensional instance segmentation method and system based on the fusion of dense and sparse convolutions. The method introduces a lightweight, high-resolution, point-based dense convolution branch into a voxel-based sparse convolution feature extraction network, which can simultaneously extract fine-grained geometric information and coarse-grained contextual information of three-dimensional point cloud data. An adaptive fusion complementary strategy is used to obtain robust and semantically rich point-level features, providing effective support for point-level clustering operations to obtain high-quality target instance masks. Utilizing the proposed dense and sparse convolution fusion module, the present invention constructs a new three-dimensional instance segmentation network framework. The three-dimensional instance segmentation network framework proposed by the present invention is used to implement 3D instance segmentation of outdoor and indoor scenes, with outstanding advantages such as high precision, low computational complexity, and few application scenario restrictions.

Description

Technical Field

[0001] The present invention belongs to the field of 3D computer vision, and in particular relates to a 3D instance segmentation method and system based on dense and sparse convolution fusion. Background Art

[0002] Computer vision is a key technology in the artificial intelligence industry, with core challenges including semantic segmentation, object detection, and instance segmentation. Instance segmentation is a fundamental and challenging task, drawing widespread attention from both academia and industry. Instance segmentation combines the requirements of semantic segmentation, requiring pixel / point-level classification, with the characteristics of object detection, requiring the ability to distinguish and identify distinct instances within the same category. Consequently, instance segmentation research has long focused on two main approaches. The first is a top-down approach based on object detection, which formulates the instance segmentation problem as two subtasks: object detection and instance mask prediction. The second is a bottom-up approach based on semantic segmentation, which typically considers instance segmentation as the result of a subsequent semantic segmentation step followed by clustering. Compared to two-dimensional images, three-dimensional point cloud data provides rich geometric, shape, and scale information, which is crucial for enabling advanced environmental interactions and context-aware services in intelligent applications such as autonomous driving, augmented reality, and robotics. However, due to the high dimensionality and unstructured nature of three-dimensional point clouds, feature extraction based on deep learning remains a significant challenge. Feature extraction networks that directly process point data have simple structures and can preserve detailed information about scene objects, but the number of points processed at a time is very limited, such as 4096. 3D scenes typically contain millions of points, which means the network can only process a portion of the input point cloud, severely impacting the representation of the scene's complete global features and contextual information. Voxel convolutional networks (VCNs) are a natural extension of 2D convolutional neural networks for learning 3D data features. However, this approach loses important geometric information during the 3D data conversion representation and subsequent learning and inference processes, resulting in high computational costs. Numerous techniques have been proposed to reduce the high memory requirements and computational costs of VCNs. Among them, index-structured submanifold sparse convolutions, by constructing hash tables, can more efficiently process large-scale spatially sparse data and have achieved excellent results on semantic segmentation tasks. Consequently, mainstream point cloud instance segmentation methods currently employ a bottom-up approach. They first utilize voxel-based spatially sparse convolutional networks to learn voxel-level features with semantic and geometric information, and then perform point-level clustering to obtain object instance masks. However, sparse convolution-based feature extractors cannot distinguish between points of different classes within the same voxel, resulting in significant boundary errors. Secondly, although sparse convolution can achieve higher resolution than ordinary convolution, due to limited computing resources, the network must actively downsample to achieve a sufficiently large receptive field. This process will cause severe loss of fine-grained information of targets in the scene, and the resolution of deep features is coarse and cannot capture small target instances.

[0003] Therefore, how to fully integrate and utilize the advantages of point-based and voxel-based feature learning, improve the ability to obtain multi-scale features and contextual information, while reducing network computing costs, and complete three-dimensional instance segmentation tasks in real time and with high precision is a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0004] In order to solve the technical problems that the current sparse convolution-based feature extractor cannot distinguish points of different categories within the same voxel, resulting in large boundary errors, and in the case of limited computing resources, excessive downsampling operations are performed to ensure resolution, resulting in the loss of fine-grained information of targets in the scene, and the deep feature resolution is coarse and cannot capture small target instances. The present invention introduces a lightweight, high-resolution, point-based dense convolution branch into the voxel-based sparse convolution feature extraction network to achieve adaptive fusion and complementarity of fine-grained and coarse-grained information to obtain robust and rich point-level semantic features, provide support for point-level clustering operations to obtain high-quality target instance masks, and construct a new 3D instance segmentation network framework using the proposed dense and sparse convolution fusion module.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is:

[0006] On the one hand, a 3D instance segmentation method based on dense and sparse convolution fusion includes the following steps:

[0007] Step 1: For the segmented scene point cloud, dense convolution and sparse convolution feature extraction networks are used to extract fine-grained point-level features and coarse-grained voxel-level features respectively;

[0008] The input format of the scene point cloud is The input features of each point include three-dimensional geometric information p i =(x i ,y i ,z i ) and color attribute f i =(r i ,g i ,b i ), the output of the dense point convolution branch is The output of the sparse voxel convolution branch is Where N is the number of scene points, D is the input feature dimension, and K p and K v are the output dimensions of the dense point convolution branch and the sparse voxel convolution branch, respectively, i∈{1,...,N}.

[0009] Step 2: Splice point-level features and voxel-level features and perform feature fusion to deeply aggregate fine-grained and coarse-grained information to obtain high-dimensional semantic features;

[0010] High-dimensional semantic features have better robustness and richness;

[0011] Step 3: Based on the high-dimensional semantic features, the semantic label prediction network and the offset vector prediction network are used to predict the semantic label of the scene point cloud and the offset vector of each point relative to the centroid of the instance to which it belongs;

[0012] Step 4: Use a hierarchical clustering algorithm to separate points with different semantic labels in the offset coordinate space, and aggregate points with the same semantic label to generate multiple groups of candidate instances;

[0013] Step 5: The high-dimensional semantic features of the corresponding points in the candidate instance are used to perform mask prediction through the mask prediction network, and the foreground and background points of the candidate instance are segmented to further refine the candidate instance and obtain the final instance mask;

[0014] Step 6: Based on the instance mask, the high-dimensional semantic features of the corresponding points are globally averaged pooled to generate features representing a single instance mask. The features of the single instance mask are input into the fully connected layer to obtain the confidence score of the instance mask. The instance mask whose confidence score meets the set threshold is assigned an instance label. The remaining instance masks are discarded, and the scene point cloud with instance labels is output to complete the segmentation.

[0015] The fully connected layer is the confidence score evaluation network; instance masks with high confidence scores (greater than 0.5) are assigned instance labels, and instance masks with low confidence scores are eliminated;

[0016] Step 7: Using the training data, perform supervised learning iterative training according to steps 1 to 6 until the overall network loss reaches a minimum or the number of iterations is reached, and the overall network parameters are updated; perform instance segmentation on the scene point cloud to be segmented according to steps 1 to 6.

[0017] The training data contains the coordinates, colors, semantic label truth values ​​and instance label truth values ​​of the points;

[0018] Furthermore, the specific process of extracting fine-grained point-level features using dense convolution and sparse convolution feature extraction networks is as follows:

[0019] In step a, each point in the input point cloud is taken as the center point, and the K nearest neighbor algorithm is used to search for the K neighboring points closest to the center point. The neighboring points are sorted according to the distance from each neighboring point to the center point to obtain the K local neighboring points of the center point.

[0020] The sorted neighborhood forms local neighborhood points, which are used to obtain local geometric features in step b to solve the point cloud permutation invariance problem;

[0021] Step b, at the center point p i The central point p is obtained by point convolution coding learning in the local neighborhood i The local geometric characteristics of

[0022] y i =Φ(W(x i -x i,j )|j=1,…K)

[0023] Among them, x i is the input feature of the i-th center point, x i,j It is the input feature of the jth local neighborhood point corresponding to the center point i. The input feature of each point includes three-dimensional geometric information p i =(x i ,y i ,z i ) and color attribute f i =(r i ,g i ,b i ), W is a learnable shared weight matrix, and Φ(·) is the maximum pooling symmetric aggregation function;

[0024] The weight matrix is ​​learned for the sorted K neighborhood points and is used to measure the influence of the neighborhood points on the center point in step a. The specific value is determined during the overall network training, and the initial value is randomly initialized.

[0025] Step c: sequentially input the local geometric features of the center point into the multi-layer perceptron to learn and obtain an adaptive high-level feature transformation function, and then sequentially input the local geometric features of the center point into the adaptive high-level feature transformation function to output fine-grained point-level features. represents the matrix, N represents the number of scene points, K p Represents the output dimension of dense convolution.

[0026] The adaptive high-level feature transformation function adds nonlinear characteristics and improves the learning ability of dense convolution and sparse convolution feature extraction networks for complex scenes.

[0027] Furthermore, for each center point p i Assign a CUDA thread and use GPU parallel acceleration to accelerate the execution of the central point p. i The central point p is obtained by point convolution coding learning in the local neighborhood i local geometric features.

[0028] Furthermore, the process of extracting coarse-grained voxel-level features using dense convolution and sparse convolution feature extraction networks is as follows:

[0029] Step ①, convert the input point cloud tensor P into a voxel tensor V by averaging the features of points within the same voxel;

[0030]

[0031]

[0032] in, It's point p k The three-dimensional space coordinates, f k p It's point p k The color features, N represents the number of scene points, D is the input feature dimension, is the voxel v m The three-dimensional discrete space coordinates of is the voxel v m The color features of , M is the number of non-empty voxels, r is the voxel resolution, floor(·) is the floor function, ζ(·) represents the point p k Whether it belongs to voxel v m , N m belongs to voxel v m The number of points;

[0033] Step 2: Input the voxel tensor V into the U-Net network constructed by stacking residual sparse convolution blocks. The data output by each 3D sparse convolution layer is then processed by batch normalization and nonlinear activation function to obtain voxel features.

[0034] Step 3: Based on the voxel-to-point mapping relationship in step 1 and the voxel features obtained in step 2, the coarse-grained voxel features of each point are obtained.

[0035] Furthermore, the feature fusion is implemented based on the following function:

[0036] F=Ψ([F point ,F voxel ])

[0037] Among them, F represents high-dimensional semantic features, K is the output dimension of feature fusion; [·] represents the concatenation operation, Ψ(·) is a learnable fusion function implemented by a multi-layer perceptron, and F point represents fine-grained point-level features, F voxel Represents coarse-grained voxel-level features.

[0038] For learnable fusion functions, see the literature: Pinkus A. Approximation theory of the MLP model in neural networks [J]. Acta numerica, 1999, 8: 143-195.

[0039] Furthermore, the overall network loss includes the semantic label prediction loss of the predicted scene point cloud, the prediction loss of the offset vector of each point in the predicted scene point cloud relative to the centroid of the instance to which it belongs, the mask prediction loss, and the confidence score evaluation loss.

[0040] Furthermore, the semantic label loss of the predicted scene point cloud is calculated using cross entropy loss, the mask prediction loss is calculated using cross entropy loss, and the confidence score evaluation loss is calculated using cross entropy loss.

[0041] The semantic label loss of the predicted scene point cloud is calculated using cross entropy loss:

[0042]

[0043] in, Represents the semantic label loss of the predicted scene point cloud, N represents the number of scene point clouds, and s i are the true value and predicted value of the semantic label of point i, respectively. w(·) is a statistical weight parameter to alleviate the problem of class imbalance, which is calculated by normalizing the average number of points of all instances of the same semantic category in the training data.

[0044] The offset vector loss of each point in the predicted scene point cloud relative to the centroid of the instance to which it belongs is calculated using the minimum absolute deviation loss:

[0045]

[0046] in, It represents the offset vector loss of each point in the predicted scene point cloud relative to the centroid of the instance to which it belongs, I is the number of true value instances, is the mask of the jth ground-truth instance, o i Represents the offset vector predicted for each point, o i =(Δx i ,Δy i ,Δz i ), Δx i ,Δy i ,Δz i The offset distances in the x, y, and z axis directions respectively; is the spatial coordinate average vector of all points in the instance, (x i ,y i ,z i ) represents point p i 3D geometric information, represents the point belonging to the jth instance, represents the points that do not belong to the jth instance.

[0047] The mask prediction loss is calculated using cross entropy loss:

[0048]

[0049] in, represents the mask prediction loss, |C| is the number of predicted instances, m j ={m j1 ,...m jN} is the mask of the j-th predicted instance, m ji =1 indicates the point belongs to this instance, m ji =0 means not belonging; and m ji are the instance mask truth and predicted value of point i belonging to the jth instance.

[0050] The confidence score evaluation loss is calculated using cross entropy loss:

[0051]

[0052] in, represents the confidence score evaluation loss, |C| is the number of predicted instances, and S i are the true and predicted confidence scores of the i-th instance, respectively.

[0053] Furthermore, the hierarchical clustering algorithm is used to separate points with different semantic labels in the offset coordinate space and to aggregate points with the same semantic label to generate multiple groups of candidate instances. The method includes two stages: point clustering and point set clustering.

[0054] Phase 1: Based on the predicted semantic labels and offset vectors, a preliminary set of instance points is clustered in the offset coordinate space using the set clustering radius.

[0055] In the offset coordinate space, points that are close to each other and have the same semantic label within the set cluster radius are clustered into one category; the original coordinates of the point are added to the obtained offset vector to obtain the offset coordinates of the point, and the offset coordinate space is determined by the offset coordinates of the point;

[0056] The second stage: cluster the points with accurate offset vectors that meet the set number threshold to form an incomplete large-scale main instance, and the points with offset vector prediction values ​​lower than the prediction threshold to form small-scale fragmented instances; then use the dynamic clustering distance d set =max(d primary ,d class ) absorbs the surrounding fragmented instances to generate complete candidate instances; where d primary is the maximum length of the master instance, d classis the average maximum length of all instances of the same semantic category in the dataset.

[0057] If the predicted value of the offset vector is lower than the prediction threshold, it means that the prediction result of the offset vector is inaccurate;

[0058] The master instance uses a dynamic clustering distance d set =max(d primary ,d class ) absorbs the surrounding fragmented instances to generate complete candidate instances. primary is the maximum length of the master instance, d class is the average maximum length of all instances of the same semantic category in the dataset;

[0059] On the other hand, a 3D instance segmentation system based on dense and sparse convolution fusion includes:

[0060] Point-level and voxel-level feature extraction units are used to extract fine-grained point-level features and coarse-grained voxel-level features from the segmented scene point cloud using dense convolutional and sparse convolutional feature extraction networks respectively.

[0061] The feature fusion unit, which deeply aggregates fine-grained and coarse-grained information by splicing point-level features and voxel-level features and performing feature fusion, thereby obtaining high-dimensional semantic features;

[0062] The semantic label and offset vector prediction unit predicts the semantic label of the scene point cloud and the offset vector of each point relative to the centroid of the instance to which it belongs based on high-dimensional semantic features and the semantic label and offset vector prediction network;

[0063] Aggregation unit, which uses a hierarchical clustering algorithm to separate points with different semantic labels in the offset coordinate space and aggregate points with the same semantic label to generate multiple groups of candidate instances;

[0064] The mask prediction unit performs mask prediction on the high-dimensional semantic features of the corresponding points in the candidate instance through the mask prediction network, segmenting the foreground and background points of the candidate instance to further refine the candidate instance and obtain the final instance mask;

[0065] The confidence evaluation unit performs global average pooling on the high-dimensional semantic features of the corresponding points based on the instance mask to generate features representing a single instance mask. The features of the single instance mask are input into the fully connected layer to obtain the confidence score of the instance mask. Instance masks whose confidence scores meet the set threshold are assigned instance labels, and the remaining instance masks are discarded. The scene point cloud with instance labels is output to complete the segmentation.

[0066] Training unit: Using the training data, the point-level and voxel-level feature extraction unit, feature fusion unit, semantic label and offset vector prediction unit, aggregation unit, mask prediction unit and confidence assessment unit are called in sequence to perform supervised learning iterative training until the overall network loss reaches the minimum or the number of iterations is reached, and the overall network parameters are updated; the remaining units of the system are used in sequence to perform instance segmentation on the scene point cloud to be segmented.

[0067] In another aspect, a readable storage medium stores a computer program, wherein the computer program is invoked by a processor to implement:

[0068] The above steps of a 3D instance segmentation method based on the fusion of dense and sparse convolutions.

[0069] Beneficial effects

[0070] The technical effect of the present invention is that the present invention can directly realize the instance segmentation of unstructured, high-dimensional and non-uniform density point cloud scenes, and has outstanding advantages such as high segmentation accuracy, high real-time performance, and diverse application scenarios. It can effectively meet the needs of indoor mobile robots, autonomous driving vehicles, industrial robots and other intelligent devices to complete three-dimensional scene perception and understanding tasks, and improve the intelligent experience of human-machine-environment interaction and services.

[0071] Compared with existing 3D instance segmentation methods, the invention is creative in the following aspects:

[0072] 1) This paper proposes a local point dense convolution layer that encodes fine-grained information. It effectively reduces memory usage through CUDA programming, can quickly process large-scale 3D point clouds on point-based branches, and is more conducive to the segmentation of small objects in the scene.

[0073] 2) This paper proposes a feature extraction module that fuses dense and sparse convolutions, and simultaneously utilizes the complementary information of point-based and voxel-based feature learning to obtain more robust and rich high-dimensional semantic features.

[0074] 3) Based on the dense and sparse convolutional fusion modules, this paper designs a novel 3D instance segmentation network DSGroup, which can be effectively applied to indoor and outdoor scenes in real time, with high instance segmentation accuracy and few application scenario restrictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is the overall flow chart of the present invention;

[0076] Figure 2 This is the structural diagram of the 3D instance segmentation network based on dense and sparse convolution fusion proposed by the present invention;

[0077] Figure 3Partial scene visualization results of semantic prediction and instance segmentation of the 3D instance segmentation method of the present invention on the STPLS3D outdoor virtual dataset;

[0078] Figure 4 Visualization results of some scenes of comparative experiments on the ScanNetv2 indoor dataset using PointGroup, HAIS, and the 3D instance segmentation method of the present invention;

[0079] Figure 5 Partial scene visualization results of semantic prediction and instance segmentation of the 3D instance segmentation method of the present invention on the S3DIS indoor dataset;

[0080] Figure 6 Schematic diagram of the hierarchical clustering process of the present invention. DETAILED DESCRIPTION

[0081] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0082] The point cloud data involved in the present invention can use the public outdoor scene dataset STPLS3D and the public indoor scene datasets ScanNetv2 and S3DIS. Among them, the STPLS3D dataset covers a virtual urban landscape of approximately 16 square kilometers, with up to 18 semantic categories and 14 instance classes, and contains the coordinate information and color information (x, y, z, r, g, b) of the points. The ScanNetv2 dataset provides a variety of indoor scenes such as offices, apartments, and bedrooms. The point cloud data is acquired by an RGB-D camera and contains the coordinate information, color information, and alpha channel (x, y, z, r, g, b, alpha) of the points. The S3DIS dataset provides 271 point cloud scenes of three different campus buildings, annotated with semantic instance masks of 13 different classes, and contains the coordinate information and color information (x, y, z, r, g, b) of the points. Specifically, in preprocessing, the STPLS3D scene point cloud will be segmented into non-overlapping blocks of 50m×50m, and the S3DIS will be randomly downsampled to one-quarter the number of points. The voxel sizes of outdoor and indoor scenes are set to 0.33m and 0.02m respectively. In the point clustering stage, the clustering radius of outdoor and indoor scenes is set to 0.9m and 0.03m respectively. The present invention provides the test results based on these three public data sets.

[0083] Example 1

[0084] like Figure 1 FIG. 1 is a flow chart of the present invention, which shows a 3D instance segmentation method based on dense and sparse convolution fusion, comprising the following steps:

[0085] Step 1: Based on the point cloud of the STPLS3D dataset, the two branches of the proposed dense and sparse convolutional feature extraction networks are used to extract fine-grained point-level features and coarse-grained voxel-level features respectively;

[0086] The input point cloud format of the network is The input features of each point include three-dimensional geometric information p i =(x i ,y i ,z i ) and color attribute f i =(r i ,g i ,b i ), the output of the dense point convolution branch is The output of the sparse voxel convolution branch is Where N is the number of scene points, D is the input feature dimension, and K p and K v are the output dimensions of the two network branches, i∈{1,...,N}.

[0087] Voxel-based methods usually voxelize points into a regular grid and apply 3D sparse convolution for feature learning. This method has high memory utilization and computational efficiency. However, this method requires very high resolution to retain fine-grained position information. On the other hand, high-resolution points can retain the accuracy of position information as well as rich local geometric features. Therefore, the present invention proposes a feature extraction method that combines dense and sparse convolution, which combines the advantages of point-based and voxel-based methods. It can not only extract global features with a large receptive field, but also utilize fine-grained information to enhance the local consistency of features.

[0088] 1) Fine-grained feature extraction based on dense point convolution;

[0089] Step a: Take each point in the input point cloud as the center point, and use the K-nearest neighbor algorithm (KNN) to search for the K nearest neighbor points to the center point. Then sort the neighbor points according to the distance from each neighbor point to the center point. The K neighbor points and the center point form the center point neighborhood, also known as the local neighborhood.

[0090] Step b: For each center point p i Allocate a CUDA thread and use GPU parallel acceleration to learn the local geometric features of the center point in the local neighborhood through point convolution coding;

[0091] y i =Φ(W(x i -x i,j )|j=1,...K)

[0092] Among them, xi is the input feature of the i-th center point, x i,j is the input feature of the jth domain point corresponding to the center point i, W is a learnable shared weight matrix, and Φ(·) is the maximum pooling symmetric aggregation function to ensure the permutation invariance of the point cloud;

[0093] The weight matrix is ​​learned for the sorted K neighborhood points and is used to measure the influence of the neighborhood points on the center point in step a. The specific value is determined during the overall network training, and the initial value is randomly initialized.

[0094] Step c: Input the local geometric features of the center point into the multi-layer perceptron to learn the adaptive high-level feature transformation function, and then input the local geometric features of the center point into the adaptive high-level feature transformation function to output fine-grained point-level features. represents the matrix, N represents the number of scene points, K p Represents the output dimension of dense convolution.

[0095] Multilayer Perceptron References: Pinkus A. Approximation theory of the MLP modelin neural networks[J]. Acta numerica, 1999, 8: 143-195;

[0096] 2) Coarse-grained feature extraction based on sparse voxel convolution;

[0097] Step a: The input point cloud tensor is transformed by averaging the features of points within the same voxel. Convert to voxel tensor

[0098]

[0099] in, It's point p k The three-dimensional space coordinates of It's point p k The color characteristics of is the voxel v m The three-dimensional discrete space coordinates of is the voxel v m The color feature of point p, r is the voxel resolution, floor(·) is the floor function, ζ(·) represents the point p k Whether it belongs to voxel v m , M is the number of non-empty voxels, N m belongs to voxel v m The number of points;

[0100] Step b: Input V into the U-Net network constructed by stacking residual sparse convolution blocks. The data output by each 3D sparse convolution layer is then processed by batch normalization and nonlinear activation function in sequence to obtain voxel features;

[0101] U-Net network reference obtained by Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. PointGroup: Dual-set point grouping for 3d instancesegmentation. In Proceedings of the IEEE / CVF conference on computer vision and Pattern recognition, pages 4867-4876, 2020;

[0102] 3D sparse convolutional layer is a component of U-Net;

[0103] Step c: Based on the voxel-to-point mapping relationship and voxel features in step a, obtain the coarse-grained voxel features of each point

[0104] Step 2: Concatenate the point-level features and voxel-level features in step 1, fuse the complementary information provided by the two branches, and realize the adaptive deep interaction of coarse-grained information and fine-grained information to obtain more robust and rich high-dimensional semantic features.

[0105] F=Ψ([F point ,F voxel ])

[0106] Where K is the output dimension of feature fusion, [·] represents the concatenation operation, and Ψ(·) is a learnable fusion function implemented by a multi-layer perceptron.

[0107] Step 3: Based on the high-dimensional semantic features obtained in step 2, two multi-layer perceptrons are used to predict the semantic labels of the points respectively. and the offset vector of each point relative to the centroid of the instance to which it belongs The semantic label and offset vector prediction are supervised by using cross entropy loss and minimum absolute deviation loss respectively.

[0108] 1) Semantic label prediction loss

[0109]

[0110] in, and s i are the true value and predicted semantic label of point i, respectively. w(·) is a statistical weight parameter to alleviate the class imbalance problem, which is calculated by normalizing the average number of points of all instances of the same semantic category in the dataset.

[0111] For example, in the training data, there are 9 instances (labeled 1-9) of 2 sofas (with points 5001 and 5003), 3 tables (with points 2000, 2001, and 2002), and 4 chairs (with points 1000, 1002, 1004, and 1006). The semantic labels (i.e., semantic categories) of the 2 sofas are both 1, the semantic labels of the 3 tables are both 2, and the semantic labels of the 4 chairs are both 3.

[0112] Then the average number of points in the same semantic category is (5001+5003) / 2=5002 for sofa, (2000+2001+2002) / 3=2001 for table, and (1000+1002+1004+1006) / 4=1003 for chair. Normalization is done by dividing by the minimum value 1003 among (5002, 2001, 1003). The final weight parameter is (5002 / 1003, 2001 / 1003, 1003 / 1003)=(4.98, 1.99, 1).

[0113] Weight parameters are from Chen M, Hu Q, Yu Z, et al. STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Dataset[J]. 2022.

[0114] 2) Offset vector prediction loss l o :

[0115]

[0116] Among them, i =(Δx i ,Δy i ,Δz i ) represents the offset vector predicted for each point, Δx i ,Δy i ,Δz i They are the offset distances in the x, y, and z axis directions respectively. is the mask of the jth ground-truth instance, represents the point belonging to the jth instance, Indicates not belonging to, is the spatial coordinate average vector of all points in the instance, and I is the number of true value instances;

[0117] Semantic label and offset vector prediction network formed by multi-layer perceptron reference Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang. Hierarchical aggregation for 3D instance segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 15467-15476, 2021.

[0118] Step 4: Use the hierarchical clustering algorithm to separate points with different semantic labels in the offset coordinate space, and aggregate points with the same semantic label to generate multiple groups of candidate instances C = {c1,...,c L}. Among them, c i is a set of instances consisting of points with the same semantics and close distances, and L is the number of instances. The hierarchical clustering algorithm includes two stages: point clustering and point set clustering. Figure 6 shown.

[0119] 1) Point clustering: Based on the predicted semantic labels and offset vectors, a preliminary set of instance points is formed by clustering with a set clustering radius of 0.9m in the offset coordinate space;

[0120] 2) Point clustering: Point clustering cannot guarantee that all points in an instance are clustered together. Most points with accurate offset vectors can be clustered together to form an incomplete large-scale main instance, while a few points with poor offset vector predictions will form small-scale fragmented instances. The main instance is clustered with a dynamic clustering distance d. set =max(d primary ,d class ) absorbs the surrounding fragmented instances to generate complete candidate instances. primary is the maximum length of the master instance, d class is the average maximum length of all instances of the same semantic category in the dataset;

[0121] Step 5: Since the candidate instances generated in step 4 are rough, they may be aggregated to points belonging to other instances due to inaccurate semantic segmentation and offset prediction. The high-dimensional semantic features of the corresponding points of the candidate instances are input into the mask prediction network to further refine the candidate instances by segmenting the foreground and background points of the candidate instances. The final instance mask M = {m1, ..., m L}. Instance mask prediction loss Cross-entropy loss is used for supervised learning.

[0122]

[0123] Among them, m j ={m j1 ,…m jN} is the mask of the j-th predicted instance, m ji =1 indicates the point belongs to this instance, m ji =0 means not belonging. and m ji are the instance mask true value and predicted value of point i belonging to the jth instance, |C| is the number of predicted instances;

[0124] Mask prediction network references Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang. Hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 15467-15476, 2021.

[0125] Step 6: Based on the instance mask obtained in step 5, the high-dimensional semantic features of its corresponding points are input into the global average pooling layer to generate features representing a single instance. The confidence score S of the instance mask is evaluated by the fully connected layer, and finally the scene point cloud with instance labels is output. Confidence score evaluation loss Cross-entropy loss is used for supervised learning.

[0126]

[0127] in, and S i are the true and predicted confidence scores of the i-th instance respectively;

[0128] Step 7: Using the training data, perform supervised learning iterative training according to steps 1 to 6 until the overall network loss reaches a minimum or the number of iterations is reached, and the overall network parameters are updated; perform instance segmentation on the scene point cloud to be segmented according to steps 1 to 6.

[0129] like Figure 2 This is the structure diagram of the 3D instance segmentation network DSGroup based on dense and sparse convolution fusion proposed by this invention. The entire 3D instance segmentation network is trained in an end-to-end manner. The total loss function The expression is as follows:

[0130]

[0131] The test results and comparison results of the present invention are as follows Figure 3-Figure 5 shown.

[0132] A detailed comparison of the algorithm of the present invention with other existing advanced methods on the STPLS3D outdoor virtual dataset is shown in Table 1. DSGroup achieved higher average performance in all indicators. It obtained the best AP in 10 of the 14 categories. For small objects such as bicycles, lampposts and others, its performance in AP is 9.3%, 5.1% and 5.6% higher than the existing method HAIS, respectively. DSGroup can perform more efficient feature aggregation operations, including coarse-grained sparse voxel-level aggregation and fine-grained dense point-level aggregation. Therefore, when there are points from different categories in a voxel, DSGroup can extract robust and rich point-level features and avoid those ambiguous predictions at the boundaries of the object, and is very beneficial for the instance segmentation of small objects that require fine-grained information in the scene.

[0133] Table 1. Test results of STPLS3D outdoor virtual dataset

[0134]

[0135] On the ScanNet v2 indoor dataset, the proposed algorithm was compared with many methods, and DSGroup achieved the highest AP score of 43.9%. Figure 3 As can be seen, for points around object boundaries, voxel-based feature extractors have difficulty learning the distinguishability between points of different classes, while DSGroup is able to obtain more accurate instance segmentation masks in these red-boxed areas. In addition, the feature extraction network fused with dense and sparse convolutions can be easily migrated to PointGroup, improving AP and AP@50 by 1.3% and 0.6%, respectively. For the results of each class, the improved PointGroup has improved accuracy in 10 of the 18 classes. Among them, the AP of small object drawings has improved by 6.9%.

[0136] Table 2 ScanNet v2 indoor dataset test results

[0137]

[0138]

[0139] Table 3 shows the experimental results on the S3DIS indoor dataset. DSGroup achieves competitive performance compared to existing methods under different evaluation metrics. DSGroup achieves 71.7% and 73.6% in Area5 and 6-fold experiments on mPrec@50, respectively.

[0140] Table 3 Test results of S3DIS indoor dataset

[0141]

[0142] To demonstrate the efficiency of our algorithm, a systematic evaluation compared the inference time of each algorithm. As shown in Table 4, DSGroup's inference time is significantly shorter than that of instance segmentation methods based on point feature learning (SGPN, ASIS, GSPN, 3D-BoNet, and GICN). The network components require 604, 166, and 83 milliseconds, respectively. Overall, DSGroup achieves accurate instance segmentation results while maintaining high efficiency. Furthermore, it can directly use the entire scene point cloud as input for training and inference, making it more flexible for practical applications.

[0143] Table 4 Reasoning time comparison test

[0144]

[0145]

[0146] According to the above experimental analysis, the algorithm of the present invention is superior to the existing algorithms in terms of accuracy and stability of 3D instance segmentation.

[0147] Example 2

[0148] A 3D instance segmentation system based on dense and sparse convolution fusion, including:

[0149] Point-level and voxel-level feature extraction units are used to extract fine-grained point-level features and coarse-grained voxel-level features from the segmented scene point cloud using dense convolutional and sparse convolutional feature extraction networks respectively.

[0150] The feature fusion unit, which deeply aggregates fine-grained and coarse-grained information by splicing point-level features and voxel-level features and performing feature fusion, thereby obtaining high-dimensional semantic features;

[0151] The semantic label and offset vector prediction unit predicts the semantic label of the scene point cloud and the offset vector of each point relative to the centroid of the instance to which it belongs based on high-dimensional semantic features and the semantic label and offset vector prediction network;

[0152] Aggregation unit, which uses a hierarchical clustering algorithm to separate points with different semantic labels in the offset coordinate space and aggregate points with the same semantic label to generate multiple groups of candidate instances;

[0153] The mask prediction unit performs mask prediction on the high-dimensional semantic features of the corresponding points in the candidate instance through the mask prediction network, segmenting the foreground and background points of the candidate instance to further refine the candidate instance and obtain the final instance mask;

[0154] The confidence evaluation unit performs global average pooling on the high-dimensional semantic features of the corresponding points based on the instance mask to generate features representing a single instance mask. The features of the single instance mask are input into the fully connected layer to obtain the confidence score of the instance mask. Instance masks whose confidence scores meet the set threshold are assigned instance labels, and the remaining instance masks are discarded. The scene point cloud with instance labels is output to complete the segmentation.

[0155] Training unit: Using the training data, the point-level and voxel-level feature extraction unit, feature fusion unit, semantic label and offset vector prediction unit, aggregation unit, mask prediction unit and confidence assessment unit are called in sequence to perform supervised learning iterative training until the overall network loss reaches the minimum or the number of iterations is reached, and the overall network parameters are updated; the remaining units of the system are used in sequence to perform instance segmentation on the scene point cloud to be segmented.

[0156] The specific algorithm process can refer to the relevant description of Example 1. The specific implementation process of each module can refer to the content of the above method, which will not be repeated here.

[0157] Example 3

[0158] A readable storage medium stores a computer program, wherein the computer program is called by a processor to implement:

[0159] The above steps of a 3D instance segmentation method based on the fusion of dense and sparse convolutions.

[0160] The specific algorithm process can refer to the relevant description of Example 1. The specific implementation process of each module can refer to the content of the above method, which will not be repeated here.

[0161] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0162] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0163] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A 3D instance segmentation method based on dense and sparse convolution fusion, characterized in that: The following steps are involved: Step 1: For the segmented scene point cloud, dense convolution and sparse convolution feature extraction networks are used to extract fine-grained point-level features and coarse-grained voxel-level features respectively; Step 2: Splice point-level features and voxel-level features and perform feature fusion to deeply aggregate fine-grained and coarse-grained information to obtain high-dimensional semantic features; Step 3: Based on the high-dimensional semantic features, the semantic label prediction network and the offset vector prediction network are used to predict the semantic label of the scene point cloud and the offset vector of each point relative to the centroid of the instance to which it belongs; Step 4: Use a hierarchical clustering algorithm to separate points with different semantic labels in the offset coordinate space, and aggregate points with the same semantic label to generate multiple groups of candidate instances; Step 5: The high-dimensional semantic features of the corresponding points in the candidate instance are used to perform mask prediction through the mask prediction network, and the foreground and background points of the candidate instance are segmented to further refine the candidate instance and obtain the final instance mask; Step 6: Based on the instance mask, the high-dimensional semantic features of the corresponding points are globally averaged pooled to generate features representing a single instance mask. The features of the single instance mask are input into the fully connected layer to obtain the confidence score of the instance mask. The instance mask whose confidence score meets the set threshold is assigned an instance label. The remaining instance masks are discarded, and the scene point cloud with instance labels is output to complete the segmentation. Step 7: Using the training data, perform supervised learning iterative training according to steps 1 to 6 until the overall network loss reaches a minimum or the number of iterations is reached, and the overall network parameters are updated; perform instance segmentation on the scene point cloud to be segmented according to steps 1 to 6.

2. The method according to claim 1, characterized in that The specific process of extracting fine-grained point-level features using dense convolution and sparse convolution feature extraction networks is as follows: In step a, each point in the input point cloud is taken as the center point, and the K nearest neighbor algorithm is used to search for the K neighboring points closest to the center point. The neighboring points are sorted according to the distance from each neighboring point to the center point to obtain the K local neighboring points of the center point. Step b, at the center point p i The central point p is obtained by point convolution coding learning in the local neighborhood i The local geometric characteristics of y i =Φ(W(x i -x i,j )|j=1,…K) Among them, x i is the input feature of the i-th center point, x i,j It is the input feature of the jth local neighborhood point corresponding to the center point i. The input feature of each point includes three-dimensional geometric information p i =(x i ,y i ,z i ) and color attribute f i =(r i ,g i ,b i ), W is a learnable shared weight matrix, and Φ(·) is the maximum pooling symmetric aggregation function; Step c: sequentially input the local geometric features of the center point into the multi-layer perceptron to learn and obtain an adaptive high-level feature transformation function, and then sequentially input the local geometric features of the center point into the adaptive high-level feature transformation function to output fine-grained point-level features. represents the matrix, N represents the number of scene points, K p Represents the output dimension of dense convolution.

3. The method according to claim 2, characterized in that For each center point p i Assign a CUDA thread and use GPU parallel acceleration to accelerate the execution of the central point p. i The central point p is obtained by point convolution coding learning in the local neighborhood i local geometric features.

4. The method according to claim 1, wherein The process of extracting coarse-grained voxel-level features using dense convolution and sparse convolution feature extraction networks is as follows: Step ①, convert the input point cloud tensor P into a voxel tensor V by averaging the features of points within the same voxel; in, It's point p k The three-dimensional space coordinates of It's point p k The color features, N represents the number of scene points, D is the input feature dimension, is the voxel v m The three-dimensional discrete space coordinates of is the voxel v m The color features of , M is the number of non-empty voxels, r is the voxel resolution, floor(·) is the floor function, ζ(·) represents the point p k Whether it belongs to voxel v m , N m belongs to voxel v m The number of points; Step 2: Input the voxel tensor V into the U-Net network constructed by stacking residual sparse convolution blocks. The data output by each 3D sparse convolution layer is then processed by batch normalization and nonlinear activation function to obtain voxel features. Step 3: Based on the voxel-to-point mapping relationship in step 1 and the voxel features obtained in step 2, the coarse-grained voxel features of each point are obtained.

5. The method according to claim 1, characterized in that The feature fusion in step 2 is implemented based on the following function: F=Ψ([F point ,F voxel ]) Among them, F represents high-dimensional semantic features, K is the output dimension of feature fusion; [·] represents the concatenation operation, Ψ(·) is a learnable fusion function implemented by a multi-layer perceptron, and F point represents fine-grained point-level features, F voxel Represents coarse-grained voxel-level features.

6. The method according to claim 1, characterized in that The overall network loss includes the semantic label prediction loss of the predicted scene point cloud, the prediction loss of the offset vector of each point in the predicted scene point cloud relative to the centroid of the instance to which it belongs, the mask prediction loss, and the confidence score evaluation loss.

7. The method according to claim 6, characterized in that The semantic label loss of the predicted scene point cloud is calculated using cross entropy loss, the mask prediction loss is calculated using cross entropy loss, and the confidence score evaluation loss is calculated using cross entropy loss.

8. The method according to claim 1, characterized in that The hierarchical clustering algorithm is used to separate points with different semantic labels in the offset coordinate space and aggregate points with the same semantic label to generate multiple groups of candidate instances. It includes two stages: point clustering and point set clustering. Phase 1: Based on the predicted semantic labels and offset vectors, a preliminary set of instance points is clustered in the offset coordinate space using the set clustering radius. The second stage: cluster the points with accurate offset vectors that meet the set number threshold to form an incomplete large-scale main instance, and the points with offset vector prediction values ​​lower than the prediction threshold to form small-scale fragmented instances; then use the dynamic clustering distance d set =max(d primary ,d class ) absorbs the surrounding fragmented instances to generate complete candidate instances; where d primary is the maximum length of the master instance, d class is the average maximum length of all instances of the same semantic category in the dataset.

9. A 3D instance segmentation system based on dense and sparse convolution fusion, characterized by: include: Point-level and voxel-level feature extraction units are used to extract fine-grained point-level features and coarse-grained voxel-level features from the segmented scene point cloud using dense convolutional and sparse convolutional feature extraction networks respectively. The feature fusion unit, which deeply aggregates fine-grained and coarse-grained information by splicing point-level features and voxel-level features and performing feature fusion, thereby obtaining high-dimensional semantic features; The semantic label and offset vector prediction unit predicts the semantic label of the scene point cloud and the offset vector of each point relative to the centroid of the instance to which it belongs based on high-dimensional semantic features and the semantic label and offset vector prediction network; Aggregation unit, which uses a hierarchical clustering algorithm to separate points with different semantic labels in the offset coordinate space and aggregate points with the same semantic label to generate multiple groups of candidate instances; The mask prediction unit performs mask prediction on the high-dimensional semantic features of the corresponding points in the candidate instance through the mask prediction network, segmenting the foreground and background points of the candidate instance to further refine the candidate instance and obtain the final instance mask; The confidence evaluation unit performs global average pooling on the high-dimensional semantic features of the corresponding points based on the instance mask to generate features representing a single instance mask. The features of the single instance mask are input into the fully connected layer to obtain the confidence score of the instance mask. Instance masks whose confidence scores meet the set threshold are assigned instance labels, and the remaining instance masks are discarded. The scene point cloud with instance labels is output to complete the segmentation. Training unit: Using the training data, the point-level and voxel-level feature extraction unit, feature fusion unit, semantic label and offset vector prediction unit, aggregation unit, mask prediction unit and confidence assessment unit are called in sequence to perform supervised learning iterative training until the overall network loss reaches the minimum or the number of iterations is reached, and the overall network parameters are updated; the remaining units of the system are used in sequence to perform instance segmentation on the scene point cloud to be segmented.

10. A readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of a three-dimensional instance segmentation method based on dense and sparse convolution fusion as described in any one of claims 1-8.

Images