Point cloud instance segmentation method based on weak supervision dynamic convolution

Abstract

The invention belongs to the technical field of point cloud data processing, and particularly relates to a point cloud instance segmentation method based on weak supervision dynamic convolution. The method comprises the following steps: firstly, acquiring three-dimensional point cloud data of a scene, and preprocessing; then, a three-dimensional point cloud instance segmentation model is constructed, the three-dimensional point cloud instance segmentation model comprises a backbone network, a prediction head, an instance encoder and an instance decoder, and the prediction head comprises a semantic segmentation prediction head and an instance kernel prediction head; the backbone network is used for extracting point cloud features, the semantic segmentation prediction head is used for semantic segmentation prediction of points, and the instance kernel prediction head is used for instance kernel prediction so as to generate an instance kernel for each instance; an instance encoder encodes all instance kernels belonging to the same semantic class to obtain instance encoding features; the instance decoder decodes the instance coding features to obtain an instance mask; each semantic class corresponds to one instance encoder and one instance decoder. According to the method, the local features of the instances are modeled to obtain instance kernels, a weak supervision learning mode of box supervision dynamic convolution is adopted, a bounding box of the marked instances is used for training the model, geometric prior is provided for the model, and the problem that the adjacent instances with the same semantic class are overlapped or densely stacked, and the segmentation effect is poor is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud processing, and in particular relates to a point cloud instance segmentation method based on weakly supervised dynamic convolution. Background Art

[0002] 3D point cloud instance segmentation aims to identify and segment different object instances from 3D point cloud data, so as to obtain information such as the relative position, direction and distance of objects in 3D space. In recent years, 3D point cloud instance segmentation methods based on convolutional neural networks have been proposed. Convolutional neural networks can automatically learn point cloud features without the need for manual design or selection of feature extractors, thereby improving segmentation efficiency. However, most of the existing 3D point cloud instance segmentation models are based on a bottom-up strategy, which has two main disadvantages: (1) The performance of the segmentation model is very sensitive to hyperparameters that need to be manually adjusted; for example, the PointGroup proposed by Li Jiang et al. [1] The method generates instances by gradually merging neighboring points with the same category label. Both the original points and the centroid offset points are explored using a manually specified search radius. The change in instance size causes this method to be particularly sensitive to the adjustment of neural network hyperparameters, resulting in poor robustness and generalization of the model. (2) It contains complex post-processing processes; 3D-MPA proposed by Francis Engelmann et al. [2] The proposed method requires an additional 10-layer graph network and a clustering post-processing step to generate the final instance segmentation mask, which makes it difficult to apply to segmentation tasks that require real-time performance.

[0003] Later, the Dyco3d proposed by researchers [3] It is a robust instance segmentation method that performs dynamic convolution on 3D point clouds. Dynamic convolution is added to the 3D point cloud instance segmentation network. Considering the sparsity of point cloud data (the points near the surface of objects in 3D scenes are dense and the rest are sparse), a sparse convolutional neural network is used to extract point cloud features, group points based on clustering, and generate instance masks by dynamically convolving the point features.

[0004] Although existing 3D point cloud instance segmentation methods have achieved good results, when multiple objects of the same semantic class overlap or are densely stacked in the scene, the segmentation effect of the model will be significantly reduced. To address this problem, the present invention proposes a point cloud instance segmentation method based on weakly supervised dynamic convolution. Summary of the invention

[0005] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a point cloud instance segmentation method based on weakly supervised dynamic convolution.

[0006] The present invention solves the technical problem by adopting the following technical solution: A point cloud instance segmentation method based on weakly supervised dynamic convolution includes the following steps: Step 1: Obtain the 3D point cloud data of the scene and perform preprocessing; Step 2: Build a 3D point cloud instance segmentation model, including a backbone network, a prediction head, an instance encoder, and an instance decoder. Each semantic class corresponds to an instance encoder and an instance decoder. The prediction head includes a semantic segmentation prediction head and an instance kernel prediction head. The backbone network is used to extract point cloud features, which are then input into the semantic segmentation prediction head for point-by-point semantic segmentation prediction to obtain the semantic class prediction label of each point. The instance kernel prediction head selects a candidate point from each instance belonging to the same semantic class according to the semantic segmentation prediction result. Candidate points, Represents the number of instances belonging to the same semantic class; based on the candidate point, the foreground point is sampled to obtain the neighboring point set of the candidate point, which satisfies the following formula: (1) In the formula, It is The neighboring point set of a candidate point, It is sampling points, It is a collection of scenic spots. It is based on The candidate point The instance mask of the sample points, is a hyperparameter; The distance between the candidate point and each neighboring point is calculated and normalized to obtain the neighborhood of the candidate point. The point cloud features and neighborhood of each candidate point are converted into instance features through a multi-layer perceptron. Each instance feature is predicted through a linear layer to obtain an instance kernel. All instance kernels belonging to the same semantic class are input into an instance encoder, and instance feature mapping is performed through parallel convolutional layers. The feature mapping process is expressed as: (3) In the formula, It is instance feature maps, represents the convolution operation, It is Point cloud features of candidate points, It is The bounding box of each instance, It is The absolute difference vector between the bounding box of the instance and the point cloud feature, It is Instance cores; All instance feature maps belonging to the same semantic class are dynamically aggregated to obtain instance encoding features, which are expressed as: (4) In the formula, It is The instance encoding features of semantic classes, It is The attention weight of the instance feature map, is the convolution kernel bias vector; The instance decoder decodes the instance encoding features to obtain the instance mask and complete the instance segmentation; Step 3: Train the 3D point cloud instance segmentation model and use the trained 3D point cloud instance segmentation model for point cloud instance segmentation.

[0007] Furthermore, the backbone network is an improvement on the 3D U-Net model, including an encoder and a decoder; the encoder includes four encoding layers, each encoding layer includes two convolutional layers, and each convolutional layer is sequentially connected to a ReLU activation function and a maximum pooling layer; the convolution kernel sizes of the four encoding layers are 8×8×2, 16×16×2, 32×32×2, and 64×64×2, respectively, 2 represents the number of channels, and the maximum pooling layer size is 2×2×2; the decoder includes four decoding layers, each decoding layer includes two convolutional layers, and each convolutional layer is sequentially connected to a ReLU activation function and a maximum pooling layer; the convolution kernel sizes of the four decoding layers are 64×64×2, 32×32×2, 16×16×2, and 8×8×2, respectively, and the maximum pooling layer size is 2×2×2.

[0008] Furthermore, the example decoder includes three convolutional layers, each of which is connected to a ReLU activation function.

[0009] Furthermore, the attention weight extraction process is as follows: globally average pooling the instance kernel to obtain the global spatial features; the global spatial features pass through two fully connected layers in sequence to obtain the attention weights.

[0010] Furthermore, during the model training process, the training loss is calculated by the following formula; (6) In the formula, represents the cross entropy loss function used to calculate the semantic segmentation prediction loss, represents the instance encoder loss, Represents the similarity loss between instance mask and real instance; The loss of the instance encoder satisfies the following formula: (8) In the formula, is the number of semantic classes, is the normalization term, is the number of points, For the The semantic class prediction label of each point, For the The semantic class label of the instance encoder; is the indicator function, When , its value is 1, otherwise it is 0; For the The instance mask of each point and the true label The cross entropy loss.

[0011] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention obtains instance kernels by modeling local features of instances and represents instances as kernels. This can effectively handle boundary blur and occlusion problems caused by densely packed objects and can better distinguish adjacent overlapping instances.

[0012] (2) The present invention proposes a weakly supervised learning method of weakly supervised dynamic convolution. Weak supervision specifically refers to using the bounding box of the labeled instance to train the model. The bounding box of the instance can serve as an additional geometric clue for the dynamic convolution, providing a geometric prior for the segmentation model and better distinguishing each instance. An instance encoder is generated for all instances belonging to the same semantic class, and then an instance encoder is designed for each semantic class. Each instance encoder corresponds to an instance decoder to decode the instance encoding features, thereby overcoming the defect of existing methods that the segmentation effect of overlapping or densely stacked adjacent instances of the same semantic class is poor.

[0013] (3) The present invention adds dynamic convolution to the 3D point cloud instance segmentation model, which has fewer hyperparameters and post-processing steps than the traditional segmentation model, and overcomes the defects of the existing 3D instance segmentation method that is sensitive to the adjustment of hyperparameters, resulting in poor robustness and generalization. The improved U-Net network is used to extract point cloud features, which enhances the model's receptive field and feature expression ability, thereby improving segmentation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall flow chart of the present invention; Figure 2 It is a structural diagram of the three-dimensional point cloud instance segmentation model of the present invention; Figure 3 This is the structural diagram of the improved 3D U-Net of the present invention. DETAILED DESCRIPTION

[0015] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to introduce the technical solutions of the present invention in detail and are not intended to limit the protection scope of the present application.

[0016] The present invention provides a point cloud instance segmentation method based on weakly supervised dynamic convolution (hereinafter referred to as method, see Figures 1 to 3 ), including the following steps: Step 1: Obtain the 3D point cloud data of the scene, downsample the 3D point cloud data, and obtain pre-processed point cloud data; Use laser radar, industrial cameras and other equipment to obtain the three-dimensional point cloud data of the scene; since the point cloud data contains a large amount of redundant data, in order to reduce the data volume and improve processing efficiency, the three-dimensional point cloud data is downsampled to achieve preprocessing of the three-dimensional point cloud data.

[0017] Step 2: Construct a 3D point cloud instance segmentation model, including a backbone network, a prediction head, an instance encoder, and an instance decoder; the prediction head includes a semantic segmentation prediction head and an instance kernel prediction head; the backbone network is used to extract point cloud features, the semantic segmentation prediction head is used for semantic segmentation prediction of points, and the instance kernel prediction head is used for instance kernel prediction to generate an instance kernel for each instance; the instance encoder encodes all instance kernels belonging to the same semantic class to obtain instance encoding features, and each semantic class corresponds to an instance encoder; the instance decoder decodes the instance encoding features to obtain an instance mask, and the instance decoder corresponds to the instance encoder one-to-one.

[0018] In this embodiment, the 3D U-Net model is selected as the backbone network, and the preprocessed point cloud data is extracted through the backbone network. , Represents the number of points, and 6 represents the point cloud position; the structure of the 3D U-Net model is symmetrical, including an encoder and a decoder; the encoder contains four encoding layers, each of which contains two convolutional layers, and each convolutional layer is sequentially connected to a ReLU activation function and a maximum pooling layer; the convolution kernel sizes of the four encoding layers are 8×8×2, 16×16×2, 32×32×2, and 64×64×2, 2 represents the number of channels, and the maximum pooling layer size is 2×2×2. The decoder includes four decoding layers, each of which contains two convolutional layers, and each of which is sequentially connected to a ReLU activation function and a maximum pooling layer; the convolution kernel sizes of the four decoding layers are 64×64×2, 32×32×2, 16×16×2, and 8×8×2, and the maximum pooling layer size is 2×2×2.

[0019] Point cloud features The input is sent to the semantic segmentation prediction head for point-by-point semantic segmentation prediction to obtain the semantic class prediction label of each point. Based on the semantic segmentation prediction result, the instance kernel prediction head selects a candidate point from each instance belonging to the same semantic class. Candidate points, Indicates the number of instances belonging to the same semantic class; based on the candidate point, the foreground point is sampled, and the sampling point is used as the neighboring point of the candidate point, and then the neighboring point set of the candidate point is obtained, and a total of The neighboring point set of a candidate point satisfies the following formula: (1) In the formula, It is The neighboring point set of a candidate point, It is sampling points, It is a collection of scenic spots. It is based on The candidate point The instance mask of the sample points, is a hyperparameter; Calculate the distance between the candidate point and each neighboring point and divide the distance by the neighborhood radius , and then normalize the distance to between -1 and 1, and then get the neighborhood of the candidate point; through the multi-layer perceptron, the point cloud features and neighborhood of each candidate point are converted into instance features, and a total of Instance features, expressed as: (2) In the formula, It is Instance features, represents a multi-layer perceptron, It is Point cloud features of candidate points, It is A field of candidate points; Each instance feature is predicted by a linear layer to obtain the instance kernel , thus generating an instance core for each instance.

[0020] An instance encoder is designed for each semantic class, that is, all instances belonging to the same semantic class correspond to an instance encoder. The instance encoder performs instance encoding based on dynamic convolution. Dynamic convolution is a convolution kernel that adaptively changes attention according to the input. Compared with ordinary convolution, it has stronger representation ability. Input into the instance encoder, and perform instance feature mapping through parallel convolutional layers. The number of convolutional layers is equal to the number of instance cores belonging to the same semantic class. The feature mapping process is expressed as: (3) In the formula, It is instance feature maps, represents the convolution operation, It is The bounding box of each instance, It is The absolute difference vector between the bounding box of an instance and the point cloud feature; All instance feature maps belonging to the same semantic class are dynamically aggregated to achieve feature fusion and obtain instance encoding features, which are expressed as: (4) In the formula, It is The instance encoding features of semantic classes, It is The attention weight of the instance feature map, is the convolution kernel bias vector; The attention weight extraction process is as follows: all instance kernels belonging to the same semantic class are globally averaged pooled to obtain global spatial features; the global spatial features are sequentially passed through two fully connected layers to obtain attention weights, and then attention weights, assigned to each instance feature map.

[0021] The instance decoder decodes the instance encoding features and outputs the instance mask to segment each instance. Each instance encoder corresponds to an instance decoder. The instance decoder includes three convolutional layers with 8 channels and a convolution kernel size of 1×1. Each layer uses ReLU as the activation function. The decoding process is expressed as: (5) In the formula, It is The instance mask of the semantic class, Represents an instance decoder.

[0022] Step 3: Train the 3D point cloud instance segmentation model, calculate the training loss according to the loss function of formula (6), and use the trained 3D point cloud instance segmentation model for point cloud instance segmentation; (6) In the formula, represents the cross entropy loss function used to calculate the semantic segmentation prediction loss, represents the instance encoder loss, Represents the similarity loss between instance mask and real instance; The semantic class prediction label set of all points is recorded as , Indicates A set of semantic class prediction labels, is the number of semantic classes, then It is expressed as: (7) In the formula, For the The semantic segmentation prediction probability of each point is For the The true semantic class label of each point, is the number of points.

[0023] The loss function of the instance encoder is: (8) In the formula, For the The semantic class prediction label of each point, For the The semantic class label of the instance encoder; is an indicator function, When , its value is 1, otherwise it is 0; is a normalization term, which limits the value of the indicator function to [0,1]. For the The instance mask of each point and the true label The cross entropy loss.

[0024] Example This example is tested on S3DIS (a large-scale indoor 3D point cloud annotation dataset), and the average precision index AP and AP 50 Evaluate the segmentation performance. Table 1 shows the impact of different backbone networks on the segmentation results, and Table 2 shows the impact of different dynamic convolutions on the segmentation results.

[0025]

[0026] Table 1 shows that the segmentation effects of point cloud features extracted by different backbone networks as inputs of subsequent models are different. When the improved 3D U-Net is used as the backbone network in the present invention, the obtained three-dimensional point cloud instance segmentation effect is the best. This is because it has the original characteristics of 3D U-Net: using jump connections to fuse the output features of the decoder with the features of the decoder, improving the situation of insufficient upsampling; by interconnecting the features extracted in the convolution and deconvolution process, the context information is passed to a higher layer to achieve information supplementation. The improved 3D U-Net performs feature extraction on point cloud data to improve the segmentation effect.

[0027]

[0028] Table 2 shows that adding weak supervision to dynamic convolution results in better point cloud instance segmentation. This is because the axisymmetric bounding boxes corresponding to different instances can be used as additional geometric clues for dynamic convolution. If two points belong to the same bounding box, it means that they belong to the same object, which provides geometric prior knowledge for the segmentation model.

[0029] In order to verify that the proposed method has a better point cloud instance segmentation effect, five widely used point cloud instance segmentation methods are selected to compare with the proposed method. The five point cloud instance segmentation methods are: Top-Down Beats Bottom-Up in 3D Instance Segmentation (TD3D [4] ), Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks (SSTNet) [5] ), SoftGroup for 3D Instance Segmentation on Point Clouds, SoftGroup [6] : 3D Point Cloud Instance Segmentation With Point-Wise Binarization (Divid and Conquer: 3D Point Cloud Instance Segmentation With Point-Wise Binarization, PBNet [7] ), Segmenting 3D Instances by Points (PointInst3D [8]). Each method uses box-supervised dynamic convolution to test on the S3DIS dataset. In this embodiment, Area1, Area2, Area3, Area4, and Area6 are selected for training in the S3DIS dataset, and Area5 is used for testing, which ensures that the training and test samples do not appear repeatedly, and the same parameters in each method are kept consistent, which ensures the fairness between the methods. The final result reflects universality and correctness. The comparison results of different point cloud instance segmentation methods are shown in Table 3.

[0030]

[0031] Table 3 shows that the method of the present invention has a better segmentation effect. This is because the method of the present invention proposes a weakly supervised learning method that uses an axisymmetric bounding box to provide geometric priors for neighboring instances with the same semantic labels in the scene, and uses it as an auxiliary method for instance segmentation training. In addition, the use of dynamic convolution reduces the sensitivity of the model to hyperparameter adjustment. By modeling the local features of the instance features to obtain the instance kernel, it can handle the occlusion problems caused by blurred boundaries and densely packed objects, and can better distinguish adjacent overlapping objects, thereby enhancing the robustness and generalization of the segmentation model.

[0032] The sources of the above methods are: [1] Jiang L, Zhao H, Shi S, et al. Pointgroup: Dual-set pointgrouping for 3d instance segmentation[C] / / Proceedings of the IEEE / CVFconference on computer vision and Pattern recognition. 2020: 4867-4876. [2] Engelmann F, Bokeloh M, Fathi A, et al. 3d-mpa: Multi-proposalaggregation for 3d semantic instance segmentation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 9031-9040. [3] He T, Shen C, Van Den Hengel A. Dyco3d: Robust instancesegmentation of 3d point clouds through dynamic convolution[C] / / Proceedingsof the IEEE / CVF conference on computer vision and pattern recognition. 2021:354-363. [4] Kolodiazhnyi M, Vorontsova A, Konushin A, et al. Top-down beatsbottom-up in 3d instance segmentation[C] / / Proceedings of the IEEE / CVF WinterConference on Applications of Computer Vision. 2024: 3566-3574. [5] Liang Z, Li Z, Xu S, et al. Instance segmentation in 3d scenesusing semantic superpoint tree networks[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision. 2021: 2783-2792. [6] Vu T, Kim K, Luu T M, et al. Softgroup for 3d instancesegmentation on point clouds[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition. 2022: 2708-2717. [7] Zhao W, Yan Y, Yang C, et al. Divide and conquer: 3d point cloudinstance segmentation with point-wise binarization[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 562-571. [8] He T, Yin W, Shen C, et al. Pointinst3d: Segmenting 3d instances by points[C] / / European Conference on Computer Vision. Cham: Springer NatureSwitzerland, 2022: 286-302. Any matters not described in the present invention are applicable to the prior art.

Claims

1. A point cloud instance segmentation method based on weakly supervised dynamic convolution, characterized in that: The method comprises the following steps: Step 1: Obtain the 3D point cloud data of the scene and perform preprocessing; Step 2: Build a 3D point cloud instance segmentation model, including a backbone network, a prediction head, an instance encoder, and an instance decoder. Each semantic class corresponds to an instance encoder and an instance decoder. The prediction head includes a semantic segmentation prediction head and an instance kernel prediction head. The backbone network is used to extract point cloud features, which are then input into the semantic segmentation prediction head for point-by-point semantic segmentation prediction to obtain the semantic class prediction label of each point. The instance kernel prediction head selects a candidate point from each instance belonging to the same semantic class according to the semantic segmentation prediction result. Candidate points, Represents the number of instances belonging to the same semantic class; based on the candidate point, the foreground point is sampled to obtain the neighboring point set of the candidate point, which satisfies the following formula: （1） In the formula, It is The neighboring point set of a candidate point, It is sampling points, It is a collection of scenic spots. It is based on The candidate point The instance mask of the sample points, is a hyperparameter; The distance between the candidate point and each neighboring point is calculated and normalized to obtain the neighborhood of the candidate point. The point cloud features and neighborhood of each candidate point are converted into instance features through a multi-layer perceptron. Each instance feature is predicted through a linear layer to obtain an instance kernel. All instance kernels belonging to the same semantic class are input into an instance encoder, and instance feature mapping is performed through parallel convolutional layers. The feature mapping process is expressed as: （3） In the formula, It is instance feature maps, represents the convolution operation, It is Point cloud features of candidate points, It is The bounding box of each instance, It is The absolute difference vector between the bounding box of the instance and the point cloud feature, It is Instance cores; All instance feature maps belonging to the same semantic class are dynamically aggregated to obtain instance encoding features, which are expressed as: （4） In the formula, It is The instance encoding features of semantic classes, It is The attention weight of the instance feature map, is the convolution kernel bias vector; The instance decoder decodes the instance encoding features to obtain the instance mask and complete the instance segmentation; Step 3: Train the 3D point cloud instance segmentation model and use the trained 3D point cloud instance segmentation model for point cloud instance segmentation.

2. The point cloud instance segmentation method based on weakly supervised dynamic convolution according to claim 1, characterized in that: The backbone network is an improvement on the 3D U-Net model, including an encoder and a decoder; the encoder includes four encoding layers, each encoding layer includes two convolutional layers, and each convolutional layer is sequentially connected to a ReLU activation function and a maximum pooling layer; the convolution kernel sizes of the four encoding layers are 8×8×2, 16×16×2, 32×32×2, and 64×64×2, respectively, 2 represents the number of channels, and the maximum pooling layer size is 2×2×2; the decoder includes four decoding layers, each decoding layer includes two convolutional layers, and each convolutional layer is sequentially connected to a ReLU activation function and a maximum pooling layer; the convolution kernel sizes of the four decoding layers are 64×64×2, 32×32×2, 16×16×2, and 8×8×2, respectively, and the maximum pooling layer size is 2×2×2.

3. The point cloud instance segmentation method based on weakly supervised dynamic convolution according to claim 1, characterized in that: The example decoder includes three convolutional layers, each followed by a ReLU activation function.

4. The point cloud instance segmentation method based on weakly supervised dynamic convolution according to any one of claims 1 to 3, characterized in that: The attention weight extraction process is as follows: globally average pool the instance kernel to obtain the global spatial features; The global spatial features pass through two fully connected layers in sequence to obtain the attention weights.

5. The point cloud instance segmentation method based on weakly supervised dynamic convolution according to claim 4, characterized in that: During model training, the training loss is calculated by the following formula; （6） In the formula, represents the cross entropy loss function used to calculate the semantic segmentation prediction loss, represents the loss of the instance encoder, Represents the similarity loss between instance mask and real instance; The loss of the instance encoder satisfies the following formula: （8） In the formula, is the number of semantic classes, is the normalization term, is the number of points, For the The semantic class prediction label of each point, For the The semantic class label of the instance encoder; is the indicator function, When , its value is 1, otherwise it is 0; For the The instance mask of each point and the true label The cross entropy loss.

Images