A three-dimensional scene instance segmentation method combined with boundary-aware loss

Abstract

This invention discloses a 3D scene instance segmentation method incorporating boundary-aware loss, belonging to the field of instance segmentation technology. The method includes the following steps: converting the original point cloud data of a 3D scene into 3D voxel data, and then into sparse tensors; generating a set of real instances based on the 3D voxel data; constructing a 3D scene segmentation network, including a 3D backbone network, a decoder, and a third MASK module; constructing a total loss, including instance classification loss, masking loss, and boundary-aware loss; and training the 3D scene segmentation network with sparse tensors to obtain a 3D scene segmentation model, which is used for instance segmentation of the original point cloud data in the 3D scene to be segmented. This invention introduces multi-scale deformable attention, boundary-aware loss, and self-supervised pre-training to solve the problems of inefficient attention, blurred boundaries, and insufficient generalization in existing technologies, improving the quality and application value of point cloud instance understanding, and showing broad application prospects, especially in autonomous driving and smart city perception.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a three-dimensional scene instance segmentation method that incorporates boundary-aware loss. Background Technology

[0002] 3D point cloud instance segmentation is a core task in computer vision, aiming to accurately identify and separate individual object instances from a 3D scene. Complex applications such as autonomous driving, robot navigation, and augmented reality often require models with high-precision boundary localization and cross-scene adaptability to meet the needs of real-time perception and large-scale 3D reconstruction. Therefore, ensuring the accuracy of masks and the quality of boundaries is crucial for performing autonomous navigation and environment modeling tasks. However, existing technologies have shortcomings in terms of low attention computation efficiency, ambiguous boundary prediction, and lack of effective pre-training mechanisms.

[0003] Definitions:

[0004] Sparse 3D U-Net is a deep learning network designed for sparse 3D data (such as voxels and point clouds), and is a sparse optimized version of the standard 3D U-Net;

[0005] The MASK module, mentioned in the paper "Mask Transformer for 3D Semantic InstanceSegmentation", is a masking module that uses refined instance queries (an instance query set containing multiple instance queries) and point features (point clouds, sparse point clouds). Based on the dot product between the point features and the instance query (for each query), it returns the class probability and mask probability map of each predicted instance. It then constructs a predicted instance set from the class probabilities and mask probability maps of all predicted instances, and finally uses Hungarian matching to map the predicted instance set to instances in the real instance set.

[0006] DCA (Deformable Cross-Attention) is an efficient attention mechanism proposed by Deformable DETR (ICLR 2021) to address the high computational complexity of standard Transformers on high-resolution feature maps. Unlike traditional cross-attention, which calculates feature similarity between the query and all locations, leading to high complexity, DCA uses sparse sampling (focusing on only a small number of sampling points, typically 4 or 8) and learnable location offsets to enable the query to adaptively focus on key regions, significantly reducing computational complexity while maintaining strong feature interaction capabilities.

[0007] Dice loss is a widely used loss function in image segmentation tasks, especially suitable for scenarios with extreme foreground-background imbalance, such as medical image segmentation. It is based on the Dice coefficient, which measures the degree of region overlap between the predicted result and the ground truth label. Summary of the Invention

[0008] The purpose of this invention is to provide a 3D scene instance segmentation method that combines boundary-aware loss to solve the problems of low attention computation efficiency, ambiguous boundary prediction, and lack of effective pre-training mechanism in the above-mentioned 3D scene instance segmentation.

[0009] To achieve the above objectives, the technical solution adopted by this invention is as follows: a 3D scene instance segmentation method combining boundary-aware loss, comprising the following steps:

[0010] S1 transforms the raw point cloud data of the 3D scene into 3D voxel data, and then into a sparse tensor X. (0) ;

[0011] S2, generating a set of real instances based on 3D voxel data. y k G k Given the label and mask of the k-th real instance, construct the initial instance query set Q. (0) , , For the j-th initial instance, query K, where K is the total number of initial instance queries;

[0012] S3 constructs a 3D scene segmentation network, including a 3D backbone network, a decoder, and a third MASK module;

[0013] The 3D backbone network is used to extract X. (0) The multi-scale features are F0, F1, and F2, with the granularity of F0 to F2 becoming progressively coarser.

[0014] The decoder includes L stacked decoding layers, wherein the l-th decoding layer includes a first MASK module, a first DCA module, a second MASK module, and a second DCA module;

[0015] The first MASK module is used to input F0 and the upper decoding layer output instance query set Q. (l-1) Output the first set of predicted instances. p j1 M j1 They are respectively The masked probability map of the class probability of the j-th predicted instance, based on Hungarian matching. Each instance in Y corresponds one-to-one with the other, and then M is determined based on the threshold τ.j Convert to first binary mask image B j1 ;

[0016] The first DCA module is used for input And F2, in B j1 Perform deformable cross attention within the region to generate intermediate queries for the l-th layer. The intermediate query set Q is obtained. (lm) , ;

[0017] The second MASK module is used to input F0 and Q. (lm) Output the second set of predicted instances. ,Will A second binary mask image B is generated by corresponding one-to-one with the instances within Y. j2 ;

[0018] The second DCA module is used for input. And F1, in B j2 Perform deformable cross attention within the region to generate instance queries for layer l. The instance query set Q of the l-th layer is obtained. (l) , ;

[0019] The third MASK module is used to input F0 and the instance query set Q of the Lth layer. (L) Output the set of predicted instances p j M j They are respectively The masked probability map of the class probability of the j-th predicted instance, based on Hungarian matching. A one-to-one correspondence is generated between each instance in Y to generate the final binary mask B. j ;

[0020] S4, construct the total loss, including the instance classification loss, masking loss, and boundary-aware loss;

[0021] S5, using sparse tensor X (0) The 3D scene segmentation network is trained to converge by minimizing the total loss, resulting in a 3D scene segmentation model.

[0022] S6: Obtain the original point cloud data of the 3D scene to be segmented, process it into a sparse tensor, and then output a set of predicted instances through the 3D scene segmentation model.

[0023] Preferably, S1 includes S11 to S13;

[0024] S11, acquire the raw point cloud data P of the 3D scene, where each point attribute includes 3D coordinates, RGB color, category label and instance label, and the 3D scene contains I instances and C instance categories;

[0025] S12, voxelize P to generate three-dimensional voxel data. M represents the total number of voxels, c j f j They are respectively the j-th voxel (c j ,f j The three-dimensional coordinates and feature vectors of the voxel, wherein the feature vectors include voxel color, height relative to the ground, category label of the voxel, and instance label;

[0026] S13, converting 3D voxel data into sparse tensor X (0) .

[0027] Preferably, the 3D backbone network adopts Sparse 3D U-Net, and F0 is a full-resolution feature, with channel dimensions of 128, 256 and 512 for F0, F1 and F2, respectively.

[0028] Preferably, the first DCA module and the second DCA module obtain Q according to the following formula. (lm) and Q (l)

[0029] ,

[0030] ,

[0031] In the formula, DCA(∙) is a deformable cross attention operation, and the inner part of DCA(∙) is a constraint condition.

[0032] As a preferred option, in S4, the classification loss L sem Mask loss L mask Boundary perception loss L bd Calculate according to the following formulas respectively:

[0033] ,

[0034] In the formula, (j,k) represents the corresponding instance after Hungarian matching by the third MASK module. The index pair with Y, For a set of indexed pairs, N match for The total number of index pairs in the middle, CE(∙,∙) is the cross-entropy loss;

[0035] ,

[0036] In the formula, BCE(∙,∙) is the pointwise binary cross-entropy loss, Dice(∙,∙) is the Dice loss, and λ bce , λ dice These are the weights of the corresponding multiplication terms;

[0037] ,

[0038] In the formula, S is M j Endothelial collection, Let S be the total number of voxels, and ∇M be the total number of voxels in S. j,i For M j The gradient magnitude on voxel i within S, min(∙,∙) is the minimum value operation, D(G k ) i For voxel i to G k The distance to the boundary, where δ is the truncation threshold.

[0039] As a preferred option, in S4, the total loss L all Calculate according to the following formula:

[0040] L all =L sem +λ1L mask +λ2L bd ,

[0041] In the formula, λ1 and λ2 are respectively L mask and L bd The weight.

[0042] Compared with the prior art, the advantages of the present invention are as follows:

[0043] (1) By introducing a multi-scale deformable attention mechanism, the model can effectively capture geometric details and contextual relationships at different scales. This approach not only improves the quality of mask prediction but also ensures cross-scale consistency, providing stronger data support for instance understanding in complex scenes.

[0044] (2) Introducing boundary-aware loss as a key step in mask supervision significantly improves instance boundary prediction. This method not only focuses on the integrity of the overall understanding but also provides precise constraints on boundary details, significantly enhancing the understanding effect. Especially in small target detection and occlusion scenarios, this technique significantly improves the practical application value of the understanding results.

[0045] (3) A self-supervised pre-training strategy is adopted as a pre-training step for the model, which enables the feature extraction network to have stronger cross-scene generalization ability. Through this method, the model can still maintain a high level of performance in applications with insufficient data or large distribution differences, which significantly enhances its application potential in large-scale 3D perception.

[0046] In summary, this solution addresses the problems of inefficient attention, blurred boundaries, and insufficient generalization in existing technologies by introducing multi-scale deformable attention, boundary-aware loss, and self-supervised pre-training. This greatly improves the quality and application value of point cloud instance understanding, providing an innovative solution for 3D scene instance segmentation, and has broad application prospects, especially in autonomous driving and smart city perception. Attached Figure Description

[0047] Figure 1 This is a flowchart of the present invention;

[0048] Figure 2 This is a diagram of the network structure for 3D scene segmentation. Detailed Implementation

[0049] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0050] Example 1: See Figure 1 and Figure 2 A 3D scene instance segmentation method incorporating boundary-aware loss includes the following steps:

[0051] S1 transforms the raw point cloud data of the 3D scene into 3D voxel data, and then into a sparse tensor X. (0) ;

[0052] S2, generating a set of real instances based on 3D voxel data. y k G k Given the label and mask of the k-th real instance, construct the initial instance query set Q. (0) , , For the j-th initial instance, query K, where K is the total number of initial instance queries;

[0053] S3 constructs a 3D scene segmentation network, including a 3D backbone network, a decoder, and a third MASK module;

[0054] The 3D backbone network is used to extract X. (0) The multi-scale features are F0, F1, and F2, with the granularity of F0 to F2 becoming progressively coarser.

[0055] The decoder includes L stacked decoding layers, wherein the l-th decoding layer includes a first MASK module, a first DCA module, a second MASK module, and a second DCA module;

[0056] The first MASK module is used to input F0 and the upper decoding layer output instance query set Q. (l-1) Output the first set of predicted instances. p j1 Mj1 They are respectively The masked probability map of the class probability of the j-th predicted instance, based on Hungarian matching. Each instance in Y corresponds one-to-one with the other, and then M is determined based on the threshold τ. j Convert to first binary mask image B j1 ;

[0057] The first DCA module is used for input And F2, in B j1 Perform deformable cross attention within the region to generate intermediate queries for the l-th layer. The intermediate query set Q is obtained. (lm) , ;

[0058] The second MASK module is used to input F0 and Q. (lm) Output the second set of predicted instances. ,Will A second binary mask image B is generated by corresponding one-to-one with the instances within Y. j2 ;

[0059] The second DCA module is used for input. And F1, in B j2 Perform deformable cross attention within the region to generate instance queries for layer l. The instance query set Q of the l-th layer is obtained. (l) , ;

[0060] The third MASK module is used to input F0 and the instance query set Q of the Lth layer. (L) Output the set of predicted instances p j M j They are respectively The masked probability map of the class probability of the j-th predicted instance, based on Hungarian matching. A one-to-one correspondence is generated between each instance in Y to generate the final binary mask B. j ;

[0061] S4, construct the total loss, including the instance classification loss, masking loss, and boundary-aware loss;

[0062] S5, using sparse tensor X (0) The 3D scene segmentation network is trained to converge by minimizing the total loss, resulting in a 3D scene segmentation model.

[0063] S6: Obtain the original point cloud data of the 3D scene to be segmented, process it into a sparse tensor, and then output a set of predicted instances through the 3D scene segmentation model.

[0064] Example 2: See Figures 1-2 Based on Example 1, a three-dimensional scene instance segmentation method combining boundary-aware loss is provided, wherein step S1 includes S11 to S13.

[0065] S11, acquire the raw point cloud data P of the 3D scene, where each point attribute includes 3D coordinates, RGB color, category label and instance label, and the 3D scene contains I instances and C instance categories;

[0066] S12, voxelize P to generate three-dimensional voxel data. M represents the total number of voxels, c j f j They are respectively the j-th voxel (c j ,f j The three-dimensional coordinates and feature vectors of the voxel, wherein the feature vectors include voxel color, height relative to the ground, category label of the voxel, and instance label;

[0067] S13, converting 3D voxel data into sparse tensor X (0) .

[0068] The 3D backbone network adopts Sparse 3D U-Net, and F0 is a full-resolution feature. The channel dimensions of F0, F1, and F2 are 128, 256, and 512, respectively.

[0069] The first DCA module and the second DCA module respectively obtain Q according to the following formula. (lm) and Q (l)

[0070] ,

[0071] ,

[0072] In the formula, DCA(∙) is a deformable cross attention operation, and the inner part of DCA(∙) is a constraint condition.

[0073] Classification loss L sem Mask loss L mask Boundary perception loss L bd Calculate according to the following formulas respectively:

[0074] ,

[0075] In the formula, (j,k) represents the corresponding instance after Hungarian matching by the third MASK module. The index pair with Y, For a set of indexed pairs, N match for The total number of index pairs in the middle, CE(∙,∙) is the cross-entropy loss;

[0076] ,

[0077] In the formula, BCE(∙,∙) is the pointwise binary cross-entropy loss, Dice(∙,∙) is the Dice loss, and λ bce , λ dice These are the weights of the corresponding multiplication terms;

[0078] ,

[0079] In the formula, S is M j Endothelial collection, Let S be the total number of voxels, and ∇M be the total number of voxels in S. j,i For M j The gradient magnitude on voxel i within S, min(∙,∙) is the minimum value operation, D(G k ) i For voxel i to G k The distance to the boundary, where δ is the truncation threshold.

[0080] In S4, the total loss L all Calculate according to the following formula:

[0081] L all =L sem +λ1L mask +λ2L bd ,

[0082] In the formula, λ1 and λ2 are respectively L mask and L bd The weight.

[0083] Example 3: See Figure 1 and Figure 2 Based on Examples 1 and 2, a more specific solution is presented: a 3D scene instance segmentation method combining boundary-aware loss, comprising the following steps:

[0084] S1 transforms the raw point cloud data of the 3D scene into 3D voxel data, and then into a sparse tensor X. (0) Specifically, it includes steps S11 to S12;

[0085] S11, assuming the original point cloud data P of the 3D scene contains N points, and P contains I instances, classifying the I instances into C instance categories, then P can be represented as... ,in Let x represent the i-th point. i ,y i ,z i Let r be the three-dimensional coordinates of the point. i ,g i ,b iThe three-channel RGB color of this point. This is the category label of the instance to which the point belongs, used to characterize which instance category the point belongs to. The category label is the instance label for this point, used to represent which instance the point belongs to. Both the category label and the instance label are semantic labels.

[0086] S12 divides the 3D scene into M voxels, extracts the 3D coordinates and feature vectors of each voxel, and obtains the coordinate set. and eigenvector set , where c j f j These are the three-dimensional coordinates and feature vector of the j-th voxel, respectively. The feature vector includes the voxel color, its height relative to the ground, the voxel's category label, and its instance label. The extraction methods for the three-dimensional coordinates and feature vector are as follows:

[0087] 3D coordinates: After dividing the 3D scene into M voxels, the 3D coordinates of each voxel can be directly quantized based on the 3D coordinates of the point cloud.

[0088] Voxel color: Since multiple points fall on the same voxel after voxel division, the average RGB color of all points within that voxel is taken as the voxel color.

[0089] Height relative to the ground: After dividing into voxels, its height relative to the ground can be obtained directly;

[0090] Voxel category label: determined by majority vote, the category labels of the points within the voxel are counted, and the category label with the most votes is taken as the voxel's category label;

[0091] Voxel instance labels: These are also determined by majority voting. The instances to which the points within the voxel belong are counted, and the instance label with the most instances is taken as the voxel instance label.

[0092] S13, construct the feature matrix from the coordinate set and the eigenvector set. Then it is transformed into a sparse tensor X. (0) .

[0093] S2, based on the feature vector of each voxel obtained from S1, can generate a set of real instances at the voxel level. y k G k Let be the label and mask of the k-th real instance, respectively, and let Q be an initial instance query set. (0) Query containing K learnable initial instances. Randomly initialize parameters.

[0094] S3~S6 are the same as steps S3~S6 in Example 1.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A 3D scene instance segmentation method incorporating boundary-aware loss, characterized in that, Includes the following steps: S1, transforming the original point cloud data of the three-dimensional scene into three-dimensional voxel data, and then into a sparse tensor X (0) ; S2, generating a set of real instances based on 3D voxel data. y k G k Given the label and mask of the k-th real instance, construct the initial instance query set Q. (0) , , For the j-th initial instance, query K, where K is the total number of initial instance queries; S3 constructs a 3D scene segmentation network, including a 3D backbone network, a decoder, and a third MASK module; The 3D backbone network is used to extract X (0) multi-scale features F0, F1, F2, F0~F2 have gradually coarser granularity; The decoder includes L stacked decoding layers, wherein the l-th decoding layer includes a first MASK module, a first DCA module, a second MASK module, and a second DCA module; The first MASK module is used to input F0 and the upper decoding layer output instance query set Q. (l-1) Output the first set of predicted instances. p j1 M j1 They are respectively The masked probability map of the class probability of the j-th predicted instance, based on Hungarian matching. Each instance in Y corresponds one-to-one with the other, and then M is determined based on the threshold τ. j Convert to first binary mask image B j1 ; The first DCA module is used for input And F2, in B j1 Perform deformable cross attention within the region to generate intermediate queries for the l-th layer. The intermediate query set Q is obtained. (lm) , ; The second MASK module is used to input F0 and Q. (lm) Output the second set of predicted instances. ,Will A second binary mask image B is generated by corresponding one-to-one with the instances within Y. j2 ; The second DCA module is used for input. And F1, in B j2 Perform deformable cross attention within the region to generate instance queries for layer l. The instance query set Q of the l-th layer is obtained. (l) , ; The third MASK module is used to input F0 and the instance query set Q of the Lth layer. (L) Output the set of predicted instances p j M j They are respectively The masked probability map of the class probability of the j-th predicted instance, based on Hungarian matching. A one-to-one correspondence is generated between each instance in Y to generate the final binary mask B. j ; S4, construct the total loss, including the instance classification loss, masking loss, and boundary-aware loss; S5, using the sparse tensor X (0) training the three-dimensional scene segmentation network to convergence with the total loss minimized, to obtain a three-dimensional scene segmentation model; S6: Obtain the original point cloud data of the 3D scene to be segmented, process it into a sparse tensor, and then output a set of predicted instances through the 3D scene segmentation model.

2. The 3D scene instance segmentation method combining boundary-aware loss according to claim 1, characterized in that, S1 includes S11~S13; S11, acquire the original point cloud data P of the 3D scene, where each point attribute includes 3D coordinates, RGB color, category label and instance label, and the 3D scene contains I instances and C instance categories; S12, voxelize P to generate three-dimensional voxel data. M represents the total number of voxels, c j f j They are respectively the j-th voxel (c j ,f j The three-dimensional coordinates and feature vectors of the voxel, wherein the feature vectors include voxel color, height relative to the ground, category label of the voxel, and instance label; S13, converting 3D voxel data into sparse tensor X (0) .

3. The 3D scene instance segmentation method combining boundary-aware loss according to claim 1, characterized in that, The 3D backbone network adopts Sparse 3D U-Net, and F0 is a full-resolution feature. The channel dimensions of F0, F1, and F2 are 128, 256, and 512, respectively.

4. The 3D scene instance segmentation method combining boundary-aware loss according to claim 1, characterized in that, The first DCA module and the second DCA module respectively obtain Q according to the following formula. (lm) and Q (l) : ， ， In the formula, DCA(∙) is a deformable cross attention operation, and the inner part of DCA(∙) is a constraint condition.

5. The 3D scene instance segmentation method combining boundary-aware loss according to claim 1, characterized in that, In S4, the classification loss L sem Mask loss L mask Boundary perception loss L bd Calculate according to the following formulas respectively: ， In the formula, (j,k) represents the corresponding instance after Hungarian matching by the third MASK module. The index pair with Y, For a set of indexed pairs, N match for The total number of index pairs in the middle, CE(∙,∙) is the cross-entropy loss; ， In the formula, BCE(∙,∙) is the pointwise binary cross-entropy loss, Dice(∙,∙) is the Dice loss, and λ bce , λ dice These are the weights of the corresponding multiplication terms; ， In the formula, S is M j Endothelial collection, Let S be the total number of voxels, and ∇M be the total number of voxels in S. j,i For M j The gradient magnitude on voxel i within S, min(∙,∙) is the minimum value operation, D(G k ) i For voxel i to G k The distance to the boundary, where δ is the truncation threshold.

6. The 3D scene instance segmentation method combining boundary-aware loss according to claim 5, characterized in that, In S4, the total loss L all Calculate according to the following formula: L all =L sem +λ1L mask +λ2L bd , In the formula, λ1 and λ2 are respectively L mask and L bd The weight.

Images