A Point Cloud Panoramic Segmentation System and Method Based on Instance-Centered Encoding

Abstract

This invention proposes a point cloud panoramic segmentation system based on instance center coding, comprising: a backbone network for extracting point cloud features from point cloud data; a semantic prediction branch for semantic prediction; an instance center proposal network for calculating and generating 3D instance centers and instance center feature codes based on point cloud data and point cloud features; a Transformer module for collecting contextual information around and between instance centers, and outputting enhanced instance center coding features; a dynamic convolutional network for generating instance mask prediction results; and a mask fusion module for fusing semantic prediction results and instance mask prediction results to obtain the final point cloud panoramic segmentation result. This invention proposes a novel detection-free and clustering-free LiDAR panoramic segmentation system that locates and segments instances by center, implying location and instance features, which can be used to represent and distinguish different instances, and exhibits excellent performance in practical panoramic segmentation.

Description

Technical Field

[0001] This invention relates to the technical field of 3D vision, and specifically to a point cloud panoramic segmentation system and method based on instance-centric coding. Background Technology

[0002] LiDAR (Light Detection and Ranging) is a key tool for perception and awareness in autonomous driving and robotics, providing high-precision 3D point cloud data of the environment. Typically, LiDAR segmentation aims to predict point-level segments for a more comprehensive understanding of 3D scenes. LiDAR panoptic segmentation (LPS) is a widespread problem in 3D scene understanding. LPS unifies semantic segmentation and instance segmentation tasks, assigning a semantic category and instance ID to each point in the LiDAR point cloud. It requires parsing objects (e.g., roads, buildings, and vegetation) and identifying objects (e.g., cars, cyclists, and pedestrians). LiDAR point clouds can be sparse, noisy, and occluded, making accurate segmentation and differentiation of different instances difficult. Despite these challenges, recent advances in deep learning have made significant progress in LPS. A key issue for achieving reliable LiDAR panoptic segmentation is the accurate localization and segmentation of instances.

[0003] Regarding instance segmentation, most existing methods take two approaches to address these challenges: detection-based and clustering-based methods. Detection-based methods employ a top-down strategy, relying on object detection as a separate branch to predict region proposals and then segmenting instances based on these proposals. However, these methods heavily depend on detector performance, and the region proposal generation process for object localization involves significant computational overhead. On the other hand, clustering-based methods implicitly localize instances using geometric offsets predicted by the offset branch and employ heuristic clustering algorithms to group instances in a bottom-up manner. However, the heuristic instance grouping process is typically time-consuming, and these methods often introduce many hand-tuned hyperparameters that are sensitive, limiting robustness and potentially leading to oversegmentation problems in practice. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a point cloud panoramic segmentation system and method based on instance center coding, and proposes a novel detection-free and cluster-free LiDAR panoramic segmentation framework (CenterLPS) that uses the center to locate and segment instances, thereby solving the above problems.

[0005] To achieve the objectives of this invention, the invention is implemented through the following technical solutions:

[0006] A point cloud panoramic segmentation system based on instance-centric coding includes:

[0007] The backbone network is used to extract point cloud features from point cloud data;

[0008] The semantic prediction branch is used to perform semantic prediction on point cloud features to obtain semantic prediction results.

[0009] The instance center proposal network calculates and generates 3D instance centers and instance center feature codes based on point cloud data and point cloud features.

[0010] The Transformer module collects contextual information around and between instance centers based on 3D instance centers and instance center feature encoding, and outputs enhanced instance center encoding features; preferably, the Transformer module is a center-aware Transformer module.

[0011] A dynamic convolutional network generates instance mask prediction results based on enhanced instance center encoding features;

[0012] The mask fusion module is used to fuse semantic prediction results and instance mask prediction results to obtain the final point cloud panoramic segmentation result.

[0013] As a preferred option, the proposed instance center network includes:

[0014] An offset head is used to predict the positional offset between the foreground point and the center of the instance to which the foreground point belongs; preferably, the offset head consists of two fully connected layers.

[0015] A 2D pseudo-heatmap is generated by dividing the foreground points into cylinders and projecting them onto the BEV plane. It is used to generate 3D instance centers and instance center feature codes.

[0016] Preferably, the Transformer module includes an encoding block, and more preferably, the Transformer module includes two cascaded encoding blocks, which include a cascaded multi-head self-attention module, a multi-head cross-attention module, and a feedforward neural network.

[0017] The multi-head self-attention module performs global modeling on the input 3D instance centers and instance center feature encodings, establishes the correlation between instance center points, and outputs instance features;

[0018] The multi-head cross-attention module takes the instance features output by the multi-head self-attention mechanism as input and uses a local mechanism to enable each instance center point to interact with the features of the points around the instance center point, thus modeling the local relationships between points within the instance.

[0019] The feedforward neural network fuses the outputs of multiple heads of the multi-head cross-attention module to output enhanced instance center encoding features;

[0020] Preferred dynamic convolutional networks include:

[0021] Dynamic kernel branching generates dynamic parameters for each instance based on the enhanced instance-centric encoding features;

[0022] A mask branch is used to extract mask features from point cloud features; preferably, the mask branch consists of two linear layers.

[0023] The shallow convolutional network is initialized based on dynamic parameters generated by dynamic kernel branches, and is used to convolve the mask features extracted by the mask branch to obtain the instance mask prediction result; preferably, the shallow convolutional network consists of two 1×1 convolutional layers.

[0024] A point cloud panoramic segmentation method based on instance-centric encoding, characterized by the following steps:

[0025] Step S1: Use the backbone network to extract point cloud features from the point cloud data, and input the point cloud features into the semantic prediction branch to perform semantic prediction and obtain the semantic prediction result;

[0026] Step S2: Input the point cloud data and point cloud features together into the instance center proposal network for calculation to obtain 3D instance centers and instance center feature codes;

[0027] Step S3: Input the 3D instance center and instance center feature encoding into the Transformer module, collect contextual information around and between the instance center point, and obtain enhanced instance center encoding features; preferably, the Transformer module is a center-aware Transformer module.

[0028] Step S4: Based on the enhanced instance center encoding features, use a dynamic convolutional network to generate instance mask prediction results;

[0029] Step S5: Use the mask fusion module to fuse the semantic prediction results and the instance mask prediction results to obtain the final point cloud panoramic segmentation results.

[0030] Preferably, step S1 specifically includes the following steps:

[0031] Step S11: Input the point cloud data P into the backbone network, first perform voxelization to obtain voxel features, and then input it into a multi-stage 3D sparse convolutional network to extract multi-scale 3D features.

[0032] Step S12: Backproject the voxel features and multi-scale 3D features to obtain point cloud features containing multi-scale and rich contextual information; the backprojection is performed on a point p in the point cloud data to be segmented. i According to its voxel index V i Voxel features can be indexed to obtain the feature f corresponding to that point. p .

[0033] Step S13: Concatenate the point cloud features containing multi-scale and rich contextual information along the channel dimension, and fuse these features using a fully connected network to obtain the final encoded point cloud feature F. p ;

[0034] Step S14: Encode the final point cloud features F p The input is fed into a semantic prediction branch consisting of two fully connected layers to perform semantic prediction and obtain the semantic prediction result.

[0035] Preferably, step S2 specifically includes the following steps:

[0036] Step S21: Input the point cloud data and point cloud features into the instance center proposal network. The point cloud features and point cloud data are first input into an offset head consisting of two fully connected layers. The offset head predicts the positional offset O between the foreground point and its corresponding instance center. Preferably, the following loss is used to optimize the prediction during training:

[0037]

[0038] Where M=(m1,…,m N ) is a binary mask indicating whether a point is a foreground point. If point p i For the foreground, then m i It is 1 otherwise it is 0; For point p i Belongs to the instance center;

[0039] Step S22: Shift the foreground point toward the instance center according to the position offset O, so that the points belonging to the same instance are closer together, and divide the shifted foreground point into cylinders and project them onto the BEV plane to generate a 2D pseudo-heatmap.

[0040] Step S23: Use window-based max pooling to select local maxima in the 2D pseudo-heatmap and find the possible cylinders at the center of the 3D instance;

[0041] Step S24: For the found cylinder, perform average pooling on the coordinates and features of the points belonging to the cylinder to obtain the corresponding 3D instance center point and instance center feature encoding;

[0042] Step S25: Perform maximum voting on the semantic prediction of the points in the cylinder to determine the semantic category corresponding to the 3D instance center and instance center feature encoding obtained in step S24.

[0043] Preferably, the Transformer module in step S3 includes an encoding block. More preferably, the Transformer module includes two cascaded encoding blocks, which include a cascaded multi-head self-attention module, a multi-head cross-attention module, and a feedforward neural network.

[0044] The Multi-Head Self-Attention Module (MHSA) globally models the input 3D instance centers and instance center feature encodings, establishes the relationships between instances, and outputs instance features, which can be represented as:

[0045]

[0046]

[0047]

[0048] Among them, MHSA(c i ) is c i The corresponding instance features are processed by multi-head self-attention, where h is the index of the multi-head attention and H is the number of attention heads. For linear layers, N c Let σ be the number of instance center proposals, σ be the softmax function, and q be the number of instance center proposals. i k represents the query features generated by the current instance center. j and v j For key-value feature pairs generated by other instance centers, D is the feature dimension; c i and c j Let i and j be the center points of the instances. and Encode the center features of the i-th and j-th instances, where δ is the position code;

[0049] The Multi-Head Cross-Attention (MHCA) module takes the instance features output by the multi-head self-attention mechanism as input and uses a local mechanism to enable each instance centroid to interact with the features of points surrounding it, modeling the local relationships between points within an instance. Specifically:

[0050]

[0051] Among them, MHCA(c i ) is c i The corresponding instance features are processed by multi-head cross-attention, where h is the index of the multi-head attention and H is the number of attention heads. For a linear layer, N(c) i ) represents the instance center point ci The set of neighborhood points, σ ​​is the softmax function, q i k is the query feature generated by the instance center feature encoding. j and v j For each key-value pair generated from the point cloud features surrounding the instance center, D represents the feature dimension.

[0052] The feedforward neural network fuses the outputs of multiple heads of the multi-head cross-attention module to output enhanced instance center encoding features;

[0053] Preferably, step S4 specifically includes the following steps:

[0054] Step S41: Input the enhanced instance center encoding features into the dynamic convolutional network. First, use the dynamic kernel branch composed of multilayer perceptrons to generate dynamic parameters w for each instance, which are used to initialize the shallow convolutional network N composed of two 1×1 convolutional layers. F ;

[0055] Step S42: Use the position offset O from step S21 as the position prior, and generate a coarse binary mask based on the 3D instance center generated in step S24 as the shape prior. Concatenate the position prior, shape prior, and point cloud features, and use a mask branch consisting of two linear layers to extract the final mask features F. m ;

[0056] Step S43: Use the shallow convolutional network N from step S41 F For the mask feature F in step S42 m Perform convolution to obtain the final instance mask prediction result.

[0057] Preferably, step S5 specifically includes the following steps:

[0058] Step S51: Calculate the IoU value between each pair of instance mask prediction results in step S43, and construct a connected graph with the mask as the node and the IoU value as the edge weight. If the edge weight is greater than the set value, it indicates that the edge is connected. Preferably, the set value is set to 0.85 by default. An edge weight above the set value indicates that the two masks overlap to a high degree.

[0059] Step S52: Based on the connected graph generated in step S51, use the connected graph labeling algorithm to obtain connected components. Each connected component represents a set of point cloud instance masks with an IoU value greater than a set value. Then, perform foreground merging on the masks in each mask set to obtain the merged instance mask prediction result.

[0060] Step S53: Fuse the merged instance mask prediction result with the semantic prediction result generated in step S14 to obtain the fused panoramic segmentation result.

[0061] Compared with the prior art, the beneficial effects of the present invention are:

[0062] This invention proposes a novel detection-free and clustering-free LiDAR panoramic segmentation framework (CenterLPS). This framework locates and segments instances using centers, addressing the inefficiencies and lack of robustness caused by current mainstream methods that rely on detection and clustering. Specifically, it constructs an instance center proposal network to predict 3D instance centers for instance location and encodes instances as instance center feature codes. The 3D instance centers and instance center feature codes generated by the instance center proposal network contain location and instance features, which can be used to represent and distinguish different instances. Rich contextual information is beneficial for instance encoding; therefore, this invention also designs a Transformer module to collect contextual information between different instance center feature embeddings and around the instance centers. Furthermore, the dynamic convolutional network designed in this invention directly predicts instance masks based on instance center feature codes of known categories, naturally avoiding semantic conflicts caused by category-independent clustering within the same cluster and correcting potential semantic prediction errors.

[0063] The instance center proposal network developed in this invention predicts the positional offset between the foreground point and the instance center to which the foreground point belongs based on the offset head. The foreground point is offset according to the positional offset. After the offset foreground point is divided into cylinders, it is projected onto the BEV image to generate a 2D pseudo-heatmap. Based on the 2D pseudo-heatmap, 3D instance centers and instance center feature codes are generated. The semantic categories of these 3D centers can be directly determined by majority voting on points in the selected cylinders, so as to explicitly decouple classification and segmentation and reduce the competition between them.

[0064] The Transformer module in this invention enhances instance center feature encoding by modeling dependencies between instances using global self-attention and local cross-attention, providing more information to facilitate subsequent instance mask prediction.

[0065] This invention creatively proposes the use of dynamic convolutional networks for LiDAR panoramic segmentation. Based on enhanced instance center encoding features, dynamic kernel branches are set to generate dynamic parameters for each instance, and a shallow convolutional network is initialized. The instance mask prediction result is then generated through this shallow convolutional network. The dynamic convolutional network, based on enhanced instance center encoding features, includes target-related information such as the instance's location, shape, and size. The instance mask prediction result obtained through the dynamic convolutional network can avoid semantic conflicts caused by category-independent clustering within the same cluster and correct potential semantic errors.

[0066] The point cloud panoramic segmentation method proposed in this invention uses instance-centric encoding to represent instances and dynamic convolution to decode instances. Compared with mainstream detection-based and clustering-based methods, the point cloud panoramic segmentation algorithm designed in this invention has higher accuracy, faster speed, and good robustness to complex situations such as large targets and crowded targets in autonomous driving 3D scenarios.

[0067] To accelerate network convergence, this invention further enhances dynamic convolution using position and shape priors.

[0068] This invention observes that due to inaccurate offset prediction, multiple centers may be generated for a single instance, especially for large objects. Therefore, this invention also designs a mask fusion module to merge the masks generated by instance centers belonging to the same instance and paste the merged mask onto the semantic prediction to unify the semantic prediction and instance mask. The proposed mask fusion module can merge potentially overlapping masks under the same instance and improve the panoramic quality. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the overall algorithm framework of a point cloud panoramic segmentation method based on instance center coding according to the present invention.

[0070] Figure 2 This is a schematic diagram of the proposed network for an example of the present invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] This invention proposes a point cloud panoramic segmentation system based on instance-centric encoding, such as... Figure 1 As shown, a novel detection-free and cluster-free framework called CenterLPS is proposed. It adopts a center-based instance encoding and decoding paradigm for LiDAR panoptic segmentation, which solves the problems of low efficiency and weak robustness caused by the reliance on detection and clustering methods in current mainstream methods.

[0073] This invention proposes a point cloud panoramic segmentation system based on instance-centric coding, comprising:

[0074] The backbone network is used to extract point cloud features from point cloud data;

[0075] The semantic prediction branch is used to perform semantic prediction on point cloud features to obtain semantic prediction results.

[0076] An instance center proposal network (IFN) calculates and generates 3D instance centers and instance center feature codes based on point cloud data and point cloud features. This invention constructs an IFN to predict 3D instance centers for instance location and encodes instances as instance center feature codes. The 3D instance centers and instance center feature codes generated by the IFN contain location and instance features, which can be used to represent and distinguish different instances. Figure 2 As shown, the instance center proposal network includes: an offset head, consisting of two fully connected layers, used to predict the positional offset between the foreground point and the instance center to which the foreground point belongs; and a 2D pseudo-heatmap, generated by projecting the offset foreground point into cylinders onto the BEV plane, used to generate 3D instance centers and instance center feature codes. The instance center proposal network developed in this invention predicts the positional offset between the foreground point and the instance center to which the foreground point belongs based on the offset head. The foreground point is offset according to the positional offset. This invention divides the offset foreground point into cylinders and projects them onto the BEV image to generate a 2D pseudo-heatmap. Based on the 2D pseudo-heatmap, 3D instance centers and instance center feature codes are generated. The semantic categories of these 3D centers are directly determined by majority voting at points in selected cylinders, thus explicitly decoupling classification and segmentation and reducing competition between them.

[0077] The Transformer module collects contextual information about the area around and between instance centers based on 3D instance centers and instance center feature encodings, and outputs enhanced instance center encoded features. In this embodiment, the Transformer module is a center-aware Transformer module; rich contextual information is beneficial for instance encoding. Therefore, this invention designs a Transformer module to collect contextual information between different instance center feature embeddings and around instance centers. In this embodiment, the Transformer module includes two cascaded encoding blocks, each consisting of a cascaded multi-head self-attention module, a multi-head cross-attention module, and a feedforward neural network. The multi-head self-attention module globally models the input 3D instance centers and instance center feature encodings, establishing relationships between instance centers and outputting instance features. The multi-head cross-attention module uses the instance features output by the multi-head self-attention mechanism as input, employing a local mechanism to enable each instance center to interact with the features of points surrounding it, modeling local relationships between points within the instance. The feedforward neural network fuses the outputs of multiple heads of the multi-head cross-attention module, outputting enhanced instance center encoded features. The Transformer module in this invention enhances instance center feature encoding by modeling dependencies between instances using global self-attention and local cross-attention, providing more information to facilitate subsequent instance mask prediction.

[0078] A dynamic convolutional network (RCN) generates instance mask prediction results based on enhanced instance center encoding features. This invention's RCN directly predicts instance masks based on instance center feature encoding of known categories, naturally avoiding semantic conflicts caused by category-independent clustering within the same cluster and correcting potential semantic errors. The RCN comprises: a dynamic kernel branch, generating dynamic parameters for each instance based on enhanced instance center encoding features; a mask branch, consisting of two linear layers, used to extract mask features from point cloud features; and a shallow convolutional network, consisting of two 1×1 convolutional layers, which convolve the mask features extracted by the mask branch to obtain the instance mask prediction results. This invention creatively proposes the use of a dynamic convolutional network for LiDAR panoramic segmentation. It sets a dynamic kernel branch based on enhanced instance center encoding features to generate dynamic parameters for each instance, initializes the shallow convolutional network, and then generates instance mask prediction results through the shallow convolutional network. Dynamic convolutional networks are based on enhanced instance center encoding features, which include target-related information such as the location, shape, and size of instances. The instance mask prediction results obtained by dynamic convolutional networks can avoid semantic conflicts caused by class-independent clustering in the same cluster and correct potential erroneous semantic predictions.

[0079] The mask fusion module is used to fuse semantic prediction results and instance mask prediction results to obtain the final point cloud panoramic segmentation result.

[0080] This invention proposes a point cloud panoramic segmentation method based on instance-centric encoding. Instances are represented by instance-centric encoding and decoded by dynamic convolution. Compared with mainstream detection-based and clustering-based methods, the point cloud panoramic segmentation algorithm designed in this invention has higher accuracy, faster speed, and good robustness to complex situations such as large targets and crowded targets in autonomous driving 3D scenarios.

[0081] This invention proposes a point cloud panoramic segmentation method based on instance-centric encoding, comprising the following steps:

[0082] Step S1: Use the backbone network to extract point cloud features from the point cloud data, and input the point cloud features into the semantic prediction branch to perform semantic prediction and obtain the semantic prediction result;

[0083] Step S1 specifically includes the following steps:

[0084] Step S11: Input the point cloud data P into the backbone network, perform voxelization first to obtain voxel features. Then, the input is fed into a multi-stage 3D sparse convolutional network to extract multi-scale 3D features.

[0085] The voxelization process is as follows:

[0086] Use p i =(x i ,y i ,z i () represents a point in the input point cloud data to be segmented, and its voxel index. Where s is the voxelization resolution. This is a floor operation;

[0087]

[0088] Indicates that the voxel index is V m Voxel characteristics of the m-th non-empty voxel; R f MLP stands for Multilayer Perceptron; A represents a fully connected layer. f It is an aggregate function; f p V represents point features. p This represents the voxel index where point p is located.

[0089] Step S12: Voxel features and multi-scale 3D features Back projection yields point cloud features containing multi-scale and rich contextual information. Back projection is for a point p in the point cloud data to be segmented. i According to its voxel index Voxel characteristics F v Indexing is performed to obtain the feature f corresponding to that point. p .

[0090] Step S13: Extract point cloud features containing multi-scale and rich contextual information. The features are stitched together along the channel dimension and then fused using a fully connected network to obtain the final encoded point cloud features F. p ;

[0091] Step S14: Encode the final point cloud features F p The input is fed into a semantic prediction branch consisting of two fully connected layers to perform semantic prediction, resulting in a semantic prediction result S. Cross-entropy loss and Lovasz loss are used for supervision during the training phase.

[0092] Step S2: Combine point cloud data P and point cloud features F p The inputs are fed into the instance center proposal network for computation to obtain 3D instance centers and instance center feature codes.

[0093] Step S2 specifically includes the following steps:

[0094] Step S21: Input the point cloud data and point cloud features into the instance center proposal network. The point cloud features and point cloud data are first input into an offset head consisting of two fully connected layers. The offset head predicts the positional offset O between the foreground point and its corresponding instance center. The following loss is used to optimize the prediction during training:

[0095]

[0096] Where M=(m1,…,m N ) is a binary mask indicating whether a point is a foreground point. If point p i For the foreground, then m i It is 1 otherwise it is 0; For point p i Belongs to the instance center;

[0097] Step S22: Shift the foreground point towards the instance center according to the position offset O, so that the points belonging to the same instance are closer together, and divide the shifted foreground point into cylinders and project them onto the BEV plane to generate a 2D pseudo-heatmap; the value corresponding to each grid in the 2D pseudo-heatmap is the number of foreground points in the cylinder corresponding to that grid.

[0098] Step S23: Use window-based max pooling to select local maxima in the 2D pseudo-heatmap and find the possible cylinders at the center of the 3D instance;

[0099] Step S24: For the found cylinder, perform average pooling on the coordinates and features of the points belonging to the cylinder to obtain the corresponding 3D instance center point and instance center feature encoding, that is:

[0100]

[0101] in and Let AvgPool be the set of coordinates and feature set of the points belonging to the k-th column; MLP is the multilayer perceptron.

[0102] Step S25: Perform maximum voting on the semantic prediction of the points in the column to determine the semantic category corresponding to the 3D instance center and instance center feature encoding obtained in step S24;

[0103] Step S3: Input the 3D instance center and instance center feature encoding into the Transformer module to collect contextual information around and between the instance center points to obtain enhanced instance center encoding features;

[0104] In step S3, the Transformer module includes an encoding block. In this embodiment, the Transformer module includes two cascaded encoding blocks, which include a cascaded multi-head self-attention module, a multi-head cross-attention module, and a feedforward neural network.

[0105] The Multi-Head Self-Attention Module (MHSA) globally models the input 3D instance centers and instance center feature encodings, establishes the relationships between instances, and outputs instance features, which can be represented as:

[0106]

[0107]

[0108]

[0109] Among them, MHSA(c i ) is c i The corresponding instance features are processed by multi-head self-attention, where h is the index of the multi-head attention and H is the number of attention heads. For linear layers, N c Let σ be the number of instance center proposals, σ be the softmax function, and q be the number of instance center proposals. i k represents the query features generated by the current instance center. j and v j For key-value feature pairs generated by other instance centers, D is the feature dimension; c i and c j Let i and j be the center points of the instances. and Encode the center features of the i-th and j-th instances, where δ is the position code;

[0110] The Multi-Head Cross-Attention (MHCA) module takes the instance features output by the multi-head self-attention mechanism as input and uses a local mechanism to enable each instance centroid to interact with the features of points surrounding it, modeling the local relationships between points within an instance. Specifically:

[0111]

[0112] Among them, MHCA(c i ) is c i The corresponding instance features are processed by multi-head cross-attention, where h is the index of the multi-head attention and H is the number of attention heads. For a linear layer, N(c) i ) represents the instance center point c iThe set of neighborhood points, σ ​​is the softmax function, q i k is the query feature generated by the instance center feature encoding. j and v j For each key-value pair generated from the point cloud features surrounding the instance center, D represents the feature dimension.

[0113] The feedforward neural network fuses the outputs of multiple heads of the multi-head cross-attention module to output enhanced instance center encoding features;

[0114] Step S4: Based on the enhanced instance center encoding features, use a dynamic convolutional network to generate instance mask prediction results;

[0115] Step S4 specifically includes the following steps:

[0116] Step S41: Input the enhanced instance center encoding features into the dynamic convolutional network. First, use the dynamic kernel branch composed of multilayer perceptrons to generate dynamic parameters w for each instance, which are used to initialize the shallow convolutional network N composed of two 1×1 convolutional layers. F ;

[0117] Step S42: Use the position offset O from step S21 as the position prior, and generate a coarse binary mask based on the 3D instance center generated in step S24 as the shape prior. Combine the position prior, shape prior, and point cloud features F p The layers are concatenated, and a mask branch consisting of two linear layers is used to extract the final mask features F. m The shape prior generation process is as follows: using the instance center as the sphere's center and the prior length of each target category as the radius, if the target point is inside the sphere, the mask value is 1; otherwise, it is 0. To accelerate network convergence, this invention uses position and shape priors to enhance dynamic convolution.

[0118] Step S43: Use the shallow convolutional network N from step S41 F For the mask feature F in step S42 m Convolution is performed to obtain the final instance mask prediction result. During training, binary cross-entropy loss and Dice loss are used to optimize the prediction result.

[0119] Step S5: Use a mask fusion module to fuse the semantic prediction results and the instance mask prediction results to obtain the final point cloud panoramic segmentation result. This invention observes that due to inaccurate offset prediction, multiple centers may be generated for a single instance, especially a large object. Therefore, this invention also designs a mask fusion module to merge the masks generated by instance centers belonging to the same instance and paste the merged mask onto the semantic prediction to unify the semantic prediction and instance mask. The proposed mask fusion module can merge potentially overlapping masks under the same instance and improve panoramic quality. Step S5 includes the following steps:

[0120] Step S51: Calculate the IoU value between each pair of instance mask prediction results in step S43, and construct a connected graph with the mask as the node and the IoU value as the edge weight. If the edge weight is greater than the set value, it indicates that the edge is connected. In this embodiment, the set value is set to 0.85 by default. An edge weight above the set value indicates that the two masks overlap to a high degree.

[0121] Step S52: Based on the connected graph generated in step S51, use the connected graph labeling algorithm to obtain connected components. Each connected component represents a set of point cloud instance masks with an IoU value greater than a set value. Then, perform foreground merging on the masks in each mask set to obtain the merged instance mask prediction result.

[0122] Step S53: Fuse the merged instance mask prediction result with the semantic prediction result generated in step S14 to obtain the fused panoramic segmentation result.

[0123] The point cloud panoramic segmentation system and method based on instance-centric encoding provided by this invention achieves state-of-the-art performance on large-scale datasets SemanticKITTI and nuScenes.

[0124] As shown in Tables 1 and 2, the methods compared in the tables include CenterLPS, a point cloud panoramic segmentation method based on instance center encoding provided in this invention. All methods are trained on SemanticKITTI and nuScenes training sets, and then tested on validation or test sets. This refers to the use of model aggregation and data augmentation techniques during testing; PQ is the panorama segmentation metric, with higher values ​​indicating better panorama quality. This indicates that the IoU of the background category will be replaced with PQ. St The average results obtained are as follows: RQ represents recognition accuracy, SQ represents segmentation precision, and PQ represents the average accuracy. St RQ St SQ St Indicators representing background categories; PQ Th RQ Th SQ ThThe metrics represent the foreground category. It can be seen that, after training under the same conditions, the test results show that the CenterLPS of this invention exhibits excellent performance.

[0125] Table 1. Results of CenterLPS on the SemanticKITTI test set.

[0126]

[0127] Table 2 shows the results of CenterLPS on the nuScenes validation set.

[0128]

[0129] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other embodiments.

Claims

1. A point cloud panoramic segmentation method based on instance-centric coding, characterized in that, Includes the following steps: Step S1: Use the backbone network to extract point cloud features from the point cloud data, and input the point cloud features into the semantic prediction branch to perform semantic prediction and obtain the semantic prediction result; Step S2: Input the point cloud data and the point cloud features together into the instance center proposal network for calculation to obtain 3D instance centers and instance center feature codes; Step S3: Input the 3D instance center and instance center feature encoding into the Transformer module, collect the context information around the instance center point and between the instance center point, and obtain the enhanced instance center encoding features; Step S4: Based on the enhanced instance center encoding features, use a dynamic convolutional network to generate instance mask prediction results; Step S5: Use the mask fusion module to fuse the semantic prediction result and the instance mask prediction result to obtain the final point cloud panoramic segmentation result; Step S2 includes the following steps: Step S21: Input the point cloud data and the point cloud features into the instance center proposal network. The point cloud features and point cloud data are first input into an offset head consisting of two fully connected layers. The offset head predicts the position offset between the foreground point and its corresponding instance center. ; Step S22: Based on the position offset The foreground points are shifted toward the instance center to make the points belonging to the same instance closer together, and the shifted foreground points are divided into cylinders and projected onto the BEV plane to generate a 2D pseudo-heatmap. Step S23: Use window-based max pooling to select local maxima in the 2D pseudo-heatmap and find the possible cylinders at the center of the 3D instance; Step S24: For the found cylinder, perform average pooling on the coordinates and features of the points belonging to the cylinder to obtain the corresponding 3D instance center and instance center feature encoding; Step S25: Perform maximum voting on the semantic prediction of the points in the column to determine the semantic category corresponding to the 3D instance center and instance center feature encoding obtained in step S24; Step S4 includes the following steps: Step S41: Input the enhanced instance center encoding features into the dynamic convolutional network. First, use the dynamic kernel branch composed of multilayer perceptrons to generate dynamic parameters for each instance. Used to initialize two layers Shallow convolutional networks composed of convolutional layers ; Step S42: Adjust the position offset mentioned in step S21 As a position prior, and based on the 3D instance center generated in step S24, a coarse binary mask is generated as a shape prior. The position prior, the shape prior, and the point cloud features are concatenated, and a mask branch consisting of two linear layers is used to extract the final mask features. ; Step S43: Use the shallow convolutional network described in step S41 Regarding the mask features in step S42 Perform convolution to obtain the final instance mask prediction result.

2. A point cloud panoramic segmentation system based on instance-centric coding, applied to the method described in claim 1, characterized in that, include: The backbone network is used to extract point cloud features from point cloud data; The semantic prediction branch is used to perform semantic prediction on the point cloud features to obtain the semantic prediction result; The instance center proposal network calculates and generates 3D instance centers and instance center feature codes based on point cloud data and the point cloud features. The Transformer module collects contextual information around and between instance centers based on 3D instance centers and instance center feature encoding, and outputs enhanced instance center encoded features. A dynamic convolutional network generates instance mask prediction results based on enhanced instance center encoding features; The mask fusion module is used to fuse the semantic prediction results and the instance mask prediction results to obtain the final point cloud panoramic segmentation results.

3. A point cloud panoramic segmentation system based on instance-centric coding according to claim 2, characterized in that, The instance center proposal network includes: Offset head, used to predict the positional offset between the foreground point and the center of the instance to which the foreground point belongs; A 2D pseudo-heatmap is generated by dividing the foreground points into cylinders and projecting them onto the BEV plane. It is used to generate 3D instance centers and instance center feature codes.

4. A point cloud panoramic segmentation system based on instance-centric coding according to claim 2, characterized in that, The Transformer module includes an encoding block, which includes a cascaded multi-head self-attention module, a multi-head cross-attention module, and a feedforward neural network. The multi-head self-attention module performs global modeling on the input 3D instance centers and instance center feature encodings, establishes the correlation between instance center points, and outputs instance features; The multi-head cross-attention module takes the instance features output by the multi-head self-attention mechanism as input and uses a local mechanism to enable each instance center point to interact with the features of the points around the instance center point, thereby modeling the local relationships between points within the instance. The feedforward neural network fuses the outputs of multiple heads of the multi-head cross-attention module to output enhanced instance center encoding features.

5. A point cloud panoramic segmentation system based on instance-centric coding according to claim 2, characterized in that, The dynamic convolutional network includes: Dynamic kernel branching generates dynamic parameters for each instance based on the enhanced instance-centric encoding features; Mask branch, used to extract mask features of point cloud features; A shallow convolutional network is initialized based on the dynamic parameters generated by the dynamic kernel branch, and is used to convolve the mask features extracted by the mask branch to obtain the instance mask prediction result.

6. The point cloud panoramic segmentation method based on instance-centric coding according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Input the point cloud data P into the backbone network, first perform voxelization to obtain voxel features, and then input it into a multi-stage 3D sparse convolutional network to extract multi-scale 3D features. Step S12: Backproject the voxel features and the multi-scale 3D features to obtain point cloud features containing multi-scale and rich contextual information; the backprojection is performed on a point in the point cloud data to be segmented. According to its voxel index Voxel features can be indexed to obtain the features corresponding to that point. ; Step S13: The point cloud features containing multi-scale and rich contextual information are concatenated along the channel dimension, and these features are fused using a fully connected network to obtain the final encoded point cloud features. ; Step S14: Convert the final encoded point cloud features The input is fed into a semantic prediction branch consisting of two fully connected layers to perform semantic prediction and obtain the semantic prediction result.

7. The point cloud panoramic segmentation method based on instance-centric coding according to claim 1, characterized in that, The Transformer module mentioned in step S3 includes an encoding block, which includes a cascaded multi-head self-attention module, a multi-head cross-attention module, and a feedforward neural network; The multi-head self-attention module performs global modeling on the input 3D instance centers and instance center feature encodings, establishes the associations between instances, and outputs instance features, represented as: ； ； ； in, for The corresponding instance features are the output of the multi-head self-attention module; h is the index of the multi-head attention. For the number of attention heads, For linear layers, The number of instance center proposals. For the softmax function, For query features generated by the current instance center, and For key-value feature pairs generated by other instance centers, For feature dimensions; and Let i and j be the center points of the instances. and Encode the center features of the i-th and j-th instances. For position encoding; The multi-head cross-attention module takes the instance features output by the multi-head self-attention mechanism as input, and uses a local mechanism to enable each instance center point to interact with the features of the points surrounding the instance center point, modeling the local relationships between points within an instance, as follows: ； in, for The corresponding instance features are the output of multi-head cross-attention; h is the index of the multi-head attention. For the number of attention heads, For linear layers, Indicates the center point of the instance The set of domain points, For the softmax function, For query features generated by instance-centric feature encoding, and These are key-value feature pairs generated from the point cloud features surrounding the instance center. For feature dimensions; The feedforward neural network fuses the outputs of multiple heads of the multi-head cross-attention module to output enhanced instance center encoding features.

8. A point cloud panoramic segmentation method based on instance-centric coding according to claim 6, characterized in that, Step S5 includes the following steps: Step S51: Calculate the IoU value between each pair of instance mask prediction results in step S43, and construct a connected graph with the mask as the node and the IoU value as the edge weight. If the edge weight is greater than the set value, it indicates that the edge is connected. Step S52: Based on the connected graph generated in step S51, use the connected graph labeling algorithm to obtain connected components. Each connected component represents a set of point cloud instance masks with an IoU value greater than a set value. Then, perform foreground merging on the masks in each mask set to obtain the merged instance mask prediction result. Step S53: Fuse the merged instance mask prediction result with the semantic prediction result generated in step S14 to obtain the fused panoramic segmentation result.

Images