A boundary-guided 3D point cloud segmentation unified transformer

Abstract

The application discloses a boundary-guided 3D point cloud segmentation unified Transformer. Firstly, the boundary-guided query initialization module is used to perform weighted aggregation and weighted sampling on semantic and instance features under the guidance of an explicit boundary pseudo label to generate an initial query containing boundary information and a corresponding geometric center; subsequently, the region decoupled attention module is used to accurately focus on a target region and suppress background interference by parallelly setting a foreground enhancement head, a background suppression head and a global interaction head; finally, the alternating update decoder is used to dynamically calibrate features and positions by alternately updating queries and corresponding geometric centers. The application aims to solve the problems that the existing unified segmentation architecture is difficult to balance content and position during processing of a complex indoor scene, foreground features are easily diluted by background noise and target features are misaligned with spatial positions. The 3D point cloud segmentation performance of the boundary-guided 3D point cloud segmentation unified Transformer is excellent.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud segmentation, and more specifically to a boundary-guided unified Transformer for 3D point cloud segmentation. Background Technology

[0002] 3D point cloud scene segmentation is the cornerstone of intelligent systems such as autonomous driving, robot navigation, and augmented reality. It primarily involves three core tasks: semantic segmentation, instance segmentation, and panoptic segmentation. In recent years, scholars have attempted to use a unified framework to simultaneously achieve semantic segmentation, instance segmentation, and panoptic segmentation by handling the semantic continuity of the "stuff" and the instance independence of the "things." However, due to the unordered and sparsity of point cloud data, as well as the different feature granularity requirements of various segmentation tasks, constructing a unified framework capable of simultaneously achieving semantic segmentation, instance segmentation, and panoptic segmentation is extremely challenging.

[0003] In recent years, with the development of Transformer technology, query-based 3D point cloud segmentation methods have gradually become mainstream. In particular, works such as QueryFormer and SPFormer have abandoned the reliance on heuristic clustering algorithms of previous methods, redefining 3D instance segmentation as a mask classification problem. OneFormer3D further introduces this idea into a unified 3D segmentation framework. However, although existing query-based methods greatly simplify the model pipeline, their performance is still limited by the following factors when dealing with complex 3D scenes.

[0004] Firstly, in query-based 3D point cloud segmentation methods, the query typically serves as the starting point for the interaction between the model and the input scene, and its initialization method directly constrains the model's convergence speed and segmentation quality. Currently, query initialization methods can be broadly categorized into parametric and non-parametric approaches. Non-parametric methods, represented by Mask3D, employ the Farthest Point Sampling (FPS) strategy, directly sampling queries from the raw point cloud data. While this approach can reduce the null sampling rate by limiting candidate queries to the region where the object resides, it cannot effectively learn cross-scene content embeddings, thus affecting the feature aggregation effect. Parametric methods, represented by SPFormer, initialize queries using a fixed set of learnable vectors. Although this method can learn common features from multiple scenes, its null sampling rate is relatively high, leading to a significant decrease in the model's recall.

[0005] Secondly, in the feature interaction stage, the query is the core carrier for parsing specific instances and semantic regions from the unordered point cloud. Most existing methods use the traditional global attention mechanism to perform undifferentiated interaction between the query and all points in the scene. Although the traditional global attention mechanism can capture long-range contextual information, in 3D space, background points (stuff) often account for the vast majority of the proportion. Furthermore, because the traditional global attention mechanism lacks explicit constraints on the interaction region, the discriminative features of foreground instances are easily diluted by background noise (such as walls and the ground), which severely weakens the model's ability to resolve weakly textured objects and subtle boundaries.

[0006] Finally, accurate geometric location awareness is crucial for achieving high-quality segmentation. During the stepwise feature extraction process of the Transformer, the features and spatial location of the target represented by the query are dynamically changing. Existing methods typically employ static geometric location encoding, injecting the geometric location information into the model all at once. However, this static encoding method cannot synchronously correct the spatial location information of the target based on dynamically optimized target features, leading to a misalignment between the target's features and their spatial location.

[0007] To address the aforementioned issues, a boundary-guided unified Transformer for 3D point cloud segmentation is proposed. Summary of the Invention

[0008] This invention proposes a boundary-guided unified Transformer for 3D point cloud segmentation, which aims to solve the problems of severe background interference in feature parsing and inaccuracy between features and geometric positions in existing 3D point cloud segmentation methods.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A boundary-guided unified Transformer for 3D point cloud segmentation includes the following steps;

[0011] Acquire the 3D point cloud data to be segmented, and extract features from the 3D point cloud data to obtain point cloud features;

[0012] The point cloud features are input into the boundary-guided query initialization module. Under the guidance of explicit boundary pseudo-labels, weighted aggregation and weighted sampling are performed on the semantic features and instance features in the point cloud features respectively to generate an initial query containing boundary information and the geometric center corresponding to the initial query.

[0013] The initial query is input into the region decoupling attention module, and feature interaction is performed through the foreground enhancement head, background suppression head and global interaction head respectively to focus on the foreground target region and suppress background interference, so as to obtain the decoupled features.

[0014] The decoupled features are input into the alternating update decoder, and the final query is obtained by alternatingly updating the query and the corresponding geometric center. Based on the final query, the segmentation results of the 3D point cloud data are output, including semantic segmentation results, instance segmentation results and panoramic segmentation results.

[0015] The boundary-guided 3D point cloud segmentation unified Transformer is trained using indoor 3D scene datasets such as ScanNet V2, ScanNet200, or S3DIS. In the application phase of the boundary-guided 3D point cloud segmentation unified Transformer, the 3D point cloud data to be segmented is input, and the trained boundary-guided 3D point cloud segmentation unified Transformer is used to segment the 3D point cloud scene, outputting corresponding semantic segmentation results, instance segmentation results, and panoramic segmentation results.

[0016] Given input point cloud ,in Indicates the first The three-dimensional coordinates of the points Indicates the first The RGB color information of each point is used; sparse voxel features of the input point cloud are extracted using a backbone network. and the sparse voxel features The data is fed into a pooling layer for aggregation to obtain superpoint features. ,in For the number of points exceeding the limit, This represents the number of feature channels; simultaneously, the coordinates of all hyperpoints are obtained by using the average of the physical coordinates of all voxels within each hyperpoint as its coordinates. ;

[0017] A k-nearest neighbor classification algorithm based on a kd-tree data structure is used to calculate point-level semantic boundary pseudo-labels and point-level instance boundary pseudo-labels in the 3D scene. The union of the point-level semantic boundary pseudo-labels and point-level instance boundary pseudo-labels is then aggregated to the superpoint level to construct the boundary pseudo-labels. ;

[0018] The boundary-guided query initialization module is used, based on the superpoint feature. With the coordinates of the super point In the boundary pseudo-label Under supervision, an initial query containing boundary information is generated. and its corresponding initial geometric center ;

[0019] The generated initial query Initial geometric center and super-point features It is fed into an alternately updated decoder containing multiple decoding layers for iterative processing; specifically, in the first... In the decoding layer, first according to the first Geometric center of decoder layer output With the coordinates of the super point Calculate the relative position offset matrix Subsequently, the region decoupling attention module is used to bias the matrix at the relative position. With the assistance of the upper layer query With the superpoint feature Perform geometry-aware interaction and obtain enhanced queries after processing. Next, based on the enhanced query Predicted geometric center offset For instance queries, the geometric center offset is... Accumulate to the current geometric center; for semantic queries, keep their geometric center unchanged; after iterative refinement through all decoding layers, obtain the final query. ;

[0020] The final query The data are fed into a prediction head for processing to obtain class scores and mask scores, and then subjected to linear mapping to obtain a set of semantic mask kernels and a set of instance mask kernels. For semantic segmentation, the semantic mask kernels are mapped to the superpoint features. Convolution is performed to obtain a semantic mask, and thresholding is applied to obtain the semantic segmentation result; for instance segmentation, the instance mask kernel is combined with the superpoint feature. Convolution is performed to obtain an instance mask, and the instance's category score is multiplied by the mask score to obtain the final ranking score. The instances are deduplicated and re-ranked based on the final ranking score to obtain the instance segmentation result. For panoptic segmentation, the filtered instance masks are superimposed on the semantic segmentation result according to the final ranking score from low to high to obtain the panoptic segmentation result.

[0021] The data processing flow of the boundary-guided query initialization module includes: First, using three parallel lightweight MLPs to process the superpoint features output by the backbone network. Decoupling is performed to obtain semantic features separately. Instance characteristics and boundary features ;

[0022] Then, in the weight calculation branch, the superpoint coordinates are... The input is mapped to high-dimensional geometric features by the position encoding module. and the boundary features With geometric features After being concatenated by channel, the data is fed into the entry control network for processing, and boundary response weight vectors are generated through an activation function. ;

[0023] Next, in the semantic branch, the semantic features are... The semantic score matrix is ​​obtained by feeding it into the auxiliary classification head. And using the boundary response weight vector For semantic score matrix Weighting is performed to filter out background noise, resulting in normalized aggregate weights. Furthermore, based on the normalized aggregation weights semantic features The initial semantic query is obtained by performing weighted aggregation. and its corresponding geometric center Initialize the entire geometry center;

[0024] Furthermore, in the instance branch, the boundary response weight vector is... Considering the importance distribution, for the coordinates of the superpoints Perform weighted farthest point sampling, and select the weighting metric during the sampling iteration process. The point that is maximized is used as the seed point, thereby selecting... Seed point index and according to the seed point index Extract the corresponding instance features and coordinates, and combine them with location encoding information to generate an initial instance query. and its corresponding initial geometric center ;

[0025] Finally, the initial semantic query Querying the initial instance The initial query is obtained by concatenating along the dimensions. At the same time, the corresponding geometric center and The initial center of the overall assembly is obtained by splicing. ,in .

[0026] The data processing flow of the region decoupling attention module includes: First, evaluating the query. With superpoint features Semantic affinity between them to generate attention masks Specifically, an MLP is used to query... Projecting onto the mask embedding space to capture object contour information, and then combining the projected query with superpoint features. Multiplying the matrices yields the affinity matrix, which is then converted into a mask using the sigmoid function. ,in Indicates the first The superpoint belongs to the first The confidence level of the foreground region of interest in each query;

[0027] Then, relative position bias matrices are introduced using the foreground enhancement head F-AH, background suppression head B-AH, and global interaction head G-AH, respectively. Perform feature interaction; use the mask in the foreground enhancement head F-AH. Non-target regions are filtered out, forcing queries to interact only with foreground superpoints of high confidence; furthermore, an inverse mask is used in the background suppression header B-AH. The query is guided to focus on non-target regions; then, no attention mask is applied in the global interaction header G-AH to maintain the query's awareness of global contextual information.

[0028] Finally, the features output by the foreground enhancement head, background suppression head, and global interaction head are concatenated along the channel dimension, and a linear transformation layer is used to reduce the dimensionality of the concatenated features to obtain the fused context features. Then, the fused context features are combined with the original input query. The summation, combined with a residual join, yields the enhanced query. .

[0029] The data processing flow of the alternating decoder update includes: in the relative position encoding stage, given the first... Query output from the decoding layer Geometric center and superpoint coordinates First calculate the first... Geometric center With the Coordinates of each super point relative coordinate vectors between Subsequently, an axial discretization strategy was adopted for... The components along each axis are quantized, and the position features are looked up in the axial embedding table according to the quantization index. Finally, the position features are fused through a linear transformation layer to obtain the relative position bias matrix. ;

[0030] During the query refinement stage, the query output from the previous layer is... With superpoint features The data is fed into the Region Decoupling Attention Module (RDAM) and a bias matrix is ​​applied at the relative position. Enhanced queries are obtained through geometry-aware interaction with the assistance of [unclear - possibly a specific technology or method]. Subsequently, a self-attention layer, a residual connection and layer normalization module, and a feedforward neural network are sequentially used to... The query is processed to obtain a more refined version. ;

[0031] During the central regression phase, the refined queries will be... The data is fed into the Center Regression (CRM) module, where an MLP is used for decoding to obtain the geometric center offset. Furthermore, a category-aware geometric update strategy is employed to calibrate the center coordinates: for instance queries, the predicted geometric center offset is adjusted accordingly. Accumulate to the current geometric center This achieves iterative approximation of the object's center; for semantic queries, gradient blocking is used to keep the geometric center unchanged.

[0032] The aforementioned boundary-guided 3D point cloud segmentation unified Transformer was trained using an indoor 3D scene dataset, including:

[0033] The total loss function of the method Due to segmentation loss Geometric consistency loss and initial loss Composition, defined as follows:

[0034] ,

[0035] Wherein, the initial loss Used to supervise the boundary-guided query initialization module, its calculation formula is:

[0036] .

[0037] In the formula, For the boundary fitting loss, a weighted binary cross-entropy loss function is used to calculate the predicted boundary response weight vector. With pre-computed boundary pseudo-labels The differences between them; To assist in semantic loss, the standard multi-class cross-entropy loss function is used to calculate the semantic score matrix output by the auxiliary classification head. The real semantic label corresponding to the superpoint The differences between them;

[0038] The geometric consistency loss The formula for updating the geometric center in the constrained alternating update decoder is as follows:

[0039] ,

[0040] In the formula, For the set of indexes for instance queries, The total number of matched instances. For the first The instance center predicted by the decoding layer, For the corresponding true geometric center, calculate the distance between the two. Distance establishes geometric consistency constraints;

[0041] The segmentation loss semantic segmentation loss With instance segmentation loss The composition and calculation formula are as follows:

[0042]

[0043] In the formula, For binary cross-entropy loss; By a binary cross-entropy loss One Dice loss And a weighted classification loss calculated based on bipartite graph matching results. The linear weighted composition is calculated using the following formula:

[0044]

[0045] in To balance the hyperparameters.

[0046] Compared with the prior art, the advantages of the present invention are as follows:

[0047] 1. This invention provides a boundary-guided unified Transformer for 3D point cloud segmentation. By injecting explicit boundary and geometric information into the entire query construction and decoding process, it achieves more robust semantic segmentation, instance segmentation, and panoramic segmentation.

[0048] 2. The boundary-guided query initialization module of the present invention can effectively provide high-quality initialization queries for 3D point cloud segmentation models;

[0049] 3. The region decoupling attention module of the present invention can effectively focus on the target region and effectively suppress the interference of background noise;

[0050] 4. The alternating update decoder of the present invention can effectively solve the problem of misalignment between target features and their spatial positions in 3D point cloud segmentation models;

[0051] 5. The boundary-guided 3D point cloud segmentation unified Transformer provided by this invention has excellent 3D point cloud segmentation performance. Attached Figure Description

[0052] Figure 1 Overall structure diagram of the unified Transformer for boundary-guided 3D point cloud segmentation;

[0053] Figure 2 Structure diagram of the boundary-guided query initialization module;

[0054] Figure 3 A structural diagram of the region decoupling attention module;

[0055] Figure 4 The structure diagram for alternating decoder updates. Detailed Implementation

[0056] Please refer to the attached document. Figure 1-4 The present invention will now be described in further detail with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0057] The overall network structure is as follows Figure 1 As shown, given the input point cloud ,in, Indicates the first The three-dimensional coordinates of the points Indicates the first The RGB color information of each point is first extracted using a sparse 3DU-Net as the backbone network to extract its sparse voxel features. Subsequently, the sparse voxel features were... The data is fed into a Flexible Pooling layer for aggregation to obtain superpoint features. ,in For the number of points exceeding the limit, This represents the number of feature channels. Simultaneously, the coordinates of all hyperpoints are obtained by using the average of the physical coordinates of all voxels within each hyperpoint as its coordinates. In addition, to provide explicit boundary information to the model, superpoint-level boundary pseudo-labels were pre-computed. As an auxiliary monitoring signal, specifically, the k-nearest neighbor (KNN) classification algorithm based on the kd-tree data structure is first used to calculate point-level semantic boundary pseudo-labels and point-level instance boundary pseudo-labels in the 3D scene. Then, by taking the union of the point-level semantic boundary pseudo-labels and the point-level instance boundary pseudo-labels and aggregating them to the super-point level, a boundary pseudo-label containing full-scene semantics and instance edge information is constructed. Next, a boundary-guided query initialization module (BGQIM) is used, based on superpoint features. With super point coordinates Under the supervision of superpoint-level boundary pseudo-labels, an initial query containing boundary information is generated. and its corresponding geometric center .

[0058] Furthermore, the generated initial query Geometric center and super-point features It is then fed into an alternating update decoder (IUD) consisting of six decoding layers for processing. Specifically, in the... In the first decoding layer, a relative position encoding module (RPEM) is first used according to the first... Geometric center of decoder layer output With super point coordinates Calculate the relative position offset matrix ; then, The data is fed into a Query Refinement Module (QRM). Within this module, a Region Decoupling Attention Module (RDAM) is used to refine the position bias matrix. With the assistance of the upper layer query With superpoint features Geometrically perceptual interaction is performed, and a self-attention layer, a residual connection and layer normalization, and a feedforward neural network are used to process the interaction results to obtain enhanced queries. Finally, the query will be enhanced. The data is fed into a center regression module (CRM) for processing to obtain the predicted geometric center offset. Furthermore, for instance queries, the predicted geometric center offset is accumulated to the current geometric center to iteratively approximate the object's center; for semantic queries, their geometric centers remain unchanged. After iterative refinement through six decoding layers, the final query is obtained. .

[0059] For the final query The data is then fed into a shared prediction head for processing to obtain the instance's class score and mask score. These are then fed into a linear mapping module for further processing to obtain a set of semantic mask kernels and a set of instance mask kernels. For semantic segmentation, the semantic mask kernels are first compared with superpoint features. Convolution is performed to obtain a semantic mask. Then, a fixed threshold is applied to the semantic mask to obtain a binary semantic segmentation result. For instance segmentation, the instance kernel and superpoint features are first combined... Convolution is performed to obtain an instance mask. Then, the instance's class score is multiplied by the mask score to obtain the instance's final ranking score. Next, based on the final ranking score, the Matrix-NMS algorithm is used to deduplicate and re-rank the instances to obtain the instance segmentation result. For panoptic segmentation, the filtered instance masks are superimposed on the semantic segmentation result in ascending order of the final ranking score to obtain the panoptic segmentation result.

[0060] A boundary-guided unified 3D point cloud segmentation Transformer was trained using an indoor 3D scene dataset, such as ScanNet V2. During training, the AdamW optimizer was used to optimize the network parameters, with an initial learning rate of 0.0001, weight decay of 0.05, and epochs of 512. For data preprocessing, a graph-based oversegmentation algorithm was used to cluster the original point cloud into superpoints. The voxel size of the point cloud was set to 2 cm. An NVIDIA A100 GPU was used to train the model on the ScanNet V2 dataset. In the application stage of the boundary-guided unified 3D point cloud segmentation Transformer, the 3D point cloud data to be segmented was input, and the trained boundary-guided unified 3D segmentation method was used to segment the 3D point cloud scene, outputting corresponding semantic segmentation results, instance segmentation results, and panoramic segmentation results.

[0061] The boundary-guided query initialization module is used to provide high-quality initialization queries for 3D point cloud segmentation models. For example... Figure 2 As shown, the boundary-guided query initialization module includes a weight calculation branch, a semantic branch, and an instance branch. Specifically, given the superpoint features output by the backbone network... and its corresponding superpoint coordinates First, three parallel lightweight MLPs are used to... Decoupling is performed to obtain semantic features separately. Instance characteristics and boundary features .

[0062] In the weight calculation branch, in order to capture geometric change information, the superpoint coordinates are first... The input is fed into a position encoding module (PE) and mapped to high-dimensional geometric features. Subsequently, the boundary features With geometric features The features are concatenated by channel and then fed into a gated MLP for processing to obtain boundary prediction scores. Finally, the Sigmoid activation function is used to... The boundary response weight vector is obtained through processing. :

[0063] ,

[0064] In the formula, This represents the Sigmoid activation function. Furthermore, to ensure... With clear boundary information, boundary pseudo-labels are used during the training phase. Supervise it.

[0065] In the semantic branch, in order to generate The prototype representation of each category, firstly... The data is fed into an auxiliary classification head for processing to obtain a semantic score matrix. Then, in order to filter out background noise and focus on the core semantic region, boundary weights are used. semantic score Perform weighting to obtain the aggregate weight. Furthermore, the Softmax function is used to perform column-wise pairing. Normalization is performed to obtain the normalized aggregate weights. Furthermore, adopting The semantic features are processed to obtain the initial semantic query. The above process can be formalized as follows:

[0066] .

[0067] In particular, to ensure the semantic score matrix The discriminative power is achieved by utilizing the real semantic labels corresponding to superpoints during the training phase. Supervise it.

[0068] Given that semantic categories are typically dispersed in a 3D scene, weighted centroids often fall into meaningless, non-entity regions and lack representativeness. Therefore, the geometric center of all semantic queries is initialized as the global geometric centroid. This strategy provides semantic queries with an unbiased global view, enabling them to aggregate category features from all parts of the scene equally.

[0069] In the instance branching, to avoid confusion between features of different instances of the same category, global aggregation is abandoned, and instead an importance sampling strategy is adopted. Specifically, firstly, Consider it as an importance distribution, and for the coordinates of the superpoints Perform weighted farthest point sampling (Weighted FPS). Unlike standard FPS, which only considers Euclidean distance, a weighted metric is selected at each iteration step. The point that is maximized is used as the new seed point (where (This is the currently selected set of points), and then filter out... A seed point index with high boundary response and spatial dispersion Then, based on the seed point index... Extract the instance features and coordinates corresponding to the (index Gather) and combine them with location encoding information to generate an initial instance query. With geometric center :

[0070] .

[0071] Ultimately, and The query is obtained by concatenating along the 0th dimension. ,in, At the same time, their corresponding geometric centers and Also, splice along the 0th dimension to obtain the geometric center. .

[0072] The aforementioned region decoupling attention module is used to focus on the target region and effectively suppress background noise interference. For example... Figure 3 As shown, given a query Super-point features extracted from the backbone network First, evaluate the query. With superpoint features The semantic affinity between them is used to generate dynamic attention masks. Specifically, an MLP is first used to... The projected query is embedded into a mask embedding space, focusing on capturing object contour information. Then, the projected query is multiplied by the superpoint features to obtain an affinity matrix that measures the semantic similarity between the two. Finally, the affinity is converted into a mask using the sigmoid function. In particular, Indicates the first The superpoint belongs to the first The confidence level of the foreground region of interest for each query. The above process can be formulated as follows:

[0073] ,

[0074] In the formula, This represents the Sigmoid function.

[0075] It is worth noting that this mask During training, it receives explicit supervision from real mask labels assigned based on bipartite matching, thereby ensuring that it can accurately distinguish object regions from background noise.

[0076] Subsequently, three attention heads—Foreground Enhancement Head (F-AH), Background Suppression Head (B-AH), and Global Interaction Head (G-AH)—are employed to achieve refined feature interaction. Specifically, to enhance the geometric awareness capability of the RDA module, the relative position bias matrix is... Three attention heads are introduced respectively.

[0077] For Foreground Enhancement Head (F-AH), use a mask. Filtering out non-target regions forces the query to interact only with foreground superpoints of high confidence. The specific calculation process of this attention head is as follows:

[0078] ,

[0079] In the formula, This represents the foreground context features output by the attention head. This represents element-wise multiplication. , , These are the query, key, and value in the attention header, respectively.

[0080] For the background suppression header (B-AH), use an inverse mask. This guides the query to focus on non-target areas. The specific calculation process for this attention head is as follows:

[0081] ,

[0082] In the formula, This represents the background context features output by the attention head. , , These are the query, key, and value in the attention header, respectively.

[0083] For the Global Interaction Header (G-AH), no attention mask is applied to maintain the query's awareness of global context information. The specific calculation process for this attention head is as follows:

[0084] ,

[0085] In the formula, This represents the global contextual features output by the attention head. , , These are the query, key, and value in the attention header, respectively.

[0086] Finally, the features output by the three attention heads are concatenated along the channel dimension, and a linear transformation layer is used to reduce the dimensionality of the concatenated features, resulting in the fused context features. To promote effective gradient propagation, the idea of ​​residual connections is adopted, and the fused context features are combined with the original input query. Add them together to get the enhanced query. The above process can be formalized as follows:

[0087] .

[0088] The alternating update decoder is used to solve the misalignment problem between target features and their spatial positions in 3D point cloud segmentation models. For example... Figure 4 As shown, the decoder's first... Each decoding layer contains a relative position encoding module, a query refinement module, and a center regression module.

[0089] Given the first Query output from the decoding layer Geometric center Super point coordinates and super-point features , No. Each decoding layer sequentially performs relative position encoding, query refinement, and center regression operations. Specifically, for the first decoding layer, the initial query output by the BGQI module is used directly. With geometric center As as well as .

[0090] In the relative position encoding module, for the first Query geometric center output by the decoding layer With super point coordinates First calculate the first Geometric center With the Coordinates of each super point relative coordinate vectors between To address the challenge of learning robust features from raw relative coordinates due to the continuity of 3D point cloud coordinates and their cross-scene scale variations, an axial discretization strategy is employed to quantize the relative coordinates. Specifically, the quantization step size is set to... The lookup table length is For relative coordinate vectors Components in each axis (in ), calculate its corresponding quantization index matrix as follows:

[0091] .

[0092] Subsequently, based on the quantization index matrix Positional features are searched in three learnable axial embedding tables. Further, these three positional features are summed element-wise and processed using a linear transformation layer to obtain the relative position bias matrix. .

[0093] In the query refinement module, for the first... Query output from the decoding layer and the output of the RPE module First, combine them with superpoint features Both are sent to the RDA module for processing to obtain the enhanced query. Subsequently, a self-attention layer, a residual connection and layer normalization (add & norm), and a feedforward neural network are used sequentially to... Processing yields a refined query. .

[0094] In the central regression module, an MLP is first used to refine the query. Decode the result relative to the geometric center. geometric offset :

[0095] .

[0096] Subsequently, a category-aware geometry update strategy is employed to calibrate the geometry center. Specifically, for instance queries, the predicted geometric offsets are accumulated to their geometric centers to iteratively approximate the object's center; for semantic queries, the backpropagation of geometric gradients is blocked, and their geometric centers remain unchanged. That is:

[0097] ,

[0098] In the formula, and These represent the index sets for instance queries and semantic queries, respectively.

[0099] The total loss function during training of the boundary-guided 3D point cloud segmentation unified Transformer. Due to segmentation loss Geometric consistency loss and initial loss Composition, namely:

[0100] ,

[0101] In the formula, , , To balance the hyperparameters.

[0102] The segmentation loss consists of instance segmentation loss and semantic segmentation loss:

[0103] .

[0104] Specifically, instance segmentation loss The weighted classification loss is calculated from the bipartite graph matching results. A binary cross-entropy loss A Dice loss Composition, namely:

[0105] ,

[0106] In the formula, A balanced hyperparameter. Semantic segmentation loss. This is a binary cross-entropy loss.

[0107] The geometric consistency loss According to the prediction instance center With the true geometric center Between Constructed using distance, i.e.:

[0108] ,

[0109] In the formula, For the set of indexes for instance queries, The total number of matched instances.

[0110] The initialization loss Used to supervise the boundary-guided query initialization module, its calculation formula is:

[0111] .

[0112] In the formula, For the boundary fitting loss, a weighted binary cross-entropy loss function is used to calculate the predicted boundary response weight vector. With pre-computed boundary pseudo-labels The differences between them; To assist in semantic loss, the standard multi-class cross-entropy loss function is used to calculate the semantic score matrix output by the auxiliary classification head. The real semantic label corresponding to the superpoint The differences between them.

[0113] Furthermore, to verify the performance of the boundary-guided unified Transformer for 3D segmentation provided in this invention, it was experimentally compared with eighteen existing 3D point cloud segmentation models on the ScanNet V2 dataset. The experimental results are shown in Table 1. As can be seen from Table 1, for instance segmentation tasks, the model provided in this invention... , as well as The average intersection-union ratios (IUU) of the network provided in this invention reached 87.3%, 78.2%, and 61.7%, respectively. For semantic segmentation tasks, the IUU of the network provided in this invention is... The panorama quality reached 77.8%. For panorama segmentation tasks, the panorama quality of the network provided by this invention ( ), prospect score ( ) and background score ( The accuracy rates reached 73.5%, 71.1%, and 88.6%, respectively. This indicates that the unified 3D segmentation architecture provided by this invention has excellent 3D point cloud scene segmentation performance.

[0114] Table 1 shows the experimental results of the network provided by this invention and eighteen existing 3D point cloud segmentation models on the ScanNet V2 dataset. Bold numbers represent the top-ranked results, and "-" indicates no available data.

[0115]

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0117] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0118] Note that the above description is merely a preferred embodiment and application of the technical principles of the present invention. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the specific embodiments described herein, and may include many other effective embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A boundary-guided unified Transformer for 3D point cloud segmentation, characterized in that, include: Acquire the 3D point cloud data to be segmented, and extract features from the 3D point cloud data to obtain point cloud features; The point cloud features are input into the boundary-guided query initialization module. Under the guidance of explicit boundary pseudo-labels, weighted aggregation and weighted sampling are performed on the semantic features and instance features in the point cloud features respectively to generate an initial query containing boundary information and the geometric center corresponding to the initial query. The initial query is input into the region decoupling attention module, and feature interaction is performed through the foreground enhancement head, background suppression head and global interaction head respectively to focus on the foreground target region and suppress background interference, so as to obtain the decoupled features. The decoupled features are input into the alternating update decoder, and the final query is obtained by alternatingly updating the query and the corresponding geometric center. Based on the final query output, the segmentation results of the 3D point cloud data are provided, including semantic segmentation results, instance segmentation results, and panoramic segmentation results. Using indoor 3D scene datasets, such as ScanNet V2, ScanNet200, or S3DIS, a boundary-guided unified Transformer for 3D point cloud segmentation is trained. In the application phase of the boundary-guided unified Transformer for 3D point cloud segmentation, the 3D point cloud data to be segmented is input, and the trained boundary-guided unified Transformer for 3D point cloud segmentation is used to segment the 3D point cloud scene, outputting the corresponding semantic segmentation results, instance segmentation results, and panoramic segmentation results.

2. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The aforementioned boundary-guided 3D point cloud segmentation unified Transformer data processing flow includes: Given input point cloud ,in Indicates the first The three-dimensional coordinates of the points Indicates the first The RGB color information of each point is used; sparse voxel features of the input point cloud are extracted using a backbone network. and the sparse voxel features The data is fed into a pooling layer for aggregation to obtain superpoint features. ,in For the number of points exceeding the limit, This represents the number of feature channels; simultaneously, the coordinates of all hyperpoints are obtained by using the average of the physical coordinates of all voxels within each hyperpoint as its coordinates. ; A k-nearest neighbor classification algorithm based on a kd-tree data structure is used to calculate point-level semantic boundary pseudo-labels and point-level instance boundary pseudo-labels in the 3D scene. The union of the point-level semantic boundary pseudo-labels and point-level instance boundary pseudo-labels is then aggregated to the superpoint level to construct the boundary pseudo-labels. ; The boundary-guided query initialization module is used, based on the superpoint feature. With the coordinates of the super point In the boundary pseudo-label Under supervision, an initial query containing boundary information is generated. and its corresponding initial geometric center ; The generated initial query Initial geometric center and super-point features It is fed into an alternately updated decoder containing multiple decoding layers for iterative processing; specifically, in the first... In the decoding layer, first according to the first Geometric center of decoder layer output With the coordinates of the super point Calculate the relative position offset matrix Subsequently, the region decoupling attention module is used to bias the matrix at the relative position. With the assistance of the upper layer query With the superpoint feature Perform geometry-aware interaction and obtain enhanced queries after processing. Next, based on the enhanced query Predicted geometric center offset For instance queries, the geometric center offset is... Accumulate to the current geometric center; for semantic queries, keep their geometric center unchanged; after iterative refinement through all decoding layers, obtain the final query. ; The final query The data are fed into a prediction head for processing to obtain class scores and mask scores, and then subjected to linear mapping to obtain a set of semantic mask kernels and a set of instance mask kernels. For semantic segmentation, the semantic mask kernels are mapped to the superpoint features. Convolution is performed to obtain a semantic mask, and thresholding is applied to obtain the semantic segmentation result; for instance segmentation, the instance mask kernel is combined with the superpoint feature. Convolution is performed to obtain an instance mask, and the instance's category score is multiplied by the mask score to obtain the final ranking score. The instances are deduplicated and re-ranked based on the final ranking score to obtain the instance segmentation result. For panoptic segmentation, the filtered instance masks are superimposed on the semantic segmentation result according to the final ranking score from low to high to obtain the panoptic segmentation result.

3. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The data processing flow of the boundary-guided query initialization module includes: First, three parallel lightweight MLPs are used to process the superpoint features output by the backbone network. Decoupling is performed to obtain semantic features separately. Instance characteristics and boundary features ; Then, in the weight calculation branch, the superpoint coordinates are... The input is mapped to high-dimensional geometric features by the position encoding module. and the boundary features With geometric features After being concatenated by channel, the data is fed into the entry control network for processing, and boundary response weight vectors are generated through an activation function. ; Next, in the semantic branch, the semantic features are... The semantic score matrix is ​​obtained by feeding it into the auxiliary classification head. And using the boundary response weight vector For semantic score matrix Weighting is performed to filter out background noise, resulting in normalized aggregate weights. Furthermore, based on the normalized aggregation weights semantic features The initial semantic query is obtained by performing weighted aggregation. and its corresponding geometric center Initialize the entire geometry center; Furthermore, in the instance branch, the boundary response weight vector is... Considering the importance distribution, for the coordinates of the superpoints Perform weighted farthest point sampling, and select the weighting metric during the sampling iteration process. The point that is maximized is used as the seed point, thereby selecting... Seed point index and according to the seed point index Extract the corresponding instance features and coordinates, and combine them with location encoding information to generate an initial instance query. and its corresponding initial geometric center ; Finally, the initial semantic query Querying the initial instance The initial query is obtained by concatenating along the dimensions. At the same time, the corresponding geometric center and The initial center of the overall assembly is obtained by splicing. ,in .

4. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The data processing flow of the region decoupling attention module includes: First, evaluate the query. With superpoint features Semantic affinity between them to generate dynamic attention masks. Specifically, MLP is used to query Projecting onto the mask embedding space to capture object contour information, and then combining the projected query with superpoint features. Multiplying the matrices yields the affinity matrix, which is then converted into a mask using the sigmoid function. ,in Indicates the first The superpoint belongs to the first The confidence level of the foreground region of interest in each query; Then, relative position bias matrices are introduced using the foreground enhancement head F-AH, background suppression head B-AH, and global interaction head G-AH, respectively. Perform feature interaction; Specifically, the mask is used in the foreground enhancement head F-AH. Filter out non-target regions, forcing queries to interact only with high-confidence foreground superpoints; Furthermore, an inverse mask is used in the background suppression header B-AH. Guide queries to focus on non-target areas; Next, in the global interaction header G-AH, no attention mask is applied to maintain the query's ability to perceive global context information; Finally, the features output by the foreground enhancement head, background suppression head, and global interaction head are concatenated along the channel dimension, and a linear transformation layer is used to reduce the dimensionality of the concatenated features to obtain the fused context features. Then, the fused context features are combined with the original input query. The summation, combined with a residual join, yields the enhanced query. .

5. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The data processing flow of the alternating decoder update includes a relative position encoding module, a query refinement module, and a center regression module.

6. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The data processing flow of the relative position encoding module includes: Given the first Query output from the decoding layer Geometric center and superpoint coordinates First calculate the first... Geometric center With the Coordinates of each super point relative coordinate vectors between Subsequently, an axial discretization strategy was adopted for... The components along each axis are quantized, and the position features are looked up in the axial embedding table according to the quantization index. Finally, the position features are fused through a linear transformation layer to obtain the relative position bias matrix. .

7. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The data processing flow of the query refinement module includes: The query output from the previous layer With superpoint features The data is fed into the Region Decoupling Attention Module (RDAM) and a bias matrix is ​​applied at the relative position. Enhanced queries are obtained through geometry-aware interaction with the assistance of [unclear - possibly a specific technology or method]. Subsequently, a self-attention layer, a residual connection and layer normalization module, and a feedforward neural network are sequentially used to... The query is processed to obtain a more refined version. .

8. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The data processing flow of the central regression module includes: Detailed query The data is fed into the Center Regression (CRM) module, where an MLP is used for decoding to obtain the geometric center offset. Furthermore, a category-aware geometric update strategy is employed to calibrate the center coordinates: for instance queries, the predicted geometric center offset is adjusted accordingly. Accumulate to the current geometric center This achieves iterative approximation of the object's center; for semantic queries, gradient blocking keeps the geometric center unchanged.

9. The boundary-guided unified Transformer for 3D point cloud segmentation according to claim 1, characterized in that, The aforementioned boundary-guided 3D point cloud segmentation unified Transformer was trained using an indoor 3D scene dataset, including: The method uses an indoor 3D scene dataset to train a boundary-guided 3D segmentation unified Transformer, including: The total loss function of the method Due to segmentation loss Geometric consistency loss and initial loss Composition, defined as follows: , Wherein, the initial loss Used to supervise the boundary-guided query initialization module, its calculation formula is: . In the formula, For the boundary fitting loss, a weighted binary cross-entropy loss function is used to calculate the predicted boundary response weight vector. With pre-computed boundary pseudo-labels The differences between them; To assist in semantic loss, the standard multi-class cross-entropy loss function is used to calculate the semantic score matrix output by the auxiliary classification head. The real semantic label corresponding to the superpoint The differences between them; The geometric consistency loss The formula for updating the geometric center in the constrained alternating update decoder is as follows: , In the formula, For the set of indexes for instance queries, The total number of matched instances. For the first The instance center predicted by the decoding layer, For the corresponding true geometric center, calculate the distance between the two. Distance establishes geometric consistency constraints; The segmentation loss semantic segmentation loss With instance segmentation loss The composition and calculation formula are as follows: ； In the formula, For binary cross-entropy loss; By a binary cross-entropy loss One Dice loss And a weighted classification loss calculated based on bipartite graph matching results. The linear weighted composition is calculated using the following formula: . in To balance the hyperparameters.

Images