A method for constructing a unified segmentation framework for large-scale 3D point cloud scenes

Through multi-dimensional feature fusion and hyper-point map technology, combined with 3D sparse U-Net and Transformer architecture, a unified segmentation framework for three-dimensional point clouds is built, solving the problem of low point cloud features and segmentation efficiency in the existing technology, and achieving high efficiency , accurate semantics, examples and panoramic segmentation.

CN118736233BInactive Publication Date: 2024-11-19JILIN AGRICULTURAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410943324.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing three-dimensional point cloud segmentation technology is difficult to comprehensively extract the original structure and key features of data, and ignores the similarity between similar objects and implicit relationships between different segmentation tasks, resulting in low segmentation efficiency and accuracy. Not high.

Method used

Using a large-scale three-dimensional point cloud unified segmentation framework based on multi-dimensional feature fusion, the point-by-point features of point clouds are extracted through 3D sparse U-Net and Transformer architecture, combining hybrid feature coding, bilateral enhancement and ternary distance attention pooling, and A unified model of semantics, instances and panoramic segmentation is constructed through hyperpoint plots and cross-attention mechanisms.

Benefits of technology

It significantly improves the accuracy and efficiency of segmentation, and can process large-scale point cloud data at the same time in real time and efficiently. It is suitable for autonomous driving, robot navigation and other fields, providing a unified and practical point cloud segmentation solution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118736233B_ABST
    Figure CN118736233B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for constructing a unified segmentation framework for large-scale three-dimensional point cloud scenes, and belongs to the field of computer image recognition and artificial intelligence technology. The method includes: extracting super-point information of point clouds; extracting local features of point clouds through hybrid feature encoding components; aggregating local neighborhoods through ternary distance attention pools to obtain point-by-point feature representations of point clouds; grouping super-point information and point-by-point features into super-point features through a super-point pooling layer as keys and values ​​of a decoder, and at the same time taking a set of learnable semantic and instance queries as inputs through random initialization, and the three capture point cloud information through a super-point cross-attention mechanism; adopting a decomposition matching strategy to train in an end-to-end manner to predict the semantic label of each point in the point cloud, and finally completing the semantic, instance and panoramic segmentation of the point cloud in the entire scene. The advantages are: being able to handle semantic, instance and panoramic segmentation tasks at the same time, avoiding the limitation of traditional methods that different frameworks are required for different tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer image recognition and artificial intelligence technology, and in particular to a method for constructing a unified segmentation framework for large-scale three-dimensional point cloud scenes, while establishing an end-to-end model framework for semantic segmentation, instance segmentation and panoptic segmentation of 3D point clouds. Background Art

[0002] With the rapid development of 3D sensor technology, the analysis of point cloud data is gradually becoming the focus of scientific research and application fields. Among them, point cloud semantic segmentation aims to give clear semantic labels to each point in the point cloud scene; instance segmentation returns a set of masks for a single object; panoramic segmentation predicts a mask for each foreground object and a semantic label for each background point. Segmentation technology occupies an important position in 3D computer vision and is widely used in many fields such as medical image analysis, smart city planning, environmental monitoring, etc. It also shows great application potential in the fields of autonomous driving and robot navigation.

[0003] 3D point cloud segmentation technology is still facing severe challenges. First, compared with structured data, the unstructured and disordered characteristics of point cloud data make it difficult to directly apply traditional image processing technology to point cloud data, and the occlusion and uneven density of point cloud data make the segmentation task more complicated. Second, most of the existing point cloud segmentation methods are based on voxelization or projection methods, which are costly and inefficient when processing large-scale data. Finally, the current semantic segmentation, instance segmentation and panoramic segmentation of point clouds rely on completely different framework structures. The semantic segmentation method relies on the U-Net network, and the instance segmentation method combines the semantic segmentation model with an aggregation scheme based on clustering, object detection or Transformer decoder. Panoramic segmentation performs panoramic segmentation in a two-dimensional image, and then lifts the predicted mask to a three-dimensional space and aggregates it point by point. Existing point cloud segmentation methods cannot comprehensively extract the original structure and key features of the point cloud effectively, ignoring the problem of feature similarity between similar objects in the point cloud, and the similarity of all segmentation tasks and the implicit relationship between them have not been effectively utilized. Therefore, there is an urgent need for a unified, simple, and effective end-to-end model framework that can solve all three types of segmentation tasks at the same time, meet real-time requirements, efficiently process large-scale point cloud data, and accurately extract local features.

[0004] Summary: In order to achieve accurate semantic, instance and panoptic segmentation in 3D point cloud scenes, the existing point cloud segmentation technology must be fundamentally improved. This paper proposes an innovative unified segmentation framework, which significantly improves the accuracy and efficiency of segmentation by fusing multi-dimensional features, optimizing feature extraction and aggregation strategies, and introducing an efficient attention mechanism. Summary of the invention

[0005] The purpose of the present invention is to provide a method for constructing a unified segmentation framework for large-scale three-dimensional point cloud scenes based on multi-dimensional feature fusion, which solves the current technical problems of failing to comprehensively and effectively extract the original structure and key features of point cloud data, ignoring the problem of feature similarity between similar objects, and failing to fully utilize the similarities and implicit relationships between different segmentation tasks. Problems with current large-scale point cloud data scene segmentation. The present invention successfully achieves accurate semantic prediction for each point, and is extremely suitable for fields such as autonomous driving, robot navigation, urban planning and environmental monitoring, and provides an efficient, unified and practical point cloud segmentation solution for them.

[0006] The above-mentioned object of the present invention is achieved by the following technical solutions:

[0007] The main idea of ​​the construction method of a unified segmentation framework for large-scale three-dimensional point cloud scenes based on multi-dimensional feature fusion is as follows: for the input raw point cloud, first use 3D sparse U-Net to extract the point cloud's point-by-point features, and optimize the hybrid feature encoding component, the hybrid bilateral enhancement component and the ternary distance attention pooling in the 3D sparse U-Net to fully exploit the color information and spatial information of the point cloud to improve the diversity and comprehensiveness of local features. At the same time, the raw point cloud input is geometrically segmented, the super-point information of the point cloud is extracted, and a super-point graph is constructed. Subsequently, the extracted point-by-point features and super-point information are simultaneously fed into the super-point pooling layer to group homogeneous adjacent points into super-point features. The super-point features are used as the keys and values ​​of the Transformer decoder, and a set of learnable semantic and instance queries are used as input. The point cloud information is captured through a cross-attention mechanism, and a set of learned kernels are output, each of which represents an instance identifier or semantic region of a single object mask, and finally a unified model framework for the three segmentation tasks of semantics, instance and panorama in point cloud scenes is constructed.

[0008] S1. Extract super-point information: Through an unsupervised algorithm, the entire point cloud is used as the input of the algorithm. Based on the k-nearest neighbor adjacency graph of the point cloud, the point cloud is divided into geometrically uniform regions through optimization problem solving, and divided into geometrically simple and meaningful shapes, called super-points; the adjacency between super-points is determined by calculating the symmetric Voronoi adjacency graph of the input point cloud to calculate the super-point graph and extract the super-point information of the point cloud.

[0009] S2. Extract and construct multi-scale features of point cloud data:

[0010] S21, hybrid feature encoding component: This hybrid feature encoding component encodes the color and spatial coordinate information of each point in the point cloud, integrating spatial information and color information. The encoding formula is as follows:

[0011]

[0012] Among them, g i and It's point p i and its neighboring points The absolute (x, y, z) coordinates of i and It's point p i and its neighboring points The color information (r,g,b), It is a cascade operation, ∥·∥ represents the calculated Euclidean distance between the center point and the neighboring points, and MLP is used for sequence encoding. After MLP further combines the spatial information and color information of the point cloud, it performs a deeper mapping transformation and obtains the mixed features of the point cloud.

[0013]

[0014] S22, hybrid bilateral enhancement component: for each point p i and its neighboring points The corresponding semantic information f i and f i k , the following formula is constructed to represent the local semantic context in the feature space:

[0015]

[0016] After the MLP operation, the dimension is reduced while the highly fine-grained local semantics are highlighted. Cascade to form a new mixed semantic variable. In order to better integrate these features, the new mixed semantic variable is further processed to optimize its representation and accurately capture the key information of the mixed features, and the final mixed feature representation is obtained:

[0017]

[0018] Similarly, while strengthening the local semantic context, highly fine-grained local geometric structure and color information are introduced for local semantics, further improving the local semantic feature's ability to perceive details. The final local semantic feature is expressed as follows:

[0019]

[0020] Enhanced hybrid features With enhanced local semantic features Cascade to form mixed bilateral enhancement features, which enhance the network's ability to perceive and extract local features. The mixed bilateral enhancement features are represented as follows:

[0021]

[0022] This representation combines local geometric structure, color information and semantic features, significantly improving the network's ability to perceive and extract local features, and providing more comprehensive information for the semantic segmentation task of point clouds.

[0023] S23, ternary distance attention pooling: The local neighborhood is aggregated through the ternary distance attention pool to obtain the point-by-point feature representation of the point cloud; the feature weights of different points are dynamically adjusted through the attention mechanism, and the influence of spatial geometric distance, color Euclidean distance and semantic feature distance are integrated at the same time. Semantic feature distance Geometric distance Euclidean distance to color The definition is as follows:

[0024]

[0025] Among them, g(i) and g(k) are points p i and its neighboring points The spatial information of point p, c(i) and c(k) are i and its neighboring points The color information of point p, f(i) and f(k) are i and its neighboring points The semantic information of , exp is the exponential function, Mean(·) represents the average function, which is used to calculate the mean of the distance. |·| is the L1 norm, and ||·|| represents the L2 norm. These three distance values ​​are cascaded and two parameters λ1 and λ2 are used to dynamically adjust and To deal with its instability, effectively balance the influence of color information and semantic features in the overall distance calculation. The final ternary distance is expressed as:

[0026]

[0027] The values ​​of λ1 and λ2 are both 0.1. The three distance values ​​are concatenated with the hybrid bilateral enhancement features of the point cloud, and the attention score is calculated through a softmax operation. The learned attention score is multiplied with the corresponding feature as the attention weighted feature.

[0028]

[0029] The super-point information and point-by-point features are grouped into super-point features through the super-point pooling layer as the key and value of the decoder. At the same time, a set of learnable semantic and instance queries are randomly initialized as input. The three capture point cloud information through the super-point cross-attention mechanism. The decomposition matching strategy is adopted to predict the semantic label of each point in the point cloud in an end-to-end training manner, and finally the semantic, instance and panoramic segmentation of the point cloud in the entire scene are completed.

[0030] S3, group homogeneous adjacent points into super-point features, and output the learned kernel through the super-point cross-attention mechanism, specifically:

[0031] S31, super point pooling layer: super points calculated by S1 process and point-level features calculated by S2 process At the same time, they are fed into the super-point pooling layer, which groups homogeneous adjacent points by averaging the point-by-point features within each super-point to obtain super-point features. At the same time, the input point cloud can be reliably downsampled into hundreds of super-points, which significantly reduces the computational overhead of subsequent processing and optimizes the representation capability of the entire network.

[0032] S32, super-point cross attention: Considering the disorder and uncertainty of the number of super-points, a super-point cross-attention mechanism is designed. The purpose is to use the Transformer's self-attention and cross-attention mechanisms to process variable-length inputs in point cloud data while capturing instance information. The super-point features and semantic and instance query vectors Z calculated in step S31 are used as inputs to the Transformer decoder. The super-point feature extraction is completed through super-point cross-attention, and the query vector is randomly initialized before training. The Transformer decoder is stacked to six layers to expand the receptive field.

[0033] S33, for the super-point features calculated in step S31, map them to S′ through linear transformation, the output of the cross-attention of the previous layer of super-points As the query vector of the next layer, the context information is captured by the superimposed point cross attention mechanism, and the formula is as follows:

[0034]

[0035] in is the output of the super-point cross attention, is the input query vector The linear projection of, K and V are different linear projections of the super point feature S′, is the super-point attention mask, given the predicted super-point mask from the previous prediction head Use the threshold τ to filter out super points, the formula is as follows:

[0036]

[0037] The i-th query vector marking the j-th superpoint, τ is set to 0.5.

[0038] S4. Use the decomposition matching strategy to train the instance kernel and predict the semantic label of each point in the point cloud. The method of training the instance kernel in an end-to-end manner is as follows:

[0039] Given the output of super-point cross attention in S3 Two independent MLPs are used to predict the classification p for each query vector i ,i∈1,…,K ins , the true value label assignment is expressed as an optimal assignment problem, and the pairwise matching cost C is formally introduced ik To evaluate the similarity between the i-th predicted value and the k-th true value. ik Matching cost by classification probability and superpoint mask Decision, defined as follows:

[0040]

[0041] where p i,ck Indicates that the i-th predicted value belongs to c k The probability of the semantic category. cls =0.5,λ mask =0.5. Superpoint matching cost It is the sum of binary cross entropy (BCE) and Dice loss with Laplace smoothing:

[0042]

[0043] m i and are the predicted mask and ground truth mask of the superpoint, respectively. A hard instance label is assigned to each superpoint based on whether more than half of the points in the superpoint belong to the instance. ik In this case, the Hungarian algorithm is used to find the optimal match between the predicted value and the true value, obtain the panoramic segmentation result from the instance and semantic output, perform loss optimization, and finally complete the semantic, instance, and panoramic segmentation of the point cloud in the entire scene.

[0044] The beneficial effects of the present invention are:

[0045] 1. This paper innovatively proposes a unified segmentation framework based on multi-dimensional feature fusion for semantic, instance and panoptic segmentation of large-scale 3D point cloud scenes. Compared with traditional methods, this paper can more comprehensively utilize the color and spatial information of point cloud data, effectively extract and aggregate features through the combination of 3D sparse U-Net and Transformer architecture, and improve the accuracy and efficiency of segmentation.

[0046] 2. The unified segmentation framework of the present invention adopts an encoder-decoder structure. Through the optimized hybrid feature encoding component and ternary distance attention pooling, as well as the superpoint pooling layer and superpoint cross attention mechanism, the ability to capture local and global features of point clouds is significantly improved, and the model's understanding of complex three-dimensional scenes is enhanced. By introducing the concept of superpoint graph and superpoint pooling technology, the present invention can maintain sensitivity to point cloud details while reducing computational complexity, which is particularly important for the processing of large-scale point cloud data, ensuring the scalability and practicality of the algorithm.

[0047] 3. The method of the present invention has high computational efficiency while improving segmentation accuracy. Through a carefully designed network structure and loss optimization function, the present invention can process large-scale point cloud data while maintaining real-time performance, meeting the needs of real-time segmentation. It also has good generalization and scalability. By adjusting the network structure and parameters, the method of the present invention can adapt to different types of point cloud data and different application scenarios, and has high practical value and flexibility.

[0048] 4. The method of the present invention can effectively solve the problem of unified segmentation of semantics, instances and panoramas in point cloud scenes, which has important application value in the field of 3D computer vision and can provide accurate 3D environment understanding for applications such as autonomous driving, robot navigation, and urban planning. It provides a new solution for 3D point cloud data processing and promotes the development of related technical fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide further understanding of the present invention and constitute a part of this application. The illustrative examples of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 A process framework diagram for establishing the present invention;

[0051] Figure 2 A diagram of the steps for establishing the present invention;

[0052] Figure 3 This is the improved 3D sparse U-Net network structure diagram of the present invention;

[0053] Figure 4It is a structural diagram of the hybrid feature encoding component of the present invention;

[0054] Figure 5 It is a structural diagram of the hybrid bilateral reinforcement assembly of the present invention;

[0055] Figure 6 It is a structural diagram of the ternary distance attention pool of the present invention;

[0056] Figure 7 This is a visualization result diagram of semantic segmentation on a large-scale outdoor scene Toronto3D dataset for a specific embodiment;

[0057] Figure 8 The visualization results of semantic, instance and panoptic segmentation of a specific embodiment on a large-scale scene Semantic3D dataset. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0059] See also Figures 1 to 8 As shown, the method for establishing a unified segmentation framework for large-scale three-dimensional point cloud scenes based on multi-dimensional feature fusion of the present invention first preprocesses the original data, including data normalization and coordinate adjustment. The training set and the validation set are divided in an 8:2 manner, and the training set is used for training. The error between the actual output result of the training and the label value is calculated, and the difference is transferred from top to bottom through the back propagation algorithm, and the weight is updated using the weight. After the training is completed, the trained neural network model is saved, and the validation set is input to adjust the parameters to make a preliminary evaluation of the training status of the network model. Specifically, the following steps are included:

[0060] S1. Through an unsupervised algorithm, the entire point cloud is used as the input of the algorithm. Based on the k-nearest neighbor adjacency graph of the point cloud, the point cloud is divided into geometrically uniform regions. In this process, k is taken as 10, and divided into geometrically simple and meaningful shapes, called superpoints. The adjacency between superpoints is determined by calculating the symmetric Voronoi adjacency graph to calculate the superpoint graph. And extract the superpoint information.

[0061] S2. For the original point cloud data, 65,536 points are randomly sampled and input into the improved 3D sparse U-Net network to extract and construct multi-scale features of the point cloud data.

[0062] The sampled points are sent to the hybrid feature encoding component to encode the color and spatial coordinate information of each point in the point cloud, integrating spatial information and color information. The encoding formula is as follows:

[0063]

[0064] Among them, g i and It's point p i and its neighboring points The absolute (x, y, z) coordinates of i and It's point p i and its neighboring points The color information (r,g,b), It is a cascade operation, ∥·∥ represents the calculated Euclidean distance between the center point and the neighboring points, and MLP is used for sequence encoding. After MLP, the spatial information and color information of the point cloud are further combined to obtain the point-by-point features of the point cloud.

[0065]

[0066] After fusion It contains both spatial and color information. Such a design maintains local color information while maintaining local geometric structure, which helps the network to effectively extract recognizable local features and more effectively deal with local information loss caused by inhomogeneity and random downsampling.

[0067] By mixing the bilateral enhancement component, the representation of local context is further enhanced. i and its neighboring points The corresponding semantic information f i and f l k , the present invention constructs the following formula to represent the local semantic context in the feature space:

[0068]

[0069] After the MLP operation, the dimension is reduced while the highly fine-grained local semantics are highlighted. Cascade to form a new hybrid semantic variable. In order to better integrate these features, the present invention further processes the new hybrid semantic variable, optimizes its representation and accurately captures the key information of the hybrid feature, and obtains the final hybrid feature representation:

[0070]

[0071] Similarly, while strengthening the local semantic context, highly fine-grained local geometric structure and color information are introduced for local semantics, further improving the local semantic feature's ability to perceive details. The final local semantic feature is expressed as follows:

[0072]

[0073] Enhanced hybrid features With enhanced local semantic features Cascade to form mixed bilateral enhancement features, which enhance the network's ability to perceive and extract local features. The mixed bilateral enhancement features are represented as follows:

[0074]

[0075] This representation combines local geometric structure, color information and semantic features, significantly improving the network's ability to perceive and extract local features, and providing more comprehensive information for the semantic segmentation task of point clouds.

[0076] The local neighborhood is aggregated through the ternary distance attention pool to obtain the point-by-point feature representation of the point cloud; the feature weights of different points are dynamically adjusted through the attention mechanism, while the effects of spatial geometric distance, color Euclidean distance and semantic feature distance are integrated. Geometric distance Euclidean distance to color The definition is as follows:

[0077]

[0078] Among them, g(i) and g(k) are points p i and its neighboring points The spatial information of point p, c(i) and c(k) are i and its neighboring points The color information of point p, f(i) and f(k) are i and its neighboring points The semantic information of , exp is the exponential function, Mean(·) represents the average function, which is used to calculate the mean of the distance. |·| is the L1 norm, and ||·|| represents the L2 norm. These three distance values ​​are cascaded and two parameters λ1 and λ2 are used to dynamically adjust and To deal with its instability, effectively balance the influence of color information and semantic features in the overall distance calculation. The final ternary distance is expressed as:

[0079]

[0080] The values ​​of λ1 and λ2 are both 0.1. The three distance values ​​are concatenated with the hybrid bilateral enhancement features of the point cloud, and the attention score is calculated through a softmax operation. The learned attention score is multiplied with the corresponding feature as the attention weighted feature.

[0081]

[0082] The number of point clouds is gradually processed to (N→N / 4→N / 16→N / 64→N / 128), and the channel size is gradually increased to (16→32→64→256→512).

[0083] S3: Construct a network recognition model based on large-scale point cloud unified segmentation to extract super-point cross-attention features;

[0084] (1) First, use the super-point pooling layer to extract super-point features:

[0085] The super points calculated by the S1 process are combined with the point-level features calculated by the S2 process. At the same time, it is sent to the super point pooling layer, where N is the number of points and C is the dimension of the feature. The super point pooling layer obtains the super point feature by performing an average pooling operation on the point features within each super point and outputs the super point feature. Where M is the number of super points, which is usually much smaller than the number of points N, which helps improve the understanding of 3D scenes and can reduce the computing resources required when processing large-scale point cloud data.

[0086] (2) Secondly, the super-point cross-attention mechanism is introduced to generate the attention features of the super-point.

[0087] For the super-point features S′ and query vector Z calculated in step S31, they are first projected through linear transformation to facilitate attention calculation. Output of the previous layer of super-point cross attention As the query vector of the next layer, the context information is captured by the superposition point cross attention mechanism, so as to obtain the attention weight of each query vector to each super point. The formula is as follows:

[0088]

[0089] in, is the output of the super-point cross attention, is the input query vector The linear projection of, K and V are different linear projections of the super point feature S′, is the super-point attention mask, given the predicted super-point mask from the previous prediction head Use the threshold τ to filter superpoints so that the query vector only focuses on the superpoint areas above the threshold in the predicted mask. The formula is as follows:

[0090]

[0091] The i-th query vector marking the j-th superpoint, τ is set to 0.5.

[0092] S4: A decomposed matching strategy is used to train instance kernels and semantic kernels in an end-to-end manner to predict the semantic label of each point in the point cloud, and finally complete the semantic, instance and panoptic segmentation of the point cloud in the entire scene.

[0093] Given the output of super-point cross attention in S3 Two independent MLPs are used to predict the classification p for each query vector i ,i∈1,…,K ins , the true value label assignment is expressed as an optimal assignment problem, and the pairwise matching cost C is formally introduced ik To evaluate the similarity between the i-th predicted value and the k-th true value. ik Matching cost by classification probability and superpoint mask Decision, defined as follows:

[0094]

[0095] where p i,ck Indicates that the i-th predicted value belongs to c k The probability of the semantic category. cls =0.5,λ mask =0.5. Superpoint matching cost It is the sum of binary cross entropy (BCE) and Dice loss with Laplace smoothing:

[0096]

[0097] m i and are the predicted mask and ground truth mask of the superpoint, respectively. A hard instance label is assigned to each superpoint based on whether more than half of the points in the superpoint belong to the instance. ik In this case, the Hungarian algorithm is used to find the optimal match between the predicted value and the true value, obtain the panoramic segmentation result from the instance and semantic output, perform loss optimization, and finally complete the semantic, instance, and panoramic segmentation of the point cloud in the entire scene.

[0098] S5: Define the network model loss function, and train and verify the network structure of the present invention in the training set and the verification set through multiple iterations. To penalize misclassification, for each match between a prediction and a ground truth instance, the superpoint mask loss is calculated as the binary cross entropy and Dice loss The sum of semantic loss is defined as the binary cross entropy. The total loss It is expressed as:

[0099]

[0100] where β is set to 0.5.

[0101] The network structure proposed in this paper is called HFA-Net. In order to fully evaluate the effect of HFA-Net in large-scale point cloud segmentation, two mainstream large-scale point cloud datasets, Toronto3D and Semantic3D, are selected as evaluation benchmarks. Corresponding semantic segmentation, instance segmentation and panoramic segmentation are performed and the results are compared with mainstream advanced methods.

[0102] 1. Dataset Introduction

[0103] The Toronto3D dataset is a large-scale urban outdoor point cloud dataset for semantic segmentation acquired using a vehicle-mounted MLS system. The data was collected on large streets in Toronto, Canada, covering a road section of about 1 km. The entire dataset is divided into four regions, each covering about 250 meters, totaling more than 78.3 million 3D points. Each point is labeled with one of 8 semantic labels. During the training process, the author's division of training and test sets is strictly followed, using region 2 as the test set and the remaining three regions as the training set.

[0104] The Semantic3D dataset is a large-scale, high-quality dataset for semantic segmentation of point clouds. It provides a large number of natural scene point clouds, totaling more than 4 billion 3D points, and each point cloud scene contains up to 10 8 The dataset contains 3D points covering various urban scenes, including churches, streets, railways, squares, villages, football fields, and castles. These point cloud data cover an area of ​​up to 160×240×30 meters in real-world 3D space. The original 3D points in the dataset belong to 8 categories, and each point cloud data is characterized by seven features including 3D coordinates, RGB information, and intensity.

[0105] 2. Evaluation Metrics

[0106] In order to quantitatively analyze the performance of the proposed network, the overall accuracy, inter-class intersection-over-union ratio and average intersection-over-union ratio are used as evaluation indicators for semantic segmentation, as shown below

[0107]

[0108] Among them, TP, FP, FN, TN represent true positives, false positives, false negatives, and true negatives, respectively, and N represents the number of categories.

[0109] For instance segmentation, mAP is used25 、mAP 50 and mAP are used as evaluation indicators. For panoramic segmentation, panoramic quality (PQ), panoramic segmentation ability of objects (PQth) and panoramic segmentation ability of things (PQst) are used as evaluation indicators.

[0110] 3. Experimental results

[0111] Toronto3D: As shown in Table 1, our HFA-Net achieves good performance (81.12%) on mIoU, 1.1% higher than MFA. In addition, our network achieves excellent performance on six of the eight categories. Figure 7 A visual comparison of RandLA-Net, BAAF-Net, SCF-Net, and the proposed HFA-Net on the Toronto3D dataset is shown to intuitively compare different methods. The red boxes indicate the parts where the segmentation is inaccurate or the boundary segmentation is not obvious in RandLA-Net, BAAF-Net, and SCF-Net. The overall experimental results show that HFA-Net has better performance than other methods, mainly because the network improves the ability of local neighbors to capture semantically similar points. In addition, the proposed method has a better segmentation ability for points that are close in location but different in semantics.

[0112] Table 1: Quantitative results on the Toronto3D dataset

[0113]

[0114] Semantic3D: As shown in Table 2, the HFA-Net of the present invention has achieved good segmentation effects in the corresponding segmentation results, and has achieved the best level compared with other segmentation methods.

[0115] Table 2: Quantitative results on the Semantic3D dataset

[0116]

[0117]

[0118] Figure 8 The results of semantic, instance and panoramic segmentation of the present invention on Semantic3D are demonstrated. It can be seen that the design of the present invention can complete the three segmentation tasks well and achieve an excellent level.

[0119] See also Figure 7 The visualization result of semantic segmentation on the large-scale outdoor scene Toronto3D dataset is shown in the figure. The present invention uses red boxes to highlight the segmentation details. The network of the present invention performs well in accurately segmenting objects such as road signs and buildings. Figure 7 As shown in the example in the red box in the upper right corner, it can effectively identify and segment each individual grid in the road sign. In addition, the network of the present invention can effectively distinguish large areas of building structures and natural vegetation, as shown in the example in the red box at the bottom. This function fully demonstrates the robustness and high accuracy of the network in handling complex scene segmentation tasks.

[0120] See also Figure 8 As shown in the visualization results of semantic, instance and panoramic segmentation on the large-scale scene Semantic3D dataset, the structure of the present invention only needs one training to complete semantic, instance and panoramic segmentation and achieves excellent segmentation results.

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made to the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a unified segmentation framework for large-scale three-dimensional point cloud scenes, characterized by: The following steps are involved: S1, extract super point information; S2, extract and construct multi-scale features of point cloud data; S3, group homogeneous adjacent points into super-point features and output the learned kernels through the super-point cross-attention mechanism; S4, using the decomposition matching strategy to train the instance kernel and predict the semantic label of each point in the point cloud; The step S2 of extracting and constructing multi-scale features of point cloud data is as follows: Extract the super-point information of the point cloud; extract the local features of the point cloud through the hybrid feature encoding component; first encode the color and spatial coordinate information of each point in the point cloud, and fuse the spatial information with the color information; the encoding formula is as follows: Among them, g i and It's point p i and its neighboring points The absolute (x, y, z) coordinates of i and It's point p i and its neighboring points The color information (r, g, b) of the point cloud is obtained by MLP. ⊕ is a cascade operation. ∥·∥ represents the calculated Euclidean distance between the center point and the neighboring points. MLP is used for sequence encoding. The spatial information and color information of the point cloud are further combined by MLP, and a deeper mapping transformation is performed to obtain the mixed features of the point cloud. By mixing the bilateral enhancement components, the representation of local context is further enhanced; for each point p i and its neighboring points The corresponding semantic information f i and The following formula is constructed to represent the local semantic context in the feature space: After the MLP operation, the dimension is reduced while highlighting the local semantics of high fine-grainedness; then Cascading is performed to form a new mixed semantic variable, and the final mixed feature representation is obtained: Similarly, while strengthening the local semantic context, highly fine-grained local geometric structure and color information are introduced for local semantics, further improving the local semantic feature's ability to perceive details. The final local semantic feature is expressed as follows: Enhanced hybrid features With enhanced local semantic features Cascade to form mixed bilateral enhancement features, which enhance the network's ability to perceive and extract local features; the mixed bilateral enhancement features are expressed as follows: The local neighborhood is aggregated through the ternary distance attention pool to obtain the point-by-point feature representation of the point cloud; the feature weights of different points are dynamically adjusted through the attention mechanism, and the influence of spatial geometric distance, color Euclidean distance and semantic feature distance are integrated at the same time; semantic feature distance Geometric distance Euclidean distance to color The definition is as follows: Among them, g(i) and g(k) are points p i and its neighboring points The spatial information of point p, c(i) and c(k) are i and its neighboring points The color information of point p, f(i) and f(k) are i and its neighboring points The semantic information of , exp is the exponential function, Mean(·) represents the average function, which is used to calculate the mean of the distance; |·| is the L1 norm, and ||·|| represents the L2 norm; these three distance values ​​are cascaded and two parameters λ1 and λ2 are used to dynamically adjust and To deal with its instability, effectively balance the influence of color information and semantic features in the overall distance calculation; the final ternary distance is expressed as: The values ​​of λ1 and λ2 are both 0.1; the three distance values ​​are cascaded with the hybrid bilateral enhancement features of the point cloud, and the attention score is calculated through the softmax operation; the learned attention score is multiplied with the corresponding feature as the attention weighted feature; The super-point information and point-by-point features are grouped into super-point features through the super-point pooling layer as the key and value of the decoder. At the same time, a set of learnable semantic and instance queries are randomly initialized as input. The three capture point cloud information through the super-point cross-attention mechanism. The decomposition matching strategy is adopted to predict the semantic label of each point in the point cloud in an end-to-end training manner, and finally the semantic, instance and panoramic segmentation of the point cloud in the entire scene are completed.

2. The method for constructing a unified segmentation framework for large-scale three-dimensional point cloud scenes according to claim 1, characterized in that: The step S1 of extracting superpoint information is as follows: Through an unsupervised algorithm, the entire point cloud is used as the input of the algorithm. Based on the k-nearest neighbor adjacency graph of the point cloud, the point cloud is divided into geometrically uniform regions and segmented into geometrically simple and meaningful shapes, called super points, by solving an optimization problem; The adjacency between superpoints is determined by calculating the symmetric Voronoi adjacency graph of the input point cloud to calculate the superpoint graph and extract the superpoint information of the point cloud.

3. The method for constructing a unified segmentation framework for large-scale three-dimensional point cloud scenes according to claim 1, characterized in that: Step S3 groups homogeneous adjacent points into super-point features, and outputs the learned kernels through the super-point cross-attention mechanism, specifically: Point-level features for superpoint computation At the same time, it is sent to the super point pooling layer, where N is the number of points and C is the dimension of the feature; The superpoint pooling layer obtains superpoint features by performing an average pooling operation on the point features within each superpoint; this means that each superpoint consists of a set of geometrically adjacent and homogeneous points, and the features of these points are aggregated to represent the features of the entire superpoint, outputting the superpoint features. Where M is the number of superpoints, which is usually much smaller than the number of points N; For the super-point feature S′ and the query vector Z, they are first projected by linear transformation to facilitate attention calculation; the output of the previous layer of super-point cross attention As the query vector of the next layer, the context information is captured by superimposing the point cross attention mechanism, so as to obtain the attention weight of each query vector to each super point; The formula is as follows: in, is the output of the super-point cross attention, is the input query vector The linear projection of, K and V are different linear projections of the super point feature S', is the super-point attention mask, given the predicted super-point mask from the previous prediction head Use the threshold τ to filter superpoints so that the query vector only focuses on the superpoint areas above the threshold in the predicted mask; the formula is as follows: The i-th query vector marking the j-th superpoint, τ is set to 0.

5.

4. The method for constructing a unified segmentation framework for large-scale three-dimensional point cloud scenes according to claim 1, characterized in that: The decomposition matching strategy described in step S4 is used to train the instance kernel to predict the semantic label of each point in the point cloud, specifically: Given the output of super-point cross attention Two independent MLPs are used to predict the classification p for each query vector i ,i∈1,…,K ins , the true value label assignment is expressed as an optimal assignment problem, and the pairwise matching cost C is formally introduced ik To evaluate the similarity between the i-th predicted value and the k-th true value; C ik Matching cost by classification probability and superpoint mask Decision, defined as follows: where p i,ck Indicates that the i-th predicted value belongs to c k The probability of semantic category; λ cls =0.5,λ mask =0.5; super point matching cost It is the sum of binary cross entropy BCE and Dice loss with Laplace smoothing: m i and are the predicted mask and ground truth mask of the superpoint respectively; a hard instance label is assigned to each superpoint according to whether more than half of the points in the superpoint belong to the instance. ik In this case, the Hungarian algorithm is used to find the optimal match between the predicted value and the true value, obtain the panoramic segmentation result from the instance and semantic output, perform loss optimization, and finally complete the semantic, instance, and panoramic segmentation of the point cloud in the entire scene.