Three-dimensional point cloud panoramic segmentation method based on deep learning and uncertainty estimation, computer device and storage medium
By incorporating deep learning and uncertainty estimation methods into a loss function for probabilistic modeling and optimizing network output, the problem of decoupling embedding learning and clustering algorithms is solved, achieving efficient and accurate 3D point cloud panoramic segmentation.
Patent Information
- Application Number
- CN202310019910.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-06
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2043-01-06
AI Technical Summary
In existing 3D point cloud panoramic segmentation technologies, the embedding learning process is uncertain, making it difficult to learn the embedding features. The clustering algorithm is decoupled from the network learning process, which cannot effectively optimize the network output, resulting in insufficient resolution.
We employ a deep learning-based approach and uncertainty estimation. By incorporating the uncertainty estimation process into the loss function for probabilistic modeling, and combining the uncertainty estimation module with a fully connected layer, we optimize the network output to achieve the matching of embedded features and prototypes.
This reduces the impact of uncertainty on network optimization, achieves an efficient and accurate clustering method, avoids the decoupling of embedding learning and clustering algorithms, and improves segmentation results.
Smart Images
Figure CN115953581B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of point cloud data processing technology, and in particular to a three-dimensional point cloud panoramic segmentation method, computer device, and storage medium based on deep learning and uncertainty estimation. Background Technology
[0002] Panoptic segmentation of 3D point clouds is crucial for 3D scene understanding. Its goal is to obtain the semantic label category for each point in a given point cloud scene, and to provide the instance number for points belonging to countable objects within that category of interest. Panoptic segmentation unifies the tasks of semantic point cloud segmentation and instance point cloud segmentation, providing detailed and complete understanding of point cloud scenes. It currently has wide applications in fields such as autonomous driving and smart cities.
[0003] Current 3D point cloud panoramic segmentation technology mainly suffers from the following problems:
[0004] Group-based methods use the distance between the network's output embedding and the corresponding prototype as a constraint for embedding learning to obtain discriminative embedding features and segment the corresponding point clouds representing these features. The loss functions used in embedding learning primarily employ distance-dependent L1 and L2 norms. The L1 norm is insensitive to distance, making network convergence difficult; the L2 norm is affected by inherent object uncertainties, producing excessively large gradients for large or noisy objects. Existing embedding learning loss functions cannot effectively optimize the network to obtain discriminative embedding feature outputs.
[0005] Due to the inherent uncertainty in embedding learning, it is difficult to learn accurate embedding features. Therefore, existing methods design clustering algorithms to cluster the embeddings for segmentation. However, the clustering algorithm is not considered in the network learning process; that is, embedding learning and clustering algorithms are decoupled, and it is impossible to guarantee through the deep learning process that the network output is suitable for the clustering algorithm.
[0006] Terminology Explanation:
[0007] Deep learning originates from research on artificial neural networks. It discovers distributed feature representations of data by combining low-level features to form more abstract high-level representations of attribute categories or features.
[0008] Uncertainty estimation: Uncertainty in deep learning frameworks can be categorized into random uncertainty and cognitive uncertainty by definition. Random uncertainty stems from the inherent uncertainty of the data itself, such as inherent noise and labeling errors. Cognitive uncertainty arises from the uncertainty generated during the learning process of the neural network, such as the randomness of parameter initialization and the randomness of the optimization process. Uncertainty estimation, then, involves measuring either random or cognitive uncertainty through certain methods.
[0009] 3D point cloud: Originating from 3D vision, it refers to the points on the surface of a target obtained by scanning and measuring objects in a scene. This massive collection of points is called a point cloud.
[0010] Panoramic segmentation is a combination of semantic segmentation and instance segmentation. It aims to segment the target at the instance level and the background content at the semantic level simultaneously. It assigns a category label and instance ID to each pixel in the input image to generate a global, uniform segmented image.
[0011] Semantic segmentation: Semantic segmentation is a fundamental task in computer vision. In semantic segmentation, we need to divide visual input into different semantically interpretable categories.
[0012] Instance segmentation: Instance segmentation is a fundamental task in computer vision. In instance segmentation, we need to assign the instance number to the visual input of a specific countable object.
[0013] Region proposal: Originating from computer vision. In computer vision, a common framework is to first obtain a series of region proposals through neural networks or other non-learning methods, and then use techniques such as nonmaximum suppression to select the required proposals to complete the visual task.
[0014] Non-maximum suppression: If the intersection-union ratio of two proposals is higher than a certain manually set hyperparameter, i.e., a threshold, then the proposal with the lower score will be deleted.
[0015] Hyperparameters: a term from deep learning, referring to parameters that are set before the model runs. Hyperparameters are not obtained through algorithms and formulas, but rather through experience.
[0016] Intersection-Union Ratio (IUR): This refers to the ratio of the intersection to the union of two sets. In computer vision, it also refers to the ratio of the intersection to the union of two regions.
[0017] Object detection: Object detection is a fundamental task in computer vision. In object detection, we need to output the detection box of a specific object in the visual input and the semantically interpretable category to which the object represented by each detection box belongs.
[0018] Embedding: a term from deep learning, refers to reducing the dimensionality of high-dimensional features to obtain low-dimensional but discriminative feature vectors.
[0019] Prototype: Derived from deep learning terminology, it refers to an expression that symbolizes something, which can be a feature vector or a specific attribute of something.
[0020] Top-view projection: refers to projecting three-dimensional spatial data onto a horizontal plane along the height direction.
[0021] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computations and has a deep structure. They are one of the representative algorithms of deep learning.
[0022] Prediction center bias: For each structured data, its prediction center is the center of its instance, while the prediction center bias refers to the offset of the structured data's position center in the structure minus the prediction center.
[0023] A heatmap is a two-dimensional image where each point has a channel representing a heat value. In the center point prediction task, the predicted heatmap is generated by a neural network, constrained to the true heatmap by the L2 norm. The true heatmap uses the instance center as the central peak of a Gaussian distribution, while other locations within a fixed radius are sampled from a Gaussian distribution with a pre-defined variance. The peak center of the true heatmap represents the instance center.
[0024] Peak search: This refers to searching for the maximum value within a fixed region of a heatmap, where the size of the fixed region is a predefined hyperparameter. The input is a heatmap, and the output is the coordinates of the peak value within the heatmap.
[0025] Clustering: The process of dividing a collection of physical or abstract objects into multiple classes composed of similar objects is called clustering. A cluster generated by clustering is a set of data objects that are similar to objects in the same cluster and different from objects in other clusters.
[0026] Backpropagation: Originating from deep learning terminology, it is an important optimization method for feedforward neural networks.
[0027] Gradient: The gradient is a vector representing the directional derivative of a function at a given point, where the directional derivative reaches its maximum value along that direction. In other words, the function changes most rapidly and at its maximum rate of change (the magnitude of the gradient) along that direction. In neural networks, the gradient is also used for backpropagation and network optimization. It is obtained by differentiating the loss function with respect to the network output, yielding gradient information about the output. This gradient is then calculated from the last layer forward and propagated to each layer that needs optimization.
[0028] Masking: refers to using a set of matrices or vectors to represent whether a selected area is selected, so that subsequent operations only affect the selected area.
[0029] Oversampling: A data augmentation technique that involves sampling data from less frequently occurring categories multiple times to balance the frequency of training different categories.
[0030] Cross-entropy is an important concept in Shannon's information theory, mainly used to measure the difference between two probability distributions.
[0031] Lovasz-Softmax: A loss function for semantic segmentation, derived from the conference paper "The Lov'asz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks".
[0032] Fully connected layer: Originating from deep learning terminology, it refers to a network structure where each node is connected to all nodes in the previous layer, used to synthesize the features extracted earlier.
[0033] Convolution is a mathematical operation that generates a third function from two functions f and g. It is essentially a special integral transformation that represents the integral of the product of the overlapping function values of functions f and g after flipping and translation with respect to the overlap length.
[0034] 3D convolution: Convolution operations performed in a 3D structure.
[0035] Sparse convolution: A type of convolution that produces the same result as convolution but reduces inference overhead when used with sparse matrices.
[0036] Submanifold convolution: Convolution is performed only at the positions where values exist in the sparse matrix. After submanifold convolution, the sparsity of the output and input matrices remains unchanged. Summary of the Invention
[0037] To address the inherent uncertainties and other technical problems in current embedded learning, the present invention aims to provide a three-dimensional point cloud panoramic segmentation method, computer device, and storage medium based on deep learning and uncertainty estimation.
[0038] On one hand, embodiments of the present invention include a 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation, comprising:
[0039] Acquire point cloud data;
[0040] The point cloud data is subjected to data normalization processing to obtain prototype points;
[0041] Structured features are extracted from the point cloud data;
[0042] Semantic segmentation is performed based on the structured features to obtain point cloud semantic labels; embedding prediction is performed based on the structured features to obtain embedding features; prototype prediction is performed based on the prototype points to obtain prototype prediction points.
[0043] Uncertainty estimation is performed based on the prototype prediction points and the embedded features to obtain prototype correction points and covariance matrices; the covariance matrix represents the Gaussian distribution covariance matrix corresponding to the prototype correction points as the Gaussian distribution center of the point cloud data.
[0044] Based on the prototype correction point and the covariance matrix, probability modeling is performed to obtain the average probability matrix;
[0045] Based on the average probability matrix, a corresponding instance label is assigned to the points in the point cloud data;
[0046] Based on the Gaussian distribution to which the points in the point cloud data belong, assign corresponding point cloud semantic labels to the points in the point cloud data.
[0047] Furthermore, the acquisition of point cloud data includes:
[0048] The scene is scanned using a LiDAR scanner to obtain the point cloud data of the current scene.
[0049] Further, the step of performing data normalization processing on the point cloud data to obtain prototype points includes:
[0050] Obtain the instance label corresponding to each frame of the point cloud scene in the point cloud data;
[0051] Statistical analysis is performed on the instance labels corresponding to each frame of the point cloud scene to obtain a set of non-overlapping labels;
[0052] Based on the set of labels, the center of each instance in the point cloud data is calculated as the prototype point.
[0053] Further, the step of performing uncertainty estimation based on the prototype prediction points and the embedded features to obtain the prototype correction points and covariance matrix includes:
[0054] After merging the prototype prediction points with the structured features, the first feature vector is obtained by processing through two fully connected layers.
[0055] After merging the first feature vector with the embedded feature, the second feature vector is obtained by processing through two fully connected layers.
[0056] The second feature vector closest to each prototype point is subjected to mean pooling to obtain instance-level feature information;
[0057] The instance-level feature information is processed by three fully connected layers to obtain the prototype correction point and the covariance matrix.
[0058] Furthermore, the formula for the covariance matrix is as follows: Among them, C i Let θ be the covariance matrix corresponding to the i-th Gaussian distribution. i Let σ be the rotation angle. i Let k be the minor axis variance. i This represents the variance ratio of the major axis and the minor axis.
[0059] Further, the step of performing probability modeling based on the prototype correction point and the covariance matrix to obtain the average probability matrix includes:
[0060] For the m-th Gaussian distribution of the point cloud data, the corresponding average probability matrix is:
[0061]
[0062] Among them, P nm Let P be the average probability matrix. nm This indicates that the embedding feature I is embedded in the m-th Gaussian distribution. P The average probability, M n C is the mask corresponding to the nth instance in the point cloud data. m Let m be the covariance matrix corresponding to the m-th Gaussian distribution. Let m be the prototype correction point corresponding to the m-th Gaussian distribution. Let j be the j-th embedded feature, and T denote the matrix transpose operation.
[0063] Further, assigning a corresponding instance label to the points in the point cloud data based on the average probability matrix includes:
[0064] Based on the average probability matrix, determine the Gaussian distribution to which the embedding features of the point representations in the point cloud data most likely belong;
[0065] The instance label corresponding to the most likely Gaussian distribution is assigned to the corresponding point in the point cloud data.
[0066] Further, assigning corresponding point cloud semantic labels to the points in the point cloud data according to the Gaussian distribution to which the points in the point cloud data belong includes:
[0067] For all Gaussian distributions corresponding to the same instance label, determine the point cloud semantic label corresponding to each Gaussian distribution, and assign the point cloud semantic label with the largest number of such labels to all Gaussian distributions corresponding to the same instance label.
[0068] On the other hand, embodiments of the present invention also include a computer device, including a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to execute the three-dimensional point cloud panoramic segmentation method based on deep learning and uncertainty estimation in the embodiments.
[0069] On the other hand, embodiments of the present invention also include a storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the three-dimensional point cloud panoramic segmentation method based on deep learning and uncertainty estimation in the embodiments.
[0070] The beneficial effects of this invention are as follows: The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation in the embodiments captures the cognitive uncertainty generated in network learning and the inherent random uncertainty of data through a unique uncertainty estimation process. In the uncertainty estimation process, uncertainty is integrated into the loss function calculation process, which can reduce the impact of uncertainty on network optimization and obtain discriminative embedded feature outputs. By probabilistically modeling the network output through uncertainty, the probability of matching between prototypes and embeddings is obtained and used to calculate the degree of matching between the network prediction results and the actual results. This avoids the decoupling of clustering algorithms and embedding learning, and also avoids the mismatch between prototype learning and embedding learning. By constructing probabilistic models through uncertainty, the probability of matching between prototypes and embeddings is obtained. In the inference stage, the relevant point cloud representing each embedding feature is directly assigned to the prototype with the highest matching probability, achieving an efficient and accurate clustering method. Attached Figure Description
[0071] Figure 1 This is a schematic diagram illustrating the steps of the 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation in the embodiment.
[0072] Figure 2 This is a schematic diagram illustrating the principle of the 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation in the embodiment. Detailed Implementation
[0073] Existing 3D point cloud panoramic segmentation methods can be categorized into region proposal-based and group-based methods based on their instance numbering assignment mechanisms. Region proposal-based methods assign instance numbers to bounding boxes obtained from object detection, unifying these with the point-by-point semantically interpretable categories obtained from semantic segmentation. For each point within a bounding box, its instance number matches the bounding box number, and the point-by-point semantically interpretable category is selected based on the semantic category with the most frequent occurrence among them. Since the non-maximum suppression threshold of the bounding boxes is not zero, partial overlap is allowed. Multiple instance numbers may be assigned to overlapping 3D point clouds. However, for panoramic segmentation tasks, each visual input has only one instance number. Therefore, to avoid instance number conflicts, group-based methods are more popular. Group-based methods constrain the embeddings of 3D point clouds to be close to instance prototypes, and then use clustering algorithms to group the embeddings, assigning a unique instance code to 3D point clouds within the same group.
[0074] Based on the above background, in this embodiment, referring to Figure 1 The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation includes the following steps:
[0075] S1. Acquire point cloud data;
[0076] S2. Perform data normalization processing on the point cloud data to obtain prototype points;
[0077] S3. Extract structured features from point cloud data;
[0078] S4. Perform semantic segmentation based on structured features to obtain point cloud semantic labels; perform embedding prediction based on structured features to obtain embedding features; perform prototype prediction based on prototype points to obtain prototype prediction points.
[0079] S5. Based on the prototype prediction points and embedded features, perform uncertainty estimation to obtain the prototype correction points and covariance matrix; the covariance matrix represents the Gaussian distribution covariance matrix when the prototype correction points are used as the Gaussian distribution center of the point cloud data.
[0080] S6. Perform probability modeling based on the prototype correction points and covariance matrix to obtain the average probability matrix;
[0081] S7. Assign a corresponding instance label to the points in the point cloud data based on the average probability matrix;
[0082] S8. Assign corresponding point cloud semantic labels to the points in the point cloud data according to the Gaussian distribution to which the points belong.
[0083] Steps S1-S2 are the data generation stage.
[0084] In step S1, the scene is scanned using a LiDAR to obtain point cloud data of the current scene. The point cloud data includes at least the three-dimensional coordinate information, semantic labels, and instance labels of each point in the point cloud. If the LiDAR supports this, additional color information, reflection intensity information, and time information from the scan frame can be added to the point cloud data.
[0085] In step S2, the point cloud data is normalized to facilitate training and testing. Specifically, the data normalization process includes the following steps: First, the point cloud data is decomposed into basic point cloud information and feature information. The basic point cloud information includes 3D coordinate information, feature information (including supplementary feature information), semantic labels S, and instance labels (or instance numbers) I. Then, the instance labels I of each frame of the point cloud scene are statistically analyzed to obtain a non-overlapping label set U. Using the label set U, the center of each instance in the point cloud data is calculated as the prototype point O, and a non-overlapping prototype set UO and the mask M of the instance corresponding to UO are obtained. Simultaneously, based on the prototype set UO, a Gaussian-modeled prototype heatmap distribution is generated on a blank map, where the center point is the prototype point in the prototype set UO. The Gaussian distribution is sampled within a set radius according to a preset variance to generate a realistic heatmap H.
[0086] In step S2, after normalizing the point cloud data, a dataset can be created. Specifically, the normalized point cloud data is divided according to a certain ratio to obtain a training set, a validation set, and a test set.
[0087] In step S2, for the data divided into the training set, several data augmentations can be randomly performed during training, including the following:
[0088] 1. Randomly rotate the point cloud coordinates along the z-axis within the range.
[0089] 2. For the point cloud coordinate information, perform random coefficient scaling on the xy plane within the range [0.95, 1.05].
[0090] 3. Gaussian noise with a variance of 0.1 was added to the point cloud coordinate information.
[0091] 4. Randomly flip the point cloud coordinate information in the xy plane.
[0092] 5. Employ an oversampling strategy to increase the sampling frequency for categories with only a small number of samples.
[0093] Steps S3-S8 constitute the model training and inference phases. In this embodiment, the principles of the model training and inference phases are as follows: Figure 2 Place
[0094] Reference Figure 2 In step S3, the point cloud data obtained in step S2 is input into the feature extraction network. The feature extraction network uses a 3D scene point cloud feature extraction network to extract features from the point cloud data, obtaining structured features P for subsequent tasks. The feature extraction network used for segmentation tasks typically employs an encoder-decoder structure. In a 3D scene, the feature extraction network can use a method based on 3D sparse sub-stream convolution, or project the data onto a 2D plane and then use a 2D convolution method for feature extraction.
[0095] In step S4, refer to Figure 2 The structured features P obtained from the feature extraction network are input into the semantic segmentation network for semantic segmentation. The semantic segmentation network outputs a point cloud semantic label S for each structured feature. P .
[0096] In step S4, refer to Figure 2 The structured features P obtained from the feature extraction network are input into the embedding prediction network for embedding prediction. The embedding prediction network outputs the embedding feature I for each structured feature. P .
[0097] In step S4, refer to Figure 2 The structured features P obtained from the feature extraction network are input into the prototype prediction network for prototype prediction. The prototype prediction network generates a heatmap H of the prototype prediction corresponding to the structured features P. P Then, the prototype prediction point O is obtained through the peak search algorithm. P .
[0098] When training semantic segmentation networks, embedding prediction networks, and prototype prediction networks, L1 norm constraints can be used to constrain instance feature I and embedding feature I. P Using L2 norm constraints, the actual heatmap H and the prototype predicted heatmap H are compared. P Using cross-entropy and Lovasz-Softmax to constrain semantic label S and point cloud semantic label S P .
[0099] In step S5, refer to Figure 2 Using the uncertainty estimation module, predict point O based on the prototype. P and embedded features I P Uncertainty estimation is performed to obtain the prototype correction point O. P′ The covariance matrix C.
[0100] In step S5, the uncertainty estimation module performs the following steps: first, it sets the prototype prediction point O... PThe prototype prediction result and the structured feature P are merged and fed into two fully connected layers in the uncertainty estimation module to obtain a first feature vector containing the corresponding prototype information; then the first feature vector obtained in the previous step is combined with the embedded feature I. P The (embedded prediction results) are merged and fed into two fully connected layers in the uncertainty estimation module to perceive the network's learning state, thus obtaining a second feature vector containing the corresponding embedding information. The second feature vector (embedded feature vector) closest to each prototype is then subjected to mean pooling to obtain instance-level feature information. This information is then passed through three fully connected layers in the uncertainty estimation module to output the covariance matrix C and the prototype correction point O. P′ Among them, the prototype correction point O P′ The significance of the point cloud data is the center of the Gaussian distribution, and the significance of the covariance matrix C is the correction of the prototype point O. P′ Given that the Gaussian distribution center of the point cloud data is the center of the point cloud data, the Gaussian distribution covariance matrix of each predicted instance of the uncertainty estimation result.
[0101] In this embodiment, the uncertainty estimation module may not directly output the covariance matrix C itself, but instead output θ. i σ i and k i These three parameters form the covariance matrix C. For the i-th predicted Gaussian distribution in the point cloud data, the uncertainty estimation module outputs a rotation angle θ. i Minor axis variance σ i And the variance ratio k of the major and minor axes i These three parameters are used to represent the covariance matrix corresponding to the i-th Gaussian distribution:
[0102]
[0103] Modeling using a Gaussian distribution and assuming that the distributions of the embeddings are independent, we obtain the set of embeddings belonging to the i-th instance. Regarding the i-th prototype correction point and the i-th covariance matrix C i The probability is:
[0104]
[0105] Among them, M i Let I be the mask corresponding to the i-th instance in the point cloud data. i P Let be the i-th embedded feature, and T denote the matrix transpose operation.
[0106] Probability can be used during the training phase. The negative log-likelihood is used as a loss function to constrain the predicted covariance matrix and prototype, and it also serves as a loss function for embedding learning, thus optimizing the network more effectively under uncertainty modeling.
[0107] In this embodiment, the loss function for embedding learning is:
[0108]
[0109] Where, N P This represents the number of instances in the scene.
[0110] In this embodiment, the uncertainty estimation module can perform uncertainty estimation for embedding learning in point cloud panorama segmentation; the uncertainty estimation module can aggregate instance-level features from point-level features and output a Gaussian distribution through multiple fully connected layers to approximate the true embedding distribution; the uncertainty loss L used by the uncertainty estimation module... um This approach addresses the challenges of distance metric loss, reducing the interference of uncertainty on network optimization while simultaneously measuring the importance of the optimization objective for a more effective optimization process. Uncertainty estimation is applied to a clustering algorithm for point cloud panorama segmentation, and probabilistic loss directly optimizes the matching degree between predicted and actual results, overcoming the mismatch between embedding learning and prototype learning. When combining probabilistic modeling using uncertainty estimation with clustering methods, the network optimization objective is ensured to be directly correlated with the final result, resolving the decoupling issue between embedding learning and clustering algorithms.
[0111] In step S6, the prototype correction point O is used. P′ Probabilistic modeling is performed using the covariance matrix C to obtain the average probability matrix. Specifically, the m-th Gaussian distribution of the point cloud data (i.e., the center point is the prototype correction point) is calculated. The covariance matrix is C m Gaussian distribution) about the nth real instance (mask M) n Embedded features I in the set of points represented P The average probability is obtained by the average probability matrix.
[0112]
[0113] Among them, P nm Let P be the average probability matrix. nm This indicates that the m-th Gaussian distribution contains embedded feature I. P The average probability, C m Let m be the covariance matrix corresponding to the m-th Gaussian distribution. Let m be the prototype correction point corresponding to the m-th Gaussian distribution. Let j be the j-th embedded feature, and T denote the matrix transpose operation.
[0114] Using the Hungarian algorithm, for a matrix P representing the average probability... nm As the cost matrix, the optimal matching scheme R is calculated. And the probabilistic loss is constructed:
[0115]
[0116] Probability loss L prob In the middle, P nm The term can be viewed as the matching probability between the m-th predicted result and the corresponding n-th true result, while the denominator can be viewed as the matching probability between all predicted results and the n-th true result. Together, the fractions express the degree of matching between the m-th predicted result and the corresponding n-th true result. By minimizing the probability loss, the matching degree between each predicted instance and its corresponding instance is maximized, directly linking the optimization objective with the final result to obtain better segmentation results (while simultaneously addressing the mismatch between prototype learning and embedding learning).
[0117] In step S7, refer to Figure 2 During the inference phase, each distribution in probabilistic modeling is assigned a unique instance label (instance number), and each point is assigned the instance number of the distribution to which its represented embedding most likely belongs. Therefore, based on the average probability matrix P... nm Assign a corresponding instance label to the points in the point cloud data.
[0118] Specifically, in step S7, firstly, based on the average probability matrix, the embedding features I of the point representations in the point cloud data are determined. P The most likely Gaussian distribution; the embedding feature I for this point representation in the point cloud data. P Find the most likely Gaussian distribution, locate the corresponding instance label, and assign the found instance label to this point in the point cloud data.
[0119] In step S8, refer to Figure 2 In point cloud data, the point cloud semantic labels of each point with the same instance label (instance number) should also be the same. For instances with the same instance label (instance number), a majority voting strategy can be adopted to assign the point cloud semantic label with the largest number of instances to all points that have that instance number.
[0120] Specifically, in step S8, firstly, for all Gaussian distributions corresponding to the same instance label I, the point cloud semantic label S corresponding to each Gaussian distribution is determined. P ; the semantic label S of the point cloud corresponding to each Gaussian distribution P Not entirely the same, meaning there are multiple point cloud semantic labels S PIn the case of each point cloud semantic label S, respectively, the semantic labels S are statistically analyzed. P The number of points cloud semantic labels S with the largest number are selected. P This will generate the most numerous point cloud semantic labels S P , and assign to all Gaussian distributions corresponding to the instance label I (i.e., assign to points in the point cloud data that may belong to these Gaussian distributions).
[0121] By executing step S8, the instance label (instance number) and point cloud semantic label of each point in the point cloud data are unified, thus achieving the result of panoramic segmentation.
[0122] In the 3D point cloud panoramic segmentation technology implemented by this method, the uncertainty of the network output is modeled, thereby reducing the impact of uncertainty on embedding learning and overcoming the decoupling problem between embedding learning and clustering algorithms.
[0123] The principle behind steps S1-S8 is as follows:
[0124] 1. Through a unique uncertainty estimation process, cognitive uncertainty and inherent random uncertainty of data generated in network learning are captured; in the uncertainty estimation process, uncertainty is integrated into the loss function calculation process, which can reduce the impact of uncertainty on network optimization and obtain discriminative embedded feature output;
[0125] 2. By probabilistically modeling the network output through uncertainty, the probability of matching between the prototype and the embedding is obtained and used to calculate the degree of matching between the network's prediction and the actual results. A global, unified loss function is used to optimize this degree of matching, incorporating the clustering algorithm into the network optimization. This avoids the decoupling of the clustering algorithm and embedding learning, and also avoids the mismatch between prototype learning and embedding learning.
[0126] 3. By constructing probabilistic models based on uncertainty, the probability of matching between prototypes and embeddings is obtained. During the inference phase, the relevant point clouds representing each embedding feature are directly assigned to the prototype with the highest matching probability, achieving an efficient and accurate clustering method;
[0127] 4. By using probabilistic modeling based on uncertainty estimation for segmentation, compared to using only distance metrics, this invention can provide higher human tolerance for directions and locations with high uncertainty, thus enabling more detailed and high-quality segmentation.
[0128] 5. Using probability loss to directly optimize the matching degree between prediction results and true results can overcome the mismatch problem between embedding learning and prototype learning, as well as the decoupling problem between embedding learning and clustering algorithms, and achieve high recognition quality.
[0129] 6. The present invention can capture the inherent uncertainties of network output and objects, provide uncertainty reasoning results, and provide corresponding modeling; for example, when the three-dimensional point cloud panoramic segmentation method in this embodiment is applied to a real autonomous driving scenario, uncertainty can assist the system in making decision-making judgments and avoid the occurrence of accidents.
[0130] Based on steps S1-S8, validation is performed on the validation set after each training epoch on the training set. In this embodiment, data such as panoptic quality and average intersection-over-union ratio are used to evaluate the execution results of steps S1-S8.
[0131]
[0132] Panoramic quality can be further broken down into segmentation quality and recognition quality. The model with the highest panoramic quality is saved for testing and use.
[0133] During testing, you can test on the test set using the best results from each training session.
[0134] A computer program that executes the 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation in this embodiment can be written into a storage medium or computer device. When the computer program is read out and run, the 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation in this embodiment is executed, thereby achieving the same technical effect as the 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation in the embodiment.
[0135] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. Furthermore, the descriptions of "upper," "lower," "left," and "right" used in this disclosure are only relative to the relative positional relationships of the various components of this disclosure in the accompanying drawings. The singular forms "a," "described," and "the" used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. Moreover, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this embodiment specification is only for describing particular embodiments and is not intended to limit the invention. The term "and / or" as used in this embodiment includes any combination of one or more of the associated listed items.
[0136] It should be understood that although the terms first, second, third, etc., may be used to describe various elements in this disclosure, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, a first element may also be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may also be referred to as a first element. The use of any and all instances or exemplary language (“e.g.,” “such as,” etc.) provided in this embodiment is intended only to better illustrate embodiments of the invention and, unless otherwise required, does not impose a limitation on the scope of the invention.
[0137] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can be implemented using standard programming techniques—including a non-transitory computer-readable storage medium configured with a computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).
[0138] Furthermore, the procedures described in this embodiment can be performed in any suitable order unless otherwise indicated by this embodiment or clearly contradicted by the context. The procedures (or variations and / or combinations thereof) described in this embodiment can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.
[0139] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described in this embodiment includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention also includes the computer itself.
[0140] A computer program can be applied to input data to perform the functions described in this embodiment, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.
[0141] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.
Claims
1. A 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation, characterized in that, The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation includes: Acquire point cloud data; The point cloud data is subjected to data normalization processing to obtain prototype points; Structured features are extracted from the point cloud data; Semantic segmentation is performed based on the structured features to obtain point cloud semantic labels; embedding prediction is performed based on the structured features to obtain embedding features; prototype prediction is performed based on the prototype points to obtain prototype prediction points. Uncertainty estimation is performed based on the prototype prediction points and the embedded features to obtain prototype correction points and covariance matrices; the covariance matrix represents the Gaussian distribution covariance matrix corresponding to the prototype correction points as the Gaussian distribution center of the point cloud data. Based on the prototype correction point and the covariance matrix, probability modeling is performed to obtain the average probability matrix; Based on the average probability matrix, a corresponding instance label is assigned to the points in the point cloud data; Based on the Gaussian distribution to which the points in the point cloud data belong, assign corresponding point cloud semantic labels to the points in the point cloud data; The step of estimating uncertainty based on the prototype prediction points and the embedded features to obtain prototype correction points and covariance matrices includes: After merging the prototype prediction points with the structured features, the first feature vector is obtained by processing through two fully connected layers. After merging the first feature vector with the embedded feature, the second feature vector is obtained by processing through two fully connected layers. The second feature vector closest to each prototype point is subjected to mean pooling to obtain instance-level feature information; The instance-level feature information is processed by three fully connected layers to obtain the prototype correction point and the covariance matrix; The step of performing probability modeling based on the prototype correction point and the covariance matrix to obtain the average probability matrix includes: For the point cloud data of the first The Gaussian distribution has the following average probability matrix: ； in, The average probability matrix is the average probability matrix. Indicates the first The embedding feature is embedded in the Gaussian distribution. The average probability, The first point in the point cloud data The mask corresponding to each instance For the first The covariance matrix corresponding to the Gaussian distribution, For the first The prototype correction point corresponding to the Gaussian distribution. For the first The aforementioned embedded features, This represents the matrix transpose operation.
2. The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation according to claim 1, characterized in that, The acquisition of point cloud data includes: The scene is scanned using a LiDAR scanner to obtain the point cloud data of the current scene.
3. The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation according to claim 1, characterized in that, The step of performing data normalization processing on the point cloud data to obtain prototype points includes: Obtain the instance label corresponding to each frame of the point cloud scene in the point cloud data; Statistical analysis is performed on the instance labels corresponding to each frame of the point cloud scene to obtain a set of non-overlapping labels; Based on the set of labels, the center of each instance in the point cloud data is calculated as the prototype point.
4. The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation according to claim 1, characterized in that, The formula for the covariance matrix is: ;in, For the first The covariance matrix corresponding to each Gaussian distribution The rotation angle is... For the minor axis variance, This represents the variance ratio of the major axis and the minor axis.
5. The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation according to claim 1, characterized in that, Assigning a corresponding instance label to the points in the point cloud data based on the average probability matrix includes: Based on the average probability matrix, determine the Gaussian distribution to which the embedding features of the point representations in the point cloud data most likely belong; The instance label corresponding to the most likely Gaussian distribution is assigned to the corresponding point in the point cloud data.
6. The 3D point cloud panoramic segmentation method based on deep learning and uncertainty estimation according to claim 1, characterized in that, The step of assigning corresponding point cloud semantic labels to points in the point cloud data according to the Gaussian distribution to which the points in the point cloud data belong includes: For all Gaussian distributions corresponding to the same instance label, determine the point cloud semantic label corresponding to each Gaussian distribution, and assign the point cloud semantic label with the largest number of such labels to all Gaussian distributions corresponding to the same instance label.
7. A computer device, characterized in that, The device includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to execute the three-dimensional point cloud panoramic segmentation method based on deep learning and uncertainty estimation as described in any one of claims 1-6.
8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the three-dimensional point cloud panoramic segmentation method based on deep learning and uncertainty estimation as described in any one of claims 1-6.