A semantic classification method and system for discovering new categories of lidar point clouds based on deep learning

Abstract

The present invention discloses a semantic classification method for discovering new categories in lidar point clouds based on deep learning, including: obtaining point cloud scene data and preprocessing it to obtain first and second coordinate data, inputting the data into an encoder, and correspondingly obtaining first and second point features; processing the first and second point features to obtain clusters; training the encoder with a target optimization function to obtain a trained encoder; inputting the point cloud scene data into the trained encoder to obtain point cloud scene features; obtaining labeled features and unlabeled features based on the point cloud scene features; obtaining category prototypes based on the labeled features; obtaining unknown category prototypes based on the unlabeled data; forming a classifier based on the category prototypes and the unknown category prototypes, and forming a prediction model with the trained encoder; and inputting the point cloud scene data to be tested into the prediction model to obtain a prediction result. This method avoids the problems of being unable to handle imbalanced number of categories and overfitting on known categories.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to a semantic classification method and system for discovering new categories of lidar point clouds based on deep learning. Background Art

[0002] Semantic segmentation of 3D LiDAR point clouds is the task of classifying each point in a point cloud into different categories. It has important applications in many fields, such as autonomous driving, robot navigation, and smart cities. With the continuous development of deep learning, the success of existing methods relies heavily on a large amount of labeled data, which is often difficult to obtain or expensive. In contrast, unlabeled data is more abundant and easy to obtain. Some scholars have introduced the novel class discovery (NCD) problem, which aims to use the knowledge of a set of labeled data to classify unlabeled data into different classes, where the classes in the labeled and unlabeled data are disjoint. Novel class discovery (NCD) is a task that attempts to discover and learn new categories from unlabeled data. Currently, there are two main types of methods to solve this problem: multi-stage learning methods and joint learning methods.

[0003] Multi-stage learning methods are learning methods that include multiple training stages. These methods usually first pre-train on labeled data, and then use the pre-trained model to fine-tune on the complete data. For the unlabeled data, before fine-tuning, the encoder of the pre-trained model is used to calculate the corresponding features and then assign pseudo-labels with the help of clustering methods. The generated pseudo-labels are then used together with the true labels to fine-tune the model. There are two problems with this type of method: First, since the pre-training is based on data of known categories, the pre-trained encoder often performs poorly in new categories, and the pseudo-labels obtained by clustering are noisy. Second, in order to generate pseudo-labels for the data, we need to obtain the features of all unknown data online and then cluster them to obtain pseudo-labels. For segmentation tasks, pixel-level or point-level features are required, and such a large amount of data is catastrophic.

[0004] Unlike multi-stage learning, joint learning methods attempt to learn both labeled and unlabeled data in a single training phase. During training, such methods use a separate classifier to generate pseudo-label information for unlabeled data, and directly use the pseudo-labels and true annotation information to train a complete classifier through cross-entropy. To improve the quality of pseudo-labels, previous methods have used deep clustering to constrain the generation of pseudo-labels. This method forces the pseudo-label results to be evenly divided to avoid result degradation (all pseudo-labels are classified into the same category). Therefore, such methods often assume that the number of unknown data of each category is relatively balanced, which is obviously not always true. At the same time, our experimental results also show that the performance of this method will drop sharply when the number of categories is unbalanced.

[0005] Therefore, how to avoid being unable to handle the imbalance in the number of categories and overfitting on known categories is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a semantic classification method and system for discovering new categories of lidar point clouds based on deep learning, which helps to avoid the problems of being unable to handle the imbalance in the number of categories and overfitting on known categories.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A semantic classification method for discovering new categories in lidar point clouds based on deep learning, including:

[0009] Acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data;

[0010] Inputting the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature respectively;

[0011] Processing the first point features and the second point features accordingly obtains a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together form a cluster set;

[0012] Training the encoder based on the cluster set and the target optimization function to obtain a trained encoder;

[0013] Inputting the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtaining labeled features and unlabeled features based on the point cloud scene features;

[0014] Calculating the average value of each category based on the labeled features to obtain a category prototype; performing clustering based on the unlabeled data to obtain an unknown category prototype;

[0015] A classifier is formed based on the category prototype and the unknown category prototype; a prediction model is formed based on the classifier and the trained encoder;

[0016] The point cloud scene data to be measured is input into the prediction model to obtain a prediction type result.

[0017] Preferably, the pretreatment includes:

[0018] Performing a rigid body change based on the point cloud scene data to obtain the first coordinate data of the point cloud scene in the first view and the second coordinate data in the second view;

[0019] The point cloud shapes and structures of the first view and the second view are exactly the same, but their positions and orientations are different.

[0020] Preferably, the processing based on the first point feature and the second point feature specifically includes:

[0021] Based on the first point feature division, a plurality of first marked point features and a plurality of first unknown point features are obtained;

[0022] Based on the second point feature division, a plurality of second marked point features and a plurality of second unknown point features are obtained;

[0023] Performing a first clustering process based on all the first annotation point features and the second annotation point features, respectively, to obtain a plurality of first annotation clusters and a plurality of second annotation clusters;

[0024] A second clustering process is performed based on all the first unknown point features and the second unknown point features, respectively, to obtain a plurality of first unlabeled clusters and a plurality of second unlabeled clusters.

[0025] Preferably, the first clustering process includes:

[0026] Based on the classification of all the first annotation point features and the second annotation point features according to category information, all point features in the same category are aggregated to obtain a plurality of the first annotation clusters and a plurality of the second annotation clusters;

[0027] The second clustering process includes:

[0028] Clustering is performed based on all the first unknown point features and the second unknown point features to obtain a first initial cluster and a second initial cluster respectively;

[0029] Obtaining a point feature histogram based on the point cloud scene data;

[0030] The first initial cluster and the second initial cluster are clustered based on the point feature histogram, and a plurality of first unlabeled clusters and a plurality of second unlabeled clusters are correspondingly obtained.

[0031] Preferably, training the encoder based on the cluster set and the target optimization function specifically includes:

[0032] Calculating cluster features based on the first labeled clusters, the first unlabeled clusters, the second labeled clusters, and the second unlabeled clusters in the cluster set, respectively, to obtain a plurality of first labeled cluster features, a plurality of first unlabeled cluster features, a plurality of second labeled cluster features, and a plurality of second unlabeled cluster features, which together form a cluster feature set;

[0033] Training the encoder based on the cluster feature set and the target optimization function;

[0034] The cluster feature calculation formulas are:

[0035]

[0036] Among them, f c Represents cluster features, K represents the number of points in the corresponding cluster, It represents the extracted features corresponding to the k-th point in the corresponding cluster C, and Q represents the feature dimension of the model output.

[0037] Preferably, the cluster feature set during the training process is cached in a buffer queue;

[0038] Splicing the current batch training data with the data in the buffer queue as the final training data;

[0039] The encoder is trained based on the final training data.

[0040] Preferably, the objective optimization function L is:

[0041] L=(1-λ)L s +λL u

[0042]

[0043]

[0044] Among them, λ represents the weight coefficient of unsupervised loss, L s represents the supervised contrastive learning objective function, L u represents the unsupervised objective function, N(i) represents the first view x in the same batch i Other points belonging to the same category, z i=φ(f ξ (x i )), f ξ represents the encoder, φ represents the multi-layer perceptron projection head, 1 n≠i represents the indicator function, z q and z n Represents the same batch and x i Features extracted from the same category and different categories, τ represents the temperature coefficient, L cluster and They represent the intra-cluster consistency constraint loss function and the inter-cluster unsupervised contrast loss function respectively, and α represents the weight coefficient of the intra-cluster consistency constraint loss function.

[0045] Preferably, obtaining a category prototype specifically includes:

[0046] Put the annotation features into a temporary cache;

[0047] During the batch-by-batch iteration process, the new features extracted from each batch are placed into the temporary cache, and the feature average of each category is updated to obtain an updated average;

[0048] Based on the temporary cache, features whose distance from the updated average value is greater than a first threshold are removed. After traversing all the point cloud scene data, an average value is obtained according to the category based on the feature data in the temporary cache as the category prototype.

[0049] Preferably, obtaining the unknown category prototype specifically includes:

[0050] Clustering the unlabeled features based on a clustering algorithm to obtain a first cluster center;

[0051] Sampling and caching the unlabeled features based on the first cluster center, obtaining a plurality of unlabeled features whose distance from the first cluster center is less than a second threshold as cached features, and obtaining a cached data set;

[0052] The unlabeled features in the cached data set are clustered based on a clustering algorithm to obtain a second cluster center as the unknown class prototype.

[0053] A semantic classification system for discovering new categories of lidar point clouds based on deep learning, comprising: a data processing module, a feature extraction module, a cluster set acquisition module, a training module, a prototype acquisition module, a prediction model construction module, and a result output module;

[0054] The data processing module is used to obtain point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data;

[0055] The feature extraction module is used to input the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature respectively;

[0056] The cluster set acquisition module is configured to process the first point feature and the second point feature to obtain a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together constitute a cluster set;

[0057] The training module is used to train the encoder based on the cluster set and the target optimization function to obtain a trained encoder;

[0058] The prototype acquisition module is used to input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtain labeled features and unlabeled features based on the point cloud scene features; calculate the average value of each category based on the labeled features to obtain a category prototype; and perform clustering based on the unlabeled data to obtain an unknown category prototype;

[0059] The prediction model construction module is used to form a classifier based on the category prototype and the unknown category prototype; and form a prediction model based on the classifier and the trained encoder;

[0060] The result output module is used to input the point cloud scene data to be measured into the prediction model to obtain a prediction type result.

[0061] Through the above technical solutions, it can be seen that compared with the existing technology, the present invention discloses a semantic classification method and system for discovering new categories of lidar point clouds based on deep learning, which helps to avoid the problem of being unable to handle the imbalance of the number of categories and overfitting on known categories, and has the following advantages:

[0062] Beneficial effects:

[0063] 1. The present invention uses a contrastive training scheme, which solves the NCD task while avoiding the class balance constraint problem caused by previous deep clustering schemes.

[0064] 2. The present invention applies unsupervised contrastive learning on all data and performs supervised contrastive learning on known data, which can make the output features of the model as aggregated as possible within the class and separated between classes.

[0065] 3. Considering that a single point cloud scene may only contain a subset of all categories, in order to enable more effective comparative learning, the present invention uses a data buffer queue to cache the feature information output by each category model during training, and introduces it into the training process to enhance the learning effect, alleviate or even avoid the problem of incomplete category information within a single batch, thereby improving the feature classification effect of the final training.

[0066] 4. The present invention adopts a cluster-based contrastive learning strategy. By using clusters for contrastive learning for training, it can effectively alleviate the storage pressure of the GPU during training while using contrastive learning to train the point cloud segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0068] Figure 1 This is a flowchart of a semantic classification method for discovering new categories of lidar point clouds based on deep learning provided by the present invention.

[0069] Figure 2 This is a clustering flow chart during the training process of the first initial cluster and the second initial cluster provided by the present invention.

[0070] Figure 3 Schematic diagram of the segmentation results in the SemanticKITTI dataset and SemanticPOSS dataset provided by the present invention.

[0071] Figure 4 A schematic diagram of the structure of a semantic classification system for discovering new categories of lidar point clouds based on deep learning provided by the present invention. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0073] Example 1

[0074] like Figure 1 As shown, an embodiment of the present invention discloses a semantic classification method for discovering new categories of lidar point clouds based on deep learning, comprising:

[0075] Acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data;

[0076] Inputting the first coordinate data and the second coordinate data into the encoder, and obtaining the first point feature and the second point feature accordingly;

[0077] Based on the first point feature and the second point feature, a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters are processed to form a cluster set;

[0078] The encoder is trained based on the cluster set and the target optimization function to obtain a trained encoder;

[0079] Input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtain labeled features and unlabeled features based on the point cloud scene features;

[0080] The average value of each category is calculated based on the labeled features to obtain the category prototype; the unknown category prototype is obtained by clustering based on the unlabeled data;

[0081] A classifier is formed based on the category prototype and the unknown category prototype; a prediction model is formed based on the classifier and the trained encoder;

[0082] Input the point cloud scene data to be tested into the prediction model to obtain the prediction type result.

[0083] Example 2

[0084] The embodiment of the present invention discloses a semantic classification method for discovering new categories of lidar point clouds based on deep learning, comprising:

[0085] Get the point cloud scene data and preprocess it to get the first coordinate data and the second coordinate data:

[0086] Preferably, the pretreatment comprises:

[0087] Performing rigid body changes based on the point cloud scene data to obtain first coordinate data of the point cloud scene in a first view and second coordinate data in a second view;

[0088] The point cloud shapes and structures of the first and second views are exactly the same, but their positions and orientations are different.

[0089] Preferably, the first coordinate data and the second coordinate data of the same point cloud scene in different coordinate spaces are obtained by performing a rigid body transformation on the point cloud scene data.

[0090] Preferably, in this embodiment, the point cloud scene data is obtained by scanning different scenes by a laser radar. The point cloud scene data consists of a known set and an unknown set. The semantic category in the point cloud is represented by C={C l ∪C u}, where C l Represents a known set of classes, C u Represents an unknown set, satisfying

[0091] Preferably, the point cloud scene data X consists of a finite but unknown number of 3D points X∈{(x,c)}, where x∈R 3 Represents the coordinates of a point, c represents its semantic category, for all points x∈X, their category c∈C l Known, but for some points belonging to unknown sets, their categories c∈C u Unknown. Here, the number of new categories to be discovered |C u |Assume that it is known, that is, |C u |=C u .

[0092] The first coordinate data and the second coordinate data are input into the encoder to obtain the first point feature and the second point feature accordingly.

[0093] Based on the first point feature and the second point feature, multiple first labeled clusters, multiple first unlabeled clusters, multiple second labeled clusters, and multiple second unlabeled clusters are obtained, which together constitute a cluster set:

[0094] Preferably, the processing based on the first feature and the second feature specifically includes:

[0095] Based on the first point feature division, a plurality of first marked point features and a plurality of first unknown point features are obtained;

[0096] Based on the second point feature division, a plurality of second marked point features and a plurality of second unknown point features are obtained;

[0097] Performing a first clustering process based on all first annotation point features and second annotation point features, respectively, to obtain a plurality of first annotation clusters and a plurality of second annotation clusters;

[0098] A second clustering process is performed based on all the first unknown point features and the second unknown point features, respectively, to obtain a plurality of first unlabeled clusters and a plurality of second unlabeled clusters.

[0099] Preferably, the first clustering process includes:

[0100] Based on all the first annotation point features and the second annotation point features, they are divided according to category information respectively, all point features in the same category are aggregated, and a cluster is constructed for each known category, correspondingly obtaining multiple first annotation clusters and multiple second annotation clusters.

[0101] Preferably, supervised comparative learning is performed on the first annotation point feature and the second annotation point to learn the feature expression.

[0102] Preferably, the second clustering process includes:

[0103] Clustering is performed based on all first unknown point features and second unknown point features, respectively, to obtain a first initial cluster and a second initial cluster;

[0104] Obtain point feature histogram based on point cloud scene data;

[0105] The first initial cluster and the second initial cluster are clustered based on the point feature histogram, and a plurality of first unlabeled clusters and a plurality of second unlabeled clusters are correspondingly obtained.

[0106] Preferably, in this embodiment, clustering is performed based on all first unknown point features and second unknown point features respectively, and the first unknown point features and second unknown point features are divided into first initial clusters and second initial clusters respectively using a Euclidean clustering algorithm.

[0107] Preferably, the Euclidean clustering algorithm is a clustering algorithm based on the Euclidean clustering metric. By applying the Euclidean distance to the point cloud data in the three-dimensional space for grouping and calculating the Euclidean distance between the unlabeled data, the algorithm can classify the qualified data into the same cluster according to the specified search radius distance and the minimum number of cluster points; this embodiment uses a search distance of 0.5m and a minimum number of cluster points of 5 for clustering, and directly discards clusters with less than the minimum number of points.

[0108] Preferably, during the comparative training, the first initial cluster and the second initial cluster are used as basic units, and when the number of clusters exceeds a specified number (ie, the number of unknown categories), the initial clusters are further clustered to a specified number based on k-means clustering.

[0109] Preferably, Figure 2 As shown in the figure, in the early stage of training: from the beginning of model training to the preset training batch, the encoder cannot obtain feature expressions containing semantic information. The point feature histogram (PFH feature) obtained based on the point cloud scene data is used to assist in clustering the first initial cluster and the second initial cluster respectively. The PFH feature provides the geometric features of each point, which can assist in effective clustering.

[0110] Late training: After exceeding the preset training batches, the encoder generates and outputs features with semantic information. The first and second initial clusters are clustered based on the point features output by the encoder. Points belonging to the same category are ultimately grouped into the same cluster. A batch is a complete training run for all input training samples.

[0111] The encoder is trained based on the cluster set and the target optimization function to obtain the trained encoder:

[0112] Preferably, the encoder is trained based on the cluster set and the target optimization function, specifically including:

[0113] Calculate cluster features based on the multiple first labeled clusters, the multiple first unlabeled clusters, the multiple second labeled clusters, and the multiple second unlabeled clusters in the cluster set, and obtain the multiple first labeled cluster features, the multiple first unlabeled cluster features, the multiple second labeled cluster features, and the multiple second unlabeled cluster features, which together form a cluster feature set;

[0114] Train the encoder based on the cluster feature set and the target optimization function;

[0115] Each cluster contains a set of point features. For each cluster, the average value of the features within the cluster is calculated as the cluster feature. The cluster feature calculation formula is:

[0116]

[0117] Among them, f c Represents cluster features, K represents the number of points in the corresponding cluster, It represents the extracted features corresponding to the k-th point in the corresponding cluster C, and Q represents the feature dimension of the model output, that is, the output feature is a matrix with 1 row and Q columns.

[0118] Preferably, the entire training is performed based on batches. In this embodiment, the batch size is set to 4. During training, contrastive learning is used to learn the feature expression of each semantic category. In order to reduce the bandwidth overhead caused by excessive data volume, contrastive learning is chosen here based on clusters rather than points.

[0119] Preferably, the cluster feature set during the training process is cached in a buffer queue;

[0120] The current batch training data is concatenated with the data in the buffer queue as the final training data;

[0121] Training the encoder based on the final training data can alleviate or even avoid the problem of incomplete category information within a single batch, thereby improving the feature classification effect of the final training.

[0122] Preferably, since the data in a single scene is usually incomplete, the data in the buffer queue is incorporated into the current batch of training data and then clustered as a whole to ensure that the category information contained in the current batch of training data is complete.

[0123] Preferably, the objective optimization function L is:

[0124] L=(1-λ)L s +λL u

[0125]

[0126] Among them, λ represents the weight coefficient of unsupervised loss, L srepresents the supervised contrastive learning objective function, L u represents the unsupervised objective function, N(i) represents the first view x in the same batch i Other points belonging to the same category, z i =φ(f ξ (x i )), f ξ represents the encoder, φ represents the multi-layer perceptron projection head, 1 n≠i Represents the indicator function, that is, when n≠i, the value is 1, otherwise it is 0, z q and z n Represents the same batch and x i Features extracted from the same category and different categories, τ represents the temperature coefficient, L cluster and They represent the intra-cluster consistency constraint loss function and the inter-cluster unsupervised contrast loss function respectively, and α represents the weight coefficient of the intra-cluster consistency constraint loss function.

[0127] Preferably, the cluster consistency constraint loss function (ClusterCCL) L cluster for:

[0128]

[0129] Among them, K c The number of clusters, represents the point features within the cluster, f c is the cluster feature corresponding to the cluster.

[0130] Preferably, unsupervised contrast training is performed based on the obtained cluster features, and the cluster unsupervised contrast loss function (ClusterUCL) for:

[0131]

[0132] Among them, z i and z i ′ respectively represent the features of the same cluster in a batch corresponding to the two views.

[0133] Preferably, by using the target optimization function L for training, the labeled data and the unlabeled data can finally achieve intra-class aggregation and inter-class separation at the semantic level, thereby obtaining a trained encoder f ξ .

[0134] Input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; based on the point cloud scene features, obtain labeled features and unlabeled features:

[0135] Preferably, a trained encoder f with semantic feature extraction capability is finally obtained through the training shown ξ , by using k-means clustering, we can obtain the category prototypes according to the cluster centers. Since the entire dataset is too large to directly perform clustering operations on the entire dataset, we use a batch sampling clustering method here.

[0136] Preferably, the point cloud scene data is input into the trained encoder f in batches. ξ , obtain point cloud scene features batch by batch, and obtain annotation features f based on point cloud scene feature division l and unlabeled features f u .

[0137] The category prototype is obtained by calculating the average value of each category based on the labeled features:

[0138] Preferably, obtaining a category prototype specifically includes:

[0139] The feature f l Put it into a temporary cache and calculate the average value for each category based on the known annotations. The average value is used as the characteristic expression of the cluster for subsequent cluster-based comparative training;

[0140] During the batch-by-batch iteration process, the new features extracted from each batch are put into a temporary cache, and the average value of the features of each category is updated to obtain the updated average value;

[0141] Based on the temporary cache, features with a distance update average value greater than the first threshold are removed to avoid excessive cache size. After traversing all point cloud scene data, the average value is obtained according to the category based on the feature data in the temporary cache as the category prototype. where |C l | represents the number of known categories.

[0142] Clustering based on unlabeled data to obtain unknown category prototypes:

[0143] Preferably, obtaining the unknown category prototype specifically includes:

[0144] Based on the k-means clustering algorithm, the unlabeled features f u Perform clustering and obtain the first cluster center;

[0145] Based on the first cluster center, the unlabeled feature f u Perform sampling caching to obtain multiple unlabeled features whose distance from the first cluster center is less than a second threshold as cache features, thereby obtaining a cache dataset. By caching a set of features closest to the cluster center and discarding other features, the cache queue is prevented from being too large and the cache features are ensured to be meaningful.

[0146] After traversing all point cloud scene data, the unlabeled features in the cached dataset are clustered based on the k-means clustering algorithm, and the second cluster center is obtained as the unknown class prototype P u ={p1,p2,....p Cu}, where |C u | represents the number of unknown categories.

[0147] Preferably, the purpose of k-means clustering is to classify a set of data. Based on the entire extraction of the data set, the iterative process of caching is to filter out a set of features from the data set that are sufficient to represent all categories, so that the final clustering is representative.

[0148] Composition of classifiers based on category prototypes and unknown category prototypes:

[0149] Preferably, P l and P u After splicing, it is used as the final classifier f s .

[0150] Based on the classifier and the trained encoder, a prediction model is formed:

[0151] Preferably, based on f ξ and f s Composition prediction model f.

[0152] Input the point cloud scene data to be tested into the prediction model to obtain the prediction type result.

[0153] Preferably, the point cloud scene data to be measured is input into the trained encoder f ξ Extract features and input the extracted features into the classifier f s , get the predicted type result for each point.

[0154] Example 3

[0155] The method of the present invention is verified:

[0156] Select an outdoor point cloud semantic segmentation dataset. In this embodiment, the public SemanticKITTI dataset and SemanticPOSS dataset are selected.

[0157] The dataset provides the coordinates of points within a scene obtained by lidar scans, along with reference labels for each point. The SemanticKITTI dataset contains 22 sequences, 43,553 scene scans, and 19 semantic categories. Training is performed on sequences 1-7, and validation is performed on sequence 8. The SemanticPOSS dataset contains 6 sequences, 2,988 scene scans, and 13 semantic categories. Sequence 3 is used as the validation set, and the remaining sequences are used for training.

[0158] For each data set, the data set is divided into labeled data and unlabeled data according to categories to verify the performance of the method of the present invention on the NCD problem, wherein the labeled data contains known categories and the unlabeled data contains unknown categories.

[0159] In order to verify the performance of the invention in different data distributions, 8 groups of category division schemes were created in the SemanticKITTI dataset and the SemanticPOSS dataset, and these division schemes were defined as and In the experiment, two partitioning methods, a and b, were tested based on the data distribution of the new categories. In the partitioning method a, the number of new category data was relatively balanced, while in the partitioning method b, the number of new category data was significantly different. The partitioning schemes are shown in Tables 1 and 2:

[0160] Table 1 SemanticKITTI dataset partitioning scheme

[0161]

[0162]

[0163] Table 2 SemanticKITTI dataset partitioning scheme

[0164]

[0165] by and For example, Five new categories are included, namely buildings, roads, sidewalks, natural surfaces, and plants. The number distribution of these five categories is relatively consistent, that is, the number distribution of points in the dataset is relatively balanced. The dataset includes five new categories: plants, fences, utility poles, pedestrians, and natural surfaces. These five categories have significantly different population distributions, with plants being significantly more numerous than the other new categories, and pedestrians being relatively fewer. By testing two different data distributions, we can verify the performance of our approach in different scenarios.

[0166] The mean intersection over union (mIoU) is used to evaluate the performance of the trained model. The mIoU is the average of the IoUs of all target categories.

[0167] Experiments were conducted on the validation sets of SemanticKITTI and SemanticPOSS. The segmentation results are shown in the figure below. Figure 3 The specific experimental results are shown in Table 3 and Table 4:

[0168] Table 3 Experimental results of SemanticKITTI validation set

[0169]

[0170] Table 4 Experimental results of SemanticPOSS validation set

[0171]

[0172] In the experiment, the segmentation capabilities of the model were tested under different new category divisions and in a fully supervised configuration. From the results, it can be seen that when there is a part of data categories that are unknown and unlabeled, the method of the present invention has a certain segmentation capability when the unknown data is in various distributions.

[0173] Example 4

[0174] like Figure 4 As shown, a semantic classification system for discovering new categories of lidar point clouds based on deep learning includes: a data processing module, a feature extraction module, a cluster set acquisition module, a training module, a prototype acquisition module, a prediction model construction module and a result output module;

[0175] A data processing module, used for acquiring point cloud scene data and performing preprocessing to obtain first coordinate data and second coordinate data;

[0176] A feature extraction module, configured to input the first coordinate data and the second coordinate data into an encoder, and obtain a first point feature and a second point feature accordingly;

[0177] A cluster set acquisition module is used to process the first point feature and the second point feature to obtain a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together constitute a cluster set;

[0178] A training module is used to train the encoder based on the cluster set and the target optimization function to obtain a trained encoder;

[0179] The prototype acquisition module is used to input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtain labeled features and unlabeled features based on the point cloud scene features; calculate the average value of each category based on the labeled features to obtain the category prototype; and perform clustering based on the unlabeled data to obtain the unknown category prototype;

[0180] A prediction model building module is used to form a classifier based on the category prototype and the unknown category prototype; and to form a prediction model based on the classifier and the trained encoder;

[0181] The result output module is used to input the point cloud scene data to be tested into the prediction model to obtain the prediction type result.

[0182] Example 5

[0183] Based on the same inventive concept, the present invention further provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0184] Memory for storing computer programs;

[0185] The processor, when used to execute the program stored in the memory, can implement a semantic classification method for discovering new categories of lidar point clouds based on deep learning as in Example 1 or 2.

[0186] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call logic instructions in the memory to execute a semantic classification method for discovering new categories of lidar point clouds based on deep learning in Example 1 or 2.

[0187] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0188] Through the above technical solutions, it can be seen that compared with the existing technology, the present invention discloses a semantic classification method and system for discovering new categories of lidar point clouds based on deep learning, which helps to avoid the problem of being unable to handle the imbalance of the number of categories and overfitting on known categories, and has the following advantages:

[0189] Beneficial effects:

[0190] 1. The present invention uses a contrastive training scheme, which solves the NCD task while avoiding the class balance constraint problem caused by previous deep clustering schemes.

[0191] 2. The present invention applies unsupervised contrastive learning on all data and performs supervised contrastive learning on known data, which can make the output features of the model as aggregated as possible within the class and separated between classes.

[0192] 3. Considering that a single point cloud scene may only contain a subset of all categories, in order to enable more effective comparative learning, the present invention uses a data buffer queue to cache the feature information output by each category model during training, and introduces it into the training process to enhance the learning effect, alleviate or even avoid the problem of incomplete category information within a single batch, thereby improving the feature classification effect of the final training.

[0193] 4. The present invention adopts a cluster-based contrastive learning strategy. By using clusters for contrastive learning for training, it can effectively alleviate the storage pressure of the GPU during training while using contrastive learning to train the point cloud segmentation model.

[0194] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0195] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic classification method for discovering new categories of lidar point clouds based on deep learning, characterized by: include: Acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data; Inputting the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature respectively; Processing the first point features and the second point features accordingly obtains a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together form a cluster set; Training the encoder based on the cluster set and the target optimization function to obtain a trained encoder; The objective optimization function L is: L=(1-λ)L s +λL u Among them, λ represents the weight coefficient of unsupervised loss, L s represents the supervised contrastive learning objective function, L u represents the unsupervised objective function, N(i) represents the first view x in the same batch i Other points belonging to the same category, z i =φ(f ξ (x i )), f ξ represents the encoder, φ represents the multi-layer perceptron projection head, 1 n≠i represents the indicator function, z q and z n Represents the same batch and x i Features extracted from the same category and different categories, τ represents the temperature coefficient, L cluster and They represent the intra-cluster consistency constraint loss function and the inter-cluster unsupervised contrast loss function respectively, and α represents the weight coefficient of the intra-cluster consistency constraint loss function; Inputting the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtaining labeled features and unlabeled features based on the point cloud scene features; Calculating the average value of each category based on the labeled features to obtain a category prototype; performing clustering based on the unlabeled features to obtain an unknown category prototype; A classifier is formed based on the category prototype and the unknown category prototype; a prediction model is formed based on the classifier and the trained encoder; The point cloud scene data to be measured is input into the prediction model to obtain a prediction type result.

2. The semantic classification method for discovering new categories of lidar point clouds based on deep learning according to claim 1, characterized in that: The pretreatment includes: Performing a rigid body change based on the point cloud scene data to obtain the first coordinate data of the point cloud scene in the first view and the second coordinate data in the second view; The point cloud shapes and structures of the first view and the second view are exactly the same, but their positions and orientations are different.

3. The semantic classification method for discovering new categories of lidar point clouds based on deep learning according to claim 1, characterized in that: The processing based on the first point feature and the second point feature specifically includes: Based on the first point feature division, a plurality of first marked point features and a plurality of first unknown point features are obtained; Based on the second point feature division, a plurality of second marked point features and a plurality of second unknown point features are obtained; Performing a first clustering process based on all the first annotation point features and the second annotation point features, respectively, to obtain a plurality of first annotation clusters and a plurality of second annotation clusters; A second clustering process is performed based on all the first unknown point features and the second unknown point features, respectively, to obtain a plurality of first unlabeled clusters and a plurality of second unlabeled clusters.

4. The semantic classification method for discovering new categories of lidar point clouds based on deep learning according to claim 3, characterized in that: The first clustering process includes: Based on the classification of all the first annotation point features and the second annotation point features according to category information, all point features in the same category are aggregated to obtain a plurality of the first annotation clusters and a plurality of the second annotation clusters; The second clustering process includes: Clustering is performed based on all the first unknown point features and the second unknown point features to obtain a first initial cluster and a second initial cluster respectively; Obtaining a point feature histogram based on the point cloud scene data; The first initial cluster and the second initial cluster are clustered based on the point feature histogram, and a plurality of first unlabeled clusters and a plurality of second unlabeled clusters are correspondingly obtained.

5. The semantic classification method for discovering new categories of lidar point clouds based on deep learning according to claim 1, characterized in that: Training the encoder based on the cluster set and the target optimization function specifically includes: Calculating cluster features based on the first labeled clusters, the first unlabeled clusters, the second labeled clusters, and the second unlabeled clusters in the cluster set, respectively, to obtain a plurality of first labeled cluster features, a plurality of first unlabeled cluster features, a plurality of second labeled cluster features, and a plurality of second unlabeled cluster features, which together form a cluster feature set; Training the encoder based on the cluster feature set and the target optimization function; The cluster feature calculation formulas are: Among them, f c Represents cluster features, K represents the number of points in the corresponding cluster, It represents the extracted features corresponding to the k-th point in the corresponding cluster c, and Q represents the feature dimension of the model output.

6. The semantic classification method for discovering new categories of lidar point clouds based on deep learning according to claim 5, characterized in that: Cache the cluster feature set during the training process into a buffer queue; Splicing the current batch training data with the data in the buffer queue as the final training data; The encoder is trained based on the final training data.

7. The semantic classification method for discovering new categories of lidar point clouds based on deep learning according to claim 1, characterized in that: Get the category prototype, including: Put the annotation features into a temporary cache; During the batch-by-batch iteration process, the new features extracted from each batch are placed into the temporary cache, and the feature average of each category is updated to obtain an updated average; Based on the temporary cache, features whose distance from the updated average value is greater than a first threshold are removed. After traversing all the point cloud scene data, an average value is obtained according to the category based on the feature data in the temporary cache as the category prototype.

8. The semantic classification method for discovering new categories of lidar point clouds based on deep learning according to claim 1, characterized in that: Get the unknown category prototype, including: Clustering the unlabeled features based on a clustering algorithm to obtain a first cluster center; Sampling and caching the unlabeled features based on the first cluster center, obtaining a plurality of unlabeled features whose distance from the first cluster center is less than a second threshold as cached features, and obtaining a cached data set; The unlabeled features in the cached data set are clustered based on a clustering algorithm to obtain a second cluster center as the unknown category prototype.

9. A semantic classification system for discovering new categories of lidar point clouds based on deep learning, characterized by: include: Data processing module, feature extraction module, cluster set acquisition module, training module, prototype acquisition module, prediction model construction module and result output module; The data processing module is used to obtain point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data; The feature extraction module is used to input the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature respectively; The cluster set acquisition module is configured to process the first point feature and the second point feature to obtain a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together constitute a cluster set; The training module is used to train the encoder based on the cluster set and the target optimization function to obtain a trained encoder; The objective optimization function L is: L=(1-λ)L s +λL u Among them, λ represents the weight coefficient of unsupervised loss, L s represents the supervised contrastive learning objective function, L u represents the unsupervised objective function, N(i) represents the first view x in the same batch i Other points belonging to the same category, z i =φ(f ξ (x i )), f ξ represents the encoder, φ represents the multi-layer perceptron projection head, 1 n≠i represents the indicator function, z q and z n Represents the same batch and x i Features extracted from the same category and different categories, τ represents the temperature coefficient, L cluster and They represent the intra-cluster consistency constraint loss function and the inter-cluster unsupervised contrast loss function respectively, and α represents the weight coefficient of the intra-cluster consistency constraint loss function; The prototype acquisition module is used to input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtain labeled features and unlabeled features based on the point cloud scene features; calculate the average value of each category based on the labeled features to obtain a category prototype; and perform clustering based on the unlabeled features to obtain an unknown category prototype; The prediction model construction module is used to form a classifier based on the category prototype and the unknown category prototype; and form a prediction model based on the classifier and the trained encoder; The result output module is used to input the point cloud scene data to be measured into the prediction model to obtain a prediction type result.

Images