Semantic classification method and system for new category discovery of laser radar point cloud based on deep learning

Abstract

The invention discloses a semantic classification method for new category discovery of laser radar point cloud based on deep learning, and the method comprises the steps: obtaining point cloud scene data, carrying out the preprocessing of the data, obtaining first coordinate data and second coordinate data, inputting the data into an encoder, and correspondingly obtaining a first point feature and a second point feature; processing based on the first point feature and the second point feature to obtain a cluster set; training the encoder with the target optimization function to obtain a trained encoder; inputting the point cloud scene data into a trained encoder to obtain point cloud scene features; obtaining labeled features and unlabeled features based on the point cloud scene features; obtaining a category prototype based on the annotation features; obtaining an unknown category prototype based on the unlabeled data; forming a classifier based on the category prototype and the unknown category prototype, and forming a prediction model with the trained encoder; and inputting the to-be-measured point cloud scene data into the prediction model to obtain a prediction result. The problems that the class number imbalance condition cannot be processed and overfitting is carried out on known classes are avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a semantic classification method and system for discovering new categories of laser radar point clouds based on deep learning. Background Art

[0002] Semantic segmentation of 3D LiDAR point clouds is a task of classifying each point in a point cloud into different categories. It has important applications in many fields such as autonomous driving, robot navigation, and smart cities. With the continuous development of deep learning, the success of existing methods depends largely on a large amount of labeled data, which is often difficult to obtain or expensive. In contrast, unlabeled data is richer and easier to obtain. Some scholars have introduced the novel class discovery (NCD) problem, which aims to use the knowledge of a set of labeled data to classify unlabeled data into different classes, where the classes in the labeled and unlabeled data are disjoint. Novel class discovery (NCD) is a task that attempts to discover and learn new categories from unlabeled data. There are currently two main types of methods to solve this problem: multi-stage learning methods and joint learning methods.

[0003] Multi-stage learning methods are learning methods that include multiple training stages. These methods usually first pre-train on labeled data, and then use the pre-trained model to tune on the complete data. Among them, for the unlabeled data part, before tuning, the encoder of the pre-trained model is used to calculate the corresponding features and then assign pseudo labels with the help of clustering methods. Then the generated pseudo labels and true labels are used to tune the model together. There are two problems with this type of method: First, since the pre-training is based on known category data, the pre-trained encoder often performs poorly in new categories, and the pseudo labels obtained by clustering are noisy. Second, in order to generate pseudo labels for data, we need to obtain the features of all unknown data online, and then cluster them to obtain pseudo labels. For segmentation tasks, it is necessary to obtain pixel-level or point-level features, and such a large amount of data is catastrophic.

[0004] Unlike multi-stage learning, joint learning methods attempt to learn both labeled and unlabeled data in one training phase. During training, such methods use a separate classifier to generate pseudo-label information for unlabeled data, and directly use the pseudo-labels and true annotation information to train a complete classifier through cross entropy. In order to improve the quality of pseudo-labels, previous methods have used deep clustering to constrain the generation of pseudo-labels. This method will force the pseudo-label results to be evenly divided to avoid the degradation of the results (all pseudo-labels are classified into the same category). Therefore, such methods often assume that the number of unknown data of each category is relatively balanced, which is obviously not always true. At the same time, our experimental results also show that the performance of this method will drop sharply when the number of categories is unbalanced.

[0005] Therefore, how to avoid being unable to handle the imbalance in the number of categories and overfitting on known categories is an urgent problem that technicians in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides a semantic classification method and system for discovering new categories of lidar point clouds based on deep learning, which helps to avoid the problem of being unable to handle the imbalance in the number of categories and overfitting on known categories.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A semantic classification method for discovering new categories of lidar point clouds based on deep learning, including:

[0009] Acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data;

[0010] Inputting the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature respectively;

[0011] Based on the first point feature and the second point feature, a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters are processed to form a cluster set;

[0012] Training the encoder based on the cluster set and the target optimization function to obtain a trained encoder;

[0013] Inputting the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtaining labeled features and unlabeled features based on the point cloud scene features;

[0014] Calculating the average value of each category based on the labeled features to obtain a category prototype; performing clustering based on the unlabeled data to obtain an unknown category prototype;

[0015] A classifier is formed based on the category prototype and the unknown category prototype; a prediction model is formed based on the classifier and the trained encoder;

[0016] The point cloud scene data to be tested is input into the prediction model to obtain a prediction type result.

[0017] Preferably, the pretreatment comprises:

[0018] Performing a rigid body change based on the point cloud scene data to obtain the first coordinate data of the point cloud scene in a first view and the second coordinate data in a second view;

[0019] The point cloud shape structure of the first view and the second view are exactly the same, but the positions and orientations are different.

[0020] Preferably, the processing based on the first point feature and the second point feature specifically includes:

[0021] Based on the first point feature division, a plurality of first marked point features and a plurality of first unknown point features are obtained;

[0022] Based on the second point feature division, a plurality of second marked point features and a plurality of second unknown point features are obtained;

[0023] Performing first clustering processing based on all the first annotation point features and the second annotation point features, respectively, to obtain a plurality of first annotation clusters and a plurality of second annotation clusters;

[0024] A second clustering process is performed based on all the first unknown point features and the second unknown point features, respectively, to obtain a plurality of first unlabeled clusters and a plurality of second unlabeled clusters accordingly.

[0025] Preferably, the first clustering process includes:

[0026] Based on the classification of all the first annotation point features and the second annotation point features according to category information, all point features in the same category are aggregated to obtain a plurality of the first annotation clusters and a plurality of the second annotation clusters;

[0027] The second clustering process comprises:

[0028] Clustering is performed based on all the first unknown point features and the second unknown point features to obtain a first initial cluster and a second initial cluster respectively;

[0029] Obtaining a point feature histogram based on the point cloud scene data;

[0030] The first initial cluster and the second initial cluster are clustered based on the point feature histogram, and a plurality of the first unlabeled clusters and a plurality of the second unlabeled clusters are correspondingly obtained.

[0031] Preferably, training the encoder based on the cluster set and the target optimization function specifically includes:

[0032] Based on the plurality of first labeled clusters, the plurality of first unlabeled clusters, the plurality of second labeled clusters, and the plurality of second unlabeled clusters in the cluster set, cluster features are calculated respectively, and a plurality of first labeled cluster features, a plurality of first unlabeled cluster features, a plurality of second labeled cluster features, and a plurality of second unlabeled cluster features are correspondingly obtained to form a cluster feature set together;

[0033] Training the encoder based on the cluster feature set and the target optimization function;

[0034] The cluster feature calculation formulas are:

[0035]

[0036] Among them, f c represents the cluster feature, K represents the number of points in the corresponding cluster, It represents the extracted features corresponding to the kth point in the corresponding cluster C, and Q represents the feature dimension of the model output.

[0037] Preferably, the cluster feature set in the training process is cached in a buffer queue;

[0038] splicing the current batch training data with the data in the buffer queue as final training data;

[0039] The encoder is trained based on the final training data.

[0040] Preferably, the objective optimization function L is:

[0041] L=(1-λ)L s +λL u

[0042]

[0043]

[0044] Among them, λ represents the weight coefficient of unsupervised loss, L s represents the supervised contrastive learning objective function, L u represents the unsupervised objective function, N(i) represents the first view x in the same batch i Other points belonging to the same category, z i=φ(f ξ (x i )),f ξ represents the encoder, φ represents the multi-layer perceptron projection head, 1 n≠i represents the indicator function, z q and z n Respectively represent the same batch and x i The features extracted from the same category and different categories, τ represents the temperature coefficient, L cluster and They represent the intra-cluster consistency constraint loss function and the inter-cluster unsupervised contrast loss function respectively, and α represents the weight coefficient of the intra-cluster consistency constraint loss function.

[0045] Preferably, obtaining a category prototype specifically includes:

[0046] Put the annotated features into a temporary cache;

[0047] In the batch-by-batch iteration process, the new features extracted from each batch are placed in the temporary cache, and the feature average of each category is updated to obtain an updated average;

[0048] Based on the temporary cache, features whose distance from the updated average value is greater than a first threshold are removed, and after traversing all the point cloud scene data, an average value is obtained according to the category based on the feature data in the temporary cache as the category prototype.

[0049] Preferably, obtaining the unknown category prototype specifically includes:

[0050] Clustering the unlabeled features based on a clustering algorithm to obtain a first cluster center;

[0051] Based on the first cluster center, the unlabeled features are sampled and cached, and a plurality of unlabeled features whose distance from the first cluster center is less than a second threshold are obtained as cached features, thereby obtaining a cached data set;

[0052] The unlabeled features in the cache data set are clustered based on a clustering algorithm to obtain a second cluster center as the unknown class prototype.

[0053] A semantic classification system for discovering new categories of lidar point clouds based on deep learning, comprising: a data processing module, a feature extraction module, a cluster set acquisition module, a training module, a prototype acquisition module, a prediction model construction module and a result output module;

[0054] The data processing module is used to acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data;

[0055] The feature extraction module is used to input the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature accordingly;

[0056] The cluster set acquisition module is used to process the first point feature and the second point feature to obtain a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together form a cluster set;

[0057] The training module is used to train the encoder based on the cluster set and the target optimization function to obtain a trained encoder;

[0058] The prototype acquisition module is used to input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtain labeled features and unlabeled features based on the point cloud scene features; calculate the average value of each category based on the labeled features to obtain a category prototype; and perform clustering based on the unlabeled data to obtain an unknown category prototype;

[0059] The prediction model building module is used to form a classifier based on the category prototype and the unknown category prototype; and to form a prediction model based on the classifier and the trained encoder;

[0060] The result output module is used to input the point cloud scene data to be tested into the prediction model to obtain a prediction type result.

[0061] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a semantic classification method and system for discovering new categories of lidar point clouds based on deep learning, which helps to avoid the problem of being unable to handle the imbalance of the number of categories and overfitting on known categories, and has the following advantages:

[0062] Beneficial effects:

[0063] 1. The present invention uses a contrastive training scheme, which not only solves the NCD task, but also avoids the class balance constraint problem caused by previous deep clustering schemes.

[0064] 2. The present invention applies unsupervised contrastive learning on all data and performs supervised contrastive learning on known data, so that the output features of the model can be aggregated within the class and separated between classes as much as possible.

[0065] 3. Considering that a single point cloud scene may only contain a subset of all categories, in order to enable more effective comparative learning, the present invention uses a data buffer queue to cache the feature information output by each category model during training, and introduces it into the training process to enhance the learning effect, alleviate or even avoid the problem of incomplete category information within a single batch, thereby improving the feature classification effect of the final training.

[0066] 4. The present invention adopts a cluster-based contrastive learning strategy. By using clusters for contrastive learning for training, it can effectively alleviate the storage pressure of the GPU during training while using contrastive learning to train the point cloud segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0068] Figure 1 A flowchart of a semantic classification method for discovering new categories of lidar point clouds based on deep learning provided by the present invention.

[0069] Figure 2 This is a clustering flow chart of the first initial cluster and the second initial cluster training process provided by the present invention.

[0070] Figure 3 Schematic diagram of segmentation results in the SemanticKITTI dataset and SemanticPOSS dataset provided by the present invention.

[0071] Figure 4 A schematic diagram of the structure of a semantic classification system for discovering new categories of lidar point clouds based on deep learning provided by the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Example 1

[0074] like Figure 1 As shown, an embodiment of the present invention discloses a semantic classification method for discovering new categories of laser radar point clouds based on deep learning, including:

[0075] Acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data;

[0076] Inputting the first coordinate data and the second coordinate data into the encoder, and obtaining the first point feature and the second point feature accordingly;

[0077] Based on the first point feature and the second point feature, a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters are processed to form a cluster set;

[0078] The encoder is trained based on the cluster set and the target optimization function to obtain a trained encoder;

[0079] Input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtain labeled features and unlabeled features based on the point cloud scene features;

[0080] The average value of each category is calculated based on the labeled features to obtain the category prototype; the unknown category prototype is obtained by clustering based on the unlabeled data;

[0081] A classifier is formed based on the category prototype and the unknown category prototype; a prediction model is formed based on the classifier and the trained encoder;

[0082] Input the point cloud scene data to be tested into the prediction model to obtain the prediction type result.

[0083] Example 2

[0084] The embodiment of the present invention discloses a semantic classification method for discovering new categories of laser radar point clouds based on deep learning, comprising:

[0085] Get the point cloud scene data and preprocess it to get the first coordinate data and the second coordinate data:

[0086] Preferably, the pretreatment comprises:

[0087] Perform rigid body changes based on the point cloud scene data to obtain first coordinate data of the point cloud scene in a first view and second coordinate data of the point cloud scene in a second view;

[0088] The point cloud shapes and structures of the first view and the second view are exactly the same, but their positions and orientations are different.

[0089] Preferably, the first coordinate data and the second coordinate data of the same point cloud scene in different coordinate spaces are obtained by performing a rigid body transformation on the point cloud scene data.

[0090] Preferably, in this embodiment, the point cloud scene data is obtained by scanning the laser radar in different scenes. The point cloud scene data consists of a known set and an unknown set. The semantic category in the point cloud is represented by C={C l ∪C u}, where C l represents the known set, C u Represents an unknown set, satisfying

[0091] Preferably, the point cloud scene data X consists of a finite but unknown number of 3D points X∈{(x,c)}, where x∈R 3 Represents the coordinates of a point, c represents its semantic category, for all points x∈X, their category c∈C l It is known that, but some points belong to unknown classes, their categories c∈C u Unknown. Here, the number of new categories to be discovered |C u |Assume that it is known, that is, |C u |=C u .

[0092] The first coordinate data and the second coordinate data are input into the encoder to obtain the first point feature and the second point feature accordingly.

[0093] Based on the first point feature and the second point feature, a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters are obtained, which together constitute a cluster set:

[0094] Preferably, the processing based on the first feature and the second feature specifically includes:

[0095] Based on the first point feature division, a plurality of first marked point features and a plurality of first unknown point features are obtained;

[0096] Based on the second point feature division, a plurality of second marked point features and a plurality of second unknown point features are obtained;

[0097] Performing first clustering processing based on all first annotation point features and second annotation point features respectively, and correspondingly obtaining a plurality of first annotation clusters and a plurality of second annotation clusters;

[0098] A second clustering process is performed based on all the first unknown point features and the second unknown point features, respectively, to obtain a plurality of first unlabeled clusters and a plurality of second unlabeled clusters accordingly.

[0099] Preferably, the first clustering process includes:

[0100] Based on the classification of all first annotation point features and second annotation point features according to category information, all point features in the same category are aggregated, and a cluster is constructed for each known category, thereby obtaining multiple first annotation clusters and multiple second annotation clusters.

[0101] Preferably, supervised comparative learning is performed on the first annotation point feature and the second annotation point to learn the feature expression.

[0102] Preferably, the second clustering process includes:

[0103] Clustering is performed based on all the first unknown point features and the second unknown point features, respectively, to obtain a first initial cluster and a second initial cluster correspondingly;

[0104] Obtain a point feature histogram based on point cloud scene data;

[0105] The first initial cluster and the second initial cluster are clustered based on the point feature histogram, and a plurality of first unlabeled clusters and a plurality of second unlabeled clusters are correspondingly obtained.

[0106] Preferably, in this embodiment, clustering is performed based on all the first unknown point features and the second unknown point features respectively, and the first unknown point features and the second unknown point features are respectively divided into the first initial cluster and the second initial cluster using the Euclidean clustering algorithm.

[0107] Preferably, the Euclidean clustering algorithm is a clustering algorithm based on the Euclidean clustering metric. By applying the Euclidean distance to the point cloud data in the three-dimensional space for grouping and by calculating the Euclidean distance between the unlabeled data, the algorithm can classify the qualified data into the same cluster according to the specified search radius distance and the minimum number of clustering points; this embodiment uses a search distance of 0.5m and a minimum number of clustering points of 5 for clustering, and directly discards clusters with less than the minimum number of points.

[0108] Preferably, during the comparison training, the first initial cluster and the second initial cluster are used as basic units, and when the number of clusters exceeds a specified number (ie, the number of unknown categories), the initial clusters are further clustered to a specified number based on k-means clustering.

[0109] Preferably, Figure 2 As shown in the figure, in the early stage of training: from the beginning of model training to the preset training batch, the encoder cannot obtain the feature expression containing semantic information. The point feature histogram (PFH feature) obtained based on the point cloud scene data is used to assist in clustering the first initial cluster and the second initial cluster respectively. The PFH feature provides the geometric features of each point, which can assist in effective clustering.

[0110] Late training stage: After exceeding the preset training batch, the encoder generates output features with semantic information. At this time, the first initial cluster and the second initial cluster are clustered based on the point features output by the encoder, and points belonging to the same category will eventually be aggregated in the same cluster. A batch means that all input training samples are trained once.

[0111] The encoder is trained based on the cluster set and the target optimization function to obtain the trained encoder:

[0112] Preferably, the encoder is trained based on the cluster set and the target optimization function, specifically including:

[0113] Based on the multiple first labeled clusters, multiple first unlabeled clusters, multiple second labeled clusters, and multiple second unlabeled clusters in the cluster set, cluster features are calculated respectively, and multiple first labeled cluster features, multiple first unlabeled cluster features, multiple second labeled cluster features, and multiple second unlabeled cluster features are correspondingly obtained to form a cluster feature set together;

[0114] Train the encoder based on the cluster feature set and the target optimization function;

[0115] Each cluster contains a set of point features. For each cluster, the average value of the features within the cluster is calculated as the cluster feature. The cluster feature calculation formula is:

[0116]

[0117] Among them, f c represents the cluster feature, K represents the number of points in the corresponding cluster, It represents the extracted features corresponding to the kth point in the corresponding cluster C, and Q represents the feature dimension of the model output, that is, the output feature is a matrix with 1 row and Q columns.

[0118] Preferably, the entire training is performed based on batches. In this embodiment, the batch size is set to 4. During training, contrastive learning is used to learn the feature expression of each semantic category. In order to reduce the bandwidth overhead problem caused by excessive data volume, contrastive learning is chosen here based on clusters rather than points.

[0119] Preferably, the cluster feature set in the training process is cached into a buffer queue;

[0120] The current batch training data is concatenated with the data in the buffer queue as the final training data;

[0121] Training the encoder based on the final training data can alleviate or even avoid the problem of incomplete category information within a single batch, thereby improving the feature classification effect of the final training.

[0122] Preferably, since the data in a single scene is usually incomplete, the data in the buffer queue is incorporated into the current batch of training data and then clustered as a whole to ensure that the category information contained in the current batch of training data is complete.

[0123] Preferably, the objective optimization function L is:

[0124] L=(1-λ)L s +λL u

[0125]

[0126] Among them, λ represents the weight coefficient of unsupervised loss, L srepresents the supervised contrastive learning objective function, L u represents the unsupervised objective function, N(i) represents the first view x in the same batch i Other points belonging to the same category, z i =φ(f ξ (x i )),f ξ represents the encoder, φ represents the multi-layer perceptron projection head, 1 n≠i represents the indicator function, that is, when n≠i, the value is 1, otherwise it is 0, z q and z n Respectively represent the same batch and x i The features extracted from the same category and different categories, τ represents the temperature coefficient, L cluster and They represent the intra-cluster consistency constraint loss function and the inter-cluster unsupervised contrast loss function respectively, and α represents the weight coefficient of the intra-cluster consistency constraint loss function.

[0127] Preferably, the cluster consistency constraint loss function (ClusterCCL) L cluster for:

[0128]

[0129] Among them, K c refers to the number of clusters, represents the point features within the cluster, f c is the cluster feature corresponding to the cluster.

[0130] Preferably, unsupervised contrast training is performed based on the obtained cluster features, and the cluster unsupervised contrast loss function (ClusterUCL) is for:

[0131]

[0132] Among them, z i and z i ′ respectively represent the features of the same cluster in a batch corresponding to the two views.

[0133] Preferably, by using the objective optimization function L for training, the labeled data and the unlabeled data can finally achieve intra-class aggregation and inter-class separation at the semantic level, thereby obtaining a trained encoder f ξ .

[0134] Input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; obtain labeled features and unlabeled features based on the point cloud scene features:

[0135] Preferably, through the training shown, a trained encoder f with semantic feature extraction capability is finally obtained. ξ , by using k-means clustering, we can get the category prototype according to the cluster center. Since the entire data set is too large to directly perform clustering operations on the entire data set, a batch sampling clustering method is used here.

[0136] Preferably, the point cloud scene data is input into the trained encoder f in batches ξ , obtain point cloud scene features batch by batch, and obtain annotation features f based on point cloud scene feature division l and unlabeled features f u .

[0137] The category prototype is obtained by calculating the average value of each category based on the labeled features:

[0138] Preferably, obtaining a category prototype specifically includes:

[0139] The feature f l Put it into a temporary cache, and calculate the average value for each category based on the known annotations. The average value is used as the characteristic expression of the cluster for subsequent cluster-based comparative training;

[0140] During the batch-by-batch iteration process, the new features extracted from each batch are put into a temporary cache, and the feature average of each category is updated to obtain the updated average;

[0141] Based on the temporary cache, features with a distance update average value greater than the first threshold are removed to avoid excessive cache size. After traversing all point cloud scene data, the average value is obtained according to the category based on the feature data in the temporary cache as the category prototype. Where |C l | represents the number of known categories.

[0142] Clustering based on unlabeled data to obtain unknown category prototypes:

[0143] Preferably, obtaining the unknown category prototype specifically includes:

[0144] Based on the k-means clustering algorithm, the unlabeled features f u Perform clustering and obtain the first cluster center;

[0145] Based on the first cluster center, the unlabeled feature f u Perform sampling cache, obtain multiple unlabeled features whose distance from the first cluster center is less than the second threshold as cache features, and obtain a cache data set; by caching a set of features closest to the cluster center and discarding other features, the cache queue is prevented from being too large and the cache features are ensured to be meaningful;

[0146] After traversing all point cloud scene data, the unlabeled features in the cached data set are clustered based on the k-means clustering algorithm to obtain the second cluster center as the unknown class prototype P u ={p 1 ,p 2 ,....p Cu}, where |C u | represents the number of unknown categories.

[0147] Preferably, the purpose of k-means clustering is to classify a set of data. Based on the entire extraction of the data set, the iterative process of caching is to filter out a set of features from the data set that are sufficient to represent all categories, so that the final clustering is representative.

[0148] Composition of classifiers based on category prototypes and unknown category prototypes:

[0149] Preferably, P l and P u After concatenation, it is used as the final classifier f s .

[0150] Based on the classifier and the trained encoder, a prediction model is formed:

[0151] Preferably, based on f ξ and f s Composition prediction model f.

[0152] Input the point cloud scene data to be tested into the prediction model to obtain the prediction type result.

[0153] Preferably, the point cloud scene data to be tested is input into the trained encoder f ξ Extract features and input the extracted features into the classifier f s , get the prediction type result for each point.

[0154] Example 3

[0155] The method of the present invention verifies:

[0156] An outdoor point cloud semantic segmentation dataset is selected. In this embodiment, the public SemanticKITTI dataset and SemanticPOSS dataset are selected.

[0157] The dataset provides the coordinates of the points obtained by the lidar scan in the scene and the reference labels of each point. The SemanticKITTI dataset contains 22 sequences, 43,553 scene scans and a total of 19 semantic categories. It is trained on sequences 01-07 and validated on sequence 08. SemanticPOSS contains 6 sequences, 2,988 scene scans and 13 semantic categories. Sequence 3 is used as the validation set, and the remaining sequences are used for training.

[0158] For each data set, the data set is divided into labeled data and unlabeled data according to categories to verify the performance of the method of the present invention on the NCD problem, wherein the labeled data contains known categories and the unlabeled data contains unknown categories.

[0159] In order to verify the performance of the invention in different data distributions, 8 groups of category division schemes were created in the SemanticKITTI dataset and the SemanticPOSS dataset, and these division schemes were defined as and In the experiment, two division methods a and b were tested based on the data distribution of the new categories. The quantity distribution of the new category data in the a-division was relatively balanced, while the quantity distribution of the new category data in the b-division was quite different. The division schemes are shown in Tables 1 and 2:

[0160] Table 1 SemanticKITTI dataset partitioning scheme

[0161]

[0162]

[0163] Table 2 SemanticKITTI dataset partitioning scheme

[0164]

[0165] by and For example, Five new categories are included, namely buildings, roads, sidewalks, natural surfaces, and plants. The quantity distribution of these five categories is relatively consistent, that is, the quantity distribution of points in the dataset is relatively balanced. Five new categories are included, namely plants, fences, electric poles, pedestrians, and natural surfaces. The number distribution of these five categories is quite different. The number of plants is significantly larger than that of other new categories, while the number of pedestrians is relatively smaller. By testing two different data distribution types, we can verify the performance of this method in different situations.

[0166] The mean intersection over union (mIoU) is used to evaluate the performance of the trained model. The mean intersection over union (mIoU) is the average of the intersection over union (IoU) of all target categories.

[0167] Experiments were conducted on the validation sets of SemanticKITTI and SemanticPOSS. The segmentation results are shown in the figure below. Figure 3 The specific experimental results are shown in Table 3 and Table 4:

[0168] Table 3 Experimental results of SemanticKITTI validation set

[0169]

[0170] Table 4 Experimental results of SemanticPOSS validation set

[0171]

[0172] In the experiment, the segmentation ability of the model based on different new category divisions and under full supervision configuration was tested. From the results, it can be seen that when there is a part of data category unknown and unlabeled, the method of the present invention has a certain segmentation ability when the unknown data is in various distributions.

[0173] Example 4

[0174] like Figure 4 As shown, a semantic classification system for discovering new categories of lidar point clouds based on deep learning includes: a data processing module, a feature extraction module, a cluster set acquisition module, a training module, a prototype acquisition module, a prediction model construction module and a result output module;

[0175] A data processing module, used for acquiring point cloud scene data and performing preprocessing to obtain first coordinate data and second coordinate data;

[0176] A feature extraction module, used for inputting the first coordinate data and the second coordinate data into the encoder, and correspondingly obtaining a first point feature and a second point feature;

[0177] A cluster set acquisition module, used for processing based on the first point feature and the second point feature, correspondingly obtaining a plurality of first labeled clusters, a plurality of first unlabeled clusters and a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together constitute a cluster set;

[0178] A training module is used to train the encoder based on the cluster set and the target optimization function to obtain a trained encoder;

[0179] The prototype acquisition module is used to input the point cloud scene data into the trained encoder in batches to obtain the point cloud scene features; obtain the labeled features and unlabeled features based on the point cloud scene features; calculate the average value of each category based on the labeled features to obtain the category prototype; cluster the unlabeled data to obtain the unknown category prototype;

[0180] A prediction model building module is used to form a classifier based on a category prototype and an unknown category prototype; and to form a prediction model based on the classifier and the trained encoder;

[0181] The result output module is used to input the point cloud scene data to be tested into the prediction model to obtain the prediction type result.

[0182] Example 5

[0183] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0184] Memory, used to store computer programs;

[0185] The processor, when used to execute the program stored in the memory, can implement a semantic classification method for discovering new categories of lidar point clouds based on deep learning as in Example 1 or 2.

[0186] The electronic device may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute a semantic classification method for discovering new categories of laser radar point clouds based on deep learning in embodiment 1 or 2.

[0187] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0188] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a semantic classification method and system for discovering new categories of lidar point clouds based on deep learning, which helps to avoid the problem of being unable to handle the imbalance of the number of categories and overfitting on known categories, and has the following advantages:

[0189] Beneficial effects:

[0190] 1. The present invention uses a contrastive training scheme, which not only solves the NCD task, but also avoids the class balance constraint problem caused by previous deep clustering schemes.

[0191] 2. The present invention applies unsupervised contrastive learning on all data and performs supervised contrastive learning on known data, so that the output features of the model can be aggregated within the class and separated between classes as much as possible.

[0192] 3. Considering that a single point cloud scene may only contain a subset of all categories, in order to enable more effective comparative learning, the present invention uses a data buffer queue to cache the feature information output by each category model during training, and introduces it into the training process to enhance the learning effect, alleviate or even avoid the problem of incomplete category information within a single batch, thereby improving the feature classification effect of the final training.

[0193] 4. The present invention adopts a cluster-based contrastive learning strategy. By using clusters for contrastive learning for training, it can effectively alleviate the storage pressure of the GPU during training while using contrastive learning to train the point cloud segmentation model.

[0194] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0195] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic classification method for discovering new categories of lidar point clouds based on deep learning, characterized in that: include: Acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data; Inputting the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature respectively; Based on the first point feature and the second point feature, a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters are processed to form a cluster set; Training the encoder based on the cluster set and the target optimization function to obtain a trained encoder; Inputting the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; Obtaining labeled features and unlabeled features based on the point cloud scene features; Calculating the average value of each category based on the labeled features to obtain a category prototype; performing clustering based on the unlabeled data to obtain an unknown category prototype; A classifier is formed based on the category prototype and the unknown category prototype; a prediction model is formed based on the classifier and the trained encoder; The point cloud scene data to be tested is input into the prediction model to obtain a prediction type result.

2. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 1, characterized in that: The pre-processing comprises: Performing a rigid body change based on the point cloud scene data to obtain the first coordinate data of the point cloud scene in a first view and the second coordinate data in a second view; The point cloud shape structure of the first view and the second view are exactly the same, but the positions and orientations are different.

3. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 1, characterized in that: The processing based on the first point feature and the second point feature specifically includes: Based on the first point feature division, a plurality of first marked point features and a plurality of first unknown point features are obtained; Based on the second point feature division, a plurality of second marked point features and a plurality of second unknown point features are obtained; Performing first clustering processing based on all the first annotation point features and the second annotation point features, respectively, to obtain a plurality of first annotation clusters and a plurality of second annotation clusters; A second clustering process is performed based on all the first unknown point features and the second unknown point features, respectively, to obtain a plurality of first unlabeled clusters and a plurality of second unlabeled clusters accordingly.

4. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 3 is characterized in that: The first clustering process comprises: Based on the classification of all the first annotation point features and the second annotation point features according to category information, all point features in the same category are aggregated to obtain a plurality of the first annotation clusters and a plurality of the second annotation clusters; The second clustering process comprises: Clustering is performed based on all the first unknown point features and the second unknown point features to obtain a first initial cluster and a second initial cluster respectively; Obtaining a point feature histogram based on the point cloud scene data; The first initial cluster and the second initial cluster are clustered based on the point feature histogram, and a plurality of the first unlabeled clusters and a plurality of the second unlabeled clusters are correspondingly obtained.

5. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 1, characterized in that: The encoder is trained based on the cluster set and the target optimization function, specifically comprising: Based on the plurality of first labeled clusters, the plurality of first unlabeled clusters, the plurality of second labeled clusters, and the plurality of second unlabeled clusters in the cluster set, cluster features are calculated respectively, and a plurality of first labeled cluster features, a plurality of first unlabeled cluster features, a plurality of second labeled cluster features, and a plurality of second unlabeled cluster features are correspondingly obtained to form a cluster feature set together; Training the encoder based on the cluster feature set and the target optimization function; The cluster feature calculation formulas are: Among them, f c represents the cluster feature, K represents the number of points in the corresponding cluster, It represents the extracted features corresponding to the kth point in the corresponding cluster C, and Q represents the feature dimension of the model output.

6. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 1, characterized in that: Cache the cluster feature set in the training process into a buffer queue; splicing the current batch training data with the data in the buffer queue as final training data; The encoder is trained based on the final training data.

7. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 1, characterized in that: The objective optimization function L is: L=(1-λ)L s +λL u Among them, λ represents the weight coefficient of unsupervised loss, L s represents the supervised contrastive learning objective function, L u represents the unsupervised objective function, N(i) represents the first view x in the same batch i Other points belonging to the same category, z i =φ(f ξ (x i )),f ξ represents the encoder, φ represents the multi-layer perceptron projection head, 1 n≠i represents the indicator function, z q and z n Respectively represent the same batch and x i The features extracted from the same category and different categories, τ represents the temperature coefficient, L cluster and They represent the intra-cluster consistency constraint loss function and the inter-cluster unsupervised contrast loss function respectively, and α represents the weight coefficient of the intra-cluster consistency constraint loss function.

8. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 1, characterized in that: Get the category prototype, including: Put the annotated features into a temporary cache; In the batch-by-batch iteration process, the new features extracted from each batch are placed in the temporary cache, and the feature average of each category is updated to obtain an updated average; Based on the temporary cache, features whose distance from the updated average value is greater than a first threshold are removed, and after traversing all the point cloud scene data, an average value is obtained according to the category based on the feature data in the temporary cache as the category prototype.

9. The semantic classification method for discovering new categories of laser radar point clouds based on deep learning according to claim 1, characterized in that: Get the unknown category prototype, including: Clustering the unlabeled features based on a clustering algorithm to obtain a first cluster center; Based on the first cluster center, the unlabeled features are sampled and cached, and a plurality of unlabeled features whose distance from the first cluster center is less than a second threshold are obtained as cached features, thereby obtaining a cached data set; The unlabeled features in the cache data set are clustered based on a clustering algorithm to obtain a second cluster center as the unknown class prototype.

10. A semantic classification system for discovering new categories of lidar point clouds based on deep learning, characterized in that: include: Data processing module, feature extraction module, cluster set acquisition module, training module, prototype acquisition module, prediction model construction module and result output module; The data processing module is used to acquire point cloud scene data and perform preprocessing to obtain first coordinate data and second coordinate data; The feature extraction module is used to input the first coordinate data and the second coordinate data into an encoder to obtain a first point feature and a second point feature accordingly; The cluster set acquisition module is used to process the first point feature and the second point feature to obtain a plurality of first labeled clusters, a plurality of first unlabeled clusters, a plurality of second labeled clusters, and a plurality of second unlabeled clusters, which together form a cluster set; The training module is used to train the encoder based on the cluster set and the target optimization function to obtain a trained encoder; The prototype acquisition module is used to input the point cloud scene data into the trained encoder in batches to obtain point cloud scene features; Obtaining labeled features and unlabeled features based on the point cloud scene features; Calculating the average value of each category based on the labeled features to obtain a category prototype; performing clustering based on the unlabeled data to obtain an unknown category prototype; The prediction model building module is used to form a classifier based on the category prototype and the unknown category prototype; and to form a prediction model based on the classifier and the trained encoder; The result output module is used to input the point cloud scene data to be tested into the prediction model to obtain a prediction type result.

Images