Large-Scene Point Cloud Incremental Segmentation Method, Device, Equipment, Medium and Program Product

Abstract

The present application relates to a large-scale scene point cloud incremental segmentation method, device, equipment, medium and program product. The method includes: obtaining point cloud data to be predicted; generating pseudo-labels for the old-class data of the point cloud data to be predicted according to a pre-trained target old-class model to obtain old-class pseudo-labels; performing momentum update on the target old-class model according to the old-class pseudo-labels and the new-class true labels corresponding to the point cloud data to be predicted to obtain a candidate new-class model; optimizing the loss of the candidate new-class model according to a pre-constructed adaptive loss control model to obtain a target new-class model; performing point cloud segmentation on the point cloud data to be predicted according to the target new-class model to obtain a segmentation result. Using this method can effectively improve the incremental prediction accuracy for large-scale scenes, and excellent results can be achieved in both cases of a large number of new classes or a small number of new classes where the number of old and new classes is unbalanced.

Description

Technical Field

[0001] This application relates to the technical field of incremental learning, and particularly to a large-scale scene point cloud incremental segmentation method, device, equipment, medium and program product. Background Art

[0002] With the development of incremental learning technology, incremental learning technology for three-dimensional point clouds has emerged. This technology is currently mainly applied to relatively simple classification tasks, presenting more complex challenges for real urban scenes. For example, the larger scale, complex shapes, and highly imbalanced semantic distributions in urban scenes are difficult problems to overcome. Therefore, how to more accurately predict the semantic distribution and thus accurately segment the three-dimensional point clouds in urban scenes has become an urgent problem in the field of incremental learning technology. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide a large-scale scene point cloud incremental segmentation method, device, computer device, computer-readable storage medium, and computer program product that can accurately segment three-dimensional point clouds in urban scenes.

[0004] In a first aspect, this application provides a large-scale scene point cloud incremental segmentation method, including:

[0005] Obtain the point cloud data to be predicted;

[0006] Generate pseudo-labels for the old-class data of the point cloud data to be predicted according to a pre-trained target old-class model, obtaining old-class pseudo-labels;

[0007] Perform momentum update on the target old-class model according to the old-class pseudo-labels and the new-class true labels corresponding to the point cloud data to be predicted, obtaining a candidate new-class model;

[0008] Optimize the loss of the candidate new-class model according to a pre-constructed adaptive loss control model, obtaining a target new-class model;

[0009] Perform point cloud segmentation on the point cloud data to be predicted according to the target new-class model, obtaining a segmentation result.

[0010] In one embodiment, the generating pseudo-labels for the old-class data of the point cloud data to be predicted according to a pre-trained target old-class model, obtaining old-class pseudo-labels, includes:

[0011] Perform class prediction on the old-class data of the point cloud data to be predicted according to the target old-class model, obtaining multiple supervoxel prediction results; wherein, the old-class data includes multiple supervoxels;

[0012] In the case where there is a candidate prediction class result greater than a preset threshold in the supervoxel prediction result, all the old class pseudo-labels of the points in the supervoxel corresponding to the supervoxel prediction result are set to the candidate prediction class result;

[0013] In the case where there is no prediction class result greater than the preset threshold in the supervoxel prediction result, all the old class pseudo-labels are set to the corresponding prediction class results.

[0014] In one embodiment, the momentum update of the target old class model according to the old class pseudo-labels and the new class true labels corresponding to the to-be-predicted point cloud data to obtain a candidate new class model includes:

[0015] Determine an old class feature prototype according to the target old class model;

[0016] Perform momentum update on the old class feature prototype according to the old class pseudo-labels and the new class true labels to obtain a new class feature prototype;

[0017] Perform distillation constraint on the new class feature prototype and the old class feature prototype to obtain a candidate new class feature prototype, and determine the candidate new class model according to the candidate new class feature prototype.

[0018] In one embodiment, the loss optimization of the candidate new class model according to a pre-constructed adaptive loss control model to obtain a target new class model includes:

[0019] Predict the to-be-predicted point cloud data according to the candidate new class model to obtain a prediction label;

[0020] Perform loss optimization on the candidate new class model according to the prediction label and the adaptive loss control model to obtain the target new class model.

[0021] In one embodiment, the method further includes:

[0022] Obtain the old and new class data of the dataset where the to-be-predicted point cloud data is located;

[0023] Determine a first loss coefficient according to the old and new class data and the momentum update rounds of the target old class model;

[0024] Perform structural constraint on the old and new class data to obtain a second loss coefficient;

[0025] Construct a loss model according to the first loss coefficient, the second loss coefficient and a pre-constructed loss control model to obtain the adaptive loss control model.

[0026] In one embodiment, the method further includes:

[0027] Obtaining training point cloud data of the data set;

[0028] Generating pseudo labels for the training point cloud data according to an initial old category model to obtain training old category pseudo labels;

[0029] Performing momentum update on the initial old category model according to the training old category pseudo labels and the new category true labels to obtain a training new category model;

[0030] Predicting the training point cloud data according to the training new category model to obtain training prediction labels;

[0031] Constructing the loss control model according to the training prediction labels, the new category true labels and the training old category pseudo labels.

[0032] In a second aspect, the present application further provides a large-scene point cloud incremental segmentation device, including:

[0033] An acquisition module, configured to acquire point cloud data to be predicted;

[0034] A pseudo label generation module, configured to generate pseudo labels for the old category data of the point cloud data to be predicted according to a pre-trained target old category model to obtain old category pseudo labels;

[0035] A momentum update module, configured to perform momentum update on the target old category model according to the old category pseudo labels and the new category true labels corresponding to the point cloud data to be predicted to obtain a candidate new category model;

[0036] A loss optimization module, configured to optimize the loss of the candidate new category model according to a pre-constructed adaptive loss control model to obtain a target new category model;

[0037] A point cloud segmentation module, configured to perform point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Obtaining point cloud data to be predicted;

[0040] Generating pseudo labels for the old category data of the point cloud data to be predicted according to a pre-trained target old category model to obtain old category pseudo labels;

[0041] Perform momentum update on the target old-class model according to the old-class pseudo labels and the new-class true labels corresponding to the to-be-predicted point cloud data, to obtain a candidate new-class model;

[0042] Optimize the loss of the candidate new-class model according to a pre-constructed adaptive loss control model, to obtain a target new-class model;

[0043] Perform point cloud segmentation on the to-be-predicted point cloud data according to the target new-class model, to obtain a segmentation result.

[0044] Fourthly, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0045] Obtain to-be-predicted point cloud data;

[0046] Generate pseudo labels for the old-class data of the to-be-predicted point cloud data according to a pre-trained target old-class model, to obtain old-class pseudo labels;

[0047] Perform momentum update on the target old-class model according to the old-class pseudo labels and the new-class true labels corresponding to the to-be-predicted point cloud data, to obtain a candidate new-class model;

[0048] Optimize the loss of the candidate new-class model according to a pre-constructed adaptive loss control model, to obtain a target new-class model;

[0049] Perform point cloud segmentation on the to-be-predicted point cloud data according to the target new-class model, to obtain a segmentation result.

[0050] Fifthly, the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0051] Obtain to-be-predicted point cloud data;

[0052] Generate pseudo labels for the old-class data of the to-be-predicted point cloud data according to a pre-trained target old-class model, to obtain old-class pseudo labels;

[0053] Perform momentum update on the target old-class model according to the old-class pseudo labels and the new-class true labels corresponding to the to-be-predicted point cloud data, to obtain a candidate new-class model;

[0054] Optimize the loss of the candidate new-class model according to a pre-constructed adaptive loss control model, to obtain a target new-class model;

[0055] Perform point cloud segmentation on the to-be-predicted point cloud data according to the target new-class model, to obtain a segmentation result.

[0056] The above large-scale scene point cloud incremental segmentation method, device, computer device, computer-readable storage medium, and computer program product obtain the point cloud data to be predicted; generate pseudo-labels for the old-class data of the point cloud data to be predicted according to the pre-trained target old-class model, obtaining old-class pseudo-labels; perform momentum update on the target old-class model according to the old-class pseudo-labels and the new-class true labels corresponding to the point cloud data to be predicted, obtaining a candidate new-class model; optimize the loss of the candidate new-class model according to the pre-constructed adaptive loss control model, obtaining a target new-class model; perform point cloud segmentation on the point cloud data to be predicted according to the target new-class model, obtaining a segmentation result. Therefore, by generating pseudo-labels for the old-class data of the point cloud data to be predicted according to the pre-trained target old-class model, the prediction ability of the target old-class model for old-class data can be utilized to generate more accurate old-class pseudo-labels, and by performing momentum update on the target old-class model according to the old-class pseudo-labels and the new-class true labels corresponding to the point cloud data to be predicted, the candidate new-class model can retain the knowledge learned by the target old-class model, and by optimizing the loss of the candidate new-class model according to the pre-constructed adaptive loss control model, the adaptive parameters in the adaptive loss control model can be used to further balance the learning processes of different classes in the candidate new-class model, thereby alleviating the prediction deviation problem caused by quantity imbalance, obtaining a target new-class model with higher incremental prediction accuracy, and then performing point cloud segmentation on the point cloud data to be predicted according to the target new-class model, obtaining a segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0058] Figure 1 It is an application environment diagram of the large-scale scene point cloud incremental segmentation method in an embodiment;

[0059] Figure 2 It is a flowchart of the large-scale scene point cloud incremental segmentation method in an embodiment;

[0060] Figure 3 It is a flowchart of the large-scale scene point cloud incremental segmentation method in an embodiment;

[0061] Figure 4 It is a modular diagram of the overall model application idea of the large-scale scene point cloud incremental segmentation method in an embodiment;

[0062] Figure 5 Schematic diagram of a voting-based pseudo-label generation module in the large-scale point cloud incremental segmentation method in an embodiment;

[0063] Figure 6 Schematic diagram of a feature prototype-based structure preservation module in the large-scale point cloud incremental segmentation method in an embodiment;

[0064] Figure 7 Schematic diagram of the application effect of the large-scale point cloud incremental segmentation method in an embodiment;

[0065] Figure 8 Schematic table of the application effect of the large-scale point cloud incremental segmentation method in an embodiment;

[0066] Figure 9 Block diagram of the structure of a large-scale point cloud incremental segmentation device in an embodiment;

[0067] Figure 10 Internal structure diagram of a computer device in an embodiment. Specific implementation manners

[0068] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] The large-scale point cloud incremental segmentation method provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or can be placed on the cloud or other network servers. The server 104 obtains the point cloud data to be predicted; generates pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model, and obtains old category pseudo labels; performs momentum update on the target old category model according to the old category pseudo labels and the new category true labels corresponding to the point cloud data to be predicted, and obtains a candidate new category model; optimizes the loss of the candidate new category model according to the pre-constructed adaptive loss control model, and obtains the target new category model; performs point cloud segmentation on the point cloud data to be predicted according to the target new category model, and obtains the segmentation result. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0070] In an exemplary embodiment, as Figure 2 shown, a large-scale scene point cloud incremental segmentation method is provided. Taking the server 104 in Figure 1 as an example, the following steps S202 to step S210 are included. Among them:

[0071] Step S202, obtain the point cloud data to be predicted.

[0072] Among them, the point cloud data to be predicted can adopt the three-dimensional point cloud data of a large-scale urban scene.

[0073] In some embodiments, the point cloud data to be predicted can be obtained from the urban scene three-dimensional point cloud data set, or the three-dimensional point cloud data of the urban scene in other cloud servers can be obtained in real time as the point cloud data to be predicted, and it is not limited to this.

[0074] Step S204, generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model, and obtain old category pseudo labels.

[0075] Among them, the target old-class model is obtained by fully supervising the training of the initial class model using all the old-class data, and it is a target old-class model capable of recognizing old classes. The pseudo-label generation process can be achieved through a supervoxel voting strategy.

[0076] In some embodiments, pseudo-labels for the old-class data of the point cloud data to be predicted are generated according to the pre-trained target old-class model, including: performing class prediction on the old-class data of the point cloud data to be predicted according to the target old-class model to obtain multiple supervoxel prediction results; where the old-class data includes multiple supervoxels; when there is a candidate prediction class result greater than a preset threshold in the supervoxel prediction results, then the old-class pseudo-labels of all points in the supervoxel corresponding to the supervoxel prediction result are set to the candidate prediction class result; when there is no prediction class result greater than the preset threshold in the supervoxel prediction results, then the old-class pseudo-labels are set to the corresponding prediction class results.

[0077] Among them, the supervoxel can refer to the background supervoxel in the point cloud data to be predicted. The supervoxel prediction results will include the semantic distribution prediction results of each class in the current supervoxel. Each semantic distribution prediction result needs to be greater than a preset minimum threshold to be recorded, and the semantic distribution prediction situation of each class in the supervoxel prediction results is stored in the corresponding candidate prediction class result. The preset threshold can be 0.7. It should be noted that when the semantic distribution prediction result of a certain class is greater than 0.7, it means that more than 70% of the point prediction class results in the current supervoxel are of this class, and it is not limited to this.

[0078] In some embodiments, class prediction is performed on each supervoxel in the old-class data of the point cloud data to be predicted by the target old-class model, and the prediction results are statistically analyzed for all points in a supervoxel to obtain multiple supervoxel prediction results.

[0079] In some embodiments, if the number of semantic distribution prediction results of a certain class in the supervoxel prediction results is greater than the preset threshold, that is, when there is a candidate prediction class result greater than the preset threshold in the supervoxel prediction results, then the old-class pseudo-labels of all points in the supervoxel corresponding to the supervoxel prediction result are set to the candidate prediction class result.

[0080] In some embodiments, if the set threshold is not exceeded, that is, when there is no prediction class result greater than the preset threshold in the supervoxel prediction results, then the old-class pseudo-labels are set to the corresponding prediction class results, and then the old-class pseudo-label corresponding to each point is the corresponding prediction result.

[0081] In this embodiment, the super-voxel voting is used to generate a pseudo-label strategy to restrict the super-voxel prediction results, which can improve the segmentation ability of the candidate new-class model obtained by momentum update with the old-class pseudo-label in the subsequent steps for the old class.

[0082] Step S206: Perform momentum update on the target old-class model according to the old-class pseudo-label and the new-class true label corresponding to the to-be-predicted point cloud data to obtain a candidate new-class model.

[0083] Among them, momentum update is the implementation process of a momentum update strategy, which is used to ensure stable and progressive adjustment of the target old-class model. The new-class true label refers to the true label corresponding to the new class in the to-be-predicted point cloud data.

[0084] In some embodiments, performing momentum update on the target old-class model according to the old-class pseudo-label and the new-class true label corresponding to the to-be-predicted point cloud data to obtain a candidate new-class model includes: determining the old-class feature prototype according to the target old-class model; performing momentum update on the old-class feature prototype according to the old-class pseudo-label and the new-class true label to obtain a new-class feature prototype; performing distillation constraint on the new-class feature prototype and the old-class feature prototype to obtain a candidate new-class feature prototype, and determining the candidate new-class model according to the candidate new-class feature prototype.

[0085] In some embodiments, use the target old-class model to generate the features of the old class, and then calculate the feature prototype of each class in the old class to obtain the old-class feature prototype. This old-class feature prototype can effectively guide the new-class feature prototype to generate the features of the old class, so that the new-class feature prototype retains the knowledge learned by the target old-class model. Calculate the old-class feature prototype p k The process is shown in the following formula (1):

[0086] (1)

[0087] Where, C k represents the number of points with semantic type k, and x i k is the input point cloud with semantic type k. ]>

[0088] In some embodiments, it should be noted that considering the possible feature differences within the same class in the to-be-predicted point cloud data, during the momentum update of the old-class feature prototype, continuously update the old-class feature prototype according to the predicted old-class pseudo-label and the new-class true label to obtain a new-class feature prototype, and determine the candidate new-class model according to the candidate new-class feature prototype, which can ensure stable and progressive adjustment of the new-class feature prototype.

[0089] In some embodiments, a momentum update strategy can be adopted to ensure stable and progressive adjustment of the new-class feature prototypes. For each class, the updated new-class feature prototype in the momentum update step is momentum-updated according to the following formula (2):

[0090] (2)

[0091] where μ is a parameter that can be set, and t is the iteration step of the momentum update.

[0092] In some embodiments, distillation constraints are imposed on the new-class feature prototypes and the old-class feature prototypes to obtain candidate new-class feature prototypes, and candidate new-class models are determined based on the candidate new-class feature prototypes, so that the candidate new-class models maintain the learning ability of the old classes.

[0093] In this embodiment, the target old-class model is momentum-updated according to the old-class pseudo labels and the new-class true labels corresponding to the point cloud data to be predicted, and a candidate new-class model is obtained. In the process of momentum-updating the target old-class model, it can be regarded as a feature structure constraint module. Through such constraints, the catastrophic forgetting problem that causes a significant decline in the performance of the old classes can be solved.

[0094] Step S208, optimize the loss of the candidate new-class model according to the pre-constructed adaptive loss control model to obtain the target new-class model.

[0095] Among them, the pre-constructed adaptive loss control model is the weight loss coefficient in the adaptive loss control model determined according to the new-class and old-class related data in the point cloud data to be predicted, so as to realize a model for adaptively adjusting the loss to solve the loss optimization deviation caused by data imbalance.

[0096] In some embodiments, optimizing the loss of the candidate new-class model according to the pre-constructed adaptive loss control model to obtain the target new-class model includes: predicting the point cloud data to be predicted according to the candidate new-class model to obtain a prediction label; optimizing the loss of the candidate new-class model according to the prediction label and the adaptive loss control model to obtain the target new-class model.

[0097] In some embodiments, first use the candidate new-class model to predict the point cloud data to be predicted to obtain a prediction label, then calculate the total loss according to the prediction label and the adaptive loss control model, and perform convergence optimization on the total loss. When the total loss converges and is optimized to the minimum, obtain the model parameters of the candidate new-class model at this time, and use the candidate new-class model under these model parameters as the target new-class model.

[0098] Step S210, perform point cloud segmentation on the to-be-predicted point cloud data according to the target new category model to obtain a segmentation result.

[0099] Among them, the segmentation result can be to mark point cloud data of different categories with different colors, so as to distinguish point clouds of different categories in the to-be-predicted point cloud data, and obtain a segmentation result containing different color markings.

[0100] In some embodiments, perform point cloud category prediction on the to-be-predicted point cloud data according to the target new category model to obtain point cloud data corresponding to different categories in the to-be-predicted point cloud data, and perform segmentation marking on the point cloud data of different categories, so as to separate the point cloud data of different categories and obtain a segmentation result.

[0101] In the above large-scale scene point cloud incremental segmentation method, by obtaining the to-be-predicted point cloud data; generating pseudo labels for the old category data of the to-be-predicted point cloud data according to the pre-trained target old category model to obtain old category pseudo labels; performing momentum update on the target old category model according to the old category pseudo labels and the new category true labels corresponding to the to-be-predicted point cloud data to obtain a candidate new category model; performing loss optimization on the candidate new category model according to the pre-constructed adaptive loss control model to obtain the target new category model; performing point cloud segmentation on the to-be-predicted point cloud data according to the target new category model to obtain a segmentation result. Therefore, by generating pseudo labels for the old category data of the to-be-predicted point cloud data according to the pre-trained target old category model, the prediction ability of the target old category model for the old category data can be utilized to generate more accurate old category pseudo labels, and by performing momentum update on the target old category model according to the old category pseudo labels and the new category true labels corresponding to the to-be-predicted point cloud data, the candidate new category model can retain the knowledge learned by the target old category model, and by performing loss optimization on the candidate new category model according to the pre-constructed adaptive loss control model, the adaptive parameters in the adaptive loss control model can be used to further balance the learning processes of different categories in the candidate new category model, so as to alleviate the prediction deviation problem caused by quantity imbalance and obtain a target new category model with higher incremental prediction accuracy, and then perform point cloud segmentation on the to-be-predicted point cloud data according to the target new category model to obtain a segmentation result.

[0102] In an exemplary embodiment, as Figure 3 shown, the large-scale scene point cloud incremental segmentation method further includes: steps S302 to S308. Among them:

[0103] Step S302, obtain the old and new category data of the data set where the to-be-predicted point cloud data is located.

[0104] Among them, the old and new category data includes: the loss weight of the old category, the loss weight of the new category, and the ratio between the new category point cloud data and the old category point cloud data.

[0105] In some embodiments, the old and new category data of the dataset where the point cloud data to be predicted is located can be directly calculated through the point cloud data in the dataset, or the old and new category data recorded in the dataset can be directly obtained, and the scope is not limited thereto.

[0106] Step S304: Determine the first loss coefficient according to the old and new category data and the momentum update rounds of the target old category model.

[0107] The first loss coefficient is a weight coefficient used for dynamically adjusting between the loss weight of the old category and the loss weight of the new category.

[0108] In some embodiments, according to the momentum update rounds of the current target old category model, the loss weight of the old category and the loss weight of the new category are dynamically adjusted by means of cosine annealing, as shown in the following formula (3):

[0109] (3)

[0110] Where, is the first loss coefficient, and epoch is the momentum update rounds.

[0111] Step S306: Perform structural constraints on the old and new category data to obtain the second loss coefficient.

[0112] The structural constraint refers to handling the imbalance between the newly added new category point cloud data and the basic old category point cloud data. The second loss coefficient is a coefficient related to the training point ratio, and the training point ratio refers to the ratio between the new category point cloud data and the old category point cloud data in the point cloud data to be predicted.

[0113] In some embodiments, the old and new category data are structurally constrained by means of parameter balancing to obtain the second loss coefficient. The specific design process of the second loss coefficient is shown in the following formula (4):

[0114] (4)

[0115] Where, is the second loss coefficient, p represents the proportion of the basic category, and a and b are parameters controlling the smoothness of the structural constraint. It should be noted that, is proportional to p. When p is small, the total loss function will focus more on the preservation of the structure, thereby avoiding the occurrence of the catastrophic forgetting problem.

[0116] Step S308: Construct a loss model according to the first loss coefficient, the second loss coefficient and the pre-constructed loss control model to obtain an adaptive loss control model.

[0117] Among them, the loss control model includes three loss calculation modules of different types.

[0118] In some embodiments, according to the first loss coefficient and the second loss coefficient, the weight coefficient corresponding to each loss calculation module in the loss control model is confirmed, and then the loss model is constructed according to each weight coefficient and each loss calculation module to obtain an adaptive loss control model.

[0119] In some embodiments, the method further includes: obtaining the training point cloud data of the data set; generating pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels; performing momentum update on the initial old category model according to the training old category pseudo labels and the new category true labels to obtain a training new category model; predicting the training point cloud data according to the training new category model to obtain training prediction labels; constructing a loss control model according to the training prediction labels, the new category true labels and the training old category pseudo labels.

[0120] Among them, the training point cloud data is a subset of the point cloud data with a certain data volume scale in the data set, and contains all types of point cloud data in the data set. The initial old category model is pre-trained and used to identify the old category point cloud data in the training point cloud data.

[0121] In some embodiments, by obtaining the training point cloud data of the data set, generating pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels, performing momentum update on the initial old category model according to the training old category pseudo labels and the new category true labels to obtain a training new category model; predicting the training point cloud data according to the training new category model to obtain training prediction labels, the loss data between the training prediction labels and the true labels can be used to measure the category prediction ability of the current initial old category model for the training point cloud data during the momentum update process, and thus a loss control model is constructed according to the training prediction labels, the new category true labels and the training old category pseudo labels.

[0122] Among them, the loss control model includes a first loss control sub-model, a second loss control sub-model, and a third loss control sub-model.

[0123] In some embodiments, first, for the background and the new category with true labels in the new category of the training new category model, the cross-entropy loss is directly calculated to determine the first loss control sub-model. The specific determination process is shown in the following formula (5):

[0124] (5)

[0125] Among them, N novel is the number of point cloud data in the training new category model, y is the true label, and p is the training prediction label.

[0126] In some embodiments, for the training old-class pseudo labels generated from the training point cloud data according to the initial old-class model, we calculate the cross entropy of the training new-class model using these training old-class pseudo labels, so as to determine the second loss control sub-model. The specific process of determining the second loss control sub-model is shown in the following formula (6):

[0127] (6)

[0128] where, is the number of old-class point cloud data of the training point cloud data, and is the true label corresponding to the i-th old-class point cloud data, is the generated training old-class pseudo label.

[0129] In some embodiments, in order to further confirm the third loss control sub-model through the training old-class pseudo labels and the training prediction labels, the specific process of determining the second loss control sub-model is shown in the following formula (7):

[0130]

[0131] where T is also a control parameter used to adjust the distribution similarity ratio, p is the calculated prototype corresponding to the training old-class pseudo label, and f is the calculated prototype corresponding to the training prediction label.

[0132] In some embodiments, finally, a loss control model is constructed according to the training prediction labels, the new-class true labels and the training old-class pseudo labels, as shown in the following formula (8):

[0133] (8).

[0134] In this embodiment, when training the initial old-class model, the point cloud data of the new class is regarded as the background. When training the new-class model, although the specific labels of the old classes are unknown, since the initial old-class model has the ability to segment the old classes, the initial old-class model can be used to segment the point cloud of the old classes to obtain the predicted values. At the same time, pseudo labels are generated through the supervoxel voting strategy and are used together with the labels of the new class to train the new-class model. In addition, we also introduce a feature structure constraint module to retain the learned knowledge using the basic-class prototypes. And an adaptive loss control model is constructed through parameter control to control the loss ratio of the new and old classes and achieve adaptive adjustment.

[0135] To more clearly understand the solution of this application, here in combination with Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 and Figure 8The description is as follows:

[0136] In some embodiments, such as Figure 4 shown, first, as shown in the upper part a of the figure, the old category dataset D base is used to train the model E base to obtain a target old category model E that can recognize old categories base . Subsequently, based on this initial model E base , the dataset D containing new category data novel is used for further training to obtain a new target new category model E novel . It should be particularly noted that the same training strategy as the 3DPC-CISS method is adopted in the preliminary training strategy of the model. When training the target old category model E base , the new category data is regarded as the background b. And when training the target new category model E novel , although the specific labels of the old categories are unknown, since the target old category model has the ability to segment old categories, the point cloud of the old categories can be segmented by the target old category model to obtain predicted values. At the same time, based on the predicted values and the supervoxel voting strategy, pseudo labels are generated and used together with the labels of the new categories to train the new model. In addition, we also introduce a feature structure constraint module into the momentum update process to retain the learned knowledge using the feature prototypes of the base categories. And, the ratio of the losses of the old and new categories is controlled for the parameters through a pre-constructed adaptive loss control model, so as to achieve adaptive adjustment.

[0137] In some embodiments, such as Figure 5 shown Figure 5 shows the process of generating pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model. Specifically, as Figure 5 shown, a background supervoxel containing old category data is selected in the point cloud data to be predicted as a prediction branch, and all the prediction results are summarized after the prediction is completed. It should be further noted that even though all the old categories have been used for fully supervised training of the old model, there are still significant differences between the prediction results of the old categories and the true labels. If the prediction results are not restricted, it will lead to a significant decline in the segmentation ability of the new model for the old categories. Therefore, the supervoxel voting strategy is adopted here to generate pseudo labels. In Figure 5 shows one of the branches in the pseudo label generation process. Specifically, for all the points in a supervoxel, the prediction results are statistically counted. If the number of prediction results of a certain category is greater than the set threshold, then the pseudo labels of all the points in this supervoxel are set to this prediction result. If it does not exceed the set threshold, then the pseudo label corresponding to each point is the corresponding prediction result. The implementation method is as Figure 5Shown: The prediction results greater than the first threshold are subjected to a second threshold judgment by means of a double threshold. If there are prediction results greater than the second threshold among the prediction results greater than the first threshold, the pseudo-labels of all points in the supervoxel are set to this prediction result.

[0138] In some embodiments, as Figure 6 shown, Figure 6 shows the in-module process of obtaining a candidate new-class model by performing momentum update on the target old-class model according to the old-class pseudo-labels and the new-class true labels corresponding to the point cloud data to be predicted. Specifically, during the training process for a new dataset, usually the true labels of all classes in the dataset are first obtained as the basis for momentum update, and then the respective feature prototypes are extracted through the target old-class model and the candidate new-class model, and the respective feature prototypes are gradually optimized through the new-class true labels and the old-class pseudo-labels, and the semantic distribution information of both is output into the Embedding Space for momentum update, thereby obtaining the candidate new-class model.

[0139] In some embodiments, Figure 7 and Figure 8 respectively show the application effects of this solution in the visual segmentation effect display and the application effects of this solution in the actual point cloud segmentation accuracy test. First, Figure 7 shows that for two different point cloud data scenarios a and b, the point cloud data results segmented by this solution (Ours) for the two datasets a and b are the closest to the original data display effects of a and b. Second, in Figure 8 for the small classification dataset scenarios (including TE, VG, WA, BR, and BU) and the large classification dataset scenarios (including BO, VG, WA, BR, and VE), the semantic segmentation prediction results (Accyarcy and mIou) of this solution (Ours) for different types (qingdao, longhua, and campus) are better than those of other point cloud segmentation solutions (JT, BL, FT, and 3DPC).

[0140] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indication of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0141] It should be noted that the feature extraction algorithm used in the present invention can be directly replaced by any other feature extraction algorithm. The specific framework of the network is not limited. For example, a neural network basic framework based on a multi-layer perceptron, grid, or hash table grid can be used. In addition, the adaptive parameter adjustment mechanism in the present invention is not limited to the declared function form, and any adjustment parameter that causes a monotonic change can be correspondingly replaced.

[0142] Based on the same inventive concept, an embodiment of the present application also provides a large-scale scene point cloud incremental segmentation device for implementing the large-scale scene point cloud incremental segmentation method described above. The implementation solution provided by this device to solve the problem is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more embodiments of the large-scale scene point cloud incremental segmentation device provided below can refer to the limitations on the large-scale scene point cloud incremental segmentation method in the above text, and will not be repeated here.

[0143] In an exemplary embodiment, as Figure 9 shown, a large-scale scene point cloud incremental segmentation device 900 is provided, including: an acquisition module 901, a pseudo-label generation module 902, a momentum update module 903, a loss optimization module 904, and a point cloud segmentation module 905, where:

[0144] The acquisition module 901 is used to acquire the point cloud data to be predicted;

[0145] The pseudo-label generation module 902 is used to generate pseudo-labels for the old-class data of the point cloud data to be predicted according to the pre-trained target old-class model, and obtain old-class pseudo-labels;

[0146] The momentum update module 903 is used to perform momentum update on the target old-class model according to the old-class pseudo-labels and the new-class true labels corresponding to the point cloud data to be predicted, and obtain a candidate new-class model;

[0147] A loss optimization module 904, configured to optimize the loss of the candidate new category model according to a pre-built adaptive loss control model, so as to obtain a target new category model;

[0148] A point cloud segmentation module 905, configured to segment the to-be-predicted point cloud data according to the target new category model, so as to obtain a segmentation result.

[0149] In some embodiments, the pseudo-label generation module 902 is further configured to perform category prediction on the old category data of the to-be-predicted point cloud data according to the target old category model, so as to obtain multiple supervoxel prediction results; wherein, the old category data includes multiple supervoxels;

[0150] In the case that there is a candidate prediction category result greater than a preset threshold in the supervoxel prediction results, all the old category pseudo-labels of the points in the supervoxel corresponding to the supervoxel prediction result are set to the candidate prediction category result;

[0151] In the case that there is no prediction category result greater than the preset threshold in the supervoxel prediction results, all the old category pseudo-labels are set to the corresponding prediction category results.

[0152] In some embodiments, the momentum update module 903 is further configured to determine an old category feature prototype according to the target old category model;

[0153] Perform momentum update on the old category feature prototype according to the old category pseudo-labels and the new category true labels, so as to obtain a new category feature prototype;

[0154] Perform distillation constraint on the new category feature prototype and the old category feature prototype, so as to obtain a candidate new category feature prototype, and determine a candidate new category model according to the candidate new category feature prototype.

[0155] In some embodiments, the loss optimization module 904 is further configured to perform prediction on the to-be-predicted point cloud data according to the candidate new category model, so as to obtain a prediction label; optimize the loss of the candidate new category model according to the prediction label and the adaptive loss control model, so as to obtain a target new category model.

[0156] In some embodiments, the apparatus further includes: a model construction module, configured to obtain old and new category data of a data set where the to-be-predicted point cloud data is located; determine a first loss coefficient according to the old and new category data and the momentum update rounds of the target old category model; perform structural constraint on the old and new category data, so as to obtain a second loss coefficient; construct a loss model according to the first loss coefficient, the second loss coefficient and a pre-built loss control model, so as to obtain an adaptive loss control model.

[0157] In some embodiments, the model construction module is further configured to obtain the training point cloud data of the data set; generate pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels; perform momentum update on the initial old category model according to the training old category pseudo labels and the new category true labels to obtain a training new category model; predict the training point cloud data according to the training new category model to obtain training prediction labels; and construct a loss control model according to the training prediction labels, the new category true labels, and the training old category pseudo labels.

[0158] In the above large-scale scene point cloud incremental segmentation device, by obtaining the point cloud data to be predicted; generating pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels; performing momentum update on the target old category model according to the old category pseudo labels and the new category true labels corresponding to the point cloud data to be predicted to obtain a candidate new category model; optimizing the loss of the candidate new category model according to the pre-constructed adaptive loss control model to obtain a target new category model; and performing point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result. Therefore, by generating pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model, the prediction ability of the target old category model for the old category data can be utilized to generate more accurate old category pseudo labels, and by performing momentum update on the target old category model according to the old category pseudo labels and the new category true labels corresponding to the point cloud data to be predicted, the candidate new category model can retain the knowledge learned by the target old category model, and by optimizing the loss of the candidate new category model according to the pre-constructed adaptive loss control model, the adaptive parameters in the adaptive loss control model can be used to further balance the learning processes of different categories in the candidate new category model, thereby alleviating the prediction deviation problem caused by the quantity imbalance and obtaining a target new category model with higher incremental prediction accuracy, and then performing point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result.

[0159] Each module in the above large-scale scene point cloud incremental segmentation device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory in the computer device in software form so that the processor can call and execute the operations corresponding to the above modules.

[0160] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 10As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the point cloud data to be predicted. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a large-scale point cloud incremental segmentation method.

[0161] Those skilled in the art can understand that Figure 10 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0162] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0163] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0164] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0167] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0168] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A large scene point cloud incremental segmentation method, characterized by: The method comprises: Acquire point cloud data to be predicted; wherein the point cloud data to be predicted is three-dimensional point cloud data of a large-scale urban scene; Generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels; Performing momentum update on the target old category model according to the old category pseudo label and the new category true label corresponding to the point cloud data to be predicted, to obtain a candidate new category model; Performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model; Performing point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result; Obtain the new and old category data of the data set where the point cloud data to be predicted is located; determining a first loss coefficient according to the new and old category data and the number of momentum update rounds of the target old category model; Structural constraints are applied to the new and old category data by means of parameter balancing to obtain a second loss coefficient; wherein the basic category ratio data of the new and old category data is proportional to the second loss coefficient; A loss model is constructed according to the first loss coefficient, the second loss coefficient and a pre-constructed loss control model to obtain the adaptive loss control model.

2. The method according to claim 1, characterized in that Generating pseudo labels for the old category data of the point cloud data to be predicted based on the pre-trained target old category model to obtain old category pseudo labels includes: Performing category prediction on old category data of the point cloud data to be predicted according to the target old category model to obtain a plurality of supervoxel prediction results; wherein the old category data includes a plurality of supervoxels; When there is a candidate prediction category result in the supervoxel prediction result that is greater than a preset threshold, the old category pseudo labels of all points in the supervoxel corresponding to the supervoxel prediction result are set as the candidate prediction category result; In the case that there is no predicted category result greater than a preset threshold in the supervoxel prediction results, the old category pseudo labels are all set to the corresponding predicted category results.

3. The method according to claim 1, characterized in that The step of performing momentum updating on the target old category model according to the old category pseudo label and the new category true label corresponding to the point cloud data to be predicted to obtain a candidate new category model includes: determining an old category feature prototype according to the target old category model; Performing momentum update on the old category feature prototype according to the old category pseudo label and the new category true label to obtain a new category feature prototype; Distillation constraints are performed on the new category feature prototype and the old category feature prototype to obtain a candidate new category feature prototype, and the candidate new category model is determined according to the candidate new category feature prototype.

4. The method according to claim 1, wherein The step of performing loss optimization on the candidate new category model according to the pre-built adaptive loss control model to obtain the target new category model includes: Predicting the point cloud data to be predicted according to the candidate new category model to obtain a predicted label; Loss optimization is performed on the candidate new category model according to the predicted label and the adaptive loss control model to obtain the target new category model.

5. The method according to claim 1, wherein The method further comprises: Obtaining training point cloud data of the data set; Generate pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels; Performing momentum update on the initial old category model according to the training old category pseudo label and the new category true label to obtain a training new category model; Predicting the training point cloud data according to the training new category model to obtain a training prediction label; The loss control model is constructed according to the training predicted labels, the new category true labels and the training old category pseudo labels.

6. A large scene point cloud incremental segmentation device, characterized in that: The device comprises: Acquisition module, used to obtain the point cloud data to be predicted; A pseudo label generation module is used to generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels; A momentum update module is used to perform momentum update on the target old category model according to the old category pseudo label and the new category true label corresponding to the point cloud data to be predicted, so as to obtain a candidate new category model; a loss optimization module, configured to perform loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model; A point cloud segmentation module is used to perform point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result; A model construction module is used to obtain new and old category data of the data set containing the point cloud data to be predicted; determine a first loss coefficient based on the new and old category data and the number of momentum update rounds of the target old category model; structurally constrain the new and old category data by parameter balancing to obtain a second loss coefficient; wherein the basic category proportion data in the new and old category data is proportional to the second loss coefficient; and construct a loss model based on the first loss coefficient, the second loss coefficient, and a pre-constructed loss control model to obtain the adaptive loss control model.

7. The device according to claim 6, characterized in that The pseudo-label generation module is further used to perform category prediction on the old category data of the point cloud data to be predicted according to the target old category model to obtain multiple supervoxel prediction results; wherein the old category data includes multiple supervoxels; when there is a candidate prediction category result greater than a preset threshold in the supervoxel prediction results, the old category pseudo labels of all points in the supervoxel corresponding to the supervoxel prediction result are set to the candidate prediction category result; when there is no prediction category result greater than the preset threshold in the supervoxel prediction results, the old category pseudo labels are all set to the corresponding prediction category results.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images