Large-scene point cloud increment segmentation method and device, equipment, medium and program product

Abstract

The invention relates to a large-scene point cloud increment segmentation method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring to-be-predicted point cloud data; according to a pre-trained target old category model, carrying out pseudo tag generation on old category data of the to-be-predicted point cloud data to obtain an old category pseudo tag; performing momentum updating on the target old category model according to the old category pseudo tag and a new category real tag corresponding to the to-be-predicted point cloud data to obtain a candidate new category model; performing loss optimization on the candidate new category model according to a pre-constructed adaptive loss control model to obtain a target new category model; and performing point cloud segmentation on the to-be-predicted point cloud data according to the target new category model to obtain a segmentation result. By adopting the method, the incremental prediction precision for a large scene can be effectively improved, and an excellent effect can be achieved under the condition that the number of new and old categories, namely more new categories or fewer new categories, is unbalanced.

Description

Technical Field

[0001] The present application relates to the field of incremental learning technology, and in particular to a method, device, equipment, medium and program product for incremental segmentation of large scene point clouds. Background Art

[0002] With the development of incremental learning technology, incremental learning technology for three-dimensional point clouds has emerged. This technology is currently mainly used for relatively simple classification tasks. It presents more complex challenges for real urban scenes. For example, the larger scale, complex shapes and highly unbalanced semantic distribution in urban scenes are difficult to overcome. Therefore, how to more accurately predict the semantic distribution and accurately segment the three-dimensional point clouds in urban scenes has become an urgent problem to be solved in the field of incremental learning technology. Summary of the invention

[0003] Based on this, it is necessary to provide a large-scene point cloud incremental segmentation method, device, computer equipment, computer-readable storage medium and computer program product that can accurately segment three-dimensional point clouds in urban scenes in response to the above-mentioned technical problems.

[0004] In a first aspect, the present application provides a large scene point cloud incremental segmentation method, comprising:

[0005] Obtain the point cloud data to be predicted;

[0006] Generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels;

[0007] Perform momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, to obtain a candidate new category model;

[0008] Performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model;

[0009] The point cloud data to be predicted is segmented according to the target new category model to obtain a segmentation result.

[0010] In one embodiment, generating pseudo labels for old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels includes:

[0011] Performing category prediction on the old category data of the point cloud data to be predicted according to the target old category model to obtain a plurality of supervoxel prediction results; wherein the old category data includes a plurality of supervoxels;

[0012] In the case that there is a candidate prediction category result greater than a preset threshold in the supervoxel prediction result, the old category pseudo labels of all points in the supervoxel corresponding to the supervoxel prediction result are set as the candidate prediction category result;

[0013] In the case that there is no predicted category result greater than a preset threshold in the supervoxel prediction results, the old category pseudo labels are all set to the corresponding predicted category results.

[0014] In one embodiment, the step of performing momentum updating on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted to obtain a candidate new category model includes:

[0015] Determining an old category feature prototype according to the target old category model;

[0016] Perform momentum update on the old category feature prototype according to the old category pseudo label and the new category real label to obtain a new category feature prototype;

[0017] Distillation constraints are performed on the new category feature prototype and the old category feature prototype to obtain a candidate new category feature prototype, and the candidate new category model is determined according to the candidate new category feature prototype.

[0018] In one embodiment, performing loss optimization on the candidate new category model according to the pre-built adaptive loss control model to obtain the target new category model includes:

[0019] Predicting the point cloud data to be predicted according to the candidate new category model to obtain a predicted label;

[0020] The candidate new category model is subjected to loss optimization according to the predicted label and the adaptive loss control model to obtain the target new category model.

[0021] In one embodiment, the method further comprises:

[0022] Obtain the new and old category data of the data set where the point cloud data to be predicted is located;

[0023] Determining a first loss coefficient according to the new and old category data and the number of momentum update rounds of the target old category model;

[0024] Performing structural constraints on the new and old category data to obtain a second loss coefficient;

[0025] A loss model is constructed according to the first loss coefficient, the second loss coefficient and a pre-constructed loss control model to obtain the adaptive loss control model.

[0026] In one embodiment, the method further comprises:

[0027] Obtaining training point cloud data of the data set;

[0028] Generate pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels;

[0029] Performing momentum update on the initial old category model according to the training old category pseudo label and the new category real label to obtain a training new category model;

[0030] Predicting the training point cloud data according to the training new category model to obtain a training prediction label;

[0031] The loss control model is constructed according to the training predicted labels, the new category true labels and the training old category pseudo labels.

[0032] In a second aspect, the present application also provides a large scene point cloud incremental segmentation device, comprising:

[0033] An acquisition module is used to obtain the point cloud data to be predicted;

[0034] A pseudo label generation module is used to generate pseudo labels for the old category data of the point cloud data to be predicted according to a pre-trained target old category model to obtain old category pseudo labels;

[0035] A momentum update module, used for performing momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, so as to obtain a candidate new category model;

[0036] A loss optimization module, used for performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model;

[0037] The point cloud segmentation module is used to perform point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Obtain the point cloud data to be predicted;

[0040] Generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels;

[0041] Perform momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, to obtain a candidate new category model;

[0042] Performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model;

[0043] The point cloud data to be predicted is segmented according to the target new category model to obtain a segmentation result.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0045] Obtain the point cloud data to be predicted;

[0046] Generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels;

[0047] Perform momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, to obtain a candidate new category model;

[0048] Performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model;

[0049] The point cloud data to be predicted is segmented according to the target new category model to obtain a segmentation result.

[0050] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0051] Obtain the point cloud data to be predicted;

[0052] Generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels;

[0053] Perform momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, to obtain a candidate new category model;

[0054] Performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model;

[0055] The point cloud data to be predicted is segmented according to the target new category model to obtain a segmentation result.

[0056] The above-mentioned large-scene point cloud incremental segmentation method, device, computer equipment, computer-readable storage medium and computer program product obtain the point cloud data to be predicted; generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain the old category pseudo labels; perform momentum update on the target old category model according to the old category pseudo labels and the new category true labels corresponding to the point cloud data to be predicted to obtain a candidate new category model; perform loss optimization on the candidate new category model according to the pre-built adaptive loss control model to obtain the target new category model; perform point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain the segmentation result. Therefore, by generating pseudo-labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model, the prediction ability of the target old category model for the old category data can be used to generate more accurate old category pseudo-labels, and the target old category model is momentum updated according to the old category pseudo-labels and the new category true labels corresponding to the point cloud data to be predicted, so that the candidate new category model can retain the knowledge learned by the target old category model, and the loss of the candidate new category model is optimized according to the pre-built adaptive loss control model, and the adaptive parameters in the adaptive loss control model can be used to further balance the learning process of different categories in the candidate new category model, thereby alleviating the prediction bias problem caused by the imbalance in quantity, and obtaining a target new category model with higher incremental prediction accuracy, and then the point cloud data to be predicted is segmented according to the target new category model to obtain the segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A diagram showing an application environment of a large scene point cloud incremental segmentation method in an embodiment;

[0059] Figure 2 A schematic diagram of a flow chart of a large scene point cloud incremental segmentation method in one embodiment;

[0060] Figure 3 A schematic diagram of a flow chart of a large scene point cloud incremental segmentation method in one embodiment;

[0061] Figure 4 A modular schematic diagram of an overall model application idea of ​​a large scene point cloud incremental segmentation method in an embodiment;

[0062] Figure 5 Schematic diagram of a voting-based pseudo-label generation module in a large scene point cloud incremental segmentation method in one embodiment;

[0063] Figure 6 A schematic diagram of a structure preservation module based on feature prototypes in a large scene point cloud incremental segmentation method in one embodiment;

[0064] Figure 7 A schematic diagram of the application effect of a large scene point cloud incremental segmentation method in one embodiment;

[0065] Figure 8 A schematic diagram showing the application effect of a large scene point cloud incremental segmentation method in one embodiment;

[0066] Fig. 9 is a structural block diagram of a large scene point cloud incremental segmentation device in one embodiment;

[0067] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] The large scene point cloud incremental segmentation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the point cloud data to be predicted; generates pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain the old category pseudo labels; updates the target old category model with momentum according to the old category pseudo labels and the new category real labels corresponding to the point cloud data to be predicted to obtain the candidate new category model; optimizes the loss of the candidate new category model according to the pre-built adaptive loss control model to obtain the target new category model; performs point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain the segmentation result. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0070] In an exemplary embodiment, Figure 2 As shown in the figure, a large scene point cloud incremental segmentation method is provided, which can be applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S202 to S210. Among them:

[0071] Step S202, obtaining point cloud data to be predicted.

[0072] Among them, the point cloud data to be predicted can adopt three-dimensional point cloud data of large-scale urban scenes.

[0073] In some embodiments, the point cloud data to be predicted can be obtained from the three-dimensional point cloud data set of the urban scene, or the three-dimensional point cloud data of the urban scene in other cloud servers can be obtained in real time as the point cloud data to be predicted, but is not limited thereto.

[0074] Step S204 , generating pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels.

[0075] Among them, the target old category model is a target old category model that can recognize old categories by fully supervising the initial category model using all the old category data. The pseudo-label generation process can be achieved through a supervoxel voting strategy.

[0076] In some embodiments, pseudo-labels are generated for old category data of the point cloud data to be predicted according to a pre-trained target old category model to obtain old category pseudo-labels, including: category prediction is performed on the old category data of the point cloud data to be predicted according to the target old category model to obtain multiple supervoxel prediction results; wherein the old category data includes multiple supervoxels; when there is a candidate prediction category result greater than a preset threshold in the supervoxel prediction result, the old category pseudo-labels of all points in the supervoxel corresponding to the supervoxel prediction result are set as the candidate prediction category result; when there is no prediction category result greater than the preset threshold in the supervoxel prediction result, the old category pseudo-labels are set as the corresponding prediction category results.

[0077] Among them, the supervoxel may refer to the background supervoxel in the point cloud data to be predicted. The supervoxel prediction result will include the semantic distribution prediction results of each category in the current supervoxel. Each semantic distribution prediction result needs to be greater than a preset minimum threshold to be recorded, and the semantic distribution prediction of each category in the supervoxel prediction result is stored in the corresponding candidate prediction category result. The preset threshold may be 0.7. It should be noted that when the semantic distribution prediction result of a category is greater than 0.7, it means that more than 70% of the point prediction category results in the current supervoxel are of this category, and is not limited to this.

[0078] In some embodiments, the target old category model is used to perform category prediction on each supervoxel in the old category data of the point cloud data to be predicted, and prediction result statistics are performed on all points in a supervoxel to obtain multiple supervoxel prediction results.

[0079] In some embodiments, if the number of semantic distribution prediction results of a certain category in the supervoxel prediction result is greater than a preset threshold, that is, if there are candidate prediction category results in the supervoxel prediction result that are greater than the preset threshold, then the old category pseudo labels of all points in the supervoxel corresponding to the supervoxel prediction result are set to the candidate prediction category result.

[0080] In some embodiments, if the set threshold is not exceeded, that is, if there is no predicted category result in the supervoxel prediction result that is greater than the preset threshold, the old category pseudo labels are set to the corresponding predicted category results, and the old category pseudo label corresponding to each point is the corresponding prediction result.

[0081] In this embodiment, the supervoxel prediction result is restricted by the supervoxel voting strategy for generating pseudo labels, which can improve the segmentation ability of the candidate new category model obtained by momentum updating the old category pseudo labels in the subsequent steps for the old category.

[0082] Step S206, momentum update is performed on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, so as to obtain a candidate new category model.

[0083] Among them, momentum update is the implementation process of a momentum update strategy. The momentum update strategy is used to ensure stable and gradual adjustment of the target old category model. The true label of the new category refers to the true label corresponding to the new category in the point cloud data to be predicted.

[0084] In some embodiments, a target old category model is momentum updated according to the old category pseudo-label and the new category true label corresponding to the point cloud data to be predicted to obtain a candidate new category model, including: determining the old category feature prototype according to the target old category model; momentum updating the old category feature prototype according to the old category pseudo-label and the new category true label to obtain a new category feature prototype; performing distillation constraints on the new category feature prototype and the old category feature prototype to obtain a candidate new category feature prototype, and determining the candidate new category model according to the candidate new category feature prototype.

[0085] In some embodiments, the target old category model is used to generate features of the old category, and then the feature prototype of each category in the old category is calculated to obtain the old category feature prototype, which can effectively guide the new category feature prototype to generate features of the old category, so that the new category feature prototype retains the knowledge learned by the target old category model, and the old category feature prototype p is calculated. k The process is shown in the following formula (1):

[0086] (1)

[0087] Among them, C k represents the number of points with semantic type k, and x i k It is the input point cloud with semantic type k.

[0088] In some embodiments, it should be noted that, taking into account the possible feature differences within the same category in the point cloud data to be predicted, during the momentum update of the old category feature prototype, the old category feature prototype is continuously updated based on the predicted old category pseudo-label and the new category true label to obtain the new category feature prototype, and the candidate new category model is determined based on the candidate new category feature prototype, which can ensure stable and gradual adjustment of the new category feature prototype.

[0089] In some embodiments, a momentum update strategy may be used to ensure a stable and progressive adjustment of the new category feature prototype. For each category, the new category feature prototype after being updated in the momentum update step is updated according to the following formula (2):

[0090] (2)

[0091] Among them, μ is a parameter that can be set, and t is the number of iterations of momentum update.

[0092] In some embodiments, distillation constraints are applied to the new category feature prototype and the old category feature prototype to obtain candidate new category feature prototypes, and a candidate new category model is determined based on the candidate new category feature prototypes, so that the candidate new category model retains the learning ability of the old category.

[0093] In this embodiment, the target old category model is momentum updated according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted to obtain a candidate new category model. The process of momentum updating the target old category model can be regarded as a feature structure constraint module. Such constraints can solve the catastrophic forgetting problem that causes a significant degradation in the performance of the old category.

[0094] Step S208, performing loss optimization on the candidate new category model according to the pre-built adaptive loss control model to obtain the target new category model.

[0095] Among them, the pre-built adaptive loss control model is a weight loss coefficient in the adaptive loss control model determined according to the new category and the related data of the old category in the point cloud data to be predicted, so as to realize the model of adaptive adjustment of loss to solve the loss optimization deviation caused by data imbalance.

[0096] In some embodiments, the loss of the candidate new category model is optimized according to a pre-built adaptive loss control model to obtain a target new category model, including: predicting the point cloud data to be predicted according to the candidate new category model to obtain a predicted label; and optimizing the loss of the candidate new category model according to the predicted label and the adaptive loss control model to obtain a target new category model.

[0097] In some embodiments, the candidate new category model is first used to predict the point cloud data to be predicted to obtain a predicted label, and then the total loss is calculated based on the predicted label and the adaptive loss control model, and the total loss is converged and optimized. When the total loss converges and optimizes to the minimum, the model parameters of the candidate new category model at this time are obtained, and the candidate new category model under this model parameter is used as the target new category model.

[0098] Step S210, performing point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result.

[0099] The segmentation result may be obtained by marking different categories of point cloud data with different colors, thereby distinguishing different categories of point clouds in the point cloud data to be predicted, and obtaining a segmentation result containing different color markings.

[0100] In some embodiments, point cloud categories are predicted for the point cloud data to be predicted according to the target new category model to obtain point cloud data corresponding to different categories in the point cloud data to be predicted, and the point cloud data of different categories are segmented and marked, so as to segment the point cloud data of different categories and obtain segmentation results.

[0101] In the above-mentioned large-scene point cloud incremental segmentation method, the point cloud data to be predicted is obtained; pseudo-labels are generated for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain the old category pseudo-labels; the target old category model is momentum updated according to the old category pseudo-labels and the new category true labels corresponding to the point cloud data to be predicted to obtain the candidate new category model; the loss of the candidate new category model is optimized according to the pre-built adaptive loss control model to obtain the target new category model; point cloud segmentation is performed on the point cloud data to be predicted according to the target new category model to obtain the segmentation result. Therefore, by generating pseudo-labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model, the prediction ability of the target old category model for the old category data can be used to generate more accurate old category pseudo-labels, and the target old category model is momentum updated according to the old category pseudo-labels and the new category true labels corresponding to the point cloud data to be predicted, so that the candidate new category model can retain the knowledge learned by the target old category model, and the loss of the candidate new category model is optimized according to the pre-built adaptive loss control model, and the adaptive parameters in the adaptive loss control model can be used to further balance the learning process of different categories in the candidate new category model, thereby alleviating the prediction bias problem caused by the imbalance in quantity, and obtaining a target new category model with higher incremental prediction accuracy, and then the point cloud data to be predicted is segmented according to the target new category model to obtain the segmentation result.

[0102] In an exemplary embodiment, Figure 3 As shown, the large scene point cloud incremental segmentation method also includes: steps S302 to S308. Among them:

[0103] Step S302, obtaining the new and old category data of the data set where the point cloud data to be predicted is located.

[0104] Among them, the new and old category data include: the loss weight of the old category, the loss weight of the new category, and the ratio between the new category point cloud data and the old category point cloud data.

[0105] In some embodiments, the new and old category data of the data set can be directly calculated from the point cloud data in the data set where the point cloud data to be predicted is located, or the new and old category data recorded in the data set can be directly obtained, but is not limited thereto.

[0106] Step S304, determining a first loss coefficient according to the new and old category data and the number of momentum update rounds of the target old category model.

[0107] Among them, the first loss coefficient is a weight coefficient used for dynamically adjusting the loss weight of the old category and the loss weight of the new category.

[0108] In some embodiments, according to the momentum update round number of the current target old category model, the loss weight of the old category and the loss weight of the new category are dynamically adjusted by cosine annealing, as shown in the following formula (3):

[0109] (3)

[0110] in, is the first loss coefficient, and epoch is the number of momentum update rounds.

[0111] Step S306, structural constraints are performed on the new and old category data to obtain a second loss coefficient.

[0112] Among them, the structural constraint refers to the imbalance between processing the newly added new category point cloud data and the basic old category point cloud data. The second loss coefficient is a coefficient related to the ratio of training points. The ratio of training points refers to the ratio between the new category point cloud data and the old category point cloud data in the point cloud data to be predicted.

[0113] In some embodiments, the new and old category data are structurally constrained by parameter balancing to obtain a second loss coefficient. The specific design process of the second loss coefficient is shown in the following formula (4):

[0114] (4)

[0115] in, is the second loss coefficient, p represents the proportion of the basic category, a and b are parameters that control the smoothness of the structural constraints. It should be noted that It is proportional to p. When p is small, the overall loss function will focus more on maintaining the structure, thereby avoiding the occurrence of catastrophic forgetting problems.

[0116] Step S308, constructing a loss model according to the first loss coefficient, the second loss coefficient and the pre-constructed loss control model to obtain an adaptive loss control model.

[0117] Among them, the loss control model contains three different types of loss calculation modules.

[0118] In some embodiments, the weight coefficient corresponding to each loss calculation module in the loss control model is confirmed according to the first loss coefficient and the second loss coefficient, and then the loss model is constructed according to each weight coefficient and each loss calculation module to obtain an adaptive loss control model.

[0119] In some embodiments, the method also includes: acquiring training point cloud data of the data set; generating pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels; performing momentum update on the initial old category model according to the training old category pseudo labels and new category true labels to obtain training new category model; predicting the training point cloud data according to the training new category model to obtain training prediction labels; and constructing a loss control model based on the training prediction labels, new category true labels, and training old category pseudo labels.

[0120] The training point cloud data is a subset of point cloud data with a certain data volume in the data set, and contains all types of point cloud data in the data set. The initial old category model is pre-trained and used to identify old category point cloud data in the training point cloud data.

[0121] In some embodiments, by acquiring training point cloud data of a data set, pseudo-labels are generated for the training point cloud data according to the initial old category model to obtain training old category pseudo-labels, and momentum is updated on the initial old category model according to the training old category pseudo-labels and new category true labels to obtain training new category models; the training point cloud data are predicted according to the training new category model to obtain training prediction labels, and the loss data between the training prediction labels and the true labels can be used to measure the category prediction ability of the current initial old category model for the training point cloud data during the momentum update process, thereby constructing a loss control model based on the training prediction labels, the new category true labels and the training old category pseudo-labels.

[0122] The loss control model includes a first loss control sub-model, a second loss control sub-model, and a third loss control sub-model.

[0123] In some embodiments, first, for the training new category model, the background and the new category with the real label in the new category are directly calculated using the cross entropy loss to determine the first loss control sub-model. The specific determination process is shown in the following formula (5):

[0124] (5)

[0125] Among them, N novel is the number of point cloud data in training the new category model, y is the true label, and p is the training prediction label.

[0126] In some embodiments, for the training old category pseudo labels generated by the initial old category model for the training point cloud data, we use these training old category pseudo labels to calculate the cross entropy of the training new category model, so as to determine the second loss control sub-model. The specific second loss control sub-model determination process is shown in the following formula (6):

[0127] (6)

[0128] in, is the number of old category point cloud data for training point cloud data, and is the true label corresponding to the i-th old category point cloud data, Pseudo labels for the generated training old categories.

[0129] In some embodiments, in order to further confirm the third loss control sub-model by training the old category pseudo-labels and training the predicted labels, the specific second loss control sub-model determination process is shown in the following formula (7):

[0130]

[0131] Among them, T is also a control parameter used to adjust the distribution similarity ratio, p is the calculation prototype corresponding to the training old category pseudo label, and f is the calculation prototype corresponding to the training prediction label.

[0132] In some embodiments, a loss control model is finally constructed based on the training prediction label, the new category true label, and the training old category pseudo label, as shown in the following formula (8):

[0133] (8).

[0134] In this embodiment, when training the initial old category model, the point cloud data of the new category is regarded as the background. When training the new category model, although the specific label of the old category is unknown, since the initial old category model has the ability to segment the old category, the initial old category model can be used to segment the point cloud of the old category to obtain the predicted value. At the same time, pseudo labels are generated through the supervoxel voting strategy, and are used together with the labels of the new category to train the new category model. In addition, we also introduce a feature structure constraint module to use the basic category prototype to retain the learned knowledge. Moreover, constructing an adaptive loss control model through parameter control can control the loss ratio of the new and old categories and achieve adaptive adjustment.

[0135] In order to understand the solution of this application more clearly, Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8The details are as follows:

[0136] In some embodiments, Figure 4 As shown in the upper part a of the figure, first, using the old category dataset D base For Model E base Train to obtain a target old category model E that can recognize old categories base Then, in this initial model E base Based on this, we use a dataset D containing new category data novel Further training is performed to obtain a new target new category model E novel It should be noted that the initial training strategy of the model is consistent with the 3DPC-CISS method. base When the new category data is regarded as background b, the new category model E novel When the old category is unknown, the target old category model has the ability to segment the old category, so the target old category model can be used to segment the point cloud of the old category to obtain the predicted value. At the same time, pseudo labels are generated based on the predicted value and supervoxel voting strategy, and used together with the new category label to train the new model. In addition, we also introduced a feature structure constraint module to apply to the momentum update process, using the feature prototype of the basic category to retain the learned knowledge. In addition, the pre-built adaptive loss control model controls the loss ratio of the new and old categories through parameters, thereby achieving adaptive adjustment.

[0137] In some embodiments, Figure 5 As shown, Figure 5 It shows the process of generating pseudo labels for old category data of predicted point cloud data according to the pre-trained target old category model. Figure 5 As shown in the figure, a background supervoxel containing old category data is selected as a prediction branch in the point cloud data to be predicted, and all the prediction results are summarized after the prediction is completed. It should be further explained that even if all the old categories have been used to train the old model for full supervision, there is still a big difference between the prediction results of the old categories and the true labels. If the prediction results are not restricted, the segmentation ability of the new model for the old categories will be greatly reduced. Therefore, the supervoxel voting strategy is used here to generate pseudo labels. Figure 5 The figure shows one of the branches in the pseudo-label generation process. Specifically, the prediction results of all points in a supervoxel are counted. If the number of prediction results of a certain category is greater than the set threshold, the pseudo-labels of all points in the supervoxel are set to the prediction result. If it does not exceed the set threshold, the pseudo-label corresponding to each point is the corresponding prediction result. The implementation method is as follows: Figure 5As shown: the prediction results greater than the first threshold are subjected to a second threshold judgment by a double threshold method. If there is a prediction result greater than the second threshold among the prediction results greater than the first threshold, the pseudo labels of all points in the supervoxel are set to the prediction result.

[0138] In some embodiments, Figure 6 As shown, Figure 6 The module process of performing momentum update on the target old category model according to the old category pseudo-label and the new category true label corresponding to the point cloud data to be predicted is shown to obtain the candidate new category model. Specifically, in the training process for a new data set, the true labels of all categories in the data set are usually obtained as the basis for momentum update, and then the target old category model and the candidate new category model are used to extract their respective feature prototypes, and the new category true label and the old category pseudo-label are used to gradually optimize their respective feature prototypes, and the semantic distribution information of the two is output into the embedding space for momentum update, so as to obtain the candidate new category model.

[0139] In some embodiments, Figure 7 and Figure 8 The application effect of this solution in visual segmentation effect display and the application effect of this solution in actual point cloud segmentation accuracy test are respectively demonstrated. Figure 7 In the figure, it is shown that for two different point cloud data scenarios, our solution (Ours) segmented the point cloud data of the two data sets a and b, which is closest to the original data display effect of a and b. Figure 8 In the scenarios of small classification datasets (including TE, VG, WA, BR and BU) and large classification datasets (including BO, VG, WA, BR and VE), the semantic segmentation prediction results (Accyarcy and mIou) of this scheme (Ours) for different types (qingdao, longhua and campus) are better than other point cloud segmentation schemes (JT, BL, FT and 3DPC).

[0140] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0141] It should be noted that the feature extraction algorithm used in the present invention can be directly replaced by any other feature extraction algorithm. The specific framework of the network is not limited, for example, a neural network basic framework based on a multi-layer perceptron, gridding, or a hash table grid can be used. In addition, the adaptive parameter adjustment mechanism in the present invention is not limited to the declared function form, and any adjustment parameter that plays a monotonic change can be replaced accordingly.

[0142] Based on the same inventive concept, the embodiment of the present application also provides a large scene point cloud incremental segmentation device for implementing the large scene point cloud incremental segmentation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more large scene point cloud incremental segmentation device embodiments provided below can refer to the limitations of the large scene point cloud incremental segmentation method above, and will not be repeated here.

[0143] In an exemplary embodiment, Fig. 9 As shown, a large scene point cloud incremental segmentation device 900 is provided, including: an acquisition module 901, a pseudo label generation module 902, a momentum update module 903, a loss optimization module 904 and a point cloud segmentation module 905, wherein:

[0144] An acquisition module 901 is used to acquire point cloud data to be predicted;

[0145] A pseudo label generation module 902 is used to generate pseudo labels for old category data of the point cloud data to be predicted according to a pre-trained target old category model to obtain old category pseudo labels;

[0146] A momentum update module 903 is used to perform momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, so as to obtain a candidate new category model;

[0147] A loss optimization module 904 is used to optimize the loss of the candidate new category model according to the pre-built adaptive loss control model to obtain a target new category model;

[0148] The point cloud segmentation module 905 is used to perform point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result.

[0149] In some embodiments, the pseudo label generation module 902 is further used to perform category prediction on the old category data of the point cloud data to be predicted according to the target old category model to obtain multiple supervoxel prediction results; wherein the old category data includes multiple supervoxels;

[0150] When there is a candidate prediction category result in the supervoxel prediction result that is greater than a preset threshold, the old category pseudo labels of all points in the supervoxel corresponding to the supervoxel prediction result are set as the candidate prediction category result;

[0151] When there is no predicted category result greater than a preset threshold in the supervoxel prediction results, the old category pseudo labels are all set to the corresponding predicted category results.

[0152] In some embodiments, the momentum update module 903 is further used to determine the old category feature prototype according to the target old category model;

[0153] Perform momentum update on the feature prototype of the old category according to the pseudo label of the old category and the real label of the new category to obtain the feature prototype of the new category;

[0154] Distillation constraints are performed on the new category feature prototype and the old category feature prototype to obtain candidate new category feature prototypes, and candidate new category models are determined based on the candidate new category feature prototypes.

[0155] In some embodiments, the loss optimization module 904 is also used to predict the point cloud data to be predicted based on the candidate new category model to obtain a predicted label; and to optimize the loss of the candidate new category model based on the predicted label and the adaptive loss control model to obtain a target new category model.

[0156] In some embodiments, the device also includes: a model construction module, used to obtain new and old category data of the data set where the point cloud data to be predicted is located; determine the first loss coefficient based on the new and old category data and the number of momentum update rounds of the target old category model; structurally constrain the new and old category data to obtain the second loss coefficient; construct a loss model based on the first loss coefficient, the second loss coefficient and a pre-constructed loss control model to obtain an adaptive loss control model.

[0157] In some embodiments, the model building module is also used to obtain training point cloud data of the data set; generate pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels; perform momentum update on the initial old category model according to the training old category pseudo labels and the new category true labels to obtain a training new category model; predict the training point cloud data according to the training new category model to obtain training prediction labels; and build a loss control model based on the training prediction labels, the new category true labels and the training old category pseudo labels.

[0158] In the above-mentioned large-scene point cloud incremental segmentation device, the point cloud data to be predicted is obtained; pseudo-labels are generated for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain the old category pseudo-labels; the target old category model is momentum updated according to the old category pseudo-labels and the new category true labels corresponding to the point cloud data to be predicted to obtain the candidate new category model; the loss of the candidate new category model is optimized according to the pre-built adaptive loss control model to obtain the target new category model; point cloud segmentation is performed on the point cloud data to be predicted according to the target new category model to obtain the segmentation result. Therefore, by generating pseudo-labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model, the prediction ability of the target old category model for the old category data can be used to generate more accurate old category pseudo-labels, and the target old category model is momentum updated according to the old category pseudo-labels and the new category true labels corresponding to the point cloud data to be predicted, so that the candidate new category model can retain the knowledge learned by the target old category model, and the loss of the candidate new category model is optimized according to the pre-built adaptive loss control model, and the adaptive parameters in the adaptive loss control model can be used to further balance the learning process of different categories in the candidate new category model, thereby alleviating the prediction bias problem caused by the imbalance in quantity, and obtaining a target new category model with higher incremental prediction accuracy, and then the point cloud data to be predicted is segmented according to the target new category model to obtain the segmentation result.

[0159] Each module in the above-mentioned large scene point cloud incremental segmentation device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0160] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store point cloud data to be predicted. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large scene point cloud incremental segmentation method is implemented.

[0161] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0164] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0166] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0168] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A large scene point cloud incremental segmentation method, characterized in that: The method comprises: Obtain the point cloud data to be predicted; Generate pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain old category pseudo labels; Perform momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, to obtain a candidate new category model; Performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model; The point cloud data to be predicted is segmented according to the target new category model to obtain a segmentation result.

2. The method according to claim 1, characterized in that The generating of pseudo labels for the old category data of the point cloud data to be predicted according to the pre-trained target old category model to obtain the old category pseudo labels includes: Performing category prediction on the old category data of the point cloud data to be predicted according to the target old category model to obtain a plurality of supervoxel prediction results; wherein the old category data includes a plurality of supervoxels; In the case that there is a candidate prediction category result greater than a preset threshold in the supervoxel prediction result, the old category pseudo labels of all points in the supervoxel corresponding to the supervoxel prediction result are set as the candidate prediction category result; In the case that there is no predicted category result greater than a preset threshold in the supervoxel prediction results, the old category pseudo labels are all set to the corresponding predicted category results.

3. The method according to claim 1, characterized in that: The step of performing momentum updating on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted to obtain a candidate new category model comprises: Determining an old category feature prototype according to the target old category model; Perform momentum update on the old category feature prototype according to the old category pseudo label and the new category real label to obtain a new category feature prototype; Distillation constraints are performed on the new category feature prototype and the old category feature prototype to obtain a candidate new category feature prototype, and the candidate new category model is determined according to the candidate new category feature prototype.

4. The method according to claim 1, characterized in that The step of performing loss optimization on the candidate new category model according to the pre-built adaptive loss control model to obtain the target new category model includes: Predicting the point cloud data to be predicted according to the candidate new category model to obtain a predicted label; The candidate new category model is subjected to loss optimization according to the predicted label and the adaptive loss control model to obtain the target new category model.

5. The method according to claim 1, characterized in that The method further comprises: Obtain the new and old category data of the data set where the point cloud data to be predicted is located; Determining a first loss coefficient according to the new and old category data and the number of momentum update rounds of the target old category model; Performing structural constraints on the new and old category data to obtain a second loss coefficient; A loss model is constructed according to the first loss coefficient, the second loss coefficient and a pre-constructed loss control model to obtain the adaptive loss control model.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining training point cloud data of the data set; Generate pseudo labels for the training point cloud data according to the initial old category model to obtain training old category pseudo labels; Performing momentum update on the initial old category model according to the training old category pseudo label and the new category real label to obtain a training new category model; Predicting the training point cloud data according to the training new category model to obtain a training prediction label; The loss control model is constructed according to the training predicted labels, the new category true labels and the training old category pseudo labels.

7. A large scene point cloud incremental segmentation device, characterized in that: The device comprises: An acquisition module is used to obtain the point cloud data to be predicted; A pseudo label generation module is used to generate pseudo labels for the old category data of the point cloud data to be predicted according to a pre-trained target old category model to obtain old category pseudo labels; A momentum update module, used for performing momentum update on the target old category model according to the old category pseudo-label and the new category real label corresponding to the point cloud data to be predicted, so as to obtain a candidate new category model; A loss optimization module, used for performing loss optimization on the candidate new category model according to a pre-built adaptive loss control model to obtain a target new category model; The point cloud segmentation module is used to perform point cloud segmentation on the point cloud data to be predicted according to the target new category model to obtain a segmentation result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images