A semantic segmentation method and device based on point cloud data and a terminal device

Abstract

The application discloses a semantic segmentation method and device based on point cloud data and a terminal device, and can make a base model and a target model predict each local geometric structure region constructed during training of a model, combine a first geometric relationship predicted by the target model with a second geometric relationship predicted by the base model to construct a loss function, and iteratively update parameters of the model through the loss function, so that the model can consider features of new and old class data during the training process, memory of old class knowledge is retained, and the new class data can be effectively learned, so that weights of new and old knowledge are effectively balanced, catastrophic forgetting of the old class knowledge in the incremental learning process is avoided, the model accurately outputs a target class result corresponding to each segmentation region, and the application can improve accuracy of class recognition on the point cloud data in an actual semantic segmentation scene.

Description

Technical Field

[0001] This invention relates to the field of point cloud data classification technology, and in particular to a semantic segmentation method, apparatus and terminal device based on point cloud data. Background Technology

[0002] The rapid development of LiDAR technology has greatly promoted the research and application of point cloud-based vision tasks. Point cloud data, with its unique three-dimensional structural information, provides crucial support for fields such as virtual reality and autonomous driving. In these applications, accurate identification and understanding of objects within a scene is essential, and point cloud semantic segmentation technology is a vital means to achieve this goal.

[0003] However, with continuous equipment updates and the addition of new data categories, enabling models to continuously learn and recognize newly added data or categories has become a crucial issue. Due to the unordered and unstructured nature of 3D point clouds, traditional 2D methods are difficult to directly extend to 3D point clouds. Furthermore, existing 3D point cloud semantic segmentation methods, during incremental learning, may suffer from the influx of new knowledge that may overwrite or interfere with the semantic representation of old knowledge. Consequently, after introducing new point cloud categories, the model gradually forgets previously learned knowledge, making it difficult to retain past knowledge and leading to catastrophic forgetting after the incremental phase. Therefore, existing technologies cannot effectively retain old category knowledge during incremental learning and cannot effectively learn new categories of data, resulting in decreased accuracy in semantic recognition and segmentation of point cloud data. This leads to inaccurate category recognition of point cloud data in real-world semantic segmentation scenarios. Summary of the Invention

[0004] This invention provides a semantic segmentation method, apparatus, and terminal device based on point cloud data. It retains the memory of old class knowledge while effectively learning new class data, thereby effectively balancing the weight of old and new knowledge during the training and updating of the target model. This avoids the catastrophic forgetting of old class knowledge during incremental learning and effectively solves the problem in existing technologies that the classification of point cloud data is not accurate enough due to the inability to better retain old class knowledge and the inability to better learn new class data.

[0005] An embodiment of the present invention provides a semantic segmentation method based on point cloud data, comprising:

[0006] Acquire the point cloud data to be identified;

[0007] The point cloud data to be identified is input into a preset point cloud classification model so that the point cloud classification model can segment the point cloud data to be identified and output the target category result corresponding to each segmented region.

[0008] The training of the point cloud classification model includes:

[0009] Acquire several sets of sample 3D point cloud data and the corresponding sample category results;

[0010] A sample of 3D point cloud data and the corresponding sample category results are used as training samples to train an initial model and generate a basic model.

[0011] Extract the 3D point cloud data of each sample and the corresponding sample category results one by one, and repeat the following model parameter update operation until the target model converges to generate the final point cloud classification model:

[0012] The extracted sample 3D point cloud data is used as input, and the corresponding sample category results are used as output to train the base model corresponding to the previous training stage to generate the target model.

[0013] Several local geometric structure regions are constructed in the currently extracted sample 3D point cloud data;

[0014] Obtain the first geometric relationship predicted by the target model for each of the local geometric structure regions, and at the same time obtain the second geometric relationship predicted by the base model corresponding to the previous training stage for each of the local geometric structure regions.

[0015] A first loss function is constructed based on each first geometric relationship and the corresponding second geometric relationship, and the network parameters of the target model are updated once based on the loss function.

[0016] If the target model is determined to have not converged, the updated target model will be used as the base model for the next model parameter update operation.

[0017] Preferably, constructing several local geometric structure regions in the currently extracted sample 3D point cloud data includes:

[0018] Several anchor points are generated in the sample 3D point cloud data according to the farthest point sampling strategy;

[0019] For each anchor point, calculate the distance between each point in the sample 3D point cloud data and the anchor point, and select several target points based on the distance between each point and the anchor point and the preset distance threshold;

[0020] Based on each anchor point and its corresponding target points, several local geometric structure regions are generated.

[0021] Preferably, the generation of the first geometric relationship includes:

[0022] A local geometric structure region is input into the target model, so that the target model calculates the geometric attributes between each point in the local geometric structure region and its adjacent points; based on the geometric attributes corresponding to each point, a first geometric relationship is generated to describe the geometric characteristics of the local geometric structure region; wherein, the geometric attributes include: position, distance, angle, and curvature;

[0023] The generation of the second geometric relation includes:

[0024] A local geometric structure region is input into the base model corresponding to the previous training stage, so that the base model calculates the geometric attributes between each point in the local geometric structure region and its adjacent points; based on the geometric attributes corresponding to each point, a second geometric relationship is generated to describe the geometric characteristics of the local geometric structure region; wherein, the geometric attributes include: position, distance, angle and curvature.

[0025] Preferably, the step of constructing a corresponding first loss function based on each first geometric relationship and the corresponding second geometric relationship includes:

[0026] The first or second geometric relation can be calculated using the following formula:

[0027]

[0028] Among them, R a This represents the a-th first geometric relation or second geometric relation. Represents the a-th first geometric relation. Represents the a-th second geometric relation; This represents the coordinates of a target point within a local geometric region. This represents the coordinates of points adjacent to the target point within the local geometric region. This represents the geometric properties of a target point within a local geometric region. This represents the geometric properties of points adjacent to the target point within a local geometric region. Indicates a cascade operation;

[0029] The first loss function is calculated using the following formula:

[0030]

[0031] in, Let represent the first loss function, and 'a' represent the number of first or second geometric relations.

[0032] Preferably, before using the updated target model as the base model for the next model parameter update operation, the method further includes:

[0033] Input an unlabeled dataset into the updated target model so that the updated target model outputs the corresponding pseudo-label category.

[0034] The extracted sample 3D point cloud data and the corresponding sample category results are input into the updated target model so that the updated target model outputs the corresponding first predicted label category.

[0035] The extracted sample 3D point cloud data and the corresponding sample category results are input into the base model corresponding to the previous training stage, so that the base model corresponding to the previous training stage outputs the corresponding second predicted label category.

[0036] The pseudo-label category and the first predicted label category are integrated to generate a hybrid label category;

[0037] The mixed label category is compared with the second predicted label category, and the target model, which has already been updated once, is updated a second time based on the comparison results.

[0038] Preferably, the pseudo-label category and the first predicted label category are integrated to generate a hybrid label category, including:

[0039] Generate mixed label categories according to the following formula:

[0040]

[0041]

[0042] in, Indicates mixed label categories, Indicates the first predicted label category. This represents the predicted probability of the i-th point corresponding to the pseudo-label. γ represents the probability of the closest point t adjacent to the i-th point, and γ represents the preset probability threshold. c represents the data of each point in the sample 3D point cloud data. bg This represents the data points in an unlabeled dataset. The Bernoulli distribution represents the weights of adjacent points. This indicates the degree of uncertainty in the predicted probability of each point.

[0043] Preferably, the network structure of the target model includes: a residual branch network and a main branch network;

[0044] The second update of the target model, which has already been updated once, based on the comparison results includes:

[0045] Based on the comparison results, the network parameters of the residual branch network in the target model that has been updated once are adjusted;

[0046] The adjusted residual branch network is integrated into the main branch network so that the target model, which has already been updated once, can be updated a second time.

[0047] Preferably, after performing a second update on the target model that has already been updated based on the comparison results, and before using the updated target model as the base model for the next model parameter update operation, the method further includes:

[0048] Based on several sets of 3D point cloud data belonging to the same sample category from the 3D point cloud data of each identified sample, multiple sets of sample data of the same category are obtained.

[0049] A set of similar sample data is used as a target category sample data, and the target category sample data is input into the base model corresponding to the previous training stage, so that the base model corresponding to the previous training stage outputs the first predicted category corresponding to the three-dimensional point cloud data of each target sample in the target category sample data;

[0050] Feature vectors are extracted for each first predicted category, and the average vector value corresponding to each feature vector is calculated; wherein, the average vector value is used to represent the feature representation corresponding to the target category sample data;

[0051] The three-dimensional point cloud data of a target sample from the target category sample data is input into the target model after a second update, so that the target model after the second update outputs the second predicted category corresponding to the three-dimensional point cloud data of the target sample.

[0052] The feature representation corresponding to the second predicted category is compared with the average vector value to generate a similarity difference;

[0053] Based on the similarity difference, the target model, which has already undergone two updates, is updated a third time.

[0054] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0055] One embodiment of the present invention provides a semantic segmentation device based on point cloud data, including: a point cloud data acquisition module and a category result recognition module;

[0056] The point cloud data acquisition module is used to acquire point cloud data to be identified;

[0057] The category result recognition module is used to input the point cloud data to be recognized into a preset point cloud classification model, so that the point cloud classification model can segment the point cloud data to be recognized and output the target category result corresponding to each segmented region.

[0058] The training of the point cloud classification model includes:

[0059] Acquire several sets of sample 3D point cloud data and the corresponding sample category results;

[0060] A sample of 3D point cloud data and the corresponding sample category results are used as training samples to train an initial model and generate a basic model.

[0061] Extract the 3D point cloud data of each sample and the corresponding sample category results one by one, and repeat the following model parameter update operation until the target model converges to generate the final point cloud classification model:

[0062] The extracted sample 3D point cloud data is used as input, and the corresponding sample category results are used as output to train the base model corresponding to the previous training stage to generate the target model.

[0063] Several local geometric structure regions are constructed in the currently extracted sample 3D point cloud data;

[0064] Obtain the first geometric relationship predicted by the target model for each of the local geometric structure regions, and at the same time obtain the second geometric relationship predicted by the base model corresponding to the previous training stage for each of the local geometric structure regions.

[0065] A first loss function is constructed based on each first geometric relationship and the corresponding second geometric relationship, and the network parameters of the target model are updated once based on the loss function.

[0066] If the target model is determined to have not converged, the updated target model will be used as the base model for the next model parameter update operation.

[0067] Based on the above method embodiments, the present invention provides corresponding terminal device embodiments.

[0068] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a semantic segmentation method based on point cloud data as described in the above-described embodiment of the invention.

[0069] The following benefits can be obtained by implementing the present invention:

[0070] This invention provides a semantic segmentation method, apparatus, and terminal device based on point cloud data. In the model training of this invention, the incremental learning process, in order to maintain the acquisition of old knowledge and avoid forgetting old class knowledge, applies the second geometric relationship predicted by the base model for each region corresponding to the current new class data each time the parameters of the target model (i.e., the new model) are iteratively updated. At the same time, in order to improve the model's ability to learn the features of new class data, this invention constructs a corresponding first loss function based on each first geometric relationship predicted by the new model and the corresponding second geometric relationship predicted by the base model, and iteratively updates the network parameters of the target model according to the first loss function. Thus, after determining that the target model has converged and using the converged target model as the point cloud classification model, the final generated point cloud classification model can retain old class knowledge while effectively learning new class data during the incremental learning process, thereby improving the performance of the model in the incremental learning stage. Compared with existing technologies, this invention can predict each constructed local geometric structure region, enabling both the base model and the target model to gain a deeper understanding of the local features of point cloud data. Furthermore, by combining the first geometric relationship predicted by the target model for each local geometric structure region with the second geometric relationships predicted by the base model in the previous training phase, a more accurate loss function can be constructed. The design of the first loss function allows the model to consider the features of both old and new classes of data during training, preserving the memory of old class knowledge while effectively learning new class data. This effectively balances the weights of old and new knowledge during the training and updating of the target model, avoiding catastrophic forgetting of old class knowledge during incremental learning. Moreover, it allows the target model to not only retain the learning of old class knowledge but also better improve its ability to learn new class data. Therefore, this invention can solve the catastrophic forgetting problem that occurs in the incremental learning process of the prior art. Based on the model training method and model update method of this invention, the accuracy of semantic recognition and semantic segmentation of point cloud data can be improved when the model is applied, so as to accurately output the target category result corresponding to each segmentation region. Thus, the model training method and semantic segmentation method of this invention can improve the accuracy of category recognition of point cloud data in actual semantic segmentation scenarios. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating a semantic segmentation method based on point cloud data provided in an embodiment of the present invention.

[0072] Figure 2 This is a schematic diagram of the incremental learning process provided in an embodiment of the present invention.

[0073] Figure 3This is a schematic diagram of the overall architecture of BRD-CIL (Balanced Residual Distillation Framework for Class Incremental Learning) provided in an embodiment of the present invention.

[0074] Figure 4 This is a schematic diagram of the structure of a semantic segmentation device based on point cloud data provided in an embodiment of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0076] like Figure 1 The diagram shown is a flowchart illustrating a semantic segmentation method based on point cloud data according to an embodiment of the present invention. The semantic segmentation method based on point cloud data includes:

[0077] Step S1: Obtain the point cloud data to be identified;

[0078] Step S2: Input the point cloud data to be identified into a preset point cloud classification model so that the point cloud classification model can segment the point cloud data to be identified and output the target category result corresponding to each segmented region;

[0079] The training of the point cloud classification model includes:

[0080] Acquire several sets of sample 3D point cloud data and the corresponding sample category results;

[0081] A sample of 3D point cloud data and the corresponding sample category results are used as training samples to train an initial model and generate a basic model.

[0082] Extract the 3D point cloud data of each sample and the corresponding sample category results one by one, and repeat the following model parameter update operation until the target model converges to generate the final point cloud classification model:

[0083] The extracted sample 3D point cloud data is used as input, and the corresponding sample category results are used as output to train the base model corresponding to the previous training stage to generate the target model.

[0084] Several local geometric structure regions are constructed in the currently extracted sample 3D point cloud data;

[0085] Obtain the first geometric relationship predicted by the target model for each of the local geometric structure regions, and at the same time obtain the second geometric relationship predicted by the base model corresponding to the previous training stage for each of the local geometric structure regions.

[0086] A first loss function is constructed based on each first geometric relationship and the corresponding second geometric relationship, and the network parameters of the target model are updated once based on the loss function.

[0087] If the target model is determined to have not converged, the updated target model will be used as the base model for the next model parameter update operation.

[0088] In a preferred embodiment, step S1 involves using a trained model to perform semantic segmentation and category recognition on the point cloud data to be identified. Illustratively, the point cloud data to be identified can come from various 3D scanning devices, such as LiDAR, depth cameras, etc. This data can capture 3D scenes, objects, or environments in the real world, forming a point cloud representation. Point cloud data typically consists of a series of 3D coordinate points, and each point may also contain other information, such as color, intensity, reflectivity, etc.

[0089] In practical applications, the point cloud data to be identified may cover multiple fields. For example, in the field of autonomous driving, point cloud data can be used to identify vehicles, pedestrians, obstacles, etc. on the road; in the field of robot vision, point cloud data can be used for environmental perception and target recognition; in Geographic Information Systems (GIS), point cloud data can be used for topographic mapping and urban planning. In short, the types of point cloud data to be identified are very diverse and can be determined according to specific application scenarios and needs. It is understood that the point cloud data to be identified in this invention conforms to the data format and features used during model training to ensure that the model can accurately perform semantic segmentation and category recognition.

[0090] In a preferred embodiment, for step S2, when the point cloud data to be identified is input into a preset point cloud classification model, the model will use the feature representation and classification ability learned during training to perform fine segmentation of the point cloud data and output the target category result corresponding to each segmentation region.

[0091] The target category result refers to the specific category label to which each segmented region in the point cloud data is classified. These category labels are usually determined according to application requirements and can be defined and labeled during the model training phase. Here are some examples of target category results:

[0092] In autonomous driving scenarios, target category results may include "vehicles," "pedestrians," "roads," "trees," and "buildings." Point cloud classification models will segment the point cloud data to be identified into different regions and identify which of the above categories each region belongs to.

[0093] In indoor environment perception applications, target category results may include "walls," "floors," "furniture," "doors and windows," etc. Point cloud classification models can distinguish different objects and surfaces in indoor spaces, providing crucial information for tasks such as robot navigation and scene understanding.

[0094] In topographic mapping applications, target category results may include "grassland," "rock," "water body," "forest," etc. Point cloud classification models can perform detailed segmentation and classification of the terrain surface, helping geographic information systems (GIS) to conduct terrain analysis and planning.

[0095] It is important to note that the specific target category results can vary depending on the needs of the actual application scenario and the characteristics of the data. In practical applications, developers can customize category labels according to specific tasks and use labeled point cloud data for supervised learning during the model training phase to train a point cloud classification model that can accurately output target category results.

[0096] To improve the accuracy of category recognition in this invention, a Balanced Residual Distillation Class-Incremental Learning Framework (BRD-CIL) for 3D point cloud semantic segmentation is proposed during model training. This framework includes two core designs: preventing the forgetting of acquired knowledge and facilitating model adaptation to new classes. Specifically, addressing the catastrophic forgetting problem caused by unstructured and disordered point clouds, this invention designs a Residual Distillation Learning Strategy (RDL). This strategy focuses on refining knowledge learned from the base model, enhancing the model's adaptability. Furthermore, this invention does not directly extend the classifier but retains the old structure composed of the base model and introduces residual branches to learn the residuals between old and new classes. Regarding the training strategy, unlike previous distillation methods that do not distinguish between old and new class regions, we only distill information from the old class and introduce residual branches to absorb information from the new class.

[0097] like Figure 2 The diagram illustrates the incremental class learning process, where D represents the dataset and C represents the class set. Incremental class learning enhances the model's ability to progressively learn and recognize new classes over time, while maintaining its performance on previously learned classes.

[0098] Specifically, during the training of the point cloud classification model, the model parameter update operation is repeatedly performed until the target model converges and the final point cloud classification model is generated.

[0099] Each time the model parameters are updated, it can be divided into three updates; for example... Figure 3 The overall architecture of BRD-CIL shown, in stage (a), the present invention is in D base The above training is done by encoder E base And classifier Y base The basic model M is composed of base In stage (b), in order to update the model to segment more categories, this invention designs a residual distillation strategy to capture data from M. base This invention incorporates knowledge learned in middle school. After training, it proposes a structural reparameterization method that can losslessly compress the model for continuous updates. Finally, it proposes a balanced pseudo-label learning strategy to promote the learning of new classes, thereby addressing training bias.

[0100] Understandably, the Balanced Residual Distillation (BRD-CIL) method based on the CIL task was used during the process of updating the model parameters; while Residual Distillation Learning (RDL) mitigates catastrophic forgetting by providing a regularization mechanism that allows the model to retain the representation of the old class, thereby leading to more robust and sustained learning over time in incremental learning scenarios.

[0101] After training the base model corresponding to the previous training stage using sample data to generate the target model, this invention first constructs several local geometric structure regions in the currently extracted sample 3D point cloud data. The specific process is as follows:

[0102] Several anchor points are generated in the sample 3D point cloud data according to the farthest point sampling strategy;

[0103] For each anchor point, calculate the distance between each point in the sample 3D point cloud data and the anchor point, and select several target points based on the distance between each point and the anchor point and the preset distance threshold;

[0104] Based on each anchor point and its corresponding target points, several local geometric structure regions are generated.

[0105] It is understood that the present invention will use P n ∈D novel Represented as a new class with S points, the current input point cloud, benefiting from the GFT module, can capture point-by-point associations to implicitly transfer learned knowledge from M. base Transfer to M novel And to represent the representation of each point, we can first use the farthest point on the input point cloud to uniformly sample a proportion r of points (there are a total of As anchor point Subsequently, the L2 distance between each point and the anchor point is calculated using xyz coordinates, and then the T nearest points are sampled using K-nearest neighbors (KNN) to create a local geometric region.

[0106] Furthermore, the present invention uses the target model to predict the first geometric relationship for each of the local geometric structure regions and uses the base model corresponding to the previous training stage to predict the second geometric relationship for each of the local geometric structure regions.

[0107] The generation of the first geometric relationship includes:

[0108] A local geometric structure region is input into the target model, so that the target model calculates the geometric attributes between each point in the local geometric structure region and its adjacent points; based on the geometric attributes corresponding to each point, a first geometric relationship is generated to describe the geometric characteristics of the local geometric structure region; wherein, the geometric attributes include: position, distance, angle, and curvature;

[0109] The generation of the second geometric relation includes:

[0110] A local geometric structure region is input into the base model corresponding to the previous training stage, so that the base model calculates the geometric attributes between each point in the local geometric structure region and its adjacent points; based on the geometric attributes corresponding to each point, a second geometric relationship is generated to describe the geometric characteristics of the local geometric structure region; wherein, the geometric attributes include: position, distance, angle and curvature.

[0111] To further achieve residual distillation learning (RDL), we utilize M base The output geometric relationships guide M novel Training with this approach provides a more flexible and adaptive method for updating the model's knowledge. This method allows for more gradual and subtle updates to the model parameters and enables the model to adjust its decision boundaries more smoothly, thus capturing pointwise relationships within the geometric domain, as expressed in:

[0112]

[0113] Among them, R a This represents the a-th first geometric relation or second geometric relation. Represents the a-th first geometric relation. Represents the a-th second geometric relation; This represents the coordinates of a target point within a local geometric region. This represents the coordinates of points adjacent to the target point within the local geometric region. This represents the geometric properties of a target point within a local geometric region. This represents the geometric properties of points adjacent to the target point within a local geometric region. This indicates a cascade operation.

[0114] Indicative, where p n P represents n The x, y, and z coordinates of a point in F. n Represent the corresponding geometric relationship features, the base model, and the new target model P. n The geometric relations on can be expressed as R. base and R novel .

[0115] The present invention can construct a corresponding first loss function based on the first geometric relationship and the corresponding second geometric relationship of each of the above-mentioned structural regions, and update the network parameters of the target model once according to the loss function; wherein, the loss function is based on a simple mean squared error (MSE) loss function:

[0116]

[0117] in, Let represent the first loss function, and 'a' represent the number of first or second geometric relations.

[0118] Through the loss function described above, we not only... base The structural and semantic knowledge contained therein is transferred to M novel In addition, it enhanced E. novel The expressive power of the dot enriches the meaning of the dot.

[0119] In a preferred embodiment, the RDL loss function of the present invention can also be expressed as:

[0120]

[0121] In the aforementioned update process, this invention predicts for each constructed local geometric structure region, enabling both the base model and the target model to gain a deeper understanding of the local features of the point cloud data. Furthermore, by combining the first geometric relationship predicted by the target model for each local geometric structure region with the second geometric relationships predicted by the base model in the previous training phase, a more accurate loss function can be constructed. By incorporating the prediction results (i.e., geometric relationships) from the base model into the loss function to guide model training, incremental learning and mitigation of catastrophic forgetting are achieved. In this process, the model updates its parameters by optimizing the loss function, including a regularization mechanism that allows the model to maintain the representation of the old classes and smoothly adjust its decision boundaries during training. This smooth adjustment helps avoid the problem of catastrophic forgetting caused by sudden changes in the decision boundaries, thus enabling the model to learn new classes more stably without forgetting old ones.

[0122] Furthermore, in a preferred embodiment, before using the updated target model as the base model for the next model parameter update operation, the method further includes:

[0123] Input an unlabeled dataset into the updated target model so that the updated target model outputs the corresponding pseudo-label category.

[0124] The extracted sample 3D point cloud data and the corresponding sample category results are input into the updated target model so that the updated target model outputs the corresponding first predicted label category.

[0125] The extracted sample 3D point cloud data and the corresponding sample category results are input into the base model corresponding to the previous training stage, so that the base model corresponding to the previous training stage outputs the corresponding second predicted label category.

[0126] The pseudo-label category and the first predicted label category are integrated to generate a hybrid label category;

[0127] The mixed label category is compared with the second predicted label category, and the target model, which has already been updated once, is updated a second time based on the comparison results.

[0128] Specifically, during the second update, the process is as follows:

[0129] The network structure of the target model of the present invention includes: a residual branch network and a main branch network;

[0130] The second update of the target model, which has already been updated once, based on the comparison results includes: adjusting the network parameters of the residual branch network in the target model that has already been updated once, according to the comparison results.

[0131] The adjusted residual branch network is integrated into the main branch network to enable a second update of the target model that has already undergone one update. This embodiment of the invention can further optimize overall performance and utilize the residual branch to absorb information from new classes for a second update of the target model.

[0132] You can use the pseudo tags generated by UPG Tags for new classes Integration yields hybrid tags It can be represented as:

[0133]

[0134]

[0135] in, Indicates mixed label categories, Indicates the first predicted label category. This represents the predicted probability of the i-th point corresponding to the pseudo-label. γ represents the probability of the closest point t adjacent to the i-th point, and γ represents the preset probability threshold. c represents the data of each point in the sample 3D point cloud data. bg This represents the data points in an unlabeled dataset. The Bernoulli distribution represents the weights of adjacent points. This indicates the degree of uncertainty in the predicted probability of each point.

[0136] Indicative This represents the Bernoulli distribution of the weights of neighboring points. To obtain... Through calculation xyz coordinates and The normalized cosine similarity between them, where t∈N(i). In Equation 5, c bg and They represent things that are different from semantic class C. base and C novel The "Background" class. M represents base The probability output, This represents the probability of the nearest point t among the adjacent points of point T. This represents the uncertainty in the predicted probability of each point. The predicted probability for each point, where γ represents the set uncertainty threshold (when the uncertainty is higher than the threshold, we consider the prediction inaccurate and use the predictions of surrounding points instead of that point's prediction). This represents the set of background classes for the new class (i.e., base class plus background class), c bgRepresents the set of background classes of the base class (i.e., the new class plus the background class);

[0137] Specifically, we use the pseudo-tags generated by UPG With the new category of labels By integrating the tags, a hybrid label was obtained. Then, we take the hybrid label as input and compare it with the label of the base class. Since the hybrid label contains high-accuracy labels of both the base class and the new class, it can guide the model's learning, maintaining the representation of the old class while absorbing knowledge of the new class.

[0138] By combining the second predicted class from the base model's output with the class information from the new target model's output to obtain a hybrid label, the model can better handle samples of new classes while maintaining accurate judgment of the base class. This design allows the model to fully utilize the knowledge of the base class when facing samples of new classes and avoids a decrease in accuracy for old classes due to the introduction of new classes. Therefore, through the design and utilization of hybrid labels, accurate segmentation for all classes can be achieved, thereby improving the model's generalization ability and adaptability.

[0139] To further integrate new information, this invention can also use M novel Output Q n and mixed tags The cross-entropy loss is as follows:

[0140]

[0141] Indicative, Y novel (·) is used to characterize a classifier that represents a new class model (it can only classify old classes), and is used to remember the old classes; Y res (·) is used to represent newly added residual branches, which are used to absorb new category information. In the secondary update process, this invention proposes a structural reparameterization method to seamlessly integrate the remaining branch information into the main branch:

[0142]

[0143] Where θ n and θ r They represent Y respectively novel (·) and Y res The parameters in parentheses (·) are represented by + and ^, which respectively indicate element-wise addition and parameter rearrangement. θ n Through zero-filling operations and linear transformations with θ r Fusion. Finally, residual branches are removed to maintain the model structure and M. base We will work together to make the next update.

[0144] It is understandable that the residual branch refers to the residual branch. The main branch refers to the part with the same structure as the base class model. In order to learn the knowledge of the new class, this invention adds a residual branch to learn the residual between the base class model and the new class model. After training, in order to maintain the smoothness of the network structure and facilitate the next incremental learning, this invention adopts a reparameterization method to maintain the uniformity and simplicity of the network model structure.

[0145] Furthermore, after updating the target model a second time based on the comparison results, and before using the updated target model as the base model for the next model parameter update operation, this embodiment of the invention also includes a three-stage update process, specifically:

[0146] Based on several sets of 3D point cloud data belonging to the same sample category from the 3D point cloud data of each identified sample, multiple sets of sample data of the same category are obtained.

[0147] A set of similar sample data is used as a target category sample data, and the target category sample data is input into the base model corresponding to the previous training stage, so that the base model corresponding to the previous training stage outputs the first predicted category corresponding to the three-dimensional point cloud data of each target sample in the target category sample data;

[0148] Feature vectors are extracted for each first predicted category, and the average vector value corresponding to each feature vector is calculated; wherein, the average vector value is used to represent the feature representation corresponding to the target category sample data;

[0149] The three-dimensional point cloud data of a target sample from the target category sample data is input into the target model after a second update, so that the target model after the second update outputs the second predicted category corresponding to the three-dimensional point cloud data of the target sample.

[0150] The feature representation corresponding to the second predicted category is compared with the average vector value to generate a similarity difference;

[0151] Based on the similarity difference, the target model, which has already undergone two updates, is updated a third time.

[0152] Understandably, for previously learned category information, in order for the model to summarize the representation of each category—that is, to represent a category using a general feature—we can input the data from the base class into the base class model. The outputs of the corresponding data belonging to each category of the base class are averaged to represent the prototype vector of each category (i.e., feature vectors are extracted for each first predicted category, and the average vector value corresponding to each feature vector is calculated). Because the model is being trained and continuously updated, we want to obtain a weight to guide the model's learning. In this formula, we input the data from the incremental process into the new model, compare the output of the new model with the prototype vector of the extracted base class, and output the similarity value between that point and all categories of the base class, which is a 1*n vector (assuming the number of base class categories is n). If the similarity is high, we consider it more likely to belong to the base class; if the similarity is low, we consider it more likely to belong to the new category. Furthermore, this invention can perform a third update on the target model after the second update based on the similarity difference, thereby further optimizing the model's performance.

[0153] Understandably, this invention proposes a BPL strategy to eliminate training bias between the base class and the new class. Specifically, by calculating C... base The average embeddings of all instances belonging to each class are used to store the average value of the base class representing the prototype. In BPL, the prototype acts as a representation of the base class, and after projecting all new samples into the learned embedding space, the normalized cosine score S between them and each prototype is calculated using standard cosine similarity. i Its description is as follows:

[0154]

[0155] Using S i As a reference for generating old and new masks, it addresses the model's preference for base classes. We believe that points with high similarity are more likely to belong to the old class, while points with low similarity are more likely to belong to the new class. To ensure that the backpropagation flow primarily originates from the new class and updates the most distinguishing positions while preserving the old representation, this invention can also generate old and new masks as follows:

[0156] m i =1-max(S) i )

[0157] Based on the generated mask, this invention can also design a weighted cross-entropy loss. And apply it to guide M novel Training:

[0158]

[0159] The goal described above is to increase the contribution of new class samples during the incremental phase, while encouraging the model to focus on learning the discriminative features of the new classes. By assigning greater weight to the new classes, their importance is prioritized during the learning process, thus better adapting to the new classes.

[0160] As can be seen from the three update schemes described above, in the first update process, this invention introduces a new paradigm called BRD-CIL (Balanced Residual Distillation Framework for Incremental Class Learning), aimed at enhancing 3D point cloud semantic segmentation. To address the challenge of catastrophic forgetting, a residual distillation learning strategy is proposed to transfer and refine acquired knowledge from previous classes. This preserves valuable information while incorporating new learning.

[0161] Furthermore, during the second update, this invention introduces a structural reparameterization technique to ensure the consistency of the network structure during continuous incremental training. The consistency of the network structure is crucial during incremental learning because it relates to the model's ability to effectively utilize previously learned knowledge. Through structural reparameterization, the model can maintain the consistency of its network structure during continuous incremental training, thereby ensuring the model's stability and generalization ability.

[0162] Finally, in the third update, to address the difficulty of learning new classes, this invention proposes a balanced pseudo-label learning strategy. This strategy effectively alleviates feature confusion and counteracts class-biased training targeting the base class. In incremental learning, learning new classes is often more difficult than learning base classes because data for them is typically scarce. The balanced pseudo-label learning strategy alleviates feature confusion, enabling the model to better learn the features of new classes. This update scheme not only improves the model's ability to recognize new classes but also further enhances the overall performance of the model.

[0163] In summary, through these three update schemes, this invention has achieved significant progress in addressing catastrophic forgetting, maintaining network structure consistency, and learning new types of data. These update schemes not only improve model performance but also make important contributions to the development of the field of 3D point cloud semantic segmentation.

[0164] like Figure 4 As shown, based on the above embodiments of various semantic segmentation methods based on point cloud data, the present invention provides corresponding embodiments of apparatus.

[0165] One embodiment of the present invention provides a semantic segmentation device based on point cloud data, including: a point cloud data acquisition module and a category result recognition module;

[0166] The point cloud data acquisition module is used to acquire point cloud data to be identified;

[0167] The category result recognition module is used to input the point cloud data to be recognized into a preset point cloud classification model, so that the point cloud classification model can segment the point cloud data to be recognized and output the target category result corresponding to each segmented region.

[0168] The training of the point cloud classification model includes:

[0169] Acquire several sets of sample 3D point cloud data and the corresponding sample category results;

[0170] A sample of 3D point cloud data and the corresponding sample category results are used as training samples to train an initial model and generate a basic model.

[0171] Extract the 3D point cloud data of each sample and the corresponding sample category results one by one, and repeat the following model parameter update operation until the target model converges to generate the final point cloud classification model:

[0172] The extracted sample 3D point cloud data is used as input, and the corresponding sample category results are used as output to train the base model corresponding to the previous training stage to generate the target model.

[0173] Several local geometric structure regions are constructed in the currently extracted sample 3D point cloud data;

[0174] Obtain the first geometric relationship predicted by the target model for each of the local geometric structure regions, and at the same time obtain the second geometric relationship predicted by the base model corresponding to the previous training stage for each of the local geometric structure regions.

[0175] A first loss function is constructed based on each first geometric relationship and the corresponding second geometric relationship, and the network parameters of the target model are updated once based on the loss function.

[0176] If the target model is determined to have not converged, the updated target model will be used as the base model for the next model parameter update operation.

[0177] It should be noted that the device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0178] Those skilled in the art will clearly understand that, for convenience and simplicity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0179] Based on the above embodiments of various semantic segmentation methods based on point cloud data, the present invention provides corresponding embodiments for terminal devices.

[0180] One embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a semantic segmentation method based on point cloud data as described in any embodiment of the present invention.

[0181] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0182] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0183] The memory can be used to store the computer program. The processor implements various functions of the terminal device by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart memory card (SMC), Secure Digital Card (SD) card, flash memory card (SD), at least one disk storage device, flash memory device or other volatile solid-state storage device.

[0184] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A semantic segmentation method based on point cloud data, characterized in that, include: Acquire the point cloud data to be identified; The point cloud data to be identified is input into a preset point cloud classification model so that the point cloud classification model can segment the point cloud data to be identified and output the target category result corresponding to each segmented region. The training of the point cloud classification model includes: Acquire several sets of sample 3D point cloud data and the corresponding sample category results; A sample of 3D point cloud data and the corresponding sample category results are used as training samples to train an initial model and generate a basic model. Extract the 3D point cloud data of each sample and the corresponding sample category results one by one, and repeat the following model parameter update operation until the target model converges to generate the final point cloud classification model: The extracted sample 3D point cloud data is used as input, and the corresponding sample category results are used as output to train the base model corresponding to the previous training stage to generate the target model. Several local geometric structure regions are constructed in the currently extracted sample 3D point cloud data; Obtain the first geometric relationship predicted by the target model for each of the local geometric structure regions, and at the same time obtain the second geometric relationship predicted by the base model corresponding to the previous training stage for each of the local geometric structure regions. A first loss function is constructed based on each first geometric relationship and the corresponding second geometric relationship, and the network parameters of the target model are updated once based on the loss function. If the target model is determined to have not converged, the updated target model will be used as the base model for the next model parameter update operation.

2. The semantic segmentation method based on point cloud data as described in claim 1, characterized in that, The process involves constructing several local geometric structure regions in the currently extracted sample 3D point cloud data, including: Several anchor points are generated in the sample 3D point cloud data according to the farthest point sampling strategy; For each anchor point, calculate the distance between each point in the sample 3D point cloud data and the anchor point, and select several target points based on the distance between each point and the anchor point and the preset distance threshold; Based on each anchor point and its corresponding target points, several local geometric structure regions are generated.

3. The semantic segmentation method based on point cloud data as described in claim 2, characterized in that... The generation of the first geometric relation includes: A local geometric structure region is input into the target model, so that the target model calculates the geometric attributes between each point in the local geometric structure region and its adjacent points; based on the geometric attributes corresponding to each point, a first geometric relationship is generated to describe the geometric characteristics of the local geometric structure region; wherein, the geometric attributes include: position, distance, angle, and curvature; The generation of the second geometric relation includes: A local geometric structure region is input into the base model corresponding to the previous training stage, so that the base model calculates the geometric attributes between each point in the local geometric structure region and its adjacent points; based on the geometric attributes corresponding to each point, a second geometric relationship is generated to describe the geometric characteristics of the local geometric structure region; wherein, the geometric attributes include: position, distance, angle and curvature.

4. The semantic segmentation method based on point cloud data as described in claim 3, characterized in that, The step of constructing a corresponding first loss function based on each first geometric relationship and the corresponding second geometric relationship includes: The first or second geometric relation can be calculated using the following formula: Among them, R a This represents the a-th first geometric relation or second geometric relation. Represents the a-th first geometric relation. Represents the a-th second geometric relation; This represents the coordinates of a target point within a local geometric region. This represents the coordinates of points adjacent to the target point within the local geometric region. This represents the geometric properties of a target point within a local geometric region. This represents the geometric properties of points adjacent to the target point within a local geometric region. Indicates a cascade operation; The first loss function is calculated using the following formula: in, Let represent the first loss function, and 'a' represent the number of first or second geometric relations.

5. The semantic segmentation method based on point cloud data as described in claim 4, characterized in that, Before using the updated target model as the base model for the next model parameter update operation, the following steps are also included: Input an unlabeled dataset into the updated target model so that the updated target model outputs the corresponding pseudo-label category. The extracted sample 3D point cloud data and the corresponding sample category results are input into the updated target model so that the updated target model outputs the corresponding first predicted label category. The extracted sample 3D point cloud data and the corresponding sample category results are input into the base model corresponding to the previous training stage, so that the base model corresponding to the previous training stage outputs the corresponding second predicted label category. The pseudo-label category and the first predicted label category are integrated to generate a hybrid label category; The mixed label category is compared with the second predicted label category, and the target model, which has already been updated once, is updated a second time based on the comparison results.

6. The semantic segmentation method based on point cloud data as described in claim 5, characterized in that, The step of integrating the pseudo-label category and the first predicted label category to generate a hybrid label category includes: Generate mixed label categories according to the following formula: in, Indicates mixed label categories, Indicates the first predicted label category. This represents the predicted probability of the i-th point corresponding to the pseudo-label. γ represents the probability of the closest point t adjacent to the i-th point, and γ represents the preset probability threshold. c represents the data of each point in the sample 3D point cloud data. bg This represents the data points in an unlabeled dataset. The Bernoulli distribution represents the weights of adjacent points. This indicates the degree of uncertainty in the predicted probability of each point.

7. The semantic segmentation method based on point cloud data as described in claim 6, characterized in that, The network structure of the target model includes: a residual branch network and a main branch network; The second update of the target model, which has already been updated once, based on the comparison results includes: Based on the comparison results, the network parameters of the residual branch network in the target model that has been updated once are adjusted; The adjusted residual branch network is integrated into the main branch network so that the target model, which has already been updated once, can be updated a second time.

8. The semantic segmentation method based on point cloud data as described in claim 7, characterized in that, After updating the target model a second time based on the comparison results, and before using the updated target model as the base model for the next model parameter update operation, the process also includes: Based on several sets of 3D point cloud data belonging to the same sample category from the 3D point cloud data of each identified sample, multiple sets of sample data of the same category are obtained. A set of similar sample data is used as a target category sample data, and the target category sample data is input into the base model corresponding to the previous training stage, so that the base model corresponding to the previous training stage outputs the first predicted category corresponding to the three-dimensional point cloud data of each target sample in the target category sample data; Feature vectors are extracted for each first predicted category, and the average vector value corresponding to each feature vector is calculated; wherein, the average vector value is used to represent the feature representation corresponding to the target category sample data; The three-dimensional point cloud data of a target sample from the target category sample data is input into the target model after a second update, so that the target model after the second update outputs the second predicted category corresponding to the three-dimensional point cloud data of the target sample. The feature representation corresponding to the second predicted category is compared with the average vector value to generate a similarity difference; Based on the similarity difference, the target model, which has already undergone two updates, is updated a third time.

9. A semantic segmentation device based on point cloud data, characterized in that, include: Point cloud data acquisition module and category result recognition module; The point cloud data acquisition module is used to acquire point cloud data to be identified; The category result recognition module is used to input the point cloud data to be recognized into a preset point cloud classification model, so that the point cloud classification model can segment the point cloud data to be recognized and output the target category result corresponding to each segmented region. The training of the point cloud classification model includes: Acquire several sets of sample 3D point cloud data and the corresponding sample category results; A sample of 3D point cloud data and the corresponding sample category results are used as training samples to train an initial model and generate a basic model. Extract the 3D point cloud data of each sample and the corresponding sample category results one by one, and repeat the following model parameter update operation until the target model converges to generate the final point cloud classification model: The extracted sample 3D point cloud data is used as input, and the corresponding sample category results are used as output to train the base model corresponding to the previous training stage to generate the target model. Several local geometric structure regions are constructed in the currently extracted sample 3D point cloud data; Obtain the first geometric relationship predicted by the target model for each of the local geometric structure regions, and at the same time obtain the second geometric relationship predicted by the base model corresponding to the previous training stage for each of the local geometric structure regions. A first loss function is constructed based on each first geometric relationship and the corresponding second geometric relationship, and the network parameters of the target model are updated once based on the loss function. If the target model is determined to have not converged, the updated target model will be used as the base model for the next model parameter update operation.

10. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a semantic segmentation method based on point cloud data as described in any one of claims 1 to 8.

Images