Point cloud segmentation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a point cloud segmentation method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a classification result of at least one second point in a target point cloud according to the position of at least one first point in the target point cloud and a truth value type marked for the first point; generating coding features used for representing the truth value type and the classification result of each second point; inputting the coding feature and the point cloud data of the target point cloud into a point cloud semantic segmentation model to obtain a point cloud segmentation result; obtaining a prediction type of each first point in the point cloud segmentation result; and adjusting the coding features in the direction of reducing the difference between the prediction type and the true value type of each first point, taking the adjusted coding features as new coding features, and returning to execute the step of obtaining the point cloud segmentation result output by the point cloud semantic segmentation model until a first preset condition is met. And taking the latest point cloud segmentation result as a point cloud segmentation result of the target point cloud. The accuracy of point cloud segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of 3D computer vision technology, and in particular to a point cloud segmentation method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, point cloud segmentation is typically achieved through projection interaction or direct interaction in 3D space. Projection interaction involves projecting 3D point cloud data onto different 2D views. Based on user annotations of point types in each 2D view, the type of each point in that view is obtained as the segmentation result. The segmentation results from each 2D view are then transferred to the 3D point cloud data, relying on the consistency of the mapping between 2D and 3D space, to obtain the final point cloud segmentation result. However, in this method, the segmentation result for the same point may differ across 2D views, making it difficult to determine the accuracy of the segmentation results for the same point in different 2D views. This results in low accuracy of the point cloud segmentation results obtained from the segmentation results of each 2D view. Direct interaction in 3D space also struggles to ensure the accuracy of the point cloud segmentation results. Therefore, improving the accuracy of point cloud segmentation has become a pressing issue. Summary of the Invention

[0003] The purpose of this application is to provide a point cloud segmentation method, apparatus, electronic device, and storage medium to improve the accuracy of point cloud segmentation. The specific technical solution is as follows:

[0004] This application provides a point cloud segmentation method, the method comprising:

[0005] Based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point, the classification result of at least one second point in the target point cloud is determined, wherein the first point is a labeled point and the second point is an unlabeled point, and the classification result is used to represent the type of the point and the confidence level of the type;

[0006] Generate features to represent the truth value type and the classification results of each of the second points, as encoded features;

[0007] The encoded features and the point cloud data of the target point cloud are input into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model.

[0008] Obtain the type of each first point in the point cloud segmentation result, and use it as the prediction type of each first point;

[0009] While keeping the parameters of the point cloud semantic segmentation model unchanged, the encoding features are adjusted in the direction of reducing the difference between the prediction type and the true value type of each first point, and the adjusted encoding features are used as new encoding features. The process of inputting the encoding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model is repeated until the first preset condition is met.

[0010] After the first preset condition is met, the latest point cloud segmentation result is used as the point cloud segmentation result of the target point cloud.

[0011] In one possible embodiment, determining the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point includes:

[0012] In the target point cloud, find points whose distance to each first point satisfies a preset proximity condition, and use them as the second points corresponding to each first point.

[0013] For each of the first points, the classification results of each of the second points corresponding to the first point are determined respectively, wherein the type represented by the classification result is the same as the truth value type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point.

[0014] In one possible embodiment, each of the first points is labeled sequentially;

[0015] The step of finding points in the target point cloud whose distances to each first point satisfy a preset proximity condition, and using these points as the second points corresponding to each first point, includes:

[0016] For each of the first points, a point in the target point cloud whose distance to the first point is less than the distance threshold corresponding to the first point is found, and this point is designated as the second point corresponding to the first point. The higher the labeling order of the first point, the larger the distance threshold, and the higher the confidence level represented by the classification result.

[0017] In one possible embodiment, the encoded features are an N×L matrix, where N is the total number of points in the target point cloud and L is the total number of types;

[0018] The value of the element in the i-th row and j-th column of the matrix is: the confidence level of the i-th point in the target point cloud for the j-th type.

[0019] In one possible embodiment, the point cloud semantic segmentation model is trained in the following manner:

[0020] Select at least one third point from the sample point cloud, wherein the sample point cloud is a point cloud with a predetermined truth value type for each point;

[0021] Based on the location and truth type of the third point, determine the classification result of at least one fourth point in the sample point cloud;

[0022] Generate features to represent the truth type of each of the third points and the classification result of each of the fourth points, as sample features;

[0023] The sample features and point cloud data of the sample point cloud are input into the model to be trained to obtain the candidate segmentation result output by the model to be trained.

[0024] In the candidate segmentation results, identify points whose type differs from the truth value type, and designate them as the fifth point;

[0025] Take the third point and the fifth point as the new third point, and return to the step of determining the classification result of at least one fourth point in the sample point cloud based on the position and truth type of the third point, until the second preset condition is met;

[0026] After the second preset condition is met, the model to be trained is trained with the latest candidate segmentation results to obtain a point cloud semantic segmentation model.

[0027] In one possible embodiment, determining points in the candidate segmentation results whose type differs from the truth type, as a fifth point, includes:

[0028] All points in the candidate segmentation results whose type differs from the truth value type are identified as candidate points;

[0029] Select a point from the candidate points that meets the preset selection criteria as the fifth point;

[0030] The preset selection conditions include any one or more of the first condition, the second condition, and the third condition;

[0031] The first condition is: located at the center of a dense region, where the density of the candidate points satisfies a preset density condition;

[0032] The second condition is: located within the neighborhood of the error boundary, wherein the error boundary is a boundary where the truth value types of the points on both sides are the same;

[0033] The third condition is: the truth value type is a preset type of interest.

[0034] This application embodiment also provides a point cloud segmentation device, the device comprising:

[0035] The classification result determination module is used to determine the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point, wherein the first point is a labeled point and the second point is an unlabeled point, and the classification result is used to represent the type of the point and the confidence level of the type;

[0036] The encoding feature generation module is used to generate features representing the truth value type and the classification results of each of the second points, as encoding features;

[0037] The segmentation result acquisition module is used to input the encoded features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model;

[0038] The prediction type acquisition module is used to acquire the type of each first point in the point cloud segmentation result, and use it as the prediction type of each first point.

[0039] The encoding feature adjustment module is used to adjust the encoding features in a direction that reduces the difference between the prediction type and the ground truth type of each first point while keeping the parameters of the point cloud semantic segmentation model unchanged, and use the adjusted encoding features as new encoding features. Then, it returns to the step of inputting the encoding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, until the first preset condition is reached.

[0040] The point cloud segmentation result module is used to take the latest point cloud segmentation result as the point cloud segmentation result of the target point cloud after the first preset condition is met.

[0041] In one possible embodiment, determining the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point includes:

[0042] In the target point cloud, find points whose distance to each first point satisfies a preset proximity condition, and use them as the second points corresponding to each first point.

[0043] For each of the first points, the classification results of each of the second points corresponding to the first point are determined respectively, wherein the type represented by the classification result is the same as the truth value type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point.

[0044] In one possible embodiment, each of the first points is labeled sequentially;

[0045] The step of finding points in the target point cloud whose distances to each first point satisfy a preset proximity condition, and using these points as the second points corresponding to each first point, includes:

[0046] For each of the first points, a point in the target point cloud whose distance to the first point is less than the distance threshold corresponding to the first point is found, and this point is designated as the second point corresponding to the first point. The higher the labeling order of the first point, the larger the distance threshold, and the higher the confidence level represented by the classification result.

[0047] In one possible embodiment, the encoded features are an N×L matrix, where N is the total number of points in the target point cloud and L is the total number of types;

[0048] The value of the element in the i-th row and j-th column of the matrix is: the confidence level of the i-th point in the target point cloud for the j-th type.

[0049] In one possible embodiment, the device further includes:

[0050] The model training module is used to train the point cloud semantic segmentation model in the following ways:

[0051] Select at least one third point from the sample point cloud, wherein the sample point cloud is a point cloud with a predetermined truth value type for each point;

[0052] Based on the location and truth type of the third point, determine the classification result of at least one fourth point in the sample point cloud;

[0053] Generate features to represent the truth type of each of the third points and the classification result of each of the fourth points, as sample features;

[0054] The sample features and point cloud data of the sample point cloud are input into the model to be trained to obtain the candidate segmentation result output by the model to be trained.

[0055] In the candidate segmentation results, identify points whose type differs from the truth value type, and designate them as the fifth point;

[0056] Take the third point and the fifth point as the new third point, and return to the step of determining the classification result of at least one fourth point in the sample point cloud based on the position and truth type of the third point, until the second preset condition is met;

[0057] After the second preset condition is met, the model to be trained is trained with the latest candidate segmentation results to obtain a point cloud semantic segmentation model.

[0058] In one possible embodiment, determining points in the candidate segmentation results whose type differs from the truth type, as a fifth point, includes:

[0059] All points in the candidate segmentation results whose type differs from the truth value type are identified as candidate points;

[0060] Select a point from the candidate points that meets the preset selection criteria as the fifth point;

[0061] The preset selection conditions include any one or more of the first condition, the second condition, and the third condition;

[0062] The first condition is: located at the center of a dense region, where the density of the candidate points satisfies a preset density condition;

[0063] The second condition is: located within the neighborhood of the error boundary, wherein the error boundary is a boundary where the truth value types of the points on both sides are the same;

[0064] The third condition is: the truth value type is a preset type of interest.

[0065] This application also provides an electronic device, including:

[0066] Memory, used to store computer programs;

[0067] The processor, when executing a program stored in memory, implements any of the point cloud segmentation methods described above.

[0068] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the point cloud segmentation methods described above.

[0069] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the point cloud segmentation methods described above.

[0070] Beneficial effects of the embodiments in this application:

[0071] The point cloud segmentation method, apparatus, electronic device, and storage medium provided in this application embodiment can be considered as follows: the point cloud semantic segmentation model is considered sufficiently accurate, and the ground truth type of the first point is considered an accurate result. The classification results of each second point are predicted based on the position and ground truth type of at least one first point. The prediction process carries a certain degree of inaccuracy, leading to inaccurate classification results for each second point, and consequently, inaccurate generated encoded features. Furthermore, since the ground truth type of the first point is considered an accurate result, the smaller the difference between the predicted type and the ground truth type of the first point in the point cloud segmentation result, the more consistent the predicted type and the marked ground truth type are, and the more accurate the predicted type of the first point is. Conversely, the larger the difference between the predicted type and the ground truth type of the first point, the less consistent the predicted type and the marked ground truth type are, and the less accurate the predicted type is. As explained above, the point cloud semantic segmentation model is sufficiently accurate, but its input (i.e., the encoded features) is not accurate enough. Therefore, if the difference between the predicted type and the ground truth type of the first point in the point cloud segmentation result is large, indicating inaccurate segmentation, this inaccuracy can be attributed to the inaccuracy of the encoded features. Based on this, even when the difference between the predicted type and the ground truth type of the first point in the segmentation result is large, the parameters of the point cloud semantic segmentation model should remain unchanged, and the encoded features should be adjusted in a direction that reduces the difference between the predicted and ground truth types of each first point. This improves the accuracy of the encoded features, thereby improving the accuracy of the point cloud segmentation result. When the first preset condition is met, the difference between the predicted and ground truth types of each first point can be considered small, meaning the predicted and ground truth types of each first point are relatively consistent. This indicates that the accuracy of the predicted types of the first points in the point cloud segmentation result is high, and thus the accuracy of the point cloud segmentation result is high. Based on this, after the first preset condition is met, the latest point cloud segmentation result is used as the point cloud segmentation result of the target point cloud. In this way, the point cloud segmentation result with higher accuracy can be used as the point cloud segmentation result of the target point cloud, thereby improving the accuracy of point cloud segmentation.

[0072] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0074] Figure 1 A schematic flowchart of a point cloud segmentation method provided in an embodiment of this application;

[0075] Figure 2 A flowchart illustrating the determination of the classification result of the second point provided in this application embodiment;

[0076] Figure 3 Another flowchart illustrating the determination of the classification result of the second point provided in an embodiment of this application;

[0077] Figure 4 Another flowchart illustrating the determination of the classification result of the second point provided in this application embodiment;

[0078] Figure 5 A schematic diagram of a process for training a point cloud semantic segmentation model provided in an embodiment of this application;

[0079] Figure 6 Another flowchart illustrating the point cloud segmentation method provided in this application embodiment;

[0080] Figure 7 A schematic diagram of a point cloud segmentation device provided in an embodiment of this application;

[0081] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0083] In the fields of computer vision and artificial intelligence, semantic segmentation plays a crucial role. The goal of semantic segmentation is to assign a precise semantic category label to each pixel in an input image or video, such as labels for people, vehicles, trees, and sky. Semantic segmentation enables machines to understand the content of the scene they see, achieving pixel-level scene parsing compared to simple object detection or classification. Semantic segmentation technology has broad application prospects in many fields, including autonomous driving, robot navigation, medical image analysis, augmented reality, and video surveillance.

[0084] With the rapid development and widespread adoption of 3D sensing technologies such as LiDAR, depth cameras, and structured light scanners, the acquisition of 3D spatial data, including 3D point clouds, mesh models, and voxel data, has become increasingly convenient and abundant. 3D sensing technology has driven the rise and development of 3D semantic segmentation (3D Semantic Segmentation). 3D semantic segmentation aims to directly understand the structure of 3D space, assigning semantic category labels to each point, facet, or voxel in the 3D data. Compared to 2D semantic segmentation, 3D semantic segmentation can more comprehensively describe the geometric structure and spatial relationships of the real world, providing a foundation for building high-precision 3D scene understanding models.

[0085] Existing 3D semantic segmentation techniques, such as PointNet++ based on point clouds, CNN-based 3D convolutional neural networks, and graph-based graph convolutional networks, typically rely on supervised learning training of network models using large-scale, high-quality 3D labeled datasets. While these methods have achieved significant progress on closed test sets, they still face the following challenges when dealing with complex and dynamic real-world scenes:

[0086] (1) High annotation cost. Compared with the annotation of two-dimensional images, the annotation of three-dimensional spatial data, especially the annotation of point clouds or dense grids, is more complex, time-consuming and costly.

[0087] (2) Limited generalization ability. The dataset required to train the network model is difficult to cover all possible scenarios, such as novel objects, complex occlusion, extreme lighting or weather conditions, which leads to a decrease in the segmentation accuracy of the model in unknown or complex scenarios, resulting in missegmentation or missed segmentation.

[0088] (3) Bottleneck of automation. Fully automated segmentation algorithms often fail to achieve satisfactory accuracy when faced with situations such as blurred boundaries, objects sticking together, small targets or background interference, and lack the necessary flexibility and controllability.

[0089] Interactive image segmentation, a technique within 2D semantic segmentation, effectively overcomes the limitations of conventional segmentation algorithms by introducing user-guided information, significantly improving segmentation accuracy for complex scenes. Representative methods of interactive image segmentation include: bounding box initialization (GrabCut), user-annotated foreground / background seed points (interactive graph cut), and recent deep learning-based Click / Stroke guided models such as RITM and f-BRS. Users only need to provide a small amount of guidance information by clicking on mis-segmented regions or outlining contours, and the network model can dynamically correct the segmentation results based on this guidance. The main application of interactive image segmentation is to expand datasets. Satisfactory segmentation results can be obtained through simple annotation, thereby reducing the cost of segmentation annotation and improving the performance of the segmentation network. Interactive image segmentation has been widely used in fields such as medical image annotation, advertising design, and film post-production.

[0090] It is evident that interactive image segmentation technology has low annotation costs, strong model generalization ability, and the ability to automatically correct segmentation results. However, as mentioned earlier, 3D semantic segmentation technology suffers from high annotation costs, limited generalization ability, and bottlenecks in automated processing. Therefore, this paper applies the aforementioned interactive image segmentation technology to the field of 3D semantic segmentation technology to achieve 3D semantic segmentation through interactive 3D segmentation, thereby solving the problems existing in 3D semantic segmentation technology.

[0091] 3D interactive segmentation technology allows users to directly provide spatial guidance information to point clouds, meshes, or voxel data through point selection, bounding box selection, and painting, thus solving the problems existing in 3D semantic segmentation technology. Currently, 3D interactive segmentation technology can be divided into two categories: projection interactive and direct 3D spatial interactive.

[0092] The projection interaction method requires projecting 3D point cloud data onto different 2D views. Based on the user's annotations of the types of some points in each 2D view, the type of each point in each 2D view is obtained as the segmentation result. The segmentation results of each 2D view are then transferred to the 3D point cloud data based on the consistency of the mapping between 2D and 3D space, resulting in the point cloud segmentation result. However, in this method, the segmentation result corresponding to the same point in different 2D views may differ, making it difficult to determine the accuracy of the segmentation result corresponding to the same point in different 2D views. This leads to low accuracy of the point cloud segmentation result obtained based on the segmentation results of each 2D view.

[0093] In direct 3D interaction methods, the InterObject3D approach can only segment a single object, or a class of targets, and does not support simultaneous segmentation of multiple target classes in a scene. The attention-guided interactive multi-object 3D segmentation method (Aglie3D) supports multi-target segmentation, but its encoding of user-annotated guidance information is relatively simple, resulting in poor segmentation performance and low accuracy of point cloud segmentation results. Therefore, improving the accuracy of point cloud segmentation has become a pressing issue.

[0094] To improve the accuracy of point cloud segmentation, this application provides a point cloud segmentation method, see [link to relevant documentation]. Figure 1 The methods include:

[0095] S101, Based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point, determine the classification result of at least one second point in the target point cloud.

[0096] The first point is a labeled point, and the second point is an unlabeled point. The classification result is used to represent the type of the point and the confidence level of the type.

[0097] S102, generate features to represent the truth type and the classification results of each second point, as encoded features.

[0098] S103, input the encoded features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model.

[0099] S104, obtain the type of each first point in the point cloud segmentation result, and use it as the prediction type of each first point.

[0100] S105, while keeping the parameters of the point cloud semantic segmentation model unchanged, adjust the encoding features in the direction of reducing the difference between the predicted type and the true value type of each first point, and use the adjusted encoding features as the new encoding features. Then, return to execute the step of inputting the encoding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, until the first preset condition is reached.

[0101] S106, after the first preset condition is met, the latest point cloud segmentation result is used as the point cloud segmentation result of the target point cloud.

[0102] Applying the embodiments of this application, the point cloud semantic segmentation model can be considered sufficiently accurate, and the ground truth type of the first point can be considered an accurate result. The classification results of each second point are predicted based on the position and ground truth type of at least one first point. The prediction process carries a certain degree of inaccuracy, leading to inaccurate classification results for each second point, and consequently, inaccurate generated encoded features. Furthermore, since the ground truth type of the first point can be considered an accurate result, the smaller the difference between the predicted type and the ground truth type of the first point in the point cloud segmentation result, the more consistent the predicted type and the marked ground truth type are, and the more accurate the predicted type of the first point is. Conversely, the larger the difference between the predicted type and the ground truth type of the first point in the point cloud segmentation result, the less consistent the predicted type and the marked ground truth type are, and the less accurate the predicted type is. As explained above, the point cloud semantic segmentation model is sufficiently accurate, but its input (i.e., the encoded features) is not accurate enough. Therefore, if the difference between the predicted type and the ground truth type of the first point in the point cloud segmentation result is large, indicating inaccurate segmentation, this inaccuracy can be attributed to the inaccuracy of the encoded features. Based on this, even when the difference between the predicted type and the ground truth type of the first point in the segmentation result is large, the parameters of the point cloud semantic segmentation model should remain unchanged, and the encoded features should be adjusted in a direction that reduces the difference between the predicted and ground truth types of each first point. This improves the accuracy of the encoded features, thereby improving the accuracy of the point cloud segmentation result. When the first preset condition is met, the difference between the predicted and ground truth types of each first point can be considered small, meaning the predicted and ground truth types of each first point are relatively consistent. This indicates that the accuracy of the predicted types of the first points in the point cloud segmentation result is high, and thus the accuracy of the point cloud segmentation result is high. Based on this, after the first preset condition is met, the latest point cloud segmentation result is used as the point cloud segmentation result of the target point cloud. In this way, the point cloud segmentation result with higher accuracy can be used as the point cloud segmentation result of the target point cloud, thereby improving the accuracy of point cloud segmentation.

[0103] The following will provide exemplary descriptions of S101-S106 respectively:

[0104] In S101, users can select at least one first point in the target point cloud by clicking, selecting with a box, or drawing, and label each first point with a truth value type. The type is the aforementioned semantic category label; for example, the type can be person, vehicle, tree, sky, etc.

[0105] Based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point, determine the classification result of at least one second point in the target point cloud using any method; wherein, if the classification result of at least one second point in the target point cloud is determined by a point cloud segmentation model, then the point cloud segmentation model should be any point cloud segmentation model other than the point cloud semantic segmentation model in S103. Specifically, the classification result of the second point is determined by the following method, such as... Figure 2 As shown, it includes:

[0106] S201, find points in the target point cloud whose distance to each first point meets the preset proximity condition, and use them as the second points corresponding to each first point.

[0107] The preset proximity condition can be: the distance to a first point is less than a preset distance threshold. The preset distance threshold can be arbitrarily set according to user needs. In this embodiment, for each first point, points in the target point cloud whose distance to that first point is less than the preset distance threshold are designated as the second points corresponding to that first point. Specifically, all points in the target point cloud whose distance to that first point is less than the preset distance threshold can be designated as the second points corresponding to that first point; alternatively, all points in the target point cloud whose distance to that first point is less than the preset distance threshold can be searched, and a subset of these points can be selected arbitrarily as the second points corresponding to that first point. The same preset distance threshold can be set for different first points, or different preset distance thresholds can be set for different first points.

[0108] The preset proximity condition can also be: when sorting all points in the target point cloud (excluding the first point) from the first point in descending order of their distance to the first point, the points that are within the first preset number of positions are considered as the second points corresponding to that first point. The preset number can be set arbitrarily according to user needs. In this embodiment, for each first point, the points within the first preset number of positions when sorting all points in the target point cloud (excluding the first point) from the first point in descending order of their distance to that first point are considered as the second points corresponding to that first point. The same preset number can be set for different first points, or different preset numbers can be set for different first points.

[0109] S202, for each first point, determine the classification result of each corresponding second point.

[0110] The classification result represents the same type as the true value type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point.

[0111] The negative correlation between the confidence level indicated by the classification result and the distance between the second and first points means that, all other factors remaining constant, the confidence level indicated by the classification result decreases as the distance between the second and first points increases, and increases as the distance between the second and first points decreases. In this application, "decreasing" can refer to monotonically decreasing or non-monotonic increasing; "increasing" can refer to monotonically increasing or non-monotonic increasing, and this application makes no limitation in this regard.

[0112] It is understandable that two points that are closer together are more likely to be points that make up the same object, and therefore, two points that are closer together are more likely to have the same type. Furthermore, since the second point corresponding to the first point is a point whose distance from the first point satisfies the preset proximity condition, it can be assumed that the second point corresponding to the first point is closer to the first point, and thus its type is the same as the truth value type of the first point. Therefore, the type represented by the classification result is the same as the truth value type of the first point.

[0113] Furthermore, for a given second point corresponding to a given first point, the closer the second point is to the first point, the greater the probability that the type of the second point is the same as the true type of the first point. In other words, the closer the second point is to the first point, the higher the confidence that the type of the second point is true. Conversely, the farther the second point is from the first point, the less likely the type of the second point is to be the same as the true type of the first point. Therefore, the greater the distance between the second point and the first point, the lower the confidence that the type of the second point is true. Thus, the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point.

[0114] By using this embodiment, points that satisfy a preset proximity condition with respect to each first point can be found in the target point cloud. These points are then identified as the second points corresponding to each first point. The classification results of each second point are determined by ensuring that the type represented by the classification result is the same as the ground truth type of the first point and that the confidence level represented by the classification result is negatively correlated with the distance between the second and first points. This improves the accuracy of the classification results of the determined second points, thereby improving the accuracy of the generated encoded features and ultimately improving the accuracy of point cloud segmentation.

[0115] In other possible embodiments, the classification result of at least one second point in the target point cloud can also be determined by any method other than the embodiments described above, based on the position of at least one first point in the target point cloud and the truth value type labeled for the first point.

[0116] Please watch it again. Figure 1In S102, the encoded features can be in any form, such as an array or a matrix. In one possible embodiment, the encoded features are an N×L matrix, where N is the total number of points in the target point cloud and L is the total number of types; the value of the element in the i-th row and j-th column of the matrix is ​​the confidence score of the i-th point in the target point cloud for the j-th type.

[0117] For example, assuming the target point cloud includes 10 points, and the point types could be tree, person, or vehicle, the encoded feature is a 10×3 matrix. Furthermore, assuming the first type is tree, the second type is person, and the third type is vehicle, the value of the element in the first row and first column of the encoded feature represents the confidence that the first point in the target point cloud is a tree; the value of the element in the fourth row and second column represents the confidence that the fourth point in the target point cloud is a person; and the value of the element in the tenth row and third column represents the confidence that the tenth point in the target point cloud is a vehicle.

[0118] In S103-S105, the point cloud data of the target point cloud is generally represented by an N×4 matrix, where N is the total number of points in the target point cloud. For example, the point cloud data of the target point cloud can be represented as [xi, yi, zi, fi], where (xi, yi, zi) represents the coordinates of the i-th point in the target point cloud in the three-dimensional coordinate system, and fi represents the basic features of the i-th point in the target point cloud, such as reflectivity and identification (id).

[0119] The encoded features and the target point cloud data can be sequentially input into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model. Alternatively, the encoded features and the target point cloud data can be concatenated and then input into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model. In this embodiment, if the encoded features are the aforementioned N×L matrix and the target point cloud data are the aforementioned N×4 matrix, then the encoded features and the target point cloud data are concatenated to obtain an N×(L+4) matrix, and this N×(L+4) matrix is ​​then input into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model.

[0120] The first preset condition can refer to the number of times the encoded features are adjusted being greater than a preset threshold, or it can refer to the difference between the predicted type and the true type of each first point being less than a preset difference threshold, etc. The preset threshold for the number of adjustments and the preset difference threshold can be set arbitrarily based on past experience or user needs.

[0121] Since the ground truth type of the first point is obtained from the annotation, it can be considered an accurate result. Therefore, if the difference between the predicted type and the ground truth type of the first point is large, it indicates that the accuracy of the predicted type of the first point is low, which also means that the accuracy of the point cloud segmentation result output by the point cloud semantic segmentation model is low. Conversely, if the difference between the predicted type and the ground truth type of the first point is small, it indicates that the accuracy of the predicted type of the first point is high, which also means that the accuracy of the point cloud segmentation result output by the point cloud semantic segmentation model is high.

[0122] Furthermore, it can be assumed that the point cloud semantic segmentation model is sufficiently accurate, and the ground truth type of the first point is an accurate result. The classification results of each second point are predicted based on the position and ground truth type of at least one first point. The prediction process inherently carries some inaccuracy, leading to inaccurate classification results for each second point, and consequently, inaccurate generated encoded features. Therefore, if the difference between the predicted type and the ground truth type of the first point in the point cloud segmentation result is significant, indicating inaccuracy, it can be concluded that the inaccuracy is caused by inaccurate encoded features. Based on this, even when the difference between the predicted type and the ground truth type of the first point in the point cloud segmentation result is significant, it is necessary to keep the parameters of the point cloud semantic segmentation model unchanged and adjust the encoded features in a direction that reduces the difference between the predicted type and the ground truth type of each first point. This improves the accuracy of the encoded features and thus enhances the accuracy of the point cloud segmentation result. Then, the encoding features are adjusted and used as the new encoding features. The process returns to step S103. Through this method, the accuracy of the encoding features is continuously improved, the difference between the predicted type and the ground truth type of the first point is reduced, and the accuracy of the predicted type of the first point is improved. This, in turn, improves the accuracy of the point cloud segmentation result output by the point cloud semantic segmentation model. Specifically, while keeping the parameters of the point cloud semantic segmentation model unchanged, the loss is calculated based on the difference between the predicted type and the ground truth type of each first point using the following formula:

[0123]

[0124] Where Loss is the loss; n is the number of first points; F c,i Let L be the prediction type for the i-th first point; c,i Let i be the truth value type of the i-th first point.

[0125] Based on the loss, the encoding features are adjusted in the direction of reducing the loss, and the adjusted encoding features are used as the new encoding features. The process then returns to the step of inputting the encoding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, until the first preset condition is met.

[0126] In S106, as explained above, if the difference between the predicted type and the ground truth type of the first point is small, it indicates that the accuracy of the predicted type of the first point is high, which also means that the accuracy of the point cloud segmentation result output by the point cloud semantic segmentation model is high. Furthermore, when the first preset condition is met, it can be considered that the difference between the predicted type and the ground truth type of each first point has been reduced sufficiently. Therefore, when the first preset condition is met, it can be considered that the accuracy of the predicted type of the first point is high enough, which also means that the accuracy of the latest point cloud segmentation result output by the point cloud semantic segmentation model is high enough. By using the latest point cloud segmentation result as the point cloud segmentation result of the target point cloud, the accuracy of the point cloud segmentation result of the target point cloud is ensured to be high enough, thereby improving the accuracy of point cloud segmentation.

[0127] S101-S106 have been described by example above. See the aforementioned section on... Figure 1 The implementation of the point cloud segmentation method provided in this application relies on the accuracy of the encoded features, which in turn depends on the accuracy of the classification result of the second point. Therefore, in order to improve the accuracy of the classification result of the determined second point, thereby improving the accuracy of point cloud segmentation, in one possible embodiment, each first point is labeled sequentially, and the classification result of the second point is determined in the following manner: Figure 3 As shown, it includes:

[0128] S2011, For each first point, find the point in the target point cloud whose distance to the first point is less than the distance threshold corresponding to the first point, and use it as the second point corresponding to the first point.

[0129] Among them, the earlier the first point is labeled, the greater the distance threshold.

[0130] S2021, For each first point, determine the classification result of each corresponding second point.

[0131] The classification result represents the same type as the true value type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point. The earlier the label of the first point appears, the higher the confidence level represented by the classification result.

[0132] In S2011-S2021, the user selects only one first point from the target point cloud for ground truth labeling each time. It's understandable that the target point cloud may contain large areas of points of the same type, or small areas. When labeling ground truth, the user generally prioritizes points from the larger areas as the first point. Since the second point is determined by finding points in the target point cloud whose distance to the first point is less than the corresponding distance threshold, using points from the larger area as the first point ensures that the determined second point is likely to be within the same area as the first point. In other words, the more likely the determined second point is to share the same type as the first point, the higher the confidence level of the determined second point's type. Therefore, by setting a higher confidence level for the first point, the confidence level of the determined second point's type can be increased.

[0133] Conversely, when a point within a small region is used as the first point, the determined second point is highly likely to be located in a different region than the first point. This means the type of the determined second point is more likely to be different from that of the first point, leading to a lower confidence level for the determined second point's type. Therefore, the confidence level of the determined second point's type can be lowered by setting a lower confidence level for the first point. In essence, the earlier the first point is labeled, the higher the confidence level of its truth type, and correspondingly, the higher the confidence level of the second point's type should be. Conversely, the later the first point is labeled, the lower the confidence level of its truth type, and correspondingly, the lower the confidence level of the second point's type should be.

[0134] When a point in a large area is taken as the first point, even if a point in the target point cloud is far away from the first point, the point and the first point are likely to be in the same large area. That is, the point and the first point are more likely to be of the same type, which makes the classification result of the point determined based on the first point more accurate. Therefore, when the first point is labeled earlier, the distance threshold corresponding to the first point needs to be set to a larger value.

[0135] Conversely, when a point in the aforementioned small region is taken as the first point, if the distance between a point in the target point cloud and the first point is slightly far, the point and the first point are likely to be in different regions. That is, the type of the point and the first point are more likely to be different, which makes the classification result of the point determined based on the first point less accurate. Therefore, when the labeling order of the first point is relatively low, the distance threshold corresponding to the first point needs to be set to a smaller value.

[0136] Specifically, such as Figure 4As shown, the distance threshold and confidence level of the truth value type of each first point can be determined as follows: the user pre-sets the maximum confidence level to 1 and the maximum distance threshold to q. When the user labels the truth value type of the first first point for the first time, the confidence level of the truth value type of the first first point is 1, and the corresponding distance threshold is q. For example, when the user clicks the first first point for the first time, the truth value type of the first first point is labeled as 0 (i.e., ...). Figure 4 In the category = 0), the confidence of the truth type of the first point is 1 (i.e., Figure 4 The weight in the value is 1.0), and the corresponding distance threshold is 1m (i.e., Figure 4 (with a radius of 1.0m).

[0137] When a user annotates the truth value type of the m-th first point for the m-th time, the confidence level of the truth value type of the m-th first point is 1-p×(m-1), and the corresponding distance threshold is (1-p×(m-1))×q. Here, p is a decreasing coefficient. For the m-th first point, the confidence level of the types of each second point corresponding to this first point ranges from (0, 1-p×(m-1)), and the confidence level of the second point's type is negatively correlated with the distance between the second point and the first point. For example, when the user clicks the 10th first point for the 10th time, the truth value type of the 10th first point is annotated as 1 (i.e., ...). Figure 4 In the category = 1), the confidence level of the truth type of the 10th point is 0.5 (i.e., Figure 4 The weight in the value is 0.5), and the corresponding distance threshold is 0.5m (i.e., Figure 4 (The radius is 0.5m). Here, category=0 and category=1 are a way to distinguish different truth value types. Category=0 and category=1 represent different categories. For example, category=0 is used to represent people, and category=1 is used to represent vehicles, etc.

[0138] In this embodiment, when determining the classification result of the second point, the influence of the size of the region composed of points of the same type in the target point cloud on the user annotation order and the confidence level of the ground truth type of the first point is fully considered. The distance threshold corresponding to the first point with a higher annotation order is set to be larger, and the confidence level represented by the classification result is negatively correlated with the distance between the second and first points. Furthermore, the higher the annotation order of the first point, the higher the confidence level represented by the classification result. Through this method, the accuracy of the determined classification result of the second point is improved, thereby improving the accuracy of point cloud segmentation.

[0139] In other possible embodiments, the user may not prioritize selecting a point from the aforementioned large area as the first point. That is, it's possible that the confidence level of the truth type of the first point, which is labeled earlier in the order, is lower, while the confidence level of the truth type of the first point, which is labeled later in the order, is higher. In this embodiment, the user can configure a confidence level for the truth type of the first point when labeling it. The larger the area where the first point is located, the higher the accuracy of the user's labeling of the truth type, and the higher the confidence level of the truth type of the first point. Conversely, the smaller the area where the first point is located, the lower the accuracy of the user's labeling of the truth type, and the lower the confidence level of the truth type of the first point.

[0140] In this embodiment, the classification result of the second point is determined as follows: For each first point, points in the target point cloud whose distance to the first point is less than the distance threshold corresponding to the first point are identified as the second points corresponding to the first point. The higher the confidence level of the ground truth type of the first point, the larger the distance threshold. For each first point, the classification result of each second point corresponding to the first point is determined. The type represented by the classification result is the same as the ground truth type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point; the higher the confidence level of the ground truth type of the first point, the higher the confidence level represented by the classification result.

[0141] The method for determining the classification result of the second point has been illustrated above. See the aforementioned section on... Figure 1 The implementation of the point cloud segmentation method provided in this application also relies on a point cloud semantic segmentation model. Therefore, the method for training the point cloud semantic segmentation model will be described exemplarily below. See [link to example]. Figure 5 The point cloud semantic segmentation model is trained in the following ways:

[0142] S501, Select at least one third point from the sample point cloud.

[0143] Among them, the sample point cloud is a point cloud with a predetermined ground truth type.

[0144] S502, based on the position and truth type of the third point, determine the classification result of at least one fourth point in the sample point cloud.

[0145] The classification result for determining at least one fourth point in S501-S502 is the same as the method for determining at least one second point described above. Therefore, the method for determining the classification result for at least one fourth point can be found in the aforementioned... Figure 2 as well as Figure 3 The relevant descriptions of the embodiments will not be repeated here.

[0146] S503, generate features to represent the truth type of each third point and the classification result of each fourth point, as sample features.

[0147] The sample features are represented in the same way as the aforementioned encoded features. For example, if the aforementioned encoded features are an N×L matrix, where N is the total number of points in the target point cloud and L is the total number of types; and the value of the element in the i-th row and j-th column of the matrix is ​​the confidence score of the i-th point in the target point cloud for the j-th type, then the sample features should be an A×L matrix, where A is the total number of points in the sample point cloud and L is the total number of types; and the value of the element in the i-th row and j-th column of the matrix is ​​the confidence score of the i-th point in the sample point cloud for the j-th type.

[0148] S504: Input the sample features and point cloud data of the sample point cloud into the model to be trained to obtain the candidate segmentation results output by the model to be trained.

[0149] Similar to the representation of target point cloud data, sample point cloud data is generally represented by an A×4 matrix, where A is the total number of points in the sample point cloud. For example, sample point cloud data can be represented as [xi, yi, zi, fi], where (xi, yi, zi) represents the coordinates of the i-th point in the sample point cloud in a three-dimensional coordinate system, and fi represents the reflectivity, line bundle ID, and other basic characteristics of the i-th point in the sample point cloud.

[0150] The method of inputting the sample features and point cloud data of the sample point cloud into the model to be trained is the same as the method of inputting the encoded features and point cloud data of the target point cloud into the point cloud semantic segmentation model in S103 above. For example, if S103 specifically involves inputting the encoded features and point cloud data of the target point cloud sequentially into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, then S504 specifically involves inputting the sample features and point cloud data of the sample point cloud sequentially into the model to be trained to obtain the candidate segmentation result output by the model to be trained.

[0151] If S103 specifically involves concatenating the encoded features and the point cloud data of the target point cloud and inputting them into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, then S504 specifically involves concatenating the sample features and the point cloud data of the sample point cloud and inputting them into the model to be trained to obtain the candidate segmentation result output by the model to be trained. In this embodiment, if the sample features are an A×L matrix and the point cloud data of the sample point cloud are an A×4 matrix, then after concatenating the sample features and the point cloud data of the sample point cloud to obtain an A×(L+4) matrix, this A×(L+4) matrix is ​​input into the model to be trained to obtain the candidate segmentation result output by the model to be trained.

[0152] S505, in the candidate segmentation results, identify points whose type is different from the true value type, and designate them as the fifth point.

[0153] In one possible embodiment, all points in the candidate segmentation results whose types are different from the truth type can be identified as candidate points, and one or more points can be arbitrarily selected from the candidate points as the fifth point.

[0154] In another possible embodiment, the fifth point can be selected by: identifying all points in the candidate segmentation results whose type differs from the truth type as candidate points; and selecting points from the candidate points that satisfy preset selection conditions as the fifth point. The preset selection conditions include any one or more of the first condition, the second condition, and the third condition.

[0155] The first condition is: located at the center of a dense region, where the density of candidate points meets the preset density condition. The preset density condition can refer to the number of candidate points in the region exceeding a preset quantity threshold; or it can refer to the ratio of the number of candidate points to the area of ​​the region exceeding a preset proportion threshold. Both the preset quantity threshold and the preset proportion threshold can be arbitrarily set based on past experience or user needs.

[0156] The second condition is: the point must be located within the neighborhood of the error boundary, and the error boundary must be a boundary where the ground value types of the points on both sides are the same. The boundary between points of different types in the candidate segmentation result is denoted as the error boundary, and the boundary where the ground value types of the points on both sides are the same is considered the error boundary. In other words, the points on both sides of the error boundary must have different types in the candidate segmentation result, but the ground value types must be the same.

[0157] The third condition is: the truth value type is a preset interest type. The preset interest type can be set arbitrarily according to different scenarios. For example, in a certain scenario, if the user is more concerned about the accuracy of people in the point cloud segmentation results, the preset interest type can be set to people, and correspondingly, candidate points with the truth value type of people can be used as the fifth condition.

[0158] In embodiments where the preset selection criteria include multiple conditions, the fifth point can be selected as follows: Priorities are assigned to each condition, and the point that satisfies the highest priority condition from the candidate points is selected as the fifth point. For example, assuming the preset selection criteria include a first condition and a second condition, and the first condition has a higher priority than the second condition, then the point that satisfies the first condition from the candidate points is selected as the fifth point. If no point that satisfies the first condition is selected from the candidate points, then the point that satisfies the second condition from the candidate points is selected as the fifth point.

[0159] In this embodiment, the fifth point can also be selected in the following way: For each candidate point, calculate the score of the candidate point based on the degree of matching between the candidate point and each condition. The higher the degree of matching between the candidate point and each condition, the higher the score of the candidate point. Sort the candidate points in descending order of their scores, and select the top b candidate points in the sort as the fifth point. b can be set arbitrarily according to the user's needs. For example, b can be set to 1, 2, 3, etc.

[0160] S506, take the third point and the fifth point as the new third point, return to execute the step of determining the classification result of at least one fourth point in the sample point cloud based on the position and truth type of the third point, until the second preset condition is met.

[0161] The second preset condition can refer to the number of times S502 is executed being greater than a second preset number threshold, or it can refer to the difference between the type of each third point in the candidate segmentation result and the truth value type of each third point being less than a second preset difference threshold, etc. The second preset number threshold and the second preset difference threshold can be arbitrarily set based on past experience or user needs.

[0162] S507. After the second preset condition is met, the model to be trained is trained with the latest candidate segmentation results to obtain the point cloud semantic segmentation model.

[0163] Specifically, after the second preset condition is met, the model parameters of the model to be trained are adjusted based on the difference between the latest candidate segmentation result and the ground truth types of all points in the sample point cloud. The model to be trained with the adjusted model parameters is then used as the new model to be trained, and the process returns to execute S501. This process continues until the third preset condition is met, at which point the semantic segmentation model is obtained. The third preset condition can refer to the number of times the model parameters are adjusted exceeding a third preset threshold, or it can refer to the difference between the latest candidate segmentation result and the ground truth types of all points in the sample point cloud being less than a third preset difference threshold, etc. Both the third preset threshold for the number of adjustments and the third preset difference threshold can be arbitrarily set based on past experience or user needs.

[0164] To facilitate understanding of the process of training the point cloud semantic segmentation model described above, the following will provide an exemplary explanation of S501-S507 with specific examples.

[0165] The user pre-sets a maximum number of clicks c. max The execution entity used to train the point cloud semantic segmentation model is located in (0, c). max A value is randomly selected from the list as the number of clicks.

[0166] For point cloud data of size A×4, create a sample feature matrix F of size A×L.click Initially, the sample feature matrix F click If empty, combine the point cloud data of the A×4 sample point cloud and the A×L sample feature matrix F. click Concatenating these matrices yields an A×(4+L) matrix, which represents the initial input features F of the model to be trained. input , will F input The input is fed into the model to be trained, and the candidate segmentation result F output by the model is obtained. output F output It is an A×1 matrix.

[0167] In candidate segmentation result F output A point whose type differs from the ground truth type is identified and designated as the fifth point. Since there is no third point initially, this fifth point is used as the new third point. Based on the position and ground truth type of the third point, the classification result of at least one fourth point in the sample point cloud is determined. The ground truth type and confidence of the third point, and the classification result of the fourth point are then added to the aforementioned sample feature matrix F. click In the process, the sample feature matrix F at the time of the first click is obtained. click The new sample feature matrix F click By concatenating the point cloud data with the sample point cloud, an A×(4+L) matrix is ​​obtained, which is the input feature F of the model to be trained at the time of the first click. input , will F input The input is fed into the model to be trained, and the candidate segmentation result F output by the model to be trained at the time of the first click is obtained. output .

[0168] The candidate segmentation result F output by the model to be trained upon the first click. output The point whose type differs from the ground truth type is identified as the fifth point. This fifth point, along with the third point from the first click, is used as the new third point. Based on the position and ground truth type of the third point, at least one fourth point in the sample point cloud is classified. The ground truth type and confidence score of the third point, and the classification result of the fourth point, are then added to the aforementioned sample feature matrix F. click In the middle, the sample feature matrix F at the time of the second click is obtained. click The new sample feature matrix F click By concatenating the point cloud data with the sample point cloud, an A×(4+L) matrix is ​​obtained, which is the input feature F of the model to be trained at the time of the second click. input , will F input The input is fed into the model to be trained, and the candidate segmentation result F output by the model to be trained is obtained when the second click occurs. output This process continues until the candidate segmentation result F output by the model to be trained is obtained at the c-th click. output .

[0169] Then, the candidate segmentation result F output by the model to be trained at the c-th click is used. outpu The model parameters of the training model are adjusted based on the differences between the ground truth types of all points in the sample point cloud and the actual ground truth types. This completes the first round of adjusting the model parameters of the training model. Subsequently, the adjusted training model is used as the new training model, and the above process is repeated to complete more rounds of adjusting the model parameters of the training model until the third preset condition is met, resulting in the point cloud semantic segmentation model.

[0170] The flowchart of the point cloud segmentation method provided in this application embodiment can also be as follows: Figure 6 As shown. See also Figure 6 First, based on the information obtained after the user clicks on the target point cloud (i.e., the truth value type of the first point mentioned above), a click encoding feature (i.e., the aforementioned encoding feature) is generated using the user click encoding module. This process is equivalent to S101-S102. Then, the click encoding feature and the point cloud data of the target point cloud are input into the point cloud semantic segmentation model to obtain the semantic segmentation result output by the point cloud semantic segmentation model (i.e., the aforementioned point cloud segmentation result). This process is equivalent to S103. Next, the original point cloud data is updated based on the semantic segmentation result. The loss function value between the information obtained after the user clicks on the target point cloud and the semantic segmentation result is calculated. The click encoding feature is updated based on this loss function value, and the updated click encoding feature is used as the new click encoding feature. The process returns to the step of inputting the click encoding feature and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the semantic segmentation result output by the point cloud semantic segmentation model, until the first preset condition is met. This process is equivalent to S104-S105. After the first preset condition is met, the latest point cloud segmentation result is used as the point cloud segmentation result of the target point cloud. This process is equivalent to S106.

[0171] Figure 6 The structure of the point cloud semantic segmentation model is also shown in the figure. Figure 6The point cloud semantic segmentation model includes initialization and S encoders. Initialization includes point cloud processing, serialization, and embedding. The encoders include grid pooling, shuffle orders, and N blocks. Each block includes conditional positional encoding (xCPE), a layer normalization layer, an attention mechanism, and a multi-layer perceptron (MLP).

[0172] This embodiment of the application updates the encoded features used to represent the ground truth types of user annotations and the predicted classification results while keeping the parameters of the point cloud semantic segmentation model unchanged. This method ensures faster consistency between the point cloud segmentation results output by the point cloud semantic segmentation model and the ground truth types annotated by the user, reducing the time required for point cloud segmentation and improving its efficiency. Furthermore, not modifying the parameters of the point cloud semantic segmentation model avoids the impact of unreasonable user annotations on the model's basic performance, eliminating the need for model parameter rollback and avoiding the additional file reading time caused by model parameter rollback. In addition, directly inputting the encoded features representing the ground truth types of user annotations and the predicted classification results, along with the target point cloud data, into the point cloud semantic segmentation model allows it to focus more on achieving multi-class, multi-target semantic segmentation, reducing the number of ground truth types that users need to annotate, lowering the annotation difficulty, and reducing the user's annotation workload.

[0173] Corresponding to the aforementioned point cloud segmentation method, this application also provides a point cloud segmentation apparatus, see [link to relevant documentation]. Figure 7 The device includes:

[0174] The classification result determination module 701 is used to determine the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point. The first point is a labeled point, and the second point is an unlabeled point. The classification result is used to represent the type of the point and the confidence level of the type.

[0175] The encoding feature generation module 702 is used to generate features that represent the truth value type and the classification results of each second point, as encoding features;

[0176] The segmentation result acquisition module 703 is used to input the encoded features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model;

[0177] The prediction type acquisition module 704 is used to acquire the type of each first point in the point cloud segmentation result, and use it as the prediction type of each first point.

[0178] The encoding feature adjustment module 705 is used to adjust the encoding features in the direction of reducing the difference between the prediction type and the true value type of each first point while keeping the parameters of the point cloud semantic segmentation model unchanged. The adjusted encoding features are used as new encoding features. The module then returns to the step of inputting the encoding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, until the first preset condition is met.

[0179] The point cloud segmentation result module 706 is used to use the latest point cloud segmentation result as the point cloud segmentation result of the target point cloud after the first preset condition is met.

[0180] In one possible embodiment, determining the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point includes:

[0181] In the target point cloud, find points whose distance to each first point meets the preset proximity condition, and use them as the second points corresponding to each first point.

[0182] For each first point, the classification result of each corresponding second point is determined. The type represented by the classification result is the same as the true value type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point.

[0183] In one possible embodiment, the first points are labeled sequentially;

[0184] In the target point cloud, find points whose distance to each first point satisfies a preset proximity condition, and use these points as the second points corresponding to each first point, including:

[0185] For each first point, find the point in the target point cloud whose distance to the first point is less than the distance threshold corresponding to the first point, and use it as the second point corresponding to the first point. The earlier the first point is labeled, the larger the distance threshold is, and the higher the confidence level represented by the classification result is.

[0186] In one possible embodiment, the encoded features are an N×L matrix, where N is the total number of points in the target point cloud and L is the total number of types;

[0187] The value of the element in the i-th row and j-th column of the matrix is: the confidence score of the i-th point in the target point cloud for the j-th type.

[0188] In one possible embodiment, the device further includes:

[0189] The model training module is used to train a point cloud semantic segmentation model in the following ways:

[0190] Select at least one third point from the sample point cloud, wherein the sample point cloud is a point cloud with a predetermined truth value type for each point;

[0191] Based on the location and ground truth type of the third point, determine the classification result of at least one fourth point in the sample point cloud;

[0192] Generate features to represent the truth type of each third point and the classification result of each fourth point, as sample features;

[0193] The sample features and point cloud data of the sample point cloud are input into the model to be trained to obtain the candidate segmentation results output by the model to be trained.

[0194] In the candidate segmentation results, identify points whose type differs from the true value type, and designate them as the fifth point;

[0195] Take the third and fifth points as the new third point, and return to execute the step of determining the classification result of at least one fourth point in the sample point cloud based on the position and truth type of the third point, until the second preset condition is met.

[0196] After the second preset condition is met, the model to be trained is trained with the latest candidate segmentation results to obtain the point cloud semantic segmentation model.

[0197] In one possible embodiment, identifying points in the candidate segmentation results whose type differs from the truth type, as a fifth point, includes:

[0198] Identify all points in the candidate segmentation results whose type differs from the true value type as candidate points;

[0199] Select the fifth point from the candidate points that meet the preset selection criteria;

[0200] The preset selection criteria include any one or more of the first, second, and third criteria.

[0201] The first condition is: located at the center of a dense region, and the dense region is a region where the density of candidate points meets the preset density condition;

[0202] The second condition is: it is located in the neighborhood of the error boundary, and the error boundary is a boundary where the truth value types of the two points are the same;

[0203] The third condition is: the truth value type is the preset type of interest.

[0204] This application also provides an electronic device, such as... Figure 8 As shown, it includes:

[0205] Memory 801 is used to store computer programs;

[0206] When processor 802 executes a program stored in memory 801, it performs the following steps:

[0207] Based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point, determine the classification result of at least one second point in the target point cloud, where the first point is a labeled point and the second point is an unlabeled point. The classification result is used to represent the type of the point and the confidence level of the type.

[0208] Generate features to represent the truth type and the classification results of each second point, as encoded features;

[0209] The encoded features and point cloud data of the target point cloud are input into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model.

[0210] Obtain the type of each first point in the point cloud segmentation result, and use it as the predicted type of each first point;

[0211] While keeping the parameters of the point cloud semantic segmentation model unchanged, the coding features are adjusted in the direction of reducing the difference between the predicted type and the true value type of each first point, and the adjusted coding features are used as the new coding features. The process is then repeated to input the coding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, until the first preset condition is met.

[0212] After the first preset condition is met, the latest point cloud segmentation result is used as the point cloud segmentation result of the target point cloud.

[0213] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 802, the communication interface, and the memory 801 communicating with each other via the communication bus.

[0214] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0215] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0216] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0217] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0218] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the point cloud segmentation methods described above.

[0219] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the point cloud segmentation methods described above.

[0220] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0221] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0222] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products containing instructions are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0223] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A point cloud segmentation method, characterized in that, The method includes: Based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point, the classification result of at least one second point in the target point cloud is determined, wherein the first point is a labeled point and the second point is an unlabeled point, and the classification result is used to represent the type of the point and the confidence level of the type; Generate features to represent the truth value type and the classification results of each of the second points, as encoded features; The encoded features and the point cloud data of the target point cloud are input into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model. Obtain the type of each first point in the point cloud segmentation result, and use it as the prediction type of each first point; While keeping the parameters of the point cloud semantic segmentation model unchanged, the encoding features are adjusted in the direction of reducing the difference between the prediction type and the true value type of each first point, and the adjusted encoding features are used as new encoding features. The process of inputting the encoding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model is repeated until the first preset condition is met. After the first preset condition is met, the latest point cloud segmentation result is used as the point cloud segmentation result of the target point cloud.

2. The method according to claim 1, characterized in that, The step of determining the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point includes: In the target point cloud, find points whose distance to each first point satisfies a preset proximity condition, and use them as the second points corresponding to each first point. For each of the first points, the classification results of each of the second points corresponding to the first point are determined respectively, wherein the type represented by the classification result is the same as the truth value type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point.

3. The method according to claim 2, characterized in that, Each of the first points is marked sequentially; The step of finding points in the target point cloud whose distances to each first point satisfy a preset proximity condition, and using these points as the second points corresponding to each first point, includes: For each of the first points, a point in the target point cloud whose distance to the first point is less than the distance threshold corresponding to the first point is found, and this point is designated as the second point corresponding to the first point. The higher the labeling order of the first point, the larger the distance threshold, and the higher the confidence level represented by the classification result.

4. The method according to claim 3, characterized in that, The encoding features are an N×L matrix, where N is the total number of points in the target point cloud and L is the total number of types; The value of the element in the i-th row and j-th column of the matrix is: the confidence level of the i-th point in the target point cloud for the j-th type.

5. The method according to claim 1, characterized in that, The point cloud semantic segmentation model is trained using the following methods: Select at least one third point from the sample point cloud, wherein the sample point cloud is a point cloud with a predetermined truth value type for each point; Based on the location and truth type of the third point, determine the classification result of at least one fourth point in the sample point cloud; Generate features to represent the truth type of each of the third points and the classification result of each of the fourth points, as sample features; The sample features and point cloud data of the sample point cloud are input into the model to be trained to obtain the candidate segmentation result output by the model to be trained. In the candidate segmentation results, identify points whose type differs from the truth value type, and designate them as the fifth point; Take the third point and the fifth point as the new third point, and return to the step of determining the classification result of at least one fourth point in the sample point cloud based on the position and truth type of the third point, until the second preset condition is met; After the second preset condition is met, the model to be trained is trained with the latest candidate segmentation results to obtain a point cloud semantic segmentation model.

6. The method according to claim 5, characterized in that, In the candidate segmentation results, points whose type differs from the truth type are identified as the fifth point, including: All points in the candidate segmentation results whose type differs from the truth value type are identified as candidate points; Select a point from the candidate points that meets the preset selection criteria as the fifth point; The preset selection conditions include any one or more of the first condition, the second condition, and the third condition; The first condition is: located at the center of a dense region, where the density of the candidate points satisfies a preset density condition; The second condition is: located within the neighborhood of the error boundary, wherein the error boundary is a boundary where the truth value types of the points on both sides are the same; The third condition is: the truth value type is a preset type of interest.

7. A point cloud segmentation device, characterized in that, The device includes: The classification result determination module is used to determine the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point, wherein the first point is a labeled point and the second point is an unlabeled point, and the classification result is used to represent the type of the point and the confidence level of the type; The encoding feature generation module is used to generate features representing the truth value type and the classification results of each of the second points, as encoding features; The segmentation result acquisition module is used to input the encoded features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model; The prediction type acquisition module is used to acquire the type of each first point in the point cloud segmentation result, and use it as the prediction type of each first point. The encoding feature adjustment module is used to adjust the encoding features in a direction that reduces the difference between the prediction type and the ground truth type of each first point while keeping the parameters of the point cloud semantic segmentation model unchanged, and use the adjusted encoding features as new encoding features. Then, it returns to the step of inputting the encoding features and the point cloud data of the target point cloud into the point cloud semantic segmentation model to obtain the point cloud segmentation result output by the point cloud semantic segmentation model, until the first preset condition is reached. The point cloud segmentation result module is used to take the latest point cloud segmentation result as the point cloud segmentation result of the target point cloud after the first preset condition is met.

8. The apparatus according to claim 7, characterized in that, The step of determining the classification result of at least one second point in the target point cloud based on the position of at least one first point in the target point cloud and the ground truth type labeled for the first point includes: In the target point cloud, find points whose distance to each first point satisfies a preset proximity condition, and use them as the second points corresponding to each first point. For each of the first points, the classification results of each of the second points corresponding to the first point are determined respectively, wherein the type represented by the classification result is the same as the truth value type of the first point, and the confidence level represented by the classification result is negatively correlated with the distance between the second point and the first point; Each of the first points is marked sequentially; The step of finding points in the target point cloud whose distances to each first point satisfy a preset proximity condition, and using these points as the second points corresponding to each first point, includes: For each of the first points, a point in the target point cloud that is less than the distance threshold corresponding to the first point is found and is designated as the second point corresponding to the first point. The higher the labeling order of the first point, the larger the distance threshold, and the higher the confidence level represented by the classification result. The encoding features are an N×L matrix, where N is the total number of points in the target point cloud and L is the total number of types; The value of the element in the i-th row and j-th column of the matrix is: the confidence score of the i-th point in the target point cloud for the j-th type; The device further includes: The model training module is used to train the point cloud semantic segmentation model in the following ways: Select at least one third point from the sample point cloud, wherein the sample point cloud is a point cloud with a predetermined truth value type for each point; Based on the location and truth type of the third point, determine the classification result of at least one fourth point in the sample point cloud; Generate features to represent the truth type of each of the third points and the classification result of each of the fourth points, as sample features; The sample features and point cloud data of the sample point cloud are input into the model to be trained to obtain the candidate segmentation result output by the model to be trained. In the candidate segmentation results, identify points whose type differs from the truth value type, and designate them as the fifth point; Take the third point and the fifth point as the new third point, and return to the step of determining the classification result of at least one fourth point in the sample point cloud based on the position and truth type of the third point, until the second preset condition is met; After the second preset condition is met, the model to be trained is trained with the latest candidate segmentation results to obtain a point cloud semantic segmentation model. In the candidate segmentation results, points whose type differs from the truth type are identified as the fifth point, including: All points in the candidate segmentation results whose type differs from the truth value type are identified as candidate points; Select a point from the candidate points that meets the preset selection criteria as the fifth point; The preset selection conditions include any one or more of the first condition, the second condition, and the third condition; The first condition is: located at the center of a dense region, where the density of the candidate points satisfies a preset density condition; The second condition is: located within the neighborhood of the error boundary, wherein the error boundary is a boundary where the truth value types of the points on both sides are the same; The third condition is: the truth value type is a preset type of interest.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images