SQN-cr based weakly supervised semantic segmentation method for three-dimensional point cloud, medium and system

Abstract

The application provides a three-dimensional point cloud weak supervision semantic segmentation method, medium and system based on SQN-CR for large-scale scenes. The method integrates contrast regularization and multi-scale classification ideas to balance the contradiction between annotation cost and segmentation accuracy. The contrast regularization method used in the model includes point-level contrast regularization and local contrast regularization. The point-level contrast regularization uses enhanced point and original point feature information for training constraint, and the local contrast regularization fully utilizes the feature information implied in the weak supervision label. In addition, multi-scale classification is used to extract multi-layer features and classify them to optimize the model, so that better segmentation results are obtained by minimizing the labeling workload, and the efficiency and effect of semantic segmentation of large-scale three-dimensional point clouds are further improved.

Description

Technical Field

[0001] This application relates to the field of 3D point cloud semantic segmentation, and in particular to a 3D point cloud weakly supervised semantic segmentation method, medium, and system based on SQN-CR. Background Technology

[0002] Large-scale 3D point cloud semantic segmentation has become a hot topic in computer vision and 3D data processing in recent years. It involves the efficient and accurate semantic analysis and understanding of massive and complex point cloud data. The goal of large-scale 3D point cloud semantic segmentation is to divide massive amounts of point cloud data into regions with different semantic meanings, such as buildings, roads, vehicles, and pedestrians, thereby providing crucial information for various advanced applications. For example, in urban planning, semantic segmentation of large-scale point cloud data can automatically identify various structures and facilities in the city, providing data support for urban planning and management. In the field of autonomous driving, vehicles need to process and analyze large-scale point cloud data in real time to accurately perceive the surrounding environment and make safe driving decisions. With the rapid development of 3D scanning and sensor technologies, the ability to acquire large-scale point cloud data is constantly improving. This data typically contains millions or even billions of points, each with 3D coordinates and possible additional information such as color and intensity.

[0003] Current methods for large-scale 3D point cloud semantic segmentation are mainly divided into fully supervised methods and weakly supervised methods. Fully supervised point cloud semantic segmentation methods rely on fully labeled training data, requiring a large amount of labeled data, which increases the cost of data collection and processing. Weakly supervised point cloud semantic segmentation methods utilize simpler and coarser labeled information for model training, effectively reducing labeling costs. Existing weakly supervised point cloud semantic segmentation methods mainly include methods based on 2D labels, methods based on pseudo-3D labels, and methods based on limited 3D labels.

[0004] However, existing weakly supervised semantic segmentation methods still have significant limitations. First, they are insufficient in considering local semantic information in large-scale scenes, resulting in the underutilization of information contained in weak annotations. Second, to achieve better segmentation accuracy, existing techniques often integrate multiple stages, which is more complex and less robust than standard end-to-end training. Third, existing methods do not perform well in balancing segmentation accuracy and annotation data volume; that is, existing weakly supervised semantic segmentation methods cannot reduce the number of annotations while maintaining segmentation accuracy without a significant decrease. Summary of the Invention

[0005] Therefore, it is necessary to provide a weakly supervised semantic segmentation method, medium, and system for 3D point clouds based on SQN-CR (Semantic Query Network-Contrastive Regularization) to address the aforementioned technical problems and improve the efficiency and effectiveness of large-scale 3D point cloud semantic segmentation.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] On one hand, embodiments of the present invention provide a weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR, comprising the following steps:

[0008] Input a 3D point cloud, mark some points in the 3D point cloud as the original point cloud, and generate an enhanced point cloud from the original point cloud;

[0009] Local feature aggregation and random downsampling are performed on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer;

[0010] Based on the feature data of layer W, the coordinates of random points in the original point cloud and the enhanced point cloud are retrieved in each layer to obtain W local feature information of the random points; the W local feature information of the random points includes the association information between the random points and their neighboring points;

[0011] Trilinear interpolation is performed on W local feature information of random points to obtain W interpolated feature vectors;

[0012] W interpolated feature vectors are concatenated into a single vector, which is then input into the first series of multilayer perceptrons for prediction to obtain the first prediction result. Point-level contrast regularization and local contrast regularization are used to optimize the first prediction result to obtain the optimized first prediction result.

[0013] The optimized first prediction result is input into the batch normalization layer for re-prediction to obtain the second prediction result;

[0014] The W interpolated feature vectors are input into the second series of multilayer perceptrons for multi-scale classification to obtain the classification prediction results.

[0015] The second prediction result is optimized based on the second prediction result and the classification prediction result to obtain the final prediction result of the random point;

[0016] Based on the calculation process of the final prediction result of random points, input the coordinates of unmarked points in the 3D point cloud, and output the prediction result of all points in the 3D point cloud.

[0017] In one embodiment, generating an enhanced point cloud from the original point cloud includes:

[0018] Data augmentation can be performed on raw point cloud data, which can be one or a combination of adding random noise, performing random rotation, scaling, and jittering.

[0019] In one embodiment, local feature aggregation and random downsampling are performed on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data, including:

[0020] Using the RandLA-Net neural network as a feature extractor, local feature aggregation and random downsampling are performed on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data.

[0021] In one embodiment, point-level contrastive regularization and local contrastive regularization are used to optimize the first prediction result, resulting in an optimized first prediction result, including:

[0022] Based on the first prediction results obtained from the original point cloud and the enhanced point cloud, calculate the point-level contrast regularization loss.

[0023] Calculate the local contrast regularization loss based on the correlation information between random points and their neighbors.

[0024] Based on point-level comparison of regularization loss Local contrast regularization loss The first prediction result is optimized to obtain the optimized first prediction result.

[0025] In one embodiment, the point-level contrast regularization loss is calculated based on the first prediction results of the original point cloud and the enhanced point cloud. include:

[0026] Obtain the first prediction result of the original points in the original point cloud;

[0027] Obtain the first prediction result of the augmented points in the augmented point cloud, where the augmented points are generated from the original points;

[0028] By merging the first prediction results of the original point with those of its generated augmented points, separating the original point from the first prediction results of other augmented points, the point-level contrastive regularization loss is calculated.

[0029] In one embodiment, the first prediction results of the original point and its generated augmented points are brought closer together, the first prediction results of the original point and other augmented points are separated, and the point-level contrast regularization loss is calculated. include:

[0030]

[0031] Where N is the total number of points in the input point cloud. and These are the prediction results for the i-th and j-th points of the original point cloud and the enhanced point cloud, respectively; 1 [j＝i] and 1 [j≠i] It is the indicator function; τ is the temperature hyperparameter.

[0032] In one embodiment, the local contrast regularization loss is calculated based on the correlation information between a random point and its neighboring points. include:

[0033] Obtain the coordinates of a random point;

[0034] Search for its K nearest enhanced neighbors based on Euclidean distance; K is a positive integer greater than 1.

[0035] Obtain the average of the first predictions from K enhanced neighbors;

[0036] The local contrast regularization loss is calculated by averaging the first prediction of the random point with the first predictions of its K augmented neighbors, separating the random point from the average of the first predictions of its K other augmented neighbors.

[0037] In one embodiment, the first prediction result of the random point is brought closer to the average of the first prediction results of its K augmented neighbors, and the average of the first prediction results of the random point and its other K augmented neighbors is separated to calculate the local contrast regularization loss. include:

[0038]

[0039] Where N is the total number of points in the input point cloud. and These are the first prediction result of the i-th random point and the average of the first prediction results of the K enhanced neighbors of the j-th random point, respectively. [j＝i] and 1 [j≠i] τ is the indicator function; τ is the temperature hyperparameter.

[0040] In one embodiment, the batch normalization layer is a hypersphere, which is a sphere in a high-dimensional space.

[0041] In one embodiment, the second prediction result is optimized based on the second prediction result and the classification prediction result to obtain the final prediction result for the random point; the degree of optimization is evaluated by a loss function, which is:

[0042]

[0043] in For cross-entropy loss, To compare regularization losses, including point-level comparative regularization losses. Local contrast regularization loss

[0044] In one embodiment, the regularization loss is compared. It can be represented as:

[0045]

[0046] On one hand, embodiments of the present invention also provide a computer-readable storage medium storing code programs thereon, characterized in that the code programs, when executed by a processor, implement the steps of any of the above-mentioned weakly supervised semantic segmentation methods for 3D point clouds based on SQN-CR.

[0047] On the one hand, embodiments of the present invention also provide a weakly supervised semantic segmentation system for three-dimensional point clouds based on SQN-CR, characterized in that it includes a point cloud input enhancement component, a local feature extraction component, a random point coordinate retrieval component, a trilinear interpolation component, a contrast regularization prediction component, a normalization layer prediction component, a multi-scale classification component, a random point final prediction component, and a full point cloud segmentation output component.

[0048] The point cloud input enhancement module is used to input a 3D point cloud, mark some points in the 3D point cloud as the original point cloud, and generate an enhanced point cloud from the original point cloud.

[0049] The local feature extraction component is used to perform local feature aggregation and random downsampling on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer;

[0050] The random point coordinate retrieval component is used to retrieve the random point coordinates of the original point cloud and the enhanced point cloud at each layer based on the W layers of feature data, and obtain W local feature information of the random point; the W local feature information of the random point includes the association information between the random point and its neighboring points;

[0051] The trilinear interpolation component is used to perform trilinear interpolation on W local feature information of random points to obtain W interpolated feature vectors;

[0052] The contrast regularization prediction component is used to concatenate W interpolated feature vectors into a single vector, which is then input into the first series of multilayer perceptrons for prediction to obtain the first prediction result. Point-level contrast regularization and local contrast regularization are used to optimize the first prediction result to obtain the optimized first prediction result.

[0053] The normalization layer prediction component is used to input the optimized first prediction result into the batch normalization layer for re-prediction to obtain the second prediction result;

[0054] The multi-scale classification component is used to input W interpolated feature vectors into the second series of multilayer perceptrons for multi-scale classification and obtain classification prediction results.

[0055] The random point final prediction component is used to optimize the second prediction result based on the second prediction result and the classification prediction result to obtain the final prediction result of the random point;

[0056] The full point cloud segmentation output component takes the coordinates of unmarked points in the 3D point cloud as input and outputs the prediction results of all points in the 3D point cloud, based on the calculation process of the final prediction results of random points.

[0057] One of the above technical solutions has the following advantages and beneficial effects:

[0058] The aforementioned weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR first encodes the original and enhanced point clouds hierarchically. Then, it uses the coordinates of arbitrary points as input to retrieve their local feature information and obtains feature vectors through trilinear interpolation. These vectors are used to calculate the multi-scale classification loss, then concatenated into a single vector, and input into a series of multilayer perceptrons for semantic prediction. The prediction results of the enhanced and original data are compared. Finally, a final batch normalization (BN) layer is designed to constrain the local contrastive regularization loss and segmentation loss in a more adaptive space. This method improves segmentation accuracy by enhancing the consideration of local information and adopting a multi-scale classification strategy, while maintaining a simple structure in an end-to-end manner. This achieves better segmentation results by minimizing the annotation workload, further improving the efficiency and effectiveness of semantic segmentation of large-scale 3D point clouds. Attached Figure Description

[0059] Figure 1 This is a flowchart of the steps of a weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR when W=4 in one embodiment;

[0060] Figure 2 This is a framework diagram for training a weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR when W=4 in one embodiment.

[0061] Figure 3 This is a visualization comparison of the effects of a weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR applied to the S3DIS large-scale indoor dataset in one embodiment.

[0062] Figure 4This is a visualization comparison of the effects of a weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR applied to SemanticKITTI in one embodiment.

[0063] Figure 5 This is a visualization comparison of the effects of a weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR applied to SensatUrban in one embodiment.

[0064] Figure 6 This is an architecture diagram of a weakly supervised semantic segmentation system for 3D point clouds based on SQN-CR in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0067] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are implemented by those skilled in the art. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0068] This application provides a weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR (Semantic Query Network-Contrastive Regularization). When W equals 4, the steps are as follows: Figure 1 As shown, it includes the following steps:

[0069] 101: Input a 3D point cloud, mark some points in the 3D point cloud as the original point cloud, and generate an enhanced point cloud from the original point cloud;

[0070] Semantic segmentation, as we understand it, refers to classifying each point element in point cloud data into different categories. In this example, semantic segmentation is performed on 3D point cloud data in an image. Through the classification process, each point in the point cloud can be assigned to a corresponding category, such as belonging to buildings, ground, trees, etc.

[0071] It is understandable that in this three-dimensional weakly supervised model, given an input point cloud P containing N points, the number of labeled points M is determined by the weak supervision ratio. For example, at a setting of 0.1%, M equals 0.1% × N, and all labeled points are randomly selected.

[0072] As is understandable, this example encodes the original point cloud and the enhanced point cloud in a hierarchical manner. The hierarchical encoding method represents and processes the information of the point cloud according to a certain hierarchical structure, with the aim of improving the model's ability to understand and model different hierarchical structures in the point cloud.

[0073] 102: Perform local feature aggregation and random downsampling on the original point cloud and the enhanced point cloud to obtain their respective 4-layer feature data; the 4-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud;

[0074] It is understandable that local feature aggregation involves selecting a center point from the original point cloud and the enhanced point cloud, and then aggregating the feature information in the local neighborhood around this center point. This allows the extraction of local features of each point in the point cloud, enabling the model to better understand the local structure of the point cloud. The aggregated features can include the coordinates, normals, colors, etc. of the points. Random downsampling reduces the density of the point cloud by randomly selecting a subset of the point cloud. By retaining a random subset of the point cloud, the computational burden is reduced while the recognition of the overall features of the point cloud is still maintained.

[0075] 103: Based on 4 layers of feature data, the coordinates of random points in the original point cloud and the enhanced point cloud are retrieved in each layer to obtain 4 local feature information of the random points; the 4 local feature information of the random points include the association information between the random points and their neighboring points;

[0076] As can be understood, random points are the points to be classified or segmented in the entire acquired 3D point cloud. In this example, the coordinates of random points can be used to search in each of the four layers to obtain the local feature information of the random point in each layer.

[0077] 104: Perform trilinear interpolation on the four local feature information of the random point to obtain four interpolated feature vectors;

[0078] It is understandable that trilinear interpolation interpolates data in three-dimensional space to obtain a continuity estimate at non-grid point locations; the four interpolation feature vectors can have different dimensions. In extracting point features, to reconstruct the true trilinear interpolation process, this embodiment chooses to set the weights to the reciprocal of the distance.

[0079] 105: The four interpolated feature vectors are concatenated into a single vector and input into the first series of multilayer perceptrons for prediction to obtain the first prediction result. Point-level contrast regularization and local contrast regularization are used to optimize the first prediction result to obtain the optimized first prediction result.

[0080] It is understandable that the four interpolated feature vectors are concatenated and input into the first series of multilayer perceptrons (MLPs). In this example, the first series of multilayer perceptrons has two hidden layers.

[0081] It is understandable that contrastive regularization is used to improve model accuracy by pulling points toward positive anchors with similarity and away from negative anchors. This method chooses to use this strategy in the prediction space, using the prediction result of each point as both a positive and a negative anchor to achieve better constraints.

[0082] 106: Input the optimized first prediction result into the batch normalization layer for re-prediction to obtain the second prediction result;

[0083] It's understandable that adding point-level contrastive regularization and local contrastive regularization to the initial segmentation loss is problematic. These two optimizations have different objectives, and the segmentation loss and local contrastive regularization loss are calculated independently; they don't always converge simultaneously. Therefore, this example uses two alternating spaces to achieve better segmentation results. Specifically, a batch normalization (BN) layer is added after the fully connected layer to construct a hypersphere to optimize the segmentation loss, while the classification before the BN layer is used for prediction, thus optimizing the contrastive regularization loss.

[0084] 107: Input the four interpolated feature vectors into the second series of multilayer perceptrons for multi-scale classification to obtain the classification prediction results;

[0085] This can be understood as performing multi-scale classification on four interpolated feature vectors to obtain local features of the query point at each scale, and then classifying these features. Specifically, the extracted features at each scale are input into a multi-layer fully connected network for classification, and the output prediction results are fed into the segmentation loss for optimization. This example achieves multi-scale classification based on multi-scale feature fusion, thereby maximizing the acquisition of more comprehensive overall information and detailed local information.

[0086] 108: Optimize the second prediction result based on the second prediction result and the classification prediction result to obtain the final prediction result for the random point;

[0087] It is understandable that, taking into account the optimization process of comparative regularization and multi-scale classification, the prediction results are finally optimized to achieve the final prediction goal and output the final prediction result.

[0088] 109: Based on the calculation process of the final prediction result of random points, input the coordinates of unmarked points in the 3D point cloud, and output the prediction result of all points in the 3D point cloud.

[0089] Understandably, in this example, with a 0.1% setting, the number of marked points M equals 0.1% × N. In this example, the segmentation model is trained according to steps 101-108 based on the information of the marked M points until it is optimized to meet the preset segmentation criteria. Then, the coordinates of the remaining 99.9% of unmarked points in the 3D point cloud are input into the model, and the final prediction result of the points is output.

[0090] It is understandable that if this model is needed to segment other point clouds, and if they are point clouds of the same type, such as point clouds containing the same classification, then the model can be directly entered into step 109 for semantic segmentation of the whole point cloud without training. If the classifications are different or the gap is too large, then the model needs to be retrained for the new point cloud according to steps 101-108 before entering the final whole point cloud segmentation step.

[0091] The above-mentioned weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR, when W=4, has the following overall architecture for its training model process (steps 102-108): Figure 2 As shown, the original and augmented point clouds are first encoded hierarchically. Then, the coordinates of arbitrary points are used as input to retrieve their local feature information, and feature vectors are obtained through trilinear interpolation. These vectors are used to calculate the multi-scale classification loss, then concatenated into a single vector, and input into a series of multilayer perceptrons for semantic prediction. The prediction results of the augmented and original data are compared. Finally, a final batch normalization (BN) layer is designed to constrain the local contrastive regularization loss and segmentation loss in a more adaptive space. This method improves the segmentation accuracy by considering local information in the augmentation and adopting a multi-scale classification strategy, while maintaining a simple structure in an end-to-end manner. This achieves better segmentation results by minimizing the annotation workload, further improving the efficiency and effectiveness of semantic segmentation of large-scale 3D point clouds.

[0092] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps; they can be executed in other orders. Figure 1At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0093] In one embodiment, generating an enhanced point cloud from the original point cloud includes:

[0094] Data augmentation can be performed on raw point cloud data, which can be one or a combination of adding random noise, performing random rotation, scaling, and jittering.

[0095] It is understood that the enhanced point cloud can be generated from the original point cloud through random sampling, rotation and translation, scaling, noise injection, dithering, and data cropping. In this embodiment, random noise, random rotation, scaling, and dithering can be selected.

[0096] In one embodiment, local feature aggregation and random downsampling are performed on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data, including:

[0097] Using the RandLA-Net neural network as a feature extractor, local feature aggregation and random downsampling are performed on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data.

[0098] It is understandable that this embodiment chooses the existing fully supervised method RandLA-Net as the feature extractor to obtain better feature embeddings. RandLA-Net is a neural network that can effectively perform large-scale semantic segmentation of point clouds, mainly relying on the Local Feature Aggregation (LFA) module and the Random Sampling (RS) method. Therefore, the LFA module and the RS method are directly applied to the four-layer encoder to achieve feature aggregation for each point.

[0099] In one embodiment, point-level contrastive regularization and local contrastive regularization are used to optimize the first prediction result, resulting in an optimized first prediction result, including:

[0100] Based on the first prediction results obtained from the original point cloud and the enhanced point cloud, calculate the point-level contrast regularization loss.

[0101] Calculate the local contrast regularization loss based on the correlation information between random points and their neighbors.

[0102] Based on point-level comparison of regularization loss Local contrast regularization loss The first prediction result is optimized to obtain the optimized first prediction result.

[0103] It's understandable that point-level comparison regularization loss... Local contrast regularization loss The calculations are performed simultaneously, and the first prediction result is optimized at the same time.

[0104] In one embodiment, the point-level contrast regularization loss is calculated based on the first prediction results of the original point cloud and the enhanced point cloud. include:

[0105] Obtain the first prediction result of the original points in the original point cloud;

[0106] Obtain the first prediction result of the augmented points in the augmented point cloud, where the augmented points are generated from the original points;

[0107] By merging the first prediction results of the original point with those of its generated augmented points, separating the original point from the first prediction results of other augmented points, the point-level contrastive regularization loss is calculated.

[0108] Understandably, point-level contrastive regularization is performed on the original and augmented data to bring the predicted results of the original and augmented points closer together and separate them from other points, and this is achieved by introducing a loss function. To describe and control this "distance".

[0109] In one embodiment, the first prediction results of the original point and its generated augmented points are brought closer together, the first prediction results of the original point and other augmented points are separated, and the point-level contrastive regularization loss is calculated. include:

[0110]

[0111] Where N is the total number of points in the input point cloud. and These are the prediction results for the i-th and j-th points of the original point cloud and the enhanced point cloud, respectively; 1 [j＝i] and 1 [j≠i] τ is the indicator function; τ is the temperature hyperparameter.

[0112] Understandable. 1 [j＝i] ∈{0,1} is an indicator function. When j = i, its value is 1; when j ≠ i, its value is the same.

[0113] In one embodiment, the local contrast regularization loss is calculated based on the correlation information between a random point and its neighboring points. include:

[0114] Obtain the coordinates of a random point;

[0115] Search for its K nearest enhanced neighbors based on Euclidean distance; K is a positive integer greater than 1.

[0116] Obtain the average of the first predictions from K enhanced neighbors;

[0117] The local contrast regularization loss is calculated by averaging the first prediction of the random point with the first predictions of its K augmented neighbors, separating the random point from the average of the first predictions of its K other augmented neighbors.

[0118] It is understandable that since semantic segmentation models for point clouds primarily predict the local features of each point, better capturing local information becomes crucial for improving segmentation accuracy. Considering that the local features of point clouds mainly originate from points and their neighborhoods, it is generally believed that the local information of point clouds in large-scale scenes typically exhibits strong semantic similarity. To better utilize this characteristic, this example proposes adding a local contrastive regularization module to optimize the modeling of local features in point clouds and introducing a loss function. This is used to describe and control the distance relationship between a random point and the average value of its surrounding neighboring points.

[0119] In one embodiment, the first prediction result of the random point is brought closer to the average of the first prediction results of its K augmented neighbors, while the average of the first prediction results of the random point and its other K augmented neighbors is separated, and the local contrastive regularization loss is calculated. include:

[0120]

[0121] Where N is the total number of points in the input point cloud. and These are the first prediction result of the i-th random point and the average of the first prediction results of the K enhanced neighbors of the j-th random point, respectively. [j＝i] and 1 [j≠i] τ is the indicator function; τ is the temperature hyperparameter.

[0122] Understandable, 1 [j＝1] ∈{0,1} is an indicator function; its value is 1 when j = i, and the value is the same when j ≠ i. In practice, given a 3D query point x in the prediction space... i Given its coordinates x, y, and z, this invention searches for its K nearest augmenting neighbors based on Euclidean distance and obtains the average of their predicted probabilities. and The calculation formula is to achieve x i The predicted value is closer to And it is far away from the predicted average of other nearest neighbors.

[0123] In one embodiment, the batch normalization layer is a hypersphere, which is a sphere in a high-dimensional space.

[0124] It is understandable that a contrastive regularization loss is added on top of the initial segmentation loss. Since point-level contrastive regularization and local contrastive regularization have different objectives, optimization can be achieved through two alternating spaces to obtain better segmentation results. This method chooses to add a batch normalization (BN) layer after the fully connected layer to construct a hypersphere to optimize the segmentation loss, while the classification before the BN layer is used for prediction to optimize the contrastive regularization loss.

[0125] In one embodiment, the second prediction result is optimized based on the second prediction result and the classification prediction result to obtain the final prediction result for the random point; the degree of optimization is evaluated by a loss function, which is:

[0126]

[0127] in For cross-entropy loss, To compare regularization losses, including point-level comparative regularization losses. Local contrast regularization loss

[0128] Understandable. It is the cross-entropy between logits and labels calculated after softmax classification, and its formula is as follows:

[0129]

[0130] Where C is the total number of label categories, M is the number of label points, H is the number of layers, and y is the total number of labels. ic Point x i The true value belonging to category c Representing point x i The predicted value for category c, This represents the predicted value of point xi belonging to category c at level h.

[0131] In one embodiment, compare the regularization loss. It can be represented as:

[0132]

[0133] Understandable. and The calculation method is as described above.

[0134] On one hand, embodiments of the present invention also provide a computer-readable storage medium storing code programs thereon, characterized in that the code programs, when executed by a processor, implement the steps of any of the above-mentioned weakly supervised semantic segmentation methods for 3D point clouds based on SQN-CR.

[0135] It is understandable that the implementation of the SQN-CR-based 3D point cloud weakly supervised semantic segmentation method does not require changes to the hardware, but only design and modification at the software level.

[0136] In some implementations, to more intuitively and comprehensively illustrate the SQN-CR-based weakly supervised semantic segmentation method for 3D point clouds, the following are experimental verification examples of the SQN-CR-based weakly supervised semantic segmentation method for 3D point clouds. These include evaluations on the S3DIS large-scale indoor dataset, on SemanticKITTI, and on SensatUrban.

[0137] All experiments used 0.1% random annotation labels to train the model end-to-end, and were conducted on a machine configured with Intel Core processors. TM The experiments were conducted on a PC with an i9-12900KF CPU and an NVIDIA RTX 3090Ti GPU (24GB GPU memory). The number of input points N varied depending on the dataset, M = 1% × N, and τ was set to 1.0 in this invention. It is worth noting that the SQN method uses a constant 1 / 3 weight in trilinear interpolation, while this experiment modifies this weight to a variable related to the Euclidean distance between nearest neighbors to achieve more accurate interpolation results.

[0138] Evaluation indicators:

[0139] This example evaluates the model's final performance across all points on the test dataset, selecting mean Intersection over Union (mIoU) and overall accuracy (OA) as evaluation metrics. Assuming K is the total number of classes, TP is the true positive, FP is the false positive, FN is the false negative, and TN is the true negative, these two evaluation metrics can be calculated as follows: To realistically evaluate the performance of the model, this example selects three commonly used public large-scale datasets for testing. They differ in terms of the sensors used, the size of the datasets, and the data features.

[0140]

[0141] (I) Evaluation on large-scale indoor datasets of S3DIS

[0142] S3DIS was created by Stanford University using Matterport from three different buildings. It consists of five areas with different characteristics, with a total area of ​​6,020 square meters. S3DIS has more than 215 million points and 13 semantic categories, including ceiling, floor, wall, beam, column, window, door, table, chair, sofa, bookcase, whiteboard, and clutter.

[0143] To achieve the segmentation performance of the SQN-CR model proposed in this invention in large-scale indoor scenes, it was compared with state-of-the-art (SOTA) methods on S3DIS Area-5. Detailed test results are shown in Table 1. It can be seen that the SQN-CR proposed in this invention achieves the highest mIoU among almost all methods, exceeding the fully supervised method PointNet by 20.37%, and exceeding Xu and Lee's methods with 10% and 0.2% weak annotations by 13.47% and 16.97%, respectively. Under the same training conditions, it is 2% higher than the baseline method of this invention. For the fully supervised methods KPCconv and RandLA-Net, with only 0.1% annotations, they are more accurate in segmentation, with this invention differing by only 5.63% and 1.53%, respectively. Furthermore, compared with Zhang et al. and PSD, with a tenfold increase in the number of annotations, the differences are only 0.33% and 2.03%, respectively. This indicates that the SQN-CR proposed in this invention can achieve results comparable to fully supervised methods on S3DIS Area-5 with only a small number of annotations.

[0144] Table 1. Quantitative results of different methods on Area-5 of S3DIS.

[0145]

[0146] "*" indicates the result of training using the official code.

[0147] Figure 3 The qualitative results of the segmentation are presented. It can be seen that SQN-CR can obtain results consistent with the ground truth and outperforms SQN, as demonstrated in the segmentation results for "table," "chair," and "wall." (II) Evaluation on SemanticKITTI

[0148] SemanticKTTI is a large outdoor scene dataset acquired using MLS (Moving Laser Scanning) technology at the University of Bonn, Germany, and densely annotated. It contains 45.49 million points and 28 semantic categories, and can distinguish between moving and non-moving vehicles and humans. The research team has made online evaluation available through a public website.

[0149] The present invention is described in Table 2 and Figure 4The performance of SQN-CR and other methods on SemanticKITTI was evaluated. Clearly, the SQN-CR of this invention achieves state-of-the-art (SOTA) performance, and with only 0.1% labeled training, it outperforms the fully supervised methods PointNet, PointNet++, HybridCR with 1% labeled training, and SQN with 0.1% labeled training by 38.3%, 32.8%, 0.6%, and 1.4%, respectively, and is only 5.2% and 1% lower than KPConv and RandLA-Net under fully supervised conditions.

[0150] Table 2 shows the quantitative results on the SemnaticKITTI dataset.

[0151]

[0152] "*" indicates the result of training using the official code.

[0153] This invention demonstrates the visualization of qualitative results on the SemanticKITTI validation set (Sequence 08), including the original point cloud, semantic labels, the results of this invention, and baseline results, presented from left to right, with red circles highlighting the areas where this invention surpasses SQN. It can be seen that the SQN-CR of this invention can better identify vegetation and sidewalks.

[0154] (III) Evaluation on SensatUrban

[0155] SensatUrban is a point cloud dataset collected over urban areas using drone photogrammetry. It contains nearly 3 billion points covering an area of ​​7.6 square kilometers across three UK cities, each labeled with 13 semantic categories. Due to the use of different sensors, the data characteristics of SensatUrban differ from those obtained using LiDAR.

[0156] like Figure 5 As shown in Table 3, for the SensatUrban dataset, the SQN-CR proposed in this invention achieves a higher mIoU than other methods with limited training data. For example, SQN-CR improves performance by 35.9%, 2%, 6.9%, and 4.6% compared to the fully supervised methods PointNet, KPConv, RandLA-Net, and the baseline method SQN, respectively. Compared to indoor datasets, the SQN-CR of this invention performs better on outdoor datasets, reaching state-of-the-art (SOTA) performance.

[0157] Table 3 在 SensatUrban 数据集的定量结果 .

[0158]

[0159] "*" indicates the result of the method trained using the official code in this invention.

[0160] To further evaluate the effectiveness of each key component in the model, including the novel trilinear interpolation strategy, contrast regularization, and multi-scale classification, several experiments were conducted on S3DIS Area-5, trained using only 0.1% of the labels. The best results were obtained by combining three experiments. It should be noted that the values ​​in Experiment I were measured based on the code provided by the SQN benchmark method of this invention, while the values ​​in Experiment VIII are the test results of SQN-CR.

[0161] To assess the effectiveness of the new trilinear interpolation strategy, this invention conducted comparative experiments between the baseline and models using the new trilinear interpolation strategy, as well as SQN-CR and models without the new trilinear interpolation strategy, to test its performance. Based on the obtained mIoU results, Experiment II showed a 0.8% improvement compared to Experiment I, and Experiment VIII showed a 1.5% improvement compared to Experiment V. This indicates that the new trilinear interpolation strategy outperforms the original strategy and significantly improves the final segmentation accuracy.

[0162] Regarding the effectiveness of contrast regularization, it includes point-level contrast regularization and local contrast regularization. After adding this component, the mIoU in Experiment III improved by 1% compared to Experiment I, and in Experiment VIII by 1.7% compared to Experiment VII. The results show that feature learning based on point-level contrast regularization enhances the feature learning capability, while local contrast regularization further improves the model's performance by utilizing nearest neighbor information.

[0163] To investigate the effectiveness of multi-scale classification, which is commonly used to improve segmentation accuracy, this invention designed comparative experiments to explore its effectiveness in SQN-CR. The results show that Experiment IV improved mIoU by 1.2% and OA by 0.7% compared to Experiment I. Experiment VIII improved mIoU by 3% and OA by 1.1% compared to Experiment VI. This verifies the improvement in segmentation accuracy achieved by introducing multi-scale classification, and the presence of the other two key components further stimulates its optimization potential, demonstrating the effectiveness and importance of multi-scale classification.

[0164] Furthermore, this invention reveals that adding a single component or using a combination of two components in experiments produces fluctuating results; however, using all three components simultaneously avoids this problem, resulting in more stable segmentation performance. Whether considering comparative results from ablation experiments or for enhanced robustness, the SQN-CR model proposed in this invention, which combines three components, significantly outperforms SQN and achieves state-of-the-art (SOTA) performance.

[0165] It should be noted that the experimental cases given in this specification are only illustrative and are not the only specific implementation cases of this invention. Those skilled in the art can use the SQN-CR three-dimensional point cloud weakly supervised semantic segmentation method provided above to achieve point cloud semantic segmentation for different application scenarios, based on the illustrative implementation cases provided by this invention.

[0166] On the one hand, embodiments of the present invention also provide a weakly supervised semantic segmentation system for three-dimensional point clouds based on SQN-CR, characterized in that it includes a point cloud input enhancement component, a local feature extraction component, a random point coordinate retrieval component, a trilinear interpolation component, a contrast regularization prediction component, a normalization layer prediction component, a multi-scale classification component, a random point final prediction component, and a full point cloud segmentation output component.

[0167] The point cloud input enhancement module is used to input a 3D point cloud, mark some points in the 3D point cloud as the original point cloud, and generate an enhanced point cloud from the original point cloud.

[0168] The local feature extraction component is used to perform local feature aggregation and random downsampling on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer;

[0169] The random point coordinate retrieval component is used to retrieve the random point coordinates of the original point cloud and the enhanced point cloud at each layer based on the W layers of feature data, and obtain W local feature information of the random point; the W local feature information of the random point includes the association information between the random point and its neighboring points;

[0170] The trilinear interpolation component is used to perform trilinear interpolation on W local feature information of random points to obtain W interpolated feature vectors;

[0171] The contrast regularization prediction component is used to concatenate W interpolated feature vectors into a single vector, which is then input into the first series of multilayer perceptrons for prediction to obtain the first prediction result. Point-level contrast regularization and local contrast regularization are used to optimize the first prediction result to obtain the optimized first prediction result.

[0172] The normalization layer prediction component is used to input the optimized first prediction result into the batch normalization layer for re-prediction to obtain the second prediction result;

[0173] The multi-scale classification component is used to input W interpolated feature vectors into the second series of multilayer perceptrons for multi-scale classification and obtain classification prediction results.

[0174] The random point final prediction component is used to optimize the second prediction result based on the second prediction result and the classification prediction result to obtain the final prediction result of the random point;

[0175] The full point cloud segmentation output component takes the coordinates of unmarked points in the 3D point cloud as input and outputs the prediction results of all points in the 3D point cloud, based on the calculation process of the final prediction results of random points.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR, characterized in that, Includes the following steps: Input a 3D point cloud, mark some points in the 3D point cloud as the original point cloud, and generate an enhanced point cloud from the original point cloud; Data augmentation is performed on the original point cloud data, wherein the data augmentation is one or a combination of adding random noise, performing random rotation, scaling, and jittering; The original point cloud and the enhanced point cloud are subjected to local feature aggregation and random downsampling to obtain their respective W-layer feature data; the RandLA-Net neural network is used as a feature extractor to perform local feature aggregation and random downsampling on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer; Based on the W layers of feature data, the coordinates of random points in the original point cloud and the enhanced point cloud are retrieved in each layer to obtain W local feature information of the random points; the W local feature information of the random points includes the association information between the random points and their neighboring points; Trilinear interpolation is performed on the W local feature information of the random point to obtain W interpolated feature vectors; The W interpolated feature vectors are concatenated into a single vector, which is then input into a first series of multilayer perceptrons for prediction to obtain a first prediction result. Point-level contrast regularization and local contrast regularization are used to optimize the first prediction result to obtain an optimized first prediction result. The optimized first prediction result is input into the batch normalization layer for re-prediction to obtain the second prediction result; the batch normalization layer is a hypersphere, which is a sphere in high-dimensional space; The W interpolated feature vectors are input into a second series of multilayer perceptrons for multi-scale classification to obtain classification prediction results. The second prediction result is optimized based on the second prediction result and the classification prediction result to obtain the final prediction result of the random point; Based on the calculation process of the final prediction result of the random points, input the coordinates of the unmarked points in the 3D point cloud, and output the prediction result of all points in the 3D point cloud.

2. The weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR according to claim 1, characterized in that, The optimization of the first prediction result using point-level contrast regularization and local contrast regularization to obtain the optimized first prediction result includes: Based on the first prediction results obtained from the original point cloud and the enhanced point cloud, the point-level contrast regularization loss is calculated. ; Calculate the local contrast regularization loss based on the correlation information between the random point and its neighboring points. ; Based on the point-level comparison regularization loss And the local comparison regularization loss The first prediction result is optimized to obtain the optimized first prediction result.

3. The weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR according to claim 2, characterized in that, Calculate the point-level contrast regularization loss based on the first prediction results of the original point cloud and the enhanced point cloud. ,include: Obtain the first prediction result of the original points in the original point cloud; Obtain the first prediction result of the enhanced points in the enhanced point cloud, wherein the enhanced points are generated from the original points; By merging the first prediction results of the original point and its generated augmented points, separating the original point from the first prediction results of other augmented points, the point-level contrast regularization loss is calculated. .

4. The weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR according to claim 3, characterized in that, By merging the first prediction results of the original point and its generated augmented points, separating the original point from the first prediction results of other augmented points, the point-level contrast regularization loss is calculated. ,include: Where N is the total number of points in the input point cloud. and These are the prediction results for the i-th and j-th points of the original point cloud and the enhanced point cloud, respectively. and τ is the indicator function; τ is the temperature hyperparameter.

5. The weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR according to claim 2, characterized in that, Calculate the local contrast regularization loss based on the correlation information between the random point and its neighboring points. ,include: Obtain the coordinates of the random point; Search for its K nearest enhanced neighbors based on Euclidean distance; K is a positive integer greater than 1. Obtain the average of the first prediction results of the K enhanced neighbors; The first prediction result of the random point is brought closer to the average of the first prediction results of its K enhanced neighbors. The average of the first prediction results of the random point and its K other enhanced neighbors is separated, and the local contrast regularization loss is calculated. .

6. The weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR according to claim 2, characterized in that, The first prediction result of the random point is brought closer to the average of the first prediction results of its K enhanced neighbors. The average of the first prediction results of the random point and its K other enhanced neighbors is separated, and the local contrast regularization loss is calculated. ,include: Where N is the total number of random points, and These are the first prediction result of the i-th random point and the average of the first prediction results of the K enhanced neighbors of the j-th random point, respectively. and τ is the indicator function; τ is the temperature hyperparameter.

7. The weakly supervised semantic segmentation method for 3D point clouds based on SQN-CR according to claim 2, characterized in that, The second prediction result is optimized based on the second prediction result and the classification prediction result to obtain the final prediction result for the random point; the degree of optimization is evaluated by a loss function, which is: in For cross-entropy loss, To compare regularization losses, including point-level comparative regularization losses. Local contrast regularization loss .

8. The SQN-CR-based weakly supervised semantic segmentation method for 3D point clouds according to claim 7, wherein the contrastive regularization loss... Represented as: 。 9. A computer-readable storage medium having code stored thereon, characterized in that, When the code program is executed by the processor, it implements the steps of the weakly supervised semantic segmentation method for three-dimensional point clouds based on SQN-CR as described in any one of claims 1 to 8.

10. A weakly supervised semantic segmentation system for 3D point clouds based on SQN-CR, characterized in that, It includes a point cloud input enhancement component, a local feature extraction component, a random point coordinate retrieval component, a trilinear interpolation component, a contrast regularization prediction component, a normalization layer prediction component, a multi-scale classification component, a random point final prediction component, and a full point cloud segmentation output component. The point cloud input enhancement module is used to input a three-dimensional point cloud, mark some points in the three-dimensional point cloud as the original point cloud, generate an enhanced point cloud of the original point cloud, and perform data enhancement on the original point cloud data. The data enhancement is one or a combination of adding random noise, performing random rotation, scaling, and jittering. The local feature extraction component is used to perform local feature aggregation and random downsampling on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; a RandLA-Net neural network is used as a feature extractor to perform local feature aggregation and random downsampling on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer; The random point coordinate retrieval component is used to retrieve the random point coordinates of the original point cloud and the enhanced point cloud at each layer based on the W layers of feature data, and obtain W local feature information of the random point; the W local feature information of the random point includes the association information between the random point and its neighboring points; The trilinear interpolation component is used to perform trilinear interpolation on the W local feature information of the random point to obtain W interpolated feature vectors; The contrast regularization prediction component is used to concatenate the W interpolated feature vectors into a single vector, input it into a first series of multilayer perceptrons for prediction, obtain a first prediction result, and use point-level contrast regularization and local contrast regularization to optimize the first prediction result to obtain an optimized first prediction result. The normalization layer prediction component is used to input the optimized first prediction result into the batch normalization layer for re-prediction to obtain the second prediction result; the batch normalization layer is a hypersphere, and the hypersphere is a sphere in high-dimensional space; The multi-scale classification component is used to input the W interpolated feature vectors into the second series of multilayer perceptrons for multi-scale classification to obtain classification prediction results. The random point final prediction component is used to optimize the second prediction result based on the second prediction result and the classification prediction result to obtain the final prediction result of the random point; The full point cloud segmentation output component, based on the calculation process of the final prediction result of the random points, inputs the coordinates of unmarked points in the 3D point cloud and outputs the prediction result of all points in the 3D point cloud.

Images