SQN-CR-based three-dimensional point cloud weak supervision semantic segmentation method, medium and system

Abstract

The invention provides an SQN-CR-based three-dimensional point cloud weak supervision semantic segmentation method, medium and system, which are used for a large-scale scene. According to the method, the ideas of contrast regularization and multi-scale classification are integrated, the contradiction between the annotation cost and the segmentation accuracy is balanced, the contrast regularization method adopted in the model comprises point-level contrast regularization and local contrast regularization, the point-level contrast regularization carries out training constraint by utilizing feature information of an enhanced point and an original point, and the local contrast regularization carries out local contrast regularization by utilizing feature information of the enhanced point and the original point. The local contrast regularization makes full use of feature information implied in a weak supervision label, and in addition, multi-scale classification is adopted to extract multi-layer features and classify the multi-layer features to optimize the model, so that a better segmentation result is obtained by minimizing the labeling workload, and the semantic segmentation efficiency and effect of the large-scale three-dimensional point cloud are further improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional point cloud semantic segmentation, and in particular to a three-dimensional point cloud weakly supervised semantic segmentation method, medium and system based on SQN-CR. Background Art

[0002] Large-scale 3D point cloud semantic segmentation has been a hot topic in the field of computer vision and 3D data processing in recent years. It involves efficient and accurate semantic analysis and understanding of large and complex point cloud data. The goal of large-scale 3D point cloud semantic segmentation is to divide massive point cloud data into regions with different semantic meanings, such as buildings, roads, vehicles, pedestrians, etc., so as to provide key information for various advanced applications. For example, in urban planning, by semantically segmenting large-scale point cloud data, various structures and facilities in the city can be automatically identified, providing data support for urban planning and management. In the field of autonomous driving, vehicles need to process and analyze large-scale point cloud data in real time to accurately perceive the surrounding environment and make safe driving decisions. With the rapid development of 3D scanning and sensor technology, the ability to obtain large-scale point cloud data has been continuously improved. These data usually contain millions or even billions of points, each with 3D coordinates and possible additional information such as color and intensity.

[0003] At present, the methods for semantic segmentation of large-scale 3D point clouds are mainly divided into fully supervised methods and weakly supervised methods. The fully supervised point cloud semantic segmentation method relies on fully labeled training data and requires a large amount of labeled data, which increases the cost of data collection and processing. The weakly supervised point cloud semantic segmentation method is a method that uses simpler and rougher annotation information to train the model, which can effectively reduce the annotation cost. The existing weakly supervised point cloud semantic segmentation methods mainly include methods based on 2D labels, methods based on pseudo 3D labels, and methods based on limited 3D labels.

[0004] However, existing weakly supervised semantic segmentation methods still have major limitations. First, local semantic information is still not sufficiently considered in large scenes, resulting in the information contained in weak annotations not being fully utilized; second, in order to obtain better segmentation accuracy, existing technologies often integrate multiple stages, which is more complex and less robust than standard end-to-end training; third, existing methods do not perform well in balancing segmentation accuracy and the amount of annotated data, that is, existing weakly supervised semantic segmentation methods cannot reduce the number of annotations while maintaining a non-significant decrease in segmentation accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide a three-dimensional point cloud weakly supervised semantic segmentation method, medium and system based on SQN-CR (Semantic Query Network-Contrastive Regularization) to address the above technical problems, so as to improve the efficiency and effect of large-scale three-dimensional point cloud semantic segmentation.

[0006] In order to achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] On the one hand, an embodiment of the present invention provides a three-dimensional point cloud weakly supervised semantic segmentation method based on SQN-CR, comprising the following steps:

[0008] Input a 3D point cloud, mark some points in the 3D point cloud as an original point cloud, and generate an enhanced point cloud of the original point cloud;

[0009] Perform local feature aggregation and random downsampling on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer;

[0010] Based on W layers of feature data, the coordinates of random points of the original point cloud and the enhanced point cloud are retrieved in each layer to obtain W local feature information of the random points; the W local feature information of the random points includes the association information between the random points and their neighboring points;

[0011] Perform trilinear interpolation on the W local feature information of random points to obtain W interpolation feature vectors;

[0012] splicing W interpolated feature vectors into a single vector, inputting the vector into a first series of multi-layer perceptrons for prediction, obtaining a first prediction result, optimizing the first prediction result using point-level contrast regularization and local contrast regularization, and obtaining an optimized first prediction result;

[0013] The optimized first prediction result is input into the batch normalization layer for re-prediction to obtain a second prediction result;

[0014] Input the W interpolated feature vectors into the second series of multi-layer perceptrons for multi-scale classification to obtain the classification prediction results;

[0015] The second prediction result is optimized according to the second prediction result and the classification prediction result to obtain a final prediction result of the random point;

[0016] According to the calculation process of the final prediction results of random points, the coordinates of unlabeled points in the three-dimensional point cloud are input, and the prediction results of all points in the three-dimensional point cloud are output.

[0017] In one embodiment, generating an enhanced point cloud of an original point cloud includes:

[0018] The original point cloud data is enhanced by adding random noise, performing random rotation, scaling, and jittering, or a combination thereof.

[0019] In one embodiment, local feature aggregation and random downsampling are performed on the original point cloud and the enhanced point cloud to obtain W layers of feature data of each, including:

[0020] Using the RandLA-Net neural network as a feature extractor, local feature aggregation and random downsampling are performed on the original point cloud and enhanced point cloud to obtain their respective W-layer feature data.

[0021] In one embodiment, the first prediction result is optimized using point-level contrast regularization and local contrast regularization to obtain the optimized first prediction result, including:

[0022] Based on the first prediction results obtained from the original point cloud and the enhanced point cloud, the point-level contrast regularization loss is calculated

[0023] Based on the association information between a random point and its neighboring points, the local contrast regularization loss is calculated

[0024] Based on point-level contrast regularization loss and local contrast regularization loss The first prediction result is optimized to obtain an optimized first prediction result.

[0025] In one embodiment, a point-level contrast regularization loss is calculated based on the first prediction results of the original point cloud and the enhanced point cloud. include:

[0026] Obtain a first prediction result of the original point in the original point cloud;

[0027] Obtaining a first prediction result of an enhanced point in the enhanced point cloud, where the enhanced point is generated from the original point;

[0028] Pull the first prediction result of the original point and the enhanced point it generates closer, separate the first prediction result of the original point and other enhanced points, and calculate the point-level contrast regularization loss

[0029] In one embodiment, the first prediction result of the original point and the enhanced point generated by it is brought closer, the first prediction result of the original point and other enhanced points is separated, and the point-level contrast regularization loss is calculated. include:

[0030]

[0031] Where N is the total number of points in the input point cloud, and are the prediction results of the i-th and j-th points of the original point cloud and enhanced point cloud respectively; 1 [j＝i] and 1 [j≠i] is the indicator function; τ is the temperature hyperparameter.

[0032] In one embodiment, the local contrast regularization loss is calculated based on the correlation information between a random point and its neighboring points. include:

[0033] Get the coordinates of a random point;

[0034] Search for the K nearest enhanced neighbors according to the Euclidean distance; K is a positive integer greater than 1;

[0035] Get the average of the first prediction results of K enhanced neighbors;

[0036] The local contrast regularization loss is calculated by bringing the first prediction result of the random point closer to the average of the first prediction results of its K enhanced neighbors, separating the random point from the average of the first prediction results of the other K enhanced neighbors.

[0037] In one embodiment, the first prediction result of the random point is brought close to the average of the first prediction results of its K enhanced neighbors, the average of the first prediction results of the random point and the other K enhanced neighbors is separated, and the local contrast regularization loss is calculated. include:

[0038]

[0039] Where N is the total number of points in the input point cloud, and are the first prediction result of the i-th random point and the average of the first prediction results of the K enhanced neighbors of the j-th random point, 1 [j＝i] and 1 [j≠i] is the indicator function; τ is the temperature hyperparameter.

[0040] In one embodiment, the batch normalization layer is a hypersphere, which is a spherical surface in a high-dimensional space.

[0041] In one embodiment, the second prediction result is optimized according to the second prediction result and the classification prediction result to obtain a final prediction result of the random point; the degree of optimization is evaluated by a loss function, and the loss function is:

[0042]

[0043] in is the cross entropy loss, To compare the regularization loss, including point-level comparison regularization loss point-level comparison regularization loss and local contrast regularization loss

[0044] In one embodiment, the contrast regularization loss It can be expressed as:

[0045]

[0046] On the one hand, an embodiment of the present invention further provides a computer-readable storage medium having a code program stored thereon, characterized in that when the code program is executed by a processor, the steps of any of the above-mentioned three-dimensional point cloud weakly supervised semantic segmentation methods based on SQN-CR are implemented.

[0047] On the one hand, an embodiment of the present invention further provides a three-dimensional point cloud weakly supervised semantic segmentation system based on SQN-CR, characterized in that it includes a point cloud input enhancement component, a local feature extraction component, a random point coordinate retrieval component, a trilinear interpolation component, a contrast regularization prediction component, a normalization layer prediction component, a multi-scale classification component, a random point final prediction component and a full point cloud segmentation output component;

[0048] The point cloud input enhancement module is used to input a three-dimensional point cloud, mark some points in the three-dimensional point cloud as an original point cloud, and generate an enhanced point cloud of the original point cloud;

[0049] The local feature extraction component is used to perform local feature aggregation and random downsampling processing on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer;

[0050] A random point coordinate retrieval component is used to retrieve the coordinates of random points of the original point cloud and the enhanced point cloud in each layer based on W layers of feature data to obtain W local feature information of the random points; the W local feature information of the random points includes the association information between the random points and their neighboring points;

[0051] A trilinear interpolation component is used to perform trilinear interpolation on W local feature information of random points to obtain W interpolation feature vectors;

[0052] A contrast regularization prediction component is used to concatenate W interpolated feature vectors into a single vector, input the vector into a first series of multi-layer perceptrons for prediction, obtain a first prediction result, and optimize the first prediction result using point-level contrast regularization and local contrast regularization to obtain an optimized first prediction result;

[0053] A normalization layer prediction component, used for inputting the optimized first prediction result into a batch normalization layer for re-prediction to obtain a second prediction result;

[0054] A multi-scale classification component is used to input the W interpolated feature vectors into the second series of multi-layer perceptrons for multi-scale classification to obtain classification prediction results;

[0055] A random point final prediction component, used to optimize the second prediction result according to the second prediction result and the classification prediction result to obtain a final prediction result of the random point;

[0056] The full point cloud segmentation output component inputs the coordinates of unlabeled points in the 3D point cloud and outputs the prediction results of all points in the 3D point cloud according to the calculation process of the final prediction results of random points.

[0057] One of the above technical solutions has the following advantages and beneficial effects:

[0058] The above-mentioned weakly supervised semantic segmentation method of 3D point cloud based on SQN-CR first encodes the original point cloud and enhanced point cloud in a hierarchical manner, then uses the coordinates of any point as input to retrieve its local feature information, and obtains the feature vector through trilinear interpolation. These vectors are used to calculate the multi-scale classification loss, and then spliced ​​into a single vector and input into a series of multi-layer perceptrons for semantic prediction. The prediction results of the enhanced and original data are compared, and finally a final batch normalization (BN) layer is designed to constrain the local contrast regularization loss and segmentation loss in a more adaptable space. This method improves the accuracy of segmentation by enhancing the consideration of local information and adopting a multi-scale classification strategy, while maintaining a simple structure in an end-to-end manner, thereby achieving better segmentation results by minimizing the annotation workload, and further improving the efficiency and effect of semantic segmentation of large-scale 3D point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart of a method for weakly supervised semantic segmentation of a three-dimensional point cloud based on SQN-CR when W=4 in one embodiment;

[0060] Figure 2 A framework diagram of a model training of a 3D point cloud weakly supervised semantic segmentation method based on SQN-CR when W=4 in one embodiment;

[0061] Figure 3 A visualization comparison diagram of a three-dimensional point cloud weakly supervised semantic segmentation method based on SQN-CR in an embodiment applied to a large-scale indoor data set of S3DIS;

[0062] Figure 4A visualization comparison diagram of a 3D point cloud weakly supervised semantic segmentation method based on SQN-CR in an embodiment applied to SemanticKITTI;

[0063] Figure 5 A visualization comparison diagram of a 3D point cloud weakly supervised semantic segmentation method based on SQN-CR in one embodiment applied to Sensat Urban;

[0064] Figure 6 The following is an architecture diagram of a 3D point cloud weakly supervised semantic segmentation system based on SQN-CR in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0067] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0068] The present application provides a 3D point cloud weakly supervised semantic segmentation method based on SQN-CR (Semantic Query Network-Contrastive Regularization). When W is equal to 4, the steps are as follows: Figure 1 As shown, the following steps are included:

[0069] 101: input a three-dimensional point cloud, mark some points in the three-dimensional point cloud as an original point cloud, and generate an enhanced point cloud of the original point cloud;

[0070] It can be understood that semantic segmentation refers to the division of each point element in the point cloud data into different categories. In this case, semantic segmentation is performed on the three-dimensional point cloud data in the image. Through the classification process, each point in the point cloud can be assigned to a corresponding category, such as buildings, ground, trees, etc.

[0071] It can be understood that in this 3D weakly supervised model, given an input point cloud P containing N points, the number of labeled points M is determined by the weak supervision ratio, for example, in the 0.1% setting, M is equal to 0.1% × N, and all labeled points are randomly selected.

[0072] It can be understood that in this example, the original point cloud and the enhanced point cloud are encoded in a hierarchical manner, and the information of the point cloud is represented and processed according to a certain hierarchical structure using a layered encoding method, with the aim of improving the model's understanding and modeling capabilities of different hierarchical structures in the point cloud.

[0073] 102: Perform local feature aggregation and random downsampling processing on the original point cloud and the enhanced point cloud to obtain four layers of feature data of each, wherein the four layers of feature data include local feature information of all points in the original point cloud and the enhanced point cloud;

[0074] It can be understood that local feature aggregation is to select a central point in the original point cloud and the enhanced point cloud, and then aggregate the feature information in the local neighborhood around the central point, so that the local features of each point in the point cloud can be extracted, so that the model can better understand the local structure of the point cloud. The aggregated features may include the coordinates, normals, colors, etc. of the points; random downsampling is to reduce the density of the point cloud by randomly selecting a subset in the point cloud. By retaining a random subset of the point cloud, the overall features of the point cloud can be recognized while reducing the computational burden.

[0075] 103: Based on the 4-layer feature data, the coordinates of the random points of the original point cloud and the enhanced point cloud are retrieved in each layer to obtain 4 local feature information of the random points; the 4 local feature information of the random points includes the association information between the random points and their neighboring points;

[0076] It can be understood that the random point is the point to be classified or segmented in the entire collected three-dimensional point cloud. In this example, the coordinates of the random point can be used to search in each of the four layers to obtain the local feature information corresponding to the random point in each layer.

[0077] 104: Perform trilinear interpolation on the four local feature information of the random points to obtain four interpolation feature vectors;

[0078] It can be understood that trilinear interpolation interpolates data in three-dimensional space to obtain continuity estimates at non-grid point locations; the four interpolated feature vectors can be of different dimensions. When extracting point features, in order to restore the true trilinear interpolation process, this embodiment chooses to set the weight to the inverse of the distance.

[0079] 105: splicing the four interpolated feature vectors into a single vector, inputting the vector into the first series of multi-layer perceptrons for prediction, obtaining a first prediction result, optimizing the first prediction result by using point-level contrast regularization and local contrast regularization, and obtaining an optimized first prediction result;

[0080] It can be understood that the four interpolated feature vectors are concatenated and input to the first series of multi-layer perceptrons (MLPs). In this example, the first series of multi-layer perceptrons has two hidden layers.

[0081] It can be understood that contrast regularization is used to improve model accuracy by pulling points towards positive anchor points with similarity and away from negative anchor points. This method chooses to use this strategy in the prediction space and uses the prediction results of each point as positive anchor points and negative anchor points to achieve better constraints.

[0082] 106: Input the optimized first prediction result into a batch normalization layer for re-prediction to obtain a second prediction result;

[0083] It can be understood that adding point-level contrast regularization and local contrast regularization on the basis of the initial segmentation loss has different optimization objectives. The segmentation loss and local contrast regularization loss are calculated independently, and they do not always converge at the same time. Therefore, in this example, the optimization is completed through two alternating spaces to obtain better segmentation results. Specifically, a batch normalization layer (BN layer) is added after the full connection to construct a hypersphere to optimize the segmentation loss, and the classification before the BN layer is used as a prediction to complete the optimization of the contrast regularization loss.

[0084] 107: Input the four interpolated feature vectors into the second series of multi-layer perceptrons for multi-scale classification to obtain classification prediction results;

[0085] It can be understood that the four interpolated feature vectors are classified at multiple scales to obtain the local features of the query points at each scale, and these features are classified. Specifically, the extracted features of each scale are input into a multi-layer fully connected network for classification, and the output prediction results are fed into the segmentation loss for optimization. This example implements multi-scale classification based on multi-scale feature fusion, thereby maximizing the acquisition of more comprehensive overall information and detailed local information.

[0086] 108: Optimizing the second prediction result according to the second prediction result and the classification prediction result to obtain a final prediction result of the random point;

[0087] It can be understood that the optimization process of contrast regularization and multi-scale classification are comprehensively considered to finally optimize the prediction results to achieve the final prediction goal and output the final prediction results.

[0088] 109: According to the calculation process of the final prediction result of the random point, the coordinates of the unlabeled points in the three-dimensional point cloud are input, and the prediction results of all the points in the three-dimensional point cloud are output.

[0089] It can be understood that in this example, under the 0.1% setting, the number of marked points M is equal to 0.1%×N. This example trains the segmentation model according to the information of the M marked points according to steps 101-108 until it is optimized to meet the preset segmentation criteria, and then inputs the coordinates of the remaining 99.9% unmarked points in the three-dimensional point cloud into the model to output the final prediction result.

[0090] It can be understood that if it is necessary to use this model to segment other point clouds, if they are the same type of point clouds, such as point clouds containing the same classification, you can directly enter step 109 for semantic segmentation of the entire point cloud without training. If the classification is different or the difference is too large, you need to re-train the model for the new point cloud according to steps 101-108, and then enter the final full point cloud segmentation step.

[0091] The above-mentioned three-dimensional point cloud weakly supervised semantic segmentation method based on SQN-CR, when W=4, the overall architecture of its training model process (steps 102-108) is as follows Figure 2 As shown in the figure, the original point cloud and the enhanced point cloud are first encoded in a hierarchical manner, and then the coordinates of any point are used as input to retrieve its local feature information, and the feature vector is obtained by trilinear interpolation. These vectors are used to calculate the multi-scale classification loss, and then spliced ​​into a single vector and input into a series of multi-layer perceptrons for semantic prediction. The prediction results of the enhanced and original data are compared, and finally a final batch normalization (BN) layer is designed to constrain the local contrast regularization loss and segmentation loss in a more adaptable space. This method improves the accuracy of segmentation by enhancing the consideration of local information and adopting a multi-scale classification strategy, while maintaining a simple structure in an end-to-end manner, thereby achieving better segmentation results by minimizing the annotation workload, and further improving the efficiency and effect of semantic segmentation of large-scale 3D point clouds.

[0092] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequentially, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0093] In one embodiment, generating an enhanced point cloud of an original point cloud includes:

[0094] The original point cloud data is enhanced by adding random noise, performing random rotation, scaling, and jittering, or a combination thereof.

[0095] It can be understood that the enhanced point cloud can be generated from the original point cloud by random sampling, rotation and translation, scaling, noise injection, jittering, and data clipping. In this embodiment, the method of adding random noise, random rotation, scaling and jittering can be selected.

[0096] In one embodiment, local feature aggregation and random downsampling are performed on the original point cloud and the enhanced point cloud to obtain W layers of feature data of each point cloud, including:

[0097] Using the RandLA-Net neural network as a feature extractor, local feature aggregation and random downsampling are performed on the original point cloud and enhanced point cloud to obtain their respective W-layer feature data.

[0098] It can be understood that this embodiment selects the existing fully supervised method RandLA-Net as a feature extractor to obtain better feature embedding. RandLA-Net is a neural network that can effectively perform large-scale semantic segmentation on point clouds, mainly relying on the local feature aggregation (LFA) module and the random sampling (RS) method. Therefore, the LFA module and the RS method are directly applied to the four-layer encoder to achieve feature aggregation for each point.

[0099] In one embodiment, the first prediction result is optimized using point-level contrast regularization and local contrast regularization to obtain an optimized first prediction result, including:

[0100] Based on the first prediction results obtained from the original point cloud and the enhanced point cloud, the point-level contrast regularization loss is calculated

[0101] Based on the association information between a random point and its neighboring points, the local contrast regularization loss is calculated

[0102] Based on point-level contrast regularization loss and local contrast regularization loss The first prediction result is optimized to obtain an optimized first prediction result.

[0103] It is understandable that the point-level contrast regularization loss and local contrast regularization loss The calculation is performed synchronously, and the first prediction result is optimized at the same time.

[0104] In one embodiment, the point-level contrast regularization loss is calculated based on the first prediction results of the original point cloud and the enhanced point cloud. include:

[0105] Obtain a first prediction result of the original point in the original point cloud;

[0106] Obtaining a first prediction result of an enhanced point in the enhanced point cloud, where the enhanced point is generated from the original point;

[0107] Pull the first prediction result of the original point and the enhanced point it generates closer, separate the first prediction result of the original point and other enhanced points, and calculate the point-level contrast regularization loss

[0108] It can be understood that for the original and enhanced data, point-level contrast regularization is performed to bring the prediction results of the original points and enhanced points closer and separate them from other points, and by introducing the loss function To describe and control this "distance".

[0109] In one embodiment, the first prediction result of the original point and the enhanced point generated by it is brought closer, the first prediction result of the original point and other enhanced points is separated, and the point-level contrast regularization loss is calculated. include:

[0110]

[0111] Where N is the total number of points in the input point cloud, and are the prediction results of the i-th and j-th points of the original point cloud and enhanced point cloud respectively; 1 [j＝i] and 1 [j≠i] is the indicator function; τ is the temperature hyperparameter.

[0112] Understandably, 1 [j＝i] ∈{0, 1} is an indicator function, and when j=i, its value is 1; when j≠i, its value is the same.

[0113] In one embodiment, the local contrast regularization loss is calculated based on the correlation information between a random point and its neighboring points. include:

[0114] Get the coordinates of a random point;

[0115] Search for the K nearest enhanced neighbors according to the Euclidean distance; K is a positive integer greater than 1;

[0116] Get the average of the first prediction results of K enhanced neighbors;

[0117] The local contrast regularization loss is calculated by bringing the first prediction result of the random point closer to the average of the first prediction results of its K enhanced neighbors, separating the random point from the average of the first prediction results of the other K enhanced neighbors.

[0118] It is understandable that since the semantic segmentation model of point clouds mainly predicts the local features of each point, how to better obtain local information becomes the key to improving segmentation accuracy. Considering that the local features of point clouds mainly come from points and their neighborhoods, it is generally believed that the local information of large-scale scene point clouds usually has strong semantic similarity. In order to better utilize this feature, this example proposes to add a local contrast regularization module to optimize the local feature modeling of point clouds, and introduce a loss function To describe and control the distance relationship between a random point and the average value of its surrounding neighboring points.

[0119] In one embodiment, the first prediction result of the random point is brought close to the average of the first prediction results of its K enhanced neighbors, and the average of the first prediction results of the random point and the other K enhanced neighbors is separated to calculate the local contrast regularization loss. include:

[0120]

[0121] Where N is the total number of points in the input point cloud, and are the first prediction result of the i-th random point and the average of the first prediction results of the K enhanced neighbors of the j-th random point, 1 [j＝i] and 1 [j≠i] is the indicator function; τ is the temperature hyperparameter.

[0122] Understandable, 1 [j＝1] ∈{0,1} is the indicator function, when j=i, its value is 1; when j≠i, its value is the same. In the actual process, given a 3D query point x in the prediction space i and its coordinates xyz, the present invention searches for its nearest K enhanced neighbors according to the Euclidean distance and obtains the average value of their predicted probabilities and The calculation formula is to achieve x i The predicted value is closer to and away from the predicted mean of the other nearest neighbors.

[0123] In one embodiment, the batch normalization layer is a hypersphere, which is a spherical surface in a high-dimensional space.

[0124] It can be understood that the contrast regularization loss is added on the basis of the initial segmentation loss, and the goals of the two losses, point-level contrast regularization and local contrast regularization, are different, so the optimization can be completed through two alternating spaces to obtain better segmentation results. This method chooses to add a BN layer after the fully connected layer to construct a hypersphere to optimize the segmentation loss, and the classification before the BN layer is used as a prediction to complete the optimization of the contrast regularization loss.

[0125] In one embodiment, the second prediction result is optimized according to the second prediction result and the classification prediction result to obtain a final prediction result of the random point; the degree of optimization is evaluated by a loss function, and the loss function is:

[0126]

[0127] in is the cross entropy loss, To compare the regularization loss, including point-level comparison regularization loss point-level comparison regularization loss and local contrast regularization loss

[0128] Understandably, It is the cross entropy between logits and labels calculated after softmax classification, and its formula is as follows:

[0129]

[0130] Among them, C is the total number of labeled categories, M is the number of labeled points, H is the number of layers, and y ic Point x i The true value belongs to category c, Represents point x i The predicted value belongs to category c, Indicates the predicted value that point xi belongs to category c at the hth layer.

[0131] In one embodiment, the contrast regularization loss It can be expressed as:

[0132]

[0133] Understandably, and The calculation method of is as described above.

[0134] On the one hand, an embodiment of the present invention further provides a computer-readable storage medium having a code program stored thereon, characterized in that when the code program is executed by a processor, the steps of any of the above-mentioned three-dimensional point cloud weakly supervised semantic segmentation methods based on SQN-CR are implemented.

[0135] It can be understood that the implementation of the 3D point cloud weakly supervised semantic segmentation method based on SQN-CR does not require changes to the hardware equipment, but only requires design and modification at the software level.

[0136] In some embodiments, in order to more intuitively and comprehensively illustrate the above-mentioned 3D point cloud weakly supervised semantic segmentation method based on SQN-CR, the following is an experimental verification example of the 3D point cloud weakly supervised semantic segmentation method based on SQN-CR, including evaluation on the S3DIS large-scale indoor dataset, evaluation on SemanticKITTI, and evaluation on SensatUrban.

[0137] All experiments were conducted using 0.1% random annotation labels to train the model end-to-end on a machine with Intel Core i7-based TM The experiments were conducted on a personal computer with an i9-12900KF CPU and an NVIDIA RTX 3090Ti GPU (24GB GPU memory). The number of input points N varies with different data sets, M = 1% × N, and the present invention sets τ to 1.0. It is worth noting that the SQN method selects a constant weight of 1 / 3 in trilinear interpolation, while this experiment modifies this weight to a variable related to the Euclidean distance between the nearest neighbor points to obtain a more accurate interpolation effect.

[0138] Evaluation Metrics:

[0139] This example evaluates the final performance of the model on all points of the test dataset, and selects mean intersection over union (mIoU) and overall accuracy (OA) as evaluation indicators. Assuming K is the total number of categories, TP is true positive, FP is false positive, FN is false negative, and TN is true negative, the two evaluation indicators can be calculated as follows: In order to truly evaluate the performance of the model of the present invention, this example chooses to test on three commonly used public large-scale datasets. They differ in terms of acquisition sensors, size, and data characteristics.

[0140]

[0141] 1. Evaluation on the S3DIS Large-Scale Indoor Dataset

[0142] S3DIS was created by Stanford University using Matterport from three different buildings and consists of five areas with different characteristics, with a total area of ​​6,020 square meters. S3DIS has more than 215 million points and 13 semantic categories, including ceilings, floors, walls, beams, columns, windows, doors, tables, chairs, sofas, bookcases, whiteboards, and clutter.

[0143] In order to obtain the segmentation effect of the SQN-CR model of the present invention in large-scale indoor scenes, the present invention compares it with the SOTA method on S3DIS Area-5, and the detailed test results are shown in Table 1. It can be seen that the SQN-CR proposed in the present invention obtains the highest mIoU among almost all methods, which is 20.37% higher than the fully supervised method PointNet, 13.47% and 16.97% higher than the 10% and 0.2% weak annotation cases of Xu and Lee, respectively, and 2% higher than the baseline method of the present invention under the same training conditions. For the fully supervised methods KPCconv and RandLA-Net, they are more accurate in segmentation with only 0.1% annotation, and the present invention only differs by 5.63% and 1.53%. At the same time, compared with Zhang et al. and PSD, the difference is 0.33% and 2.03% respectively when the number of annotations increases tenfold, which shows that the SQN-CR proposed in the present invention can be trained on S3DIS Area-5 with only a small amount of annotations to obtain results comparable to those of fully supervised methods.

[0144] Table 1 Quantitative results of different methods on Area-5 of S3DIS.

[0145]

[0146] “*” indicates the results of the method trained with the official code.

[0147] Figure 3 The qualitative results of the segmentation are shown. It can be seen that SQN-CR can obtain results consistent with the ground truth and outperforms SQN, which is shown in the segmentation results of "table", "chair" and "wall". (II) Evaluation on SemanticKITTI

[0148] SemanticKTTI is a large outdoor scene dataset collected by the University of Bonn in Germany using MLS (Mobile Laser Scanning) technology, which is densely annotated. It contains 45.49 million points and 28 semantic categories, and can distinguish between moving and non-moving vehicles and humans. The research team provides online evaluation through a public website.

[0149] The present invention is in Table 2 and Figure 4The performance of SQN-CR and other methods on SemanticKITTI was evaluated in . Obviously, the SQN-CR of the present invention can reach the SOTA level, and the present invention is 38.3%, 32.8%, 0.6% and 1.4% higher than the fully supervised methods PointNet, PointNet++, 1% labeled HybridCR and 0.1% labeled SQN respectively with only 0.1% labeled training, and is only 5.2% and 1% lower than the fully supervised conditions of KPConv and RandLA-Net.

[0150] Table 2 Quantitative results on the SemnaticKITTI dataset.

[0151]

[0152] “*” indicates the results of the method trained with the official code.

[0153] The present invention shows the visualization of qualitative results on the validation set (Sequence 08) of SemanticKITTI, including the original point cloud, semantic labels, the results of the present invention and the baseline results, which are presented from left to right, and the red circles are used to highlight the positions where the present invention surpasses SQN. It can be seen that the SQN-CR of the present invention can better identify vegetation and sidewalks.

[0154] (III) Evaluation on Sensat Urban

[0155] SensatUrban is a point cloud dataset collected in urban areas using drone photogrammetry. It contains nearly 3 billion points covering an area of ​​7.6 square kilometers from three cities in the UK, each with a label of 13 semantic categories. Due to the use of different sensors, the data characteristics of SensatUrban are different from those obtained by LiDAR.

[0156] like Figure 5 As shown in Table 3, for the SensatUrban dataset, the proposed SQN-CR obtains higher mIoU than other methods with limited training data. For example, SQN-CR improves by 35.9%, 2%, 6.9% and 4.6% compared with the fully supervised methods PointNet, KPConv, RandLA-Net and the baseline method SQN, respectively. Compared with the indoor dataset, the proposed SQN-CR performs better on the outdoor dataset and reaches the SOTA level.

[0157] Table 3 在 SensatUrban 数据集的定量结果 .

[0158]

[0159] “*” indicates the results of the method of the present invention trained using the official code.

[0160] To further evaluate the effectiveness of each key component in the model, including the new trilinear interpolation strategy, contrast regularization, and multi-scale classification, several experiments were performed on S3DIS Area-5, using only 0.1% of the labels for training. The best results were obtained by combining three experiments together. It should be noted that the values ​​of Experiment I are measured based on the code provided by the benchmark method SQN of this invention, while the values ​​of Experiment VIII are the test results of SQN-CR.

[0161] For the effectiveness of the new trilinear interpolation strategy, the present invention conducted comparative experiments between the baseline and the model using the new trilinear interpolation strategy, as well as SQN-CR and the model not using the new trilinear interpolation strategy to test its performance. According to the obtained mIoU results, Experiment II improved by 0.8% compared with Experiment I, and Experiment VIII improved by 1.5% compared with Experiment V. This shows that the new trilinear interpolation strategy is better than the original strategy and significantly improves the final segmentation accuracy.

[0162] Regarding the effectiveness of contrast regularization, contrast regularization includes point-level contrast regularization and local contrast regularization. After adding this component, the mIoU of Experiment III is improved by 1% compared with Experiment I, and Experiment VIII is improved by 1.7% compared with Experiment VII. The results show that feature learning based on point-level contrast regularization enhances the ability of feature learning, while local contrast regularization further improves the performance of the model by utilizing the nearest neighbor information.

[0163] As for the effectiveness of multi-scale classification, multi-scale classification is often used to improve segmentation accuracy. In order to explore its effectiveness in SQN-CR, the present invention designed a comparative experiment. The results show that the mIoU of Experiment IV is improved by 1.2% compared with Experiment I, and OA is improved by 0.7% compared with Experiment I. Experiment VIII improves the mIoU of Experiment VI by 3%, and OA improves by 1.1%. This verifies the improvement of segmentation accuracy by introducing multi-scale classification, and the existence of the other two key components further stimulates its optimization potential, proving the effectiveness and importance of multi-scale classification.

[0164] In addition, the present invention finds that adding one component alone or using a combination of two components for experiments will produce fluctuating results, but using three sets of components at the same time can avoid this problem, thereby having a more stable segmentation performance. Whether from the comparison results of the ablation experiment or for the purpose of having a more robust performance, the SQN-CR model proposed by the present invention that combines three components is significantly better than SQN and reaches the SOTA level.

[0165] It should be noted that the experimental cases given in this specification are only for illustration and are not the sole limitation for the specific implementation cases of the present invention. Those skilled in the art can, under the guidance of the implementation cases provided by the present invention, adopt the three-dimensional point cloud weakly supervised semantic segmentation method of SQN-CR provided above to realize point cloud semantic segmentation for different application scenarios.

[0166] On the one hand, an embodiment of the present invention further provides a three-dimensional point cloud weakly supervised semantic segmentation system based on SQN-CR, characterized in that it includes a point cloud input enhancement component, a local feature extraction component, a random point coordinate retrieval component, a trilinear interpolation component, a contrast regularization prediction component, a normalization layer prediction component, a multi-scale classification component, a random point final prediction component and a full point cloud segmentation output component;

[0167] The point cloud input enhancement module is used to input a three-dimensional point cloud, mark some points in the three-dimensional point cloud as an original point cloud, and generate an enhanced point cloud of the original point cloud;

[0168] The local feature extraction component is used to perform local feature aggregation and random downsampling processing on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer;

[0169] A random point coordinate retrieval component is used to retrieve the coordinates of random points of the original point cloud and the enhanced point cloud in each layer based on W layers of feature data to obtain W local feature information of the random points; the W local feature information of the random points includes the association information between the random points and their neighboring points;

[0170] A trilinear interpolation component is used to perform trilinear interpolation on W local feature information of random points to obtain W interpolation feature vectors;

[0171] A contrast regularization prediction component is used to concatenate W interpolated feature vectors into a single vector, input the vector into a first series of multi-layer perceptrons for prediction, obtain a first prediction result, and optimize the first prediction result using point-level contrast regularization and local contrast regularization to obtain an optimized first prediction result;

[0172] A normalization layer prediction component, used for inputting the optimized first prediction result into a batch normalization layer for re-prediction to obtain a second prediction result;

[0173] A multi-scale classification component is used to input the W interpolated feature vectors into the second series of multi-layer perceptrons for multi-scale classification to obtain classification prediction results;

[0174] A random point final prediction component, used to optimize the second prediction result according to the second prediction result and the classification prediction result to obtain a final prediction result of the random point;

[0175] The full point cloud segmentation output component inputs the coordinates of unlabeled points in the 3D point cloud and outputs the prediction results of all points in the 3D point cloud according to the calculation process of the final prediction results of random points.

[0176] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A 3D point cloud weakly supervised semantic segmentation method based on SQN-CR, characterized in that: The following steps are involved: Input a three-dimensional point cloud, mark some points in the three-dimensional point cloud as an original point cloud, and generate an enhanced point cloud of the original point cloud; Performing data enhancement on the original point cloud data, wherein the data enhancement may be one or a combination of adding random noise, performing random rotation, scaling, and jittering; The original point cloud and the enhanced point cloud are subjected to local feature aggregation and random downsampling processing to obtain W-layer feature data of each; the RandLA-Net neural network is used as a feature extractor to perform local feature aggregation and random downsampling processing on the original point cloud and the enhanced point cloud to obtain W-layer feature data of each; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer; Based on the W layers of feature data, the coordinates of the random points of the original point cloud and the enhanced point cloud are retrieved in each layer to obtain W local feature information of the random points; the W local feature information of the random points includes the association information between the random points and their neighboring points; Perform trilinear interpolation on the W local feature information of the random point to obtain W interpolation feature vectors; splicing the W interpolated feature vectors into a single vector, inputting the vector into a first series of multi-layer perceptrons for prediction, obtaining a first prediction result, and optimizing the first prediction result using point-level contrast regularization and local contrast regularization to obtain an optimized first prediction result; Inputting the optimized first prediction result into a batch normalization layer for re-prediction to obtain a second prediction result; the batch normalization layer is a hypersphere, and the hypersphere is a spherical surface in a high-dimensional space; Inputting the W interpolation feature vectors into a second series of multi-layer perceptrons for multi-scale classification to obtain classification prediction results; Optimizing the second prediction result according to the second prediction result and the classification prediction result to obtain a final prediction result of the random point; According to the calculation process of the final prediction result of the random point, the coordinates of the unmarked points in the three-dimensional point cloud are input, and the prediction results of all the points in the three-dimensional point cloud are output.

2. The SQN-CR-based 3D point cloud weakly supervised semantic segmentation method according to claim 1, characterized in that: The step of optimizing the first prediction result by using point-level contrast regularization and local contrast regularization to obtain the optimized first prediction result includes: Calculate the point-level contrast regularization loss based on the first prediction result obtained from the original point cloud and the enhanced point cloud According to the association information between the random point and its neighboring points, the local contrast regularization loss is calculated According to the point-level contrast regularization loss and the local contrast regularization loss The first prediction result is optimized to obtain an optimized first prediction result.

3. The SQN-CR-based 3D point cloud weakly supervised semantic segmentation method according to claim 2, characterized in that: Calculate the point-level contrast regularization loss according to the first prediction results of the original point cloud and the enhanced point cloud include: Obtaining a first prediction result of an original point in the original point cloud; Obtaining a first prediction result of an enhanced point in the enhanced point cloud, wherein the enhanced point is generated from the original point; The first prediction result of the original point and the enhanced point generated by it is brought closer, the first prediction result of the original point and other enhanced points is separated, and the point-level contrast regularization loss is calculated.

4. The SQN-CR-based 3D point cloud weakly supervised semantic segmentation method according to claim 5, characterized in that: The first prediction result of the original point and the enhanced point generated by it is brought closer, the first prediction result of the original point and other enhanced points is separated, and the point-level contrast regularization loss is calculated. include: Where N is the total number of points in the input point cloud, and are the prediction results of the i-th and j-th points of the original point cloud and the enhanced point cloud respectively; 1 [j＝i] and 1 [j≠i] is the indicator function; τ is the temperature hyperparameter.

5. The SQN-CR-based 3D point cloud weakly supervised semantic segmentation method according to claim 2, characterized in that: Calculate the local contrast regularization loss based on the association information between the random point and its neighboring points include: Obtaining the coordinates of the random point; Search for the K nearest enhanced neighbors according to the Euclidean distance; K is a positive integer greater than 1; Obtaining an average value of the first prediction results of the K enhanced neighbors; The first prediction result of the random point is brought closer to the average of the first prediction results of its K enhanced neighbors, and the average of the first prediction results of the random point and other K enhanced neighbors is separated to calculate the local contrast regularization loss.

6. The SQN-CR-based 3D point cloud weakly supervised semantic segmentation method according to claim 2, characterized in that: The first prediction result of the random point is brought closer to the average of the first prediction results of its K enhanced neighbors, and the average of the first prediction results of the random point and other K enhanced neighbors is separated to calculate the local contrast regularization loss. include: Where N is the total number of random points, and are the first prediction result of the i-th random point and the average of the first prediction results of the K enhanced neighbors of the j-th random point, 1 [j＝i] and 1 [j≠i] is the indicator function; τ is the temperature hyperparameter.

7. The SQN-CR-based 3D point cloud weakly supervised semantic segmentation method according to claim 2, characterized in that: The second prediction result is optimized according to the second prediction result and the classification prediction result to obtain the final prediction result of the random point; the degree of optimization is evaluated by a loss function, and the loss function is: in is the cross entropy loss, To compare the regularization loss, including point-level comparison regularization loss point-level comparison regularization loss and local contrast regularization loss 8. According to the SQN-CR-based 3D point cloud weakly supervised semantic segmentation method of claim 7, the contrast regularization loss It can be expressed as: 。 9. A computer-readable storage medium having a code program stored thereon, characterized in that: When the code program is executed by a processor, the steps of the three-dimensional point cloud weakly supervised semantic segmentation method based on SQN-CR described in any one of claims 1 to 11 are implemented.

10. A 3D point cloud weakly supervised semantic segmentation system based on SQN-CR, characterized in that: It includes point cloud input enhancement component, local feature extraction component, random point coordinate retrieval component, trilinear interpolation component, contrast regularization prediction component, normalization layer prediction component, multi-scale classification component, random point final prediction component and full point cloud segmentation output component; The point cloud input enhancement module is used to input a three-dimensional point cloud, mark some points in the three-dimensional point cloud as an original point cloud, and generate an enhanced point cloud of the original point cloud; Performing data enhancement on the original point cloud data, wherein the data enhancement may be one or a combination of adding random noise, performing random rotation, scaling, and jittering; The local feature extraction component is used to perform local feature aggregation and random downsampling processing on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; using the RandLA-Net neural network as a feature extractor, performing local feature aggregation and random downsampling processing on the original point cloud and the enhanced point cloud to obtain their respective W-layer feature data; the W-layer feature data includes the local feature information of all points in the original point cloud and the enhanced point cloud; W is a positive integer; The random point coordinate retrieval component is used to retrieve the random point coordinates of the original point cloud and the enhanced point cloud in each layer based on the W layers of feature data to obtain W local feature information of the random point; the W local feature information of the random point includes the association information between the random point and its neighboring points; The trilinear interpolation component is used to perform trilinear interpolation on the W local feature information of the random point to obtain W interpolation feature vectors; The contrast regularization prediction component is used to splice the W interpolation feature vectors into a single vector, input the vector into a first series of multi-layer perceptrons for prediction, obtain a first prediction result, and optimize the first prediction result using point-level contrast regularization and local contrast regularization to obtain an optimized first prediction result; The normalization layer prediction component is used to input the optimized first prediction result into the batch normalization layer for re-prediction to obtain a second prediction result; the batch normalization layer is a hypersphere, and the hypersphere is a spherical surface in a high-dimensional space; The multi-scale classification component is used to input the W interpolation feature vectors into a second series of multi-layer perceptrons for multi-scale classification to obtain a classification prediction result; The random point final prediction component is used to optimize the second prediction result according to the second prediction result and the classification prediction result to obtain the final prediction result of the random point; The full point cloud segmentation output component inputs the coordinates of unmarked points in the three-dimensional point cloud according to the calculation process of the final prediction results of the random points, and outputs the prediction results of all points in the three-dimensional point cloud.

Images