Three-dimensional point cloud semantic segmentation and measurement reference surface intelligent fitting system and method thereof

Abstract

The invention relates to the technical field of computer vision and three-dimensional measurement, in particular to a three-dimensional point cloud semantic segmentation and measurement reference surface intelligent fitting system and a method thereof, which are mainly applied to scenes needing accurate three-dimensional space understanding, such as topographic survey, mining, urban planning, intelligent driving and industrial detection. The system comprises a data acquisition module, a preprocessing module, a feature extraction module, a reference surface acquisition module and a pixel feature conversion module. Three-dimensional point cloud data are collected through multi-sensor fusion, and the data quality and integrity are improved. A cascade filtering preprocessing process is innovatively designed, and the data quality is remarkably improved. The feature extraction module generates semantic tags, the reference surface acquisition module calculates a probability optimal fitting plane, and the pixel feature conversion module realizes multi-scale feature output. According to the method, the three-dimensional point cloud processing precision and efficiency are effectively improved, and powerful technical support is provided for the field of computer vision and three-dimensional measurement.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and three-dimensional measurement technology, and specifically to a system and method for intelligent fitting of three-dimensional point cloud semantic segmentation and measurement reference planes. The system is mainly used in scenarios requiring accurate three-dimensional spatial understanding, such as topographic surveying, mining, urban planning, intelligent driving, and industrial inspection. Background Art

[0002] With the rapid development of 3D perception technologies such as LiDAR and stereo cameras, 3D point cloud data has become an important information carrier for spatial measurement and scene understanding. 3D point cloud data contains rich geometric and semantic information, enabling precise measurement and environmental perception. However, extracting effective information from this chaotic point cloud data and performing accurate measurement, particularly determining a reliable measurement reference surface, remains a key technical challenge.

[0003] Existing technologies typically approach the tasks of semantic segmentation and ground truth fitting independently. For semantic segmentation, traditional methods rely primarily on hand-crafted feature extractors, such as geometric feature-based region growing and cluster analysis. These methods are sensitive to noise and uneven point cloud distribution, and their robustness is limited. Deep learning-based point cloud segmentation methods, such as PointNet and PointNet++, have made significant progress in recent years, but effectively processing complex point clouds in large-scale outdoor scenes remains a challenge.

[0004] Traditional methods such as the RANSAC (Random Sample Consensus) algorithm are widely used for datum plane fitting. However, these methods often require numerous iterations to find a suitable plane in complex environments and are prone to falling into local optima. Furthermore, existing methods typically separate semantic and geometric information, failing to fully exploit their complementary nature, resulting in insufficient fitting accuracy and reliability.

[0005] In particular, in complex environments like mining and forested areas, point cloud data often contains a large amount of noise and outliers, and the terrain structure is complex and varied, making it difficult for existing technologies to accurately identify and fit a reliable measurement datum. Furthermore, due to the scarcity of annotated 3D point cloud data, achieving high-precision semantic segmentation and datum fitting with limited annotated data is an urgent problem.

[0006] Therefore, there is an urgent need for an intelligent system that can organically integrate semantic segmentation and datum surface fitting, improve the accuracy and reliability of datum surface fitting by assisting geometric processing with semantic information, and at the same time use semi-supervised and weakly supervised learning methods to reduce dependence on labeled data and adapt to complex and changing environments. Summary of the Invention

[0007] The purpose of the present invention is to provide a three-dimensional point cloud semantic segmentation and measurement reference surface intelligent fitting system and method, aiming to solve the problem of insufficient accuracy caused by the separation of semantic segmentation and reference surface fitting in the existing technology, and to achieve high-precision and high-reliability measurement reference surface intelligent fitting through the organic integration of deep learning and geometric processing.

[0008] The present invention proposes a 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system, including:

[0009] Data acquisition module, used to collect three-dimensional point cloud data through multi-sensor fusion;

[0010] A preprocessing module, configured to perform denoising and downsampling processing on the three-dimensional point cloud data;

[0011] A feature extraction module, configured to extract features from the preprocessed point cloud data and generate semantic labels;

[0012] A reference plane acquisition module, configured to calculate a probabilistic optimal fitting plane based on the semantic label;

[0013] The pixel feature conversion module is used to map the point cloud features into the pixel feature space to achieve multi-scale feature output.

[0014] Preferably, the data acquisition module includes a depth camera and a lidar, wherein the depth camera adopts a binocular camera, including a data acquisition unit, an image stitching unit, an image correction unit and a data storage unit, and the depth camera and lidar are installed on the drone for scanning and imaging the target working area.

[0015] Preferably, the preprocessing module includes:

[0016] The geometric denoising submodule is used to remove flying spots generated during data acquisition and uses a straight-through filtering algorithm and a voxel grid filtering algorithm to downsample the data;

[0017] The outlier removal submodule is used to filter outliers outside the point cloud distribution area using a statistical filtering algorithm to reduce the number of subsequent calculations.

[0018] Preferably, the feature extraction module is implemented using the PointNet++ convolutional computing network, including point cloud feature extraction, feature upsampling, feature fusion and semantic label generation processes, in which the local feature enhancement mechanism and context-aware module are used to capture the hierarchical structural information of the point cloud.

[0019] Preferably, the feature extraction module further includes a generative adversarial network for receiving the semantic label vector and generating a lithologic label, wherein the generative adversarial network includes:

[0020] Generator, used to convert semantic label vectors into lithologic distribution features;

[0021] The discriminator is used to distinguish the generated labels from the true labels and continuously optimize the generated features through error feedback.

[0022] Preferably, the reference plane acquisition module is used to:

[0023] Calculate the ratio of the number of point clouds of each semantic category to the total number of point clouds;

[0024] When the ratio is not greater than a preset threshold, feature extraction is performed on each semantic category, including the centroid, normal vector, local surface normal vector, covariance matrix, point pair, principal component and distance from point to fitting surface of the point cloud;

[0025] The hybrid RANSAC algorithm is used to fit semantic categories and features, and multiple groups of different semantic planes and the confidence probability of each semantic plane are obtained;

[0026] A final reference surface is determined based on the confidence probability.

[0027] Preferably, the pixel feature conversion module is implemented using a Unet architecture, wherein the Unet architecture includes two parts, encoding and decoding. The encoding part is composed of convolution, pooling and downsampling, and the decoding part is implemented using transposed convolution. A jump connection is used in the middle of the Unet architecture to fuse the high-level semantics of each scale in the encoding process with the detailed features of the low-level decoding.

[0028] Preferably, the adversarial generative network is learned based on physical constraints, including:

[0029] Construct adversarial learning between semantic models and physical models;

[0030] Design new semantic segmentation model labels based on terrain label information and color map information;

[0031] Optimizing the semantic model through a cross entropy loss function to enable it to learn physical properties;

[0032] A progressive constraint strategy is adopted, focusing on feature learning in the early stage of training and gradually strengthening the physical constraint weights in the later stage.

[0033] Preferably, the formula for calculating the plane confidence of the reference plane acquisition module is:

[0034] Confidence = semantic consistency × geometric fit × physical rationality,

[0035] Among them, semantic consistency represents the degree of consistency of the semantic labels of points in the plane, geometric fit represents the residual error of plane fitting, and physical rationality represents the rationality of evaluating the plane based on physical rules.

[0036] The method based on the three-dimensional point cloud semantic segmentation and measurement reference surface intelligent fitting system includes:

[0037] The step of collecting 3D point cloud data is to use a depth camera and a laser radar to scan and image the target operation area to obtain 3D point cloud data;

[0038] A preprocessing step of denoising and downsampling the three-dimensional point cloud data;

[0039] Feature extraction step: Use the PointNet++ network to extract features from the preprocessed point cloud data and generate semantic label vectors;

[0040] a lithologic label generation step, inputting the semantic label vector into a generative adversarial network to generate a lithologic label;

[0041] a reference plane acquisition step, calculating a probabilistic optimal fitting plane based on the lithologic labels, determining a confidence level of the plane, and determining the plane as a measurement reference plane if the confidence level is higher than a preset threshold;

[0042] In the online reasoning step, the three-dimensional point cloud data is input into the trained model for processing to obtain the measurement reference surface in the target area.

[0043] The beneficial effects of the present invention include: 1) improving data quality and integrity through multi-sensor fusion acquisition; 2) innovatively designing a cascade filtering preprocessing process to significantly improve data quality; 3) organically integrating deep learning and geometric processing to achieve complementary enhancement of semantic and geometric information; 4) introducing a probabilistic optimal fitting plane algorithm and a confidence-based evaluation mechanism to greatly improve the reference plane fitting accuracy; 5) adopting semi-supervised and weakly supervised learning strategies to reduce dependence on labeled data; 6) improving model generalization ability and physical rationality through physical constraint adversarial learning; 7) adapting to a variety of complex environments and having strong environmental adaptability and anti-interference ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the overall architecture diagram of the 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system of the present invention;

[0045] Figure 2 Flowchart of data preprocessing of the system of the present invention;

[0046] Figure 3 This is a diagram of the feature extraction module architecture of the system of the present invention;

[0047] Figure 4 This is a workflow diagram of the reference plane acquisition module of the system of the present invention;

[0048] Figure 5This is the PointNet++ network structure diagram of the present invention;

[0049] Figure 6 This is the Unet architecture diagram of the present invention;

[0050] Figure 7 Flowchart of the hybrid RANSAC algorithm of the present invention;

[0051] Figure 8 is a method flow chart of the system of the present invention;

[0052] Figure 9 This is a diagram showing the effect of the present invention in actual application scenarios. DETAILED DESCRIPTION

[0053] Please refer to the attached Figure 1-9 , the specific implementation of the present invention is further described in detail below with reference to the accompanying drawings.

[0054] like Figure 1 As shown, the three-dimensional point cloud semantic segmentation and measurement reference plane intelligent fitting system provided by the present invention includes: a data acquisition module 10, a preprocessing module 20, a feature extraction module 30, a reference plane acquisition module 40 and a pixel feature conversion module 50.

[0055] The data acquisition module 10 is used to collect three-dimensional point cloud data through multi-sensor fusion. Figure 1 As shown, the data acquisition module 10 includes a depth camera 11 and a laser radar 12. The depth camera 11 is a binocular camera and includes a data acquisition unit 111, an image stitching unit 112, an image correction unit 113, and a data storage unit 114. The depth camera 11 and the laser radar 12 are installed on the drone 13 to scan and image the target operation area.

[0056] Preferably, the depth camera 11 of the present invention utilizes a binocular stereo camera with a resolution of 1920×1080 pixels, a baseline distance of 120 mm, a field of view of 60°×45°, and an operating frequency of 30 Hz. The lidar 12 preferably utilizes a 16- or 32-line lidar with a scanning frequency of 10 Hz, a measurement range of 0.5-100 m, and an accuracy better than ±2 cm. The drone 13 can be a drone or unmanned vehicle, carrying sensors to scan the target area.

[0057] In one embodiment of the present invention, data acquisition module 10 acquires high-quality 3D point cloud data through the following steps: First, depth camera 11 and lidar 12 synchronously acquire data; then, data acquisition unit 111 transmits the acquired raw data to image stitching unit 112 for image stitching; then, image correction unit 113 performs distortion and geometric correction on the stitched image; finally, the corrected data is stored in data storage unit 114. This multi-sensor fusion approach can simultaneously acquire high-resolution texture information and accurate distance measurements, providing high-quality input data for subsequent processing.

[0058] like Figure 2 As shown, the preprocessing module 20 is used to denoise and downsample the 3D point cloud data. The preprocessing module 20 includes a geometric denoising submodule 21 and an outlier removal submodule 22. The geometric denoising submodule 21 removes flying points generated during data acquisition by downsampling the data using a straight-through filtering algorithm and a voxel grid filtering algorithm. The outlier removal submodule 22 uses a statistical filtering algorithm to filter out outliers outside the point cloud distribution area, reducing the number of subsequent calculations.

[0059] In a preferred embodiment of the present invention, the through-filtering algorithm filters out-of-range points by setting the effective range of the point cloud on the three coordinate axes of x, y, and z. For example, in a mining application scenario, the x-axis range can be set to [-100m, 100m], the y-axis range can be set to [-100m, 100m], and the z-axis range can be set to [0m, 50m]. These parameters can be flexibly adjusted according to the specific application scenario. The voxel grid filtering algorithm divides the point cloud space into several voxel grids (such as a cubic grid with a side length of 0.05m), and then replaces all points in each voxel grid with the centroid of all points in the grid, thereby achieving uniform downsampling of the point cloud, which not only retains the geometric characteristics of the point cloud, but also greatly reduces the number of points and improves processing efficiency.

[0060] The statistical filtering algorithm removes outliers based on the statistical properties of the point cloud. The statistical filtering algorithm employed in this invention first calculates the average distance from each point to its k nearest neighbors (preferably k = 50), and then calculates the global mean μ and standard deviation σ of all these average distances. If the average distance of a point exceeds the range μ ± α·σ (preferably α = 2), the point is considered an outlier and is removed. This method can effectively remove outliers caused by sensor noise or environmental interference, improving the accuracy and efficiency of subsequent processing.

[0061] like Figure 3As shown, the feature extraction module 30 is implemented using the PointNet++ convolutional computing network, which includes a point cloud feature extraction unit 31, a feature upsampling unit 32, a feature fusion unit 33 and a semantic label generation unit 34, in which a local feature enhancement mechanism and a context-aware module are used to capture the hierarchical structure information of the point cloud.

[0062] In the present invention, the PointNet++ network processes point cloud data through multi-level sampling and grouping operations. Specifically, the point cloud feature extraction unit 31 first selects a series of center points from the input point cloud using the Farthest Point Sampling (FPS) algorithm, then constructs local regions around each center point (using sphere query or K-nearest neighbor query), and then applies a shared PointNet layer to each local region to extract local features. Through this hierarchical approach, the network can effectively capture local and global geometric features of the point cloud.

[0063] This paper makes several key improvements to the PointNet++ network. First, a multi-scale sampling strategy is introduced, using sphere queries of different radii at each level (e.g., r1 = 0.1m, r2 = 0.2m, r3 = 0.4m) to enhance its adaptability to point clouds of varying densities. Second, a context-aware module is designed to capture the hierarchical structure of point clouds through multi-layer receptive field fusion. These improvements enable the network to better handle complex point cloud data from large-scale outdoor scenes.

[0064] The feature upsampling unit 32 is responsible for upsampling the low-resolution feature map to the original point cloud resolution. This invention employs a feature propagation method based on distance interpolation, ensuring a smooth transition of features through an interpolation strategy where weights are inversely proportional to distance. The feature fusion unit 33 fuses features from different levels through skip connections, preserving high-level semantic information and low-level geometric details. The semantic label generation unit 34 ultimately outputs a semantic label vector for each point through a fully connected layer and a softmax classifier.

[0065] like Figure 6 As shown, the feature extraction module 30 also includes a lithology classification network 35 for receiving semantic label vectors and generating lithology labels. The lithology classification network 35 adopts a multi-layer perceptron (MLP) structure, including three fully connected layers, and outputs the lithology category probability distribution of each point through a softmax classifier.

[0066] like Figure 4As shown, the reference plane acquisition module 40 is used to calculate the probabilistic optimal fitting plane based on the semantic labels. The workflow of the reference plane acquisition module 40 includes: calculating the ratio of the number of point clouds of each semantic category to the total number of point clouds 41; when the ratio is not greater than a preset threshold, performing feature extraction 42 for each semantic category, including the point cloud centroid, normal vector, local surface normal vector, covariance matrix, point pairs, principal components, and the distance from the point to the fitting surface; fitting the semantic categories and features using a hybrid RANSAC algorithm 43, and obtaining multiple sets of different semantic planes and the confidence probability of each semantic plane 44; and determining the final reference plane based on the confidence probability 45.

[0067] In an embodiment of the present invention, the preset threshold is preferably 30%. When the point cloud of a certain semantic category accounts for no more than 30%, comprehensive feature extraction is required to ensure fitting accuracy. When the proportion exceeds 30%, only key features such as normal vectors need to be extracted, which not only ensures fitting accuracy but also improves computational efficiency.

[0068] The calculation formula of point cloud centroid is:

[0069]

[0070] Among them, C represents the centroid of the point cloud, Pi represents the coordinates of the i-th point in the point cloud, and n represents the number of points in the point cloud.

[0071] Normal vector calculation is based on principal component analysis (PCA) of the local area of the point cloud. The specific formula is:

[0072]

[0073] in, Represents the unit normal vector of the point cloud, sgn(x), sgn(y), and sgn(z) represent the sign functions of the point normal in the x, y, and z directions respectively. A unit vector representing a coordinate axis.

[0074] The formula for calculating the covariance matrix is:

[0075]

[0076] Among them, Cov represents the covariance matrix, P i represents the local neighborhood point of the point cloud, and C represents the local neighborhood mean of the point cloud.

[0077] This paper innovatively proposes a hybrid RANSAC algorithm, which uses semantic information to guide geometric processing, improving fitting efficiency and accuracy. The core concept of the hybrid RANSAC algorithm is to adjust the sampling strategy and model evaluation criteria based on the semantic labels of the point cloud. Different sampling weights and inlier thresholds are used for point clouds of different semantic categories, allowing the algorithm to focus more on the distribution of point clouds in key semantic areas. Furthermore, the algorithm simultaneously evaluates multiple candidate planes to avoid falling into local optimal solutions.

[0078] Confidence calculation is a key step in the reference surface acquisition module 40. The present invention proposes a confidence calculation formula that comprehensively considers semantic consistency, geometric fit, and physical rationality:

[0079] Confidence(plane)=(InlierCount / TotalPoints)×(1-ResidualError)×SemanticConsistency,

[0080] InlierCount represents the number of points within the plane, TotalPoints represents the total number of points, ResidualError represents the residual error of the plane fitting, and SemanticConsistency represents the degree of semantic label consistency among the points within the plane. This multi-dimensional evaluation mechanism ensures the high quality and reliability of the reference plane. Specifically, when the calculated confidence level exceeds a preset threshold (preferably 0.85), the plane is determined to be a reliable measurement reference plane; otherwise, the search for a better plane continues.

[0081] like Figure 6 As shown, the pixel feature conversion module 50 is implemented using the Unet architecture and includes an encoding part 51 and a decoding part 52. The encoding part 51 is composed of convolution, pooling, and downsampling, and the decoding part 52 is implemented using transposed convolution. Skip connections 53 are used in the middle of the Unet architecture to fuse the high-level semantics of each scale in the encoding process with the detailed features of the low-level decoding.

[0082] In an embodiment of the present invention, the encoding part 51 of the Unet architecture consists of 5 downsampling blocks, each of which contains two 3×3 convolutional layers (padding 1, stride 1) and a 2×2 max pooling layer (stride 2). The convolutional layers are followed by batch normalization and ReLU activation functions. The number of channels in the encoding part is 64, 128, 256, 512, and 1024, respectively. The decoding part 52 consists of 4 upsampling blocks, each of which contains a 2×2 deconvolutional layer (stride 2) and two 3×3 convolutional layers (padding 1, stride 1). The number of channels in the upsampling blocks is 512, 256, 128, and 64, respectively.

[0083] The skip connection 53 is a key innovation of the Unet architecture. It connects the feature maps of the encoding layer directly to the corresponding layers of the decoding layer. This preserves both high-level semantic information and low-level detail, effectively addressing the information loss problem that may occur during upsampling. In this paper, the skip connection is implemented using concatenation, which concatenates the feature maps of the encoding layer with the feature maps of the decoding layer in the channel dimension.

[0084] The Generative Adversarial Network 35 learns based on physical constraints, including constructing adversarial learning between semantic and physical models; designing new semantic segmentation model labels based on terrain label information and color map information; optimizing the semantic model through the cross-entropy loss function to enable it to learn physical properties; and adopting a progressive constraint strategy, focusing on feature learning in the early stages of training and gradually strengthening the physical constraint weights in the later stages.

[0085] In one embodiment of the present invention, physical constraint adversarial learning uses a bidirectional constraint mechanism. First, the physical model generates pseudo labels that conform to physical laws to guide the training of the semantic model. Second, the semantic model's prediction results are used to evaluate the rationality of the physical model, forming a bidirectional feedback optimization. The mathematical expression of the physical constraint is:

[0086]

[0087] Among them, λ p is the physical constraint weight (initial value is 0.1, gradually increases to 1.0 as training progresses), φ i represents the i-th physical constraint function, m is the total number of physical constraint functions, x is the input data, y pred Predict the results for the model.

[0088] The physical constraint functions mainly include: 1) surface continuity constraint, which ensures that the surface height changes smoothly between adjacent areas; 2) geometric consistency constraint, which ensures that areas of the same semantic category have similar geometric characteristics; 3) gravity direction constraint, which ensures that the normal vectors of vertical structures (such as walls) are approximately horizontal, and the normal vectors of horizontal structures (such as the ground) are approximately vertical; 4) material property constraint, which ensures that the reflectivity and texture characteristics of different materials (such as rocks and soil) conform to physical laws.

[0089] The formula for calculating the plane confidence by the reference plane acquisition module 40 is:

[0090] Confidence(plane)=SemanticConsistency×GeometricFitness×PhysicalReasonability,

[0091] Among them, SemanticConsistency represents the consistency of semantic labels of points in the plane, and the calculation formula is:

[0092]

[0093] Among them: C i represents the number of points in the plane that belong to the i-th category, n represents the total number of semantic categories, N inlier Represents the total number of points in the plane This formula takes the semantic category with the largest proportion in the plane as the numerator, ensuring that the output value is between 0 and 1, and when all points in the plane belong to the same category, the semantic consistency reaches the maximum value of 1.

[0094] GeometricFitness represents the geometric conformity of plane fitting. To handle semantically ambiguous areas, we introduce a soft semantic consistency measure that considers the probability distribution of point categories:

[0095] The calculation formula is:

[0096]

[0097] Where: P i,dominant represents the probability that the i-th point belongs to the dominant semantic category of the plane, N inlier Indicates the total number of points in the plane. When using soft semantic consistency, the system combines the consistency evaluation of hard labels and soft labels to better handle the semantic boundary fuzzy areas. The final semantic consistency used is:

[0098]

[0099] Where: α is the weight coefficient, the value range is [0, 1], and the preferred value is 0.7, which is used to balance the contribution of hard labels and soft labels.

[0100] Geometric Fit (GeometricFitness) indicates the geometric conformity of plane fitting. The revised calculation formula is:

[0101]

[0102] Where: d(P i , Π) represents the orthogonal distance from point Px to plane Π, t is the distance threshold (preferably 0.05m), N inlier is the number of points in the plane. This formula changes the original subtraction relationship to a fractional relationship, ensuring that the output value is always between 0 and 1 and decreases smoothly as the distance from the point to the plane increases. When the distance from all points to the plane is 0, the geometric fit reaches its maximum value of 1; when the average distance is much larger than the threshold t, the geometric fit approaches 0. The formula for calculating the distance from a point to a plane is:

[0103]

[0104] Among them, (a, b, c) are the components of the plane normal vector, d is the constant term in the plane equation ax+by+cz+d=0, (x i ,y i ,z i ) is point P i 's coordinates.

[0105] Physical Reasonability means evaluating the rationality of a plane based on physical rules. The revised calculation formula is:

[0106]

[0107] Among them, w i is the weight coefficient of the i-th physical property, represents the measured value of the ith physical property of the plane, represents the ideal value of the physical property, ||·|| represents the Euclidean norm, and k is the total number of physical properties. This formula uses a sigmoid function instead of the original exponential decay function to ensure that the evaluation score of each physical property varies smoothly between 0 and 1, and that slight deviations in a single property will not lead to a sharp drop in the overall score.

[0108] The main physical characteristics include: the direction of the plane's normal vector, that is, the relationship between the plane's posture and the direction of gravity; the plane's height distribution, which evaluates whether the height changes of points on the plane are smooth; the geometric relationship between the plane and the surrounding structures, such as the angle with adjacent planes.

[0109] For the evaluation of the plane normal direction, the following formula is used:

[0110]

[0111] in, is the unit normal vector of the plane, is the normalized gravity direction vector (usually (0,0,1) or (0,0,-1)). For a horizontal plane, the ideal value is Close to 1; for vertical planes, the ideal value is close to 0.

[0112] To improve the robustness of the confidence calculation, we change the product form of the three indicators to a weighted average form:

[0113]

[0114] Among them, w1, w2, and w3 are the weight coefficients for semantic consistency, geometric fit, and physical rationality, with preferred values of 0.3, 0.5, and 0.2, respectively. By using a weighted average rather than a product form, the problem of a single poor evaluation metric leading to a sharp drop in overall confidence is avoided, making the evaluation results more robust and reliable.

[0115] When the calculated confidence level is higher than a preset threshold (preferably 0.75), the plane is determined to be a reliable measurement reference surface; otherwise, the search for a better plane continues. The system uses an iterative optimization strategy to generate multiple sets of candidate planes through a hybrid RANSAC algorithm and selects the plane with the highest confidence level as the final reference surface.

[0116] The core process of the hybrid RANSAC algorithm is as follows:

[0117] 1. Initialization parameters: maximum number of iterations N_iter (preferably 1000), inlier distance threshold t (preferably 0.05m), minimum inlier ratio r_min (preferably 30%).

[0118] 2. Semantic-guided sampling: Assign sampling weights based on the semantic labels of points, giving priority to sampling points of key semantic categories. The sampling weight calculation formula is:

[0119]

[0120] Where: S(L i ) is point P i Semantic label L i The importance score of point cloud is N, and N is the total number of point clouds.

[0121] 3. Plane Model Estimation: For a randomly sampled set of points, the plane parameters are estimated using the least squares method. The plane equation is ax + by + cz + d = 0, and the parameters (a, b, c, d) are obtained using analytical solutions or SVD decomposition.

[0122] 4. Inlier evaluation: Calculate the distance from all points to the plane, and points with a distance less than a threshold t are considered inliers.

[0123] 5. Model evaluation: Calculate the confidence of the plane, including a comprehensive score of semantic consistency, geometric fit and physical rationality.

[0124] 6. Iterative optimization: Repeat steps 2-5 and retain the plane model with the highest confidence.

[0125] 7. Result verification: Check whether the interior point ratio of the optimal plane is greater than the minimum interior point ratio r min , and whether the confidence is higher than the threshold.

[0126] Through this semantically guided RANSAC algorithm, the system can more effectively find the optimal plane that meets semantic and geometric constraints, greatly improving the accuracy and reliability of reference plane fitting.

[0127] The method for semantic segmentation of three-dimensional point clouds and intelligent fitting of measurement reference surfaces of the present invention comprises the following steps:

[0128] Step S1: collecting three-dimensional point cloud data, using the depth camera 11 and the laser radar 12 to scan and image the target working area to obtain three-dimensional point cloud data;

[0129] Step S2: preprocessing step, denoising and downsampling the 3D point cloud data;

[0130] Step S3: Feature extraction step, using the PointNet++ network to extract features from the preprocessed point cloud data and generate semantic label vectors;

[0131] Step S4: lithologic label generation step, inputting the semantic label vector into the adversarial generative network 35 to generate lithologic labels;

[0132] Step S5: a reference plane acquisition step, calculating a probabilistic optimal fitting plane based on the lithologic labels, determining the confidence level of the plane, and determining the plane as the measurement reference plane if the confidence level is higher than a preset threshold;

[0133] Step S6: Online reasoning step, inputting the 3D point cloud data into the trained model for processing to obtain the measurement reference surface in the target area.

[0134] In step S1, the 3D point cloud data acquisition process involves simultaneous data acquisition by the depth camera 11 and the lidar 12. The depth camera 11 is responsible for obtaining high-resolution texture information, while the lidar 12 is responsible for obtaining precise distance measurements. During the acquisition process, the drone 13 scans the target area along a pre-set path to ensure complete data coverage. Preferably, the scanning frequency of the lidar 12 is set to 10 Hz, the acquisition frequency of the depth camera 11 is set to 30 Hz, and the flight altitude of the drone 13 can be set to 20-100 meters depending on the scenario.

[0135] In preprocessing step S2, a straight-through filtering algorithm is first used to remove obvious outliers. Downsampling is then performed using a voxel grid filtering algorithm. Finally, a statistical filtering algorithm is applied to remove outliers. The parameters for the straight-through filtering vary depending on the scene. The voxel size for the voxel grid filtering is preferably 0.05m, the number of neighborhood points for the statistical filtering is set to 50, and the standard deviation multiplier is set to 2.0. These parameter settings, based on extensive experimental results, effectively remove noise and outliers while preserving important geometric features.

[0136] In the feature extraction step S3, the PointNet++ network processes the point cloud data through multi-level sampling and grouping operations. Preferably, the network sets 4 layers of downsampling, the number of sampling points is 1024, 256, 64 and 16, and the sampling radius is 0.1m, 0.2m, 0.4m and 0.8m, respectively. Each layer uses a shared PointNet to extract local features, and the feature dimensions are 64, 128, 256 and 512, respectively. In the feature upsampling stage, a distance-based interpolation strategy is used to restore the features to the original point cloud resolution, and features at different levels are fused through jump connections. Finally, the semantic label of each point is output through the fully connected layer and the softmax classifier. The label category may include ground, vegetation, building, vehicle, etc. according to the application scenario.

[0137] In the lithologic label generation step S4, the Generative Adversarial Network 35 receives the semantic label vector as input. The generator 351 is responsible for converting the semantic label vector into lithologic distribution characteristics, and the discriminator 352 is responsible for distinguishing the generated labels from the true labels. The generator 351 and the discriminator 352 are continuously optimized through alternating training, ultimately generating high-quality lithologic labels. Preferably, the generator 351 adopts a 5-layer U-Net structure with an initial number of 64 channels and a maximum number of 512 channels; the discriminator 352 adopts a 4-layer PatchGAN structure with a 70×70 receptive field, an initial number of 64 channels, and a maximum number of 512 channels.

[0138] In the reference plane acquisition step S5, the point cloud proportion of each semantic category is first calculated, and then the feature extraction strategy is determined based on the proportion. When the proportion does not exceed 30%, comprehensive features are extracted for each semantic category; when the proportion exceeds 30%, only key features such as normal vectors are extracted. Next, the hybrid RANSA C algorithm is used to fit the semantic categories and features to obtain multiple groups of different semantic planes and the confidence probability of each semantic plane. The number of iterations of the hybrid RANSA C algorithm is preferably set to 1000, the inlier threshold is set to 0.05m, and the minimum inlier ratio is set to 30%. Finally, the final reference plane is determined based on the confidence probability, and the confidence threshold is preferably set to 0.85.

[0139] In the online inference step S6, the optimal model trained by the system is deployed in a real-world application environment. When new point cloud data is input, the system processes it in real time, including preprocessing, feature extraction, lithologic label generation, and datum fitting, ultimately outputting a measured datum within the target area. The system's online processing speed ideally reaches 5-10 frames per second (depending on the point cloud size and hardware configuration), meeting the requirements of most real-time application scenarios.

[0140] This method achieves high-precision intelligent fitting of measurement datum surfaces through semantic information-assisted geometric processing. Compared with traditional methods, this method has the following significant advantages: First, multi-sensor fusion acquisition improves data quality and integrity; second, the cascaded filtering preprocessing process significantly enhances data quality; third, the organic integration of deep learning and geometric processing achieves complementary enhancement of semantic and geometric information; in addition, the probabilistic optimal fitting plane algorithm and confidence-based evaluation mechanism significantly improve the accuracy of datum surface fitting; finally, semi-supervised and weakly supervised learning strategies reduce reliance on annotated data, and physical constraint adversarial learning improves model generalization and physical rationality.

[0141] In actual testing, the present invention has performed well in a variety of complex environments. In mining applications, the system can accurately identify different lithologies and geological structures, with a reference plane fitting accuracy of ±1cm. In forest applications, the system can effectively separate vegetation and terrain, accurately identifying the ground reference plane. In urban applications, the system can accurately identify buildings, roads, and the ground, providing a reliable measurement benchmark for urban planning. In addition, the system demonstrates strong robustness to interference factors such as lighting changes, complex occlusions, changes in point cloud density, and sensor noise, and can operate stably in a variety of complex environments.

[0142] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. 3D point cloud semantic segmentation and measurement reference surface intelligent fitting system, characterized by: include: Data acquisition module, used to collect three-dimensional point cloud data through multi-sensor fusion; A preprocessing module, configured to perform denoising and downsampling processing on the three-dimensional point cloud data; A feature extraction module, configured to extract features from the preprocessed point cloud data and generate semantic labels; A reference plane acquisition module, configured to calculate a probabilistic optimal fitting plane based on the semantic label; The pixel feature conversion module is used to map the point cloud features into the pixel feature space to achieve multi-scale feature output.

2. The 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 1, characterized in that: The data acquisition module includes a depth camera and a laser radar, wherein the depth camera adopts a binocular camera, including a data acquisition unit, an image stitching unit, an image correction unit and a data storage unit. The depth camera and the laser radar are installed on the drone for scanning and imaging the target operation area.

3. The 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 1, characterized in that: The pre-processing module comprises: The geometric denoising submodule is used to remove flying spots generated during data acquisition and uses a straight-through filtering algorithm and a voxel grid filtering algorithm to downsample the data; The outlier removal submodule is used to filter outliers outside the point cloud distribution area using a statistical filtering algorithm to reduce the number of subsequent calculations.

4. The 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 1, characterized in that: The feature extraction module is implemented using the PointNet++ convolutional computing network, including point cloud feature extraction, feature upsampling, feature fusion and semantic label generation processes, in which the local feature enhancement mechanism and context-aware module are used to capture the hierarchical structure information of the point cloud.

5. The 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 4, characterized in that: The feature extraction module further includes a generative adversarial network for receiving the semantic label vector and generating a lithologic label. The generative adversarial network includes: Generator, used to convert semantic label vectors into lithologic distribution features; The discriminator is used to distinguish the generated labels from the true labels and continuously optimize the generated features through error feedback.

6. The 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 1, characterized in that: The reference plane acquisition module is used for: Calculate the ratio of the number of point clouds of each semantic category to the total number of point clouds; When the ratio is not greater than a preset threshold, feature extraction is performed on each semantic category, including the centroid, normal vector, local surface normal vector, covariance matrix, point pair, principal component and distance from point to fitting surface of the point cloud; The hybrid RANSAC algorithm is used to fit semantic categories and features, and multiple groups of different semantic planes and the confidence probability of each semantic plane are obtained; A final reference surface is determined based on the confidence probability.

7. The 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 1, characterized in that: The pixel feature conversion module is implemented using the Unet architecture, which includes two parts: encoding and decoding. The encoding part consists of convolution, pooling and downsampling, and the decoding part is implemented using transposed convolution. Skip connections are used in the middle of the Unet architecture to fuse high-level semantics at various scales in the encoding process with detailed features at the low level of decoding.

8. The 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 5, characterized in that: The adversarial generative network learns based on physical constraints, including: Construct adversarial learning between semantic models and physical models; Design new semantic segmentation model labels based on terrain label information and color map information; Optimizing the semantic model through a cross entropy loss function to enable it to learn physical properties; A progressive constraint strategy is adopted, focusing on feature learning in the early stage of training and gradually strengthening the physical constraint weights in the later stage.

9. The three-dimensional point cloud semantic segmentation and measurement reference plane intelligent fitting system according to claim 6, characterized in that: The formula for calculating the plane confidence of the reference plane acquisition module is: Confidence = semantic consistency × geometric fit × physical rationality, Among them, semantic consistency represents the degree of consistency of the semantic labels of points in the plane, geometric fit represents the residual error of plane fitting, and physical rationality represents the rationality of evaluating the plane based on physical rules.

10. A method for the 3D point cloud semantic segmentation and measurement reference plane intelligent fitting system according to any one of claims 1 to 9, comprising: The step of collecting 3D point cloud data is to use a depth camera and a laser radar to scan and image the target operation area to obtain 3D point cloud data; A preprocessing step of denoising and downsampling the three-dimensional point cloud data; Feature extraction step: Use the PointNet++ network to extract features from the preprocessed point cloud data and generate semantic label vectors; a lithologic label generation step, inputting the semantic label vector into a generative adversarial network to generate a lithologic label; a reference plane acquisition step, calculating a probabilistic optimal fitting plane based on the lithologic labels, determining a confidence level of the plane, and determining the plane as a measurement reference plane if the confidence level is higher than a preset threshold; In the online reasoning step, the three-dimensional point cloud data is input into the trained model for processing to obtain the measurement reference surface in the target area.

Images