Multi-view cloud high-precision fusion splicing method based on semantic geometry collaborative optimization

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a multi-view cloud high-precision fusion splicing method based on semantic geometry collaborative optimization, which comprises the following steps: extracting and quantifying features of multi-view cloud, endowing each point cloud with a semantic label, calculating the confidence coefficient of classification of each point cloud, and simultaneously calculating the normal vector and curvature geometric features of each point cloud. And performing robust initial registration based on semantic constraint, and outputting a robust initial transformation matrix. And constructing a collaborative energy function, carrying out dynamic quantitative coupling on semantic confidence and geometric stability, and carrying out iterative registration. And realizing anisotropic point cloud fusion based on semantic guidance, and outputting a fused and spliced three-dimensional point cloud model. And finally, verifying the performance of the method, and measuring fusion splicing errors. According to the method, the semantic features of the point cloud data are effectively extracted and utilized, the precision and robustness of point cloud fusion splicing are improved, and geometric details of a final model are reserved to the maximum extent.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a multi-view cloud high-precision fusion splicing method based on semantic geometric collaborative optimization. BACKGROUND

[0002] With the rapid development of three-dimensional sensing technology, devices such as laser radar and depth camera can efficiently obtain three-dimensional point cloud data of a target scene. As a direct carrier of three-dimensional spatial information, point cloud has the advantages of high precision, rich details, and no texture dependence, and has been widely used in key fields such as autonomous driving, three-dimensional reconstruction, industrial detection, and cultural heritage protection.

[0003] However, the field of view and measurement range of a single sensor are limited, and it is difficult to obtain complete three-dimensional information of a large scene or a complex target at one time. Point cloud splicing technology emerges as the times require, and its core goal is to generate a globally consistent and high-precision complete three-dimensional model by registering and fusing point cloud data collected from multiple different viewpoints with position and attitude differences. It is a key technical bridge connecting local point cloud and global scene.

[0004] Current point cloud splicing technology mainly includes two core stages: registration and fusion. The registration stage is responsible for eliminating the position and attitude deviation between multi-view point clouds, and is the basis for splicing accuracy. The fusion stage processes the overlapping area of the registered point cloud, eliminates redundancy, fills gaps, and ensures the integrity and consistency of the model.

[0005] In terms of registration technology, the mainstream methods can be divided into three categories: first, geometric-based methods, represented by the Iterative Closest Point (ICP) algorithm, which realizes registration by iteratively optimizing point-to-point or point-to-plane distance errors. It has the characteristics of simple principle and high precision, but is sensitive to the initial alignment attitude and easy to fall into local optimum. Second, feature-based methods, which extract three-dimensional feature descriptors such as FPFH, SIFT3D, and SHOT, realize feature matching between different point clouds, and then use RANSAC algorithm to remove false matches and solve the transformation matrix. It is suitable for scenes with large initial attitude differences, but the feature matching rate decreases significantly under low texture, occlusion or noise interference. Third, deep learning-based methods, which use network models such as PointNet and PointTransformer to automatically learn deep features of point cloud, or directly regress registration transformation parameters, have strong feature expression ability, but are highly dependent on training data and have large computational overhead, and have insufficient generalization in small sample or unstructured scenes.

[0006] In the aspect of fusion technology, the traditional methods mainly include weighted average fusion, Poisson fusion, region growing fusion, etc. The weighted average fusion realizes smooth transition by assigning distance-related weights to the overlapping area point cloud, but it is easy to cause detail blur; the Poisson fusion can generate smooth surface based on the implicit function reconstruction principle, but it is sensitive to registration error and easy to produce geometric distortion; the region growing fusion screens effective points through the geometric consistency of neighborhood point cloud, but it is easy to appear holes in the sparse point cloud area.

[0007] Although the point cloud stitching technology has made significant progress, in complex actual scenes, the existing methods still have many problems to be solved, and it is difficult to meet the application requirements of high precision and high robustness, and the specific deficiencies are as follows: (1) In the registration stage, the existing technology either simply relies on geometry and image features for matching, or relies on specific external calibration equipment to assist alignment. The former fails to deeply optimize the reliability of semantic information and the stability of geometric features, and the latter limits the flexibility and universality of the method. These ways lead to poor robustness and insufficient precision of registration when facing complex general scenes such as noise, occlusion, weak texture or repeated structure.

[0008] (2) In the fusion stage, the existing technology generally only focuses on the simple merging after alignment, adopts the isotropic processing method, and treats all areas "the same", which cannot differentiate the processing according to the actual content of the point cloud (such as plane, edge, corner, etc.), resulting in that the sharp edge profile is blunted and the fine geometric structure details are lost in the final generated three-dimensional model, and the model fidelity is not high.

[0009] Therefore, the technical problems to be solved in this field are how to deeply and dynamically couple the semantic and geometric information of the point cloud itself to improve the robustness and precision of the point cloud registration in complex general scenes, and how to adaptively process different areas according to the semantic content in the fusion process to maximize the retention of geometric details of the final model.

[0010] A point cloud splicing method and device are disclosed in prior art CN110120013A, which includes obtaining a first group and a second group of images of a measured object, each group of images containing left / right views and point cloud graphs, extracting 2D feature points in the images and 3D feature points in the point cloud graphs, respectively, calculating the three-dimensional coordinates of the 2D feature points in the point cloud graphs, and merging them with the 3D feature points to obtain a fused feature point pair set, and transforming the two groups of point cloud graphs to a unified coordinate system to complete splicing according to the feature point pair set. The purpose of this method is to fuse 2D and 3D feature points to avoid complex iterative calculations and improve splicing accuracy and efficiency. However, the shortcomings of this method are that its registration process completely depends on 2D image features and 3D geometric feature points. In the face of complex scenes with weak texture (such as large-area white walls) or repeated geometric structures (such as identical windows and corridors), it is difficult to extract sufficient and robust feature points, which can easily lead to matching ambiguity and cause registration failure or precision decline. In addition, this method does not use high-level semantic information of the scene to assist in eliminating matching ambiguity, and does not optimize the point cloud processing method in the fusion stage, which can cause the final generated three-dimensional model to be blurred or distorted in fine structures (such as object edges). A three-dimensional point cloud splicing method, device and equipment are disclosed in prior art CN117710584A, which mainly obtains a device coordinate system by acquiring a calibration board image and a raster picture and accurately calibrating a structured light scanning device; then converts the collected target object three-dimensional point cloud from the device coordinate system to the calibrated indoor positioning system coordinate system to obtain a transformation initial value; and finally performs registration fusion on the transformation initial value. The disadvantage of this method is that its high-precision splicing effect is heavily dependent on a specific structured light scanning device and a controlled environment containing a calibration board, a positioning system and other auxiliary equipment. This greatly limits the flexibility and universality of its application, making it difficult to be applied to more general scenarios (such as quickly reconstructing three-dimensional models of existing indoor and outdoor environments) without calibration objects. At the same time, this method also does not deeply utilize the semantic and geometric features of the point cloud for intelligent registration optimization and differential fusion processing, so it has similar problems as the above-mentioned prior art in dealing with complex structures and preserving model details. SUMMARY

[0011] To solve the above problems, the present application proposes a multi-view point cloud high-precision fusion splicing method based on semantic and geometric collaborative optimization, which realizes high-precision and high-robustness splicing of multi-view point cloud data by deeply fusing semantic and geometric information. To achieve the above scheme, the present application provides the following technical solutions: A multi-view point cloud high-precision fusion splicing method based on semantic and geometric collaborative optimization, comprising the following steps: S1: Feature extraction and quantization of multi-view point clouds, assigning semantic labels to each point cloud and calculating its classification confidence, while calculating the normal vector and curvature geometric features of each point cloud. S2: Based on the multi-viewpoint cloud features extracted in step S1, perform robust initial registration based on semantic constraints and output a robust initial transformation matrix. S3: Iterative registration based on the co-energy function. Construct the co-energy function, dynamically quantize and couple the semantic confidence and geometric stability of the corresponding point cloud pairs, and perform iterative registration based on the co-energy function in step S2. S4: Semantic-guided anisotropic point cloud fusion unifies all point clouds into a global coordinate system and voxels them. Based on the dominant semantic label of the point cloud within each voxel, a unique anisotropic fusion strategy is selected from a model library containing planar projection, principal axis projection, and weighted average strategies and executed. The fused and stitched 3D point cloud model is then output. S5: Verify the performance of the method and measure the fusion and splicing error.

[0012] Further, step S1 includes: S11: Assign semantic labels to each point cloud data and calculate the confidence score of its classification by using a pre-trained 3D point cloud semantic segmentation network; S12: Calculate the geometric features of the point cloud. The normal vector and curvature geometric features of each point cloud data are calculated by principal component analysis, and the discrete point cloud data is transformed into a high-dimensional feature point cloud with quantization reliability.

[0013] Further, step S2 includes: S21: Based on the quantization features extracted in step S1, convert all point cloud data into source point clouds. and target point cloud To pair up; S22: Select points from the point cloud pairs that meet the semantic confidence level after pairing in step S21. and geometric curvature Key points of the conditions Including source key points and key target points ,in The semantic confidence threshold. Both are geometric curvature thresholds, which are variables that can be determined independently and are used as the lower limit of the point cloud screening criteria. S23: For each key point selected in step S22 Generate fast point feature histogram descriptors; S24: At the source critical point and key target points Finding a correspondence between them is defined as if and only if the source key points and key target points semantic tags and Only when they are completely identical are they considered a potential match; S25: Input the set of all potential matching pairs that satisfy the semantic constraints into the random sampling consensus algorithm, and output the initial transformation matrix. .

[0014] Further, step S3 includes: S31: Construct a dynamic corresponding point set for the source point cloud. Each point in Through matrix After transformation, finding the closest spatial point, and semantic label verification, the points are included in the set of valid corresponding points. S32: Construct the cooperative energy function, and calculate its expression as follows:

[0015] in, K For a valid set of corresponding points; For each pair of corresponding points in a point set; R Transformation matrix T The rotation matrix part; t This is the translation vector part; The normal vector of the target point; Represents the dot product of vectors; Represents matrix-vector multiplication; The square of the scalar; The dynamic weighting function is calculated using the following formula:

[0016] in, and These are the semantic confidence scores for the source and target points, respectively. and These are the geometric curvatures of the source point and the target point, respectively. For balancing parameters, Scale factor; It is a natural exponential function; S33: Introduce a lower limit threshold for weighting Correction This is used to prevent weights from being affected by excessively low semantic confidence or excessively high geometric curvature. Approaching 0, after correction The calculation expression is:

[0017] in, It is a positive number close to 0; S34: Put all point pairs and its corresponding Substitute the energy function constructed in step S32 By iteratively minimizing the energy function, a tiny transformation increment can be obtained. and update Repeat this process until the transformation matrix is ​​obtained. T If the update amount is less than the convergence threshold or the maximum number of iterations is reached, output the registration matrix. .

[0018] Further, step S4 includes: S41: Global unified voxelization, creates a 3D voxel grid to divide the entire 3D space; S42: Implement semantically guided anisotropic fusion; S43: Attribute information is fused synchronously. While calculating the final geometric position of each voxel, the color and normal vector of the point cloud are fused simultaneously.

[0019] Further, step S42 includes: S421: Determine the dominant semantics. Statistically analyze the semantic labels of all points within the voxel constructed in step S41 obtained in step S11, and determine the label with the highest frequency as the dominant semantic label of the voxel. S422: Perform model selection, construct a configurable lookup table to store semantic category mapping rules, define the mapping relationship from the dominant semantic label to a specific fusion model in the lookup table, select and execute a unique corresponding fusion model for the voxel by looking up the table.

[0020] Further, step S5 includes: S51: In the dataset m Random selection n A registration experiment was conducted on viewpoint clouds with overlapping regions, and the rotation and translation errors after registration were calculated. m , n All of these are variables that can be determined independently; S52: After registering all viewpoint clouds, model fusion is performed using the anisotropic fusion strategy described in step S4.

[0021] Compared with the prior art, the beneficial effects of the present invention are: 1. By constructing a collaborative energy function, the reliability of semantics and the stability of geometry are deeply and dynamically coupled, which improves the tolerance of the registration process to local noise and individual errors in semantic segmentation, and achieves robustness and accuracy far exceeding existing technologies.

[0022] 2. By adopting a semantically guided anisotropic fusion model, differentiated and intelligent processing of point clouds is achieved, supporting the recognition of semantic attributes in different regions, jointly matching and calling the optimal geometric processing strategy, effectively improving geometric accuracy and detail fidelity.

[0023] 3. It achieves a high degree of automation and intelligence in the end-to-end process, reducing the complexity of human operation and the errors that may be introduced, and also improving the efficiency and universality of processing large-scale and complex scenario data. Attached Figure Description

[0024] Figure 1 This is a flowchart of the multi-viewpoint cloud high-precision fusion and stitching method based on semantic geometric collaborative optimization of the present invention.

[0025] Figure 2 This is a flowchart of the collaborative energy function construction process of the present invention.

[0026] Figure 3 This is a flowchart of the high-precision iterative registration process based on the cooperative energy function of this invention. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0029] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0030] In this embodiment of the invention, a high-precision fusion and stitching method for multi-viewpoint clouds based on semantic-geometric collaborative optimization is described, as detailed in the appendix. Figure 1 Specifically, it includes: S1: Feature extraction and quantization of multi-view point clouds, assigning semantic labels to each point cloud and calculating its classification confidence, while calculating the normal vector and curvature geometric features of each point cloud. S2: Based on the multi-viewpoint cloud features extracted in step S1, perform robust initial registration based on semantic constraints and output a robust initial transformation matrix. S3: Iterative registration based on the co-energy function. Construct the co-energy function, dynamically quantize and couple the semantic confidence and geometric stability of the corresponding point cloud pairs, and perform iterative registration based on the co-energy function in step S2. S4: Semantic-guided anisotropic point cloud fusion unifies all point clouds into a global coordinate system and voxels them. Based on the dominant semantic label of the point cloud within each voxel, a unique anisotropic fusion strategy is selected from a model library containing planar projection, principal axis projection, and weighted average strategies and executed. The fused and stitched 3D point cloud model is then output. S5: Verify the performance of the method and measure the fusion and splicing error.

[0031] In a specific embodiment, to address the problems of poor registration and fusion accuracy, inability to effectively distinguish and process differentiated point cloud content, and insufficient dynamism and adaptability in existing point cloud fusion and stitching methods, point cloud fusion and stitching engineers propose a high-precision multi-view point cloud fusion and stitching method based on semantic-geometric co-optimization. First, each point cloud is assigned a semantic label and its classification confidence is calculated. Simultaneously, the normal vector and curvature geometric features of each point cloud are calculated, achieving feature extraction and quantization of multi-view point clouds. Second, robust initial registration based on semantic constraints is performed based on the extracted multi-view point cloud features, outputting a robust initial transformation matrix. Third, a co-energy function is constructed to dynamically quantize and couple the semantic confidence and geometric stability of corresponding point cloud pairs, and iterative registration is performed based on this co-energy function. Then, based on semantically guided anisotropic point cloud fusion, all point clouds are unified to a global coordinate system and voxelized. According to the dominant semantic label of the point cloud within each voxel, a unique anisotropic fusion strategy is selected from a model library containing planar projection, principal axis projection, and weighted averaging strategies, and executed to output the fused and stitched 3D point cloud model. Finally, the performance of the proposed method is verified, and the fusion and stitching error is measured.

[0032] Example 1 In one embodiment, as shown in the appendix Figure 1 As shown, to address the problem of poor registration and fusion accuracy caused by the inability to consider the semantic features of point cloud data in existing point cloud fusion and stitching methods, point cloud fusion and stitching engineers propose a feature extraction and quantization method for viewpoint clouds, specifically including: S11: For each point cloud data segmentation dataset, a pre-trained 3D point cloud semantic segmentation network, BEVFormer, is used. To perform pointwise semantic segmentation, BEVFormer is pre-trained using the large-scale public dataset ScanNet, and the output layer of BEVFormer is set to a Softmax layer. For each point cloud The BEVFormer output layer outputs a K-dimensional probability distribution vector. Where K is the total number of predefined semantic categories, It is a point The probability of belonging to the k-th class; the semantic label of the point cloud. The category with the highest probability is identified; simultaneously, the maximum value in this probability vector is extracted as the semantic confidence score for that point. The calculation formula is:

[0033] in, Let be the function that takes the maximum value in the vector. The value ranges from 0 to 1. It is a direct quantification of the network classification accuracy and is the basis for subsequent dynamic weighted optimization. It elevates semantic information from a discrete label to a continuous value with a reliability metric. S12: Calculate the geometric features of the point cloud for each point in the point cloud. Calculate its local geometric descriptor. First, for each point, construct a kd-tree data structure. Efficiently query all neighboring points within a fixed radius to form a neighborhood point set, where the fixed radius is set to... rice; Subsequently, principal component analysis (PCA) was performed on the neighborhood point set to obtain three eigenvalues. And its corresponding eigenvectors. We will compare it with the smallest eigenvalue. The corresponding feature vector is defined as a point. normal vector Simultaneously, the three-dimensional curvature of the point is calculated using these three eigenvalues. This curvature value effectively measures the flatness of the local surface at a point and is a key indicator for assessing geometric stability. The calculation formula is as follows: .

[0034] Example 2 In one embodiment, as shown in the appendix Figure 1 As shown, point cloud fusion and stitching engineers constructed a robust initial registration method based on semantic constraints. Based on the extracted quantized features, it can perform initial registration for any pair of source point clouds. and target point cloud Calculate a reliable initial alignment transformation matrix Specifically, it includes: S21: Based on the extracted quantization features, convert all point cloud data into source point clouds. and target point cloud To pair up; S22: Select points from the paired point cloud pairs that meet the semantic confidence level. and geometric curvature Key points of the conditions Including source key points and key target points Among them, a threshold is set based on the characteristics of point cloud data and semantic confidence requirements. The filtering process here directly uses the output of Example 1 above. and This joint screening strategy ensures that key points are not only semantically clear, but also located in areas rich in geometric features, such as object edges and corners. S23: For each selected key point Generate Fast Point Feature Histogram (FPFH) descriptors; S24: At the source critical point and key target points Find the corresponding relationship between them, enforce semantic consistency constraints, and agree that the source key point is valid if and only if the source key point is valid. and key target points semantic tags and Only when they are completely identical are they considered a potential match; S25: Input the set of all potential matching pairs that satisfy the semantic constraints into the Random Sample Consensus (RANSAC) algorithm. RANSAC estimates the optimal initial rigid body transformation matrix from the matching set containing a large number of outliers through iterative sampling. .

[0035] Example 3 A preferred embodiment, as shown in the appendix Figure 2 Appendix Figure 3 As shown, to effectively improve the registration accuracy of point cloud data fusion and stitching methods and fully utilize point cloud semantic features, point cloud fusion and stitching engineers performed high-precision iterative registration based on the collaborative energy function, specifically including: S31: Construct a dynamic corresponding point set for the source point cloud. Each point in First, use the current transformation matrix. Transform it to the coordinate system of the target point cloud to obtain Secondly, in the target point cloud Searching for and The closest point in space Corresponding points for each pair of candidates It must pass semantic tag validation if and only if Only then are these points confirmed as valid correspondences and included in the set of valid correspondences for the current iteration. K ; S32: Construct the cooperative energy function, and calculate its expression as follows:

[0036] in, K For a valid set of corresponding points; For each pair of corresponding points in a point set; R Transformation matrix T The 3×3 rotation matrix part; t The translation vector part is 3×1; The normal vector of the target point; Represents the dot product of vectors; Represents matrix-vector multiplication; It is the square of the scalar; this function calculates the transformed source point. To the target point The distance to the tangent plane is the weighted sum of squares of the distances from the point to the plane; This is a dynamic weighting function used to intelligently evaluate the reliability of each corresponding point pair. The calculation formula is as follows:

[0037] in, and These are the semantic confidence scores for the source and target points, respectively. and These are the geometric curvatures of the source point and the target point, respectively; For balancing parameters, Scale factor; It is a natural exponential function; S33: In addition, to ensure the numerical stability of the optimization process and prevent issues arising from semantic confidence... Extremely low (e.g., in mis-segmented regions) or geometric curvature Extremely high (e.g., at outlier noise points) leads to weights The value approaches 0, rendering the point pair completely ineffective in the optimization process. Therefore, a lower bound threshold for the weights is introduced. Correction After correction The calculation expression is:

[0038] in, It is a positive number close to 0, and here the value is 0.05; S34: Collect all point pairs and their corresponding pairs. Substitute into the energy function The Levenberg-Marquardt nonlinear least squares algorithm is used to iteratively minimize the energy function, thus solving for a small transformation increment. and update Repeat this process until the transformation matrix is ​​obtained. T The update amount is less than the convergence threshold It can reach a maximum of 100 iterations, ultimately outputting a high-precision registration matrix. .

[0039] Example 4 A preferred embodiment, as shown in the appendix Figure 1 As shown, to improve the efficiency of utilizing the semantic features of point cloud data and enhance the precision and detail of the generated 3D point cloud data model, point cloud fusion and stitching engineers implemented semantically guided anisotropic point cloud fusion. Specific steps include: S41: Global unified voxelization, creating a 3D voxel grid to divide the entire 3D space; setting the side length of the voxels. A fixed value of 0.01 meters is used; all transformed and aligned point cloud data are inserted into this raster to effectively organize nearby points in space into the same voxel; for each non-empty voxel containing at least one data point... The set of points inside it is denoted as This point set will serve as the basic unit for subsequent fusion processing; S42: Implement semantically guided anisotropic fusion; Determine the dominant semantics for voxels V All points within are obtained through Semantic tags are statistically analyzed, and the tag with the highest frequency is determined as the dominant semantic tag for that voxel. ; The process involves performing model selection and constructing a configurable lookup table to store semantic category mapping rules. The lookup table defines rules derived from the dominant semantic label. The mapping relationship to a specific fusion model is used to select and execute a unique corresponding fusion model for a voxel by looking up a table; like If a category belongs to a predefined set of planar semantics, such as walls, floors, and ceilings—categories with explicit planar attributes—then the system must and uniquely employ Model 1: Planar Projection Fusion. This model applies to the point set within a voxel. Perform principal component analysis (PCA) to determine the best-fit plane. (From the center of mass) and normal vector (Define); then, for each point within the voxel All are projected onto this plane Above; the projection point of each point The calculation formula is:

[0040] in, It is the dot product of vectors; The coordinates of the projection point of the original point onto the best-fit plane; It is an original three-dimensional point within a voxel; It is the centroid (geometric center) of all points within the voxel; The unit normal vector representing the plane, calculated using PCA; the final output voxel represents the point. Determined as all these projection points The arithmetic mean; like For categories belonging to a predefined set of structural edge semantics, such as door frames, window frames, table edges, and stair edges—those with explicit linear structural attributes—the system must and uniquely employ Model 2: principal axis projection fusion. This model also applies to the point set within the voxel. We perform PCA, but focus on the eigenvector corresponding to the largest eigenvalue representing the principal direction (i.e., the edge direction) of the point set. ; Put all points Projected onto physical fitness and the principal axis vector The projection point of each point on the defined straight line. The calculation formula is:

[0041] in, For vector dot product, The coordinates of the projection of the original point onto the principal axis line. The unit principal direction vector representing the edge direction, calculated using PCA; the final output voxel represents the point. Determined as all these projection points The arithmetic mean; like If a data point does not belong to any of the predefined sets mentioned above, such as chairs, table lamps, or sofas, which have complex non-planar or non-linear structures, then the system must and uniquely adopt Model 3: Confidence-Geometric Weighted Average Fusion. This model calculates the final point by weighting all points within a voxel. Its formula is:

[0042] in, This represents the unique three-dimensional point coordinates of the final output of the voxel; It is a primitive three-dimensional point within a voxel; each point weight The innovative idea from step three was directly reused, combining the semantic confidence and geometric stability of this point. The calculation formula is as follows:

[0043] in, It is a point semantic confidence, It is its local geometric curvature; This is a balancing parameter between 0 and 1, set to 0.6 in this embodiment, to reconcile the importance of semantics and geometry; It is a positive scaling factor, which is set to 0.1 in this embodiment to control the sensitivity of curvature to weights.

[0044] S43: Attribute information is synchronously fused during the calculation of the final geometric position of each voxel. Simultaneously, the color and normal vector attributes of the point cloud are fused to ensure the integrity of the final output model. Specifically, when executing the uniquely selected fusion model, the color and normal vector attributes of the points are also fused using a logically consistent strategy. If a planar projection fusion model is used, the final output point The normal vector is directly taken as the normal vector of the fitted plane. Its color is the color of all points within the voxel. The arithmetic mean of colors; If the principal axis projection fusion model is used, its color is the color of all points within the voxel. The color is the arithmetic mean; its normal vector (representing the edge normal) is calculated by averaging all the original normal vectors within the voxel and removing them along the principal axis. The components in the direction are obtained, that is, the normal vector is projected onto a plane orthogonal to the principal axis and normalized to ensure that the normal vector is orthogonal to the edge direction; If a weighted average fusion model is used, the final color and normal vector will also use the exact same weights. The weighted average of the corresponding attributes of all points within the voxel is obtained.

[0045] Example 5 A preferred embodiment, as shown in the appendix Figure 1 As shown, to quantitatively verify the effectiveness and advancement of the method of this invention, the point cloud fusion and stitching engineers selected the 'scene0000_00' sequence from the publicly available standard dataset ScanNet. This sequence contains multiple indoor scene viewpoint clouds and ground truth models generated by a high-precision laser scanner. The point cloud fusion and stitching engineers compared it with two mainstream existing technologies: In contrast to technique one, the geometric feature method: FPFH-RANSAC-ICP, this method first uses Fast Point Feature Histogram (FPFH) descriptors for feature matching, combines this with Random Sample Consensus (RANSAC) for initial registration, and finally employs the classic Iterative Closest Point (ICP) algorithm for fine alignment. This method represents a technical approach that relies purely on geometric features for registration. In contrast to technique two, the basic semantic method, Semantic-ICP, this method requires that matching point pairs must have the same semantic label during the iteration of ICP, but does not use semantic confidence or geometric stability for weighting. This method represents a technical approach that imposes simple hard constraints on semantic information. The specific process includes: S51: In this embodiment, 100 pairs of viewpoint clouds with overlapping regions are randomly selected from the ScanNet dataset for registration experiments, and the rotation error (RE, unit: degrees) and translation error (TE, unit: meters) after registration are calculated. Rotational error

[0046] Translation error

[0047] in, The transformation matrix estimated by the algorithm. These are the ground truth values. The mean and median of the experimental results are shown in the table below:

[0048] Compared to the first comparative technique, the average rotation and translation errors of the method in this invention are reduced by approximately 69.8% and 73.3%, respectively. This is because the first comparative technique has insufficient discriminative power of FPFH features when facing weakly textured walls or repetitive furniture common in ScanNet scenarios, resulting in a high initial matching error rate and ICP easily getting trapped in local optima. This invention performs initial registration through semantic constraints and performs fine registration using a co-energy function, which greatly improves robustness and accuracy in complex scenarios.

[0049] Compared to Comparative Technique 2, the mean rotation and translation errors of the method of this invention are reduced by approximately 50.3% and 55.6%, respectively. This is because although Comparative Technique 2 introduces semantic constraints, they are only used as hard filtering conditions and cannot handle semantic segmentation errors. The original collaborative energy function of this invention dynamically weights semantic confidence and geometric curvature, which can intelligently suppress the influence of unreliable matching pairs, such as points with low semantic segmentation confidence or located in noisy regions, thus obtaining more accurate and robust registration results.

[0050] S52: After registering all viewpoint clouds, the anisotropic fusion strategy described in step S4 is used for model fusion. Both techniques 1 and 2 use the standard truncated signed distance function (TSDF) for fusion. The reconstructed global model is compared with the ground truth model provided by the dataset to calculate accuracy (mean distance from the reconstructed point to the ground truth model, in centimeters) and completeness (mean distance from the ground truth model to the reconstructed point, in centimeters).

[0051] The accuracy error of the method of the present invention (0.93 cm) is much lower than that of comparative technique one (2.15 cm) and comparative technique two (1.68 cm). This is mainly due to two factors: First, the higher registration accuracy of the present invention provides better initial conditions for fusion; second, the anisotropic fusion strategy of the present invention, such as using planar projection on the "wall", can actively eliminate noise perpendicular to the surface, making the surface closer to the real geometry, thereby greatly improving the accuracy of the model.

[0052] The method of this invention also performs best in terms of integrity index, thanks to its robust registration, which ensures that the viewpoint clouds can be correctly stitched together and reduces model holes caused by registration failure.

[0053] Through 3D visualization comparison, it was found that the models generated by TSDF fusion in comparison techniques one and two exhibit obvious rounding and blurring at the edges of objects, such as table corners and door frames, and there are also minor bumps and depressions on large planes such as walls due to sensor noise; while the model generated by the method of this invention has extremely smooth walls, clear and sharp edges of structural elements such as table corners and door frames, and significantly higher fidelity of geometric details.

[0054] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0055] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A high-precision fusion and stitching method for multi-viewpoint clouds based on semantic-geometric collaborative optimization, characterized in that, Includes the following steps: S1: Feature extraction and quantization of multi-view point clouds, assigning semantic labels to each point cloud and calculating its classification confidence, while calculating the normal vector and curvature geometric features of each point cloud. S2: Based on the multi-viewpoint cloud features extracted in step S1, perform robust initial registration based on semantic constraints and output a robust initial transformation matrix. S3: Iterative registration based on the co-energy function. Construct the co-energy function, dynamically quantize and couple the semantic confidence and geometric stability of the corresponding point cloud pairs, and perform iterative registration based on the co-energy function in step S2. S4: Semantic-guided anisotropic point cloud fusion unifies all point clouds into a global coordinate system and voxels them. Based on the dominant semantic label of the point cloud within each voxel, a unique anisotropic fusion strategy is selected from a model library containing planar projection, principal axis projection, and weighted average strategies and executed. The fused and stitched 3D point cloud model is then output. S5: Verify the performance of the method and measure the fusion and splicing error.

2. The multi-viewpoint cloud high-precision fusion and stitching method based on semantic-geometric collaborative optimization according to claim 1, characterized in that, Step S1 includes: S11: Assign semantic labels to each point cloud data and calculate the confidence score of its classification by using a pre-trained 3D point cloud semantic segmentation network; S12: Calculate the geometric features of the point cloud. The normal vector and curvature geometric features of each point cloud data are calculated by principal component analysis, and the discrete point cloud data is transformed into a high-dimensional feature point cloud with quantization reliability.

3. The multi-viewpoint cloud high-precision fusion and stitching method based on semantic-geometric collaborative optimization according to claim 1, characterized in that, Step S2 includes: S21: Based on the quantization features extracted in step S1, convert all point cloud data into source point clouds. and target point cloud To pair up; S22: Select points from the point cloud pairs that meet the semantic confidence level after pairing in step S21. and geometric curvature Key points of the conditions Including source key points and key target points ,in The semantic confidence threshold. Both are geometric curvature thresholds, which are variables that can be determined independently and are used as the lower limit of the point cloud screening criteria. S23: For each key point selected in step S22 Generate fast point feature histogram descriptors; S24: At the source critical point and key target points Finding a correspondence between them is defined as if and only if the source key points and key target points semantic tags and Only when they are completely identical are they considered a potential match; S25: Input the set of all potential matching pairs that satisfy the semantic constraints into the random sampling consensus algorithm, and output the initial transformation matrix. .

4. The multi-viewpoint cloud high-precision fusion and stitching method based on semantic-geometric collaborative optimization according to claim 1, characterized in that, Step S3 includes: S31: Construct a dynamic corresponding point set for the source point cloud. Each point in Through matrix After transformation, finding the closest spatial point, and semantic label verification, the points are included in the set of valid corresponding points. S32: Construct the cooperative energy function, and calculate its expression as follows: in, For a valid set of corresponding points; For each pair of corresponding points in a point set; Transformation matrix The rotation matrix part; t This is the translation vector part; The normal vector of the target point; Represents the dot product of vectors; Represents matrix-vector multiplication; The square of the scalar; The dynamic weighting function is calculated using the following formula: in, and These are the semantic confidence scores for the source and target points, respectively. and These are the geometric curvatures of the source point and the target point, respectively. For balancing parameters, Scale factor; It is a natural exponential function; S33: Introduce a lower limit threshold for weighting Correction This is used to prevent weights from being affected by excessively low semantic confidence or excessively high geometric curvature. Approaching 0, after correction The calculation expression is: in, It is a positive number close to 0; S34: Put all point pairs and its corresponding Substitute the energy function constructed in step S32 By iteratively minimizing the energy function, a tiny transformation increment can be obtained. and update Repeat this process until the transformation matrix is ​​obtained. T If the update amount is less than the convergence threshold or the maximum number of iterations is reached, output the registration matrix. .

5. The multi-viewpoint cloud high-precision fusion and stitching method based on semantic-geometric collaborative optimization according to claim 1, characterized in that, Step S4 includes: S41: Global unified voxelization, creates a 3D voxel grid to divide the entire 3D space; S42: Implement semantically guided anisotropic fusion; S43: Attribute information is fused synchronously. While calculating the final geometric position of each voxel, the color and normal vector of the point cloud are fused simultaneously.

6. The multi-viewpoint cloud high-precision fusion and stitching method based on semantic-geometric collaborative optimization according to claim 5, characterized in that, Step S42 includes: S421: Determine the dominant semantics. Statistically analyze the semantic labels of all points within the voxel constructed in step S41 obtained in step S11, and determine the label with the highest frequency as the dominant semantic label of the voxel. S422: Perform model selection, construct a configurable lookup table to store semantic category mapping rules, define the mapping relationship from the dominant semantic label to a specific fusion model in the lookup table, select and execute a unique corresponding fusion model for the voxel by looking up the table.

7. The multi-viewpoint cloud high-precision fusion and stitching method based on semantic-geometric collaborative optimization according to claim 1, characterized in that, Step S5 includes: S51: In the dataset m Random selection n A registration experiment was conducted on viewpoint clouds with overlapping regions, and the rotation and translation errors after registration were calculated. m , n All of these are variables that can be determined independently; S52: After registering all viewpoint clouds, model fusion is performed using the anisotropic fusion strategy described in step S4.

Images