A method for semantic point cloud boundary extraction and vectorization based on multimodal fusion

Abstract

This invention discloses a semantic point cloud boundary extraction and vectorization method based on multimodal fusion, aiming to solve the problems of low boundary extraction accuracy, poor adaptability to complex scenes, and lack of structured output in existing technologies. The method comprises five stages: multimodal boundary confidence modeling, fusing semantic uncertainty, geometric abrupt changes, and probabilistic gradient features to generate confidence scores; adaptive segmentation and type classification, automatically classifying boundary types; graph neural network boundary connection optimization, repairing breaks and removing noise; structured polyline generation and curve optimization, obtaining smooth vector boundaries; and trajectory-guided priority sorting, outputting a priority-based structured boundary list. This invention improves boundary detection accuracy and robustness, outputs vector boundaries conforming to high-precision map standards, and is suitable for scenarios such as autonomous driving and intelligent robot navigation, demonstrating strong engineering practicality.

Description

Technical Field

[0001] This invention relates to the field of 3D environment perception and high-precision map construction technology, specifically a method for semantic point cloud boundary extraction and vectorization based on multimodal fusion. Background Technology

[0002] In cutting-edge applications such as autonomous driving, intelligent robot navigation, and high-precision map construction, accurate perception and structured representation of the 3D environment are crucial. Semantic point clouds, as a form of 3D data that integrates geometric information (3D spatial coordinates, normal vectors, etc.) and semantic labels (such as "ground," "curb," "wall," etc.), have become the core carrier of environmental modeling. Among them, boundary information (such as road edges, curbs, wall junctions, etc.) not only reflects the geometric structure of the scene but also carries key semantic separation meanings. It is an important prerequisite for tasks such as path planning, obstacle recognition, and map updating. Therefore, accurately extracting and vectorizing boundaries from semantic point clouds has core technical value.

[0003] In current technologies, boundary extraction methods mainly suffer from two core paths and corresponding shortcomings: Extraction methods based on pure geometric features rely on geometric information such as abrupt changes in normal vectors, curvature variations, height jumps, or discontinuities in plane segmentation to achieve boundary recognition. While these methods perform stably in regular structural scenes such as building walls, they are susceptible to point cloud noise, occlusion, or uneven sampling in complex urban scenes (such as intersections or areas obscured by vegetation), leading to misjudgments of boundaries. More importantly, they cannot distinguish between "semantically identical but geometrically continuous regions" and "the semantic boundaries that truly need attention" (such as the intersection of lane lines and shoulders), lacking semantic-level discriminative capabilities.

[0004] Extraction methods based on single semantic segmentation results locate boundaries by detecting changes in semantic labels within the neighborhood. However, semantic segmentation itself has uncertainties, especially at object boundaries where "fuzzy bands" or misclassification problems often occur, directly leading to boundary positioning offsets or the generation of a large number of false boundaries. Although some methods attempt to combine geometric and semantic information, they only use simple threshold superposition or rule judgment, lacking deep fusion and adaptive weighting mechanisms for multi-source features, making it difficult to cope with diverse real-world scenarios (such as temporary boundaries in construction areas and noise interference from point clouds on rainy days).

[0005] In addition, existing technologies also have three major common defects: Unstructured boundary output: Most methods only output discrete boundary point sets without modeling the continuity, direction, and topological structure of the boundaries. Such point sets cannot be directly used for vectorized storage of high-precision maps or path planning systems. They usually require additional clustering, fitting, or manual correction, which is inefficient and prone to introducing quadratic errors. Although some studies in recent years have introduced graph models or deep learning to optimize boundary connections, they have ignored trajectory priors, differences in boundary types, and the need for multi-scale representation, resulting in insufficient accuracy in key areas or overfitting of noise in non-key areas.

[0006] Lack of boundary importance quantification mechanism: In scenarios such as autonomous driving, the importance of boundaries varies significantly (e.g., the edge of the road near the vehicle's trajectory is more valuable for navigation than the boundary of a wall in the distance). However, existing technologies treat all boundaries "equally" and cannot achieve on-demand optimization and priority output, which limits their application performance in resource-constrained or real-time systems (such as in-vehicle embedded platforms).

[0007] Insufficient overall adaptability: Existing solutions are unable to simultaneously meet the four core requirements of "high-precision detection", "robustness in complex scenarios", "structured output" and "task-oriented prioritization", and cannot support the high reliability requirements of advanced autonomous driving and intelligent robots in understanding the environment.

[0008] In summary, existing semantic point cloud boundary extraction methods have significant shortcomings in terms of feature fusion depth, boundary connectivity robustness, vector representation structure, and task-oriented priority modeling, and new technical solutions are urgently needed to overcome these bottlenecks. Summary of the Invention

[0009] The purpose of this invention is to provide a semantic point cloud boundary extraction and vectorization method based on multimodal fusion to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a semantic point cloud boundary extraction and vectorization method based on multimodal fusion includes the following steps: (1) Multimodal boundary confidence modeling: For the input 3D point cloud with semantic labels, class probabilities and normal vectors, semantic uncertainty features, geometric mutation features and semantic probability gradient features are extracted respectively, and the boundary confidence score of each point is generated by normalization and weighted fusion. (2) Adaptive boundary point segmentation and type classification: The multi-threshold Otsu algorithm is used to automatically segment strong boundaries, weak boundaries and non-boundary points. The threshold of key areas is dynamically adjusted in combination with trajectory information. Then, based on the degree of geometric change, semantic confusion and height change characteristics, the boundary points are divided into four categories: roadside boundary, wall boundary, complex semantic boundary or weak boundary. (3) Boundary connection optimization based on graph neural network: a weighted graph is constructed with strong boundary points as nodes. The edge weights are fused with feature similarity, spatial distance and trajectory proximity. The connection relationship between points is learned through two layers of graph convolutional network to repair broken boundaries and remove noisy edges. (4) Structured polyline generation and curve optimization: Multi-scale depth-first search is used to extract and merge boundary polylines. For each polyline, a smooth vector boundary is generated by cubic B-spline least squares fitting. (5) Priority ranking of trajectory-guided boundaries: Calculate the comprehensive score of the boundary based on four indicators: average distance, direction consistency, trajectory coverage and boundary strength, and output a structured list of boundaries with type labels and priorities in descending order of scores.

[0011] Preferably, the semantic uncertainty features in step (1) include the label entropy and the proportion of the largest non-class labels calculated through the neighborhood label distribution, the geometric mutation features include the normal angle, the height step and the difference in flatness, and the semantic probability gradient features are obtained by calculating the gradient magnitude of the local neighborhood probability matrix through the Sobel operator.

[0012] Preferably, in step (1), the multimodal features are weighted and fused according to preset weights after min-max normalization to generate boundary confidence scores.

[0013] Preferably, in step (2), the adaptive threshold is solved by the multi-threshold Otsu algorithm to find the optimal dual threshold.

[0014] Preferably, in step (3), the graph node features are 6-dimensional multimodal fusion features. The forward propagation of the graph convolutional network uses adjacency matrix with self-loop and degree matrix normalization, and outputs edge probability retention to form an optimized connection graph.

[0015] Preferably, the multi-scale polyline extraction in step (4) includes fine scale, medium scale and coarse scale, and the path with endpoint distance less than the set value is merged by topological consistency check during fusion.

[0016] Preferably, in step (4), the B spline fitting uses the cumulative chord length normalized parameter node vector, and the control points are solved by least squares optimization. The objective function includes a geometric fidelity term and a third-order difference smoothing term, with a smoothing factor λ=0.1.

[0017] Preferably, the comprehensive score calculation formula in step (5) is as follows, where Dm is the average distance, Sm is the directional consistency, Cm is the trajectory coverage, and Bm is the boundary strength.

[0018] Secondly, embodiments of the present invention provide a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0019] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above embodiments.

[0020] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention constructs a unified boundary confidence model by deeply integrating multi-dimensional information such as semantic uncertainty, geometric mutation features and semantic probability gradients. This effectively overcomes the problem that single features are easily affected by noise or scene limitations, and significantly improves the accuracy and robustness of boundary detection, especially in complex urban scenes (such as intersections, construction areas, and vegetation-covered edges).

[0021] 2. This invention introduces a graph neural network to optimize the connection of boundary points. This not only automatically repairs boundary breaks caused by sparse point clouds or occlusion, but also effectively suppresses isolated noise points, generating coherent and complete boundary segments, thus solving the problem of discrete and fragmented output in traditional methods. Simultaneously, through multi-scale polyline extraction and B-spline curve smoothing, the original point set is transformed into a smooth, compact, and geometrically accurate vector boundary, meeting the format requirements of high-precision maps for structured data and can be directly used for path planning and map updating.

[0022] 3. This invention innovatively introduces vehicle or robot trajectories as prior guidance, incorporating trajectory perception mechanisms into each stage of boundary extraction, connection optimization, and final sorting to ensure that the boundaries of key navigation areas are preferentially preserved and accurately represented. Combined with refined boundary type classification and comprehensive evaluation of multiple indicators, the system can output a structured list of boundaries with semantic labels and priorities, greatly improving the efficiency and safety of downstream tasks (such as local path planning and obstacle avoidance).

[0023] 4. This solution has good scalability and engineering practicality, supports feature weight self-learning, online incremental update and multi-resolution output, and is suitable for different sensor configurations and application scenarios. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the overall process of semantic point cloud boundary extraction and vectorization based on multimodal fusion according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the multimodal boundary confidence modeling module according to an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the semantic point cloud boundary type classification based on multi-feature combination according to an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the semantic point cloud boundary connection optimization process based on graph neural networks according to an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the principle of semantic point cloud multi-scale polyline extraction and B-spline curve optimization technology in an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating the principle of the trajectory-aware semantic point cloud boundary importance assessment and ranking technology according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figure 1 , Figure 2 This invention provides a technical solution: a method for semantic point cloud boundary extraction and vectorization based on multimodal fusion, comprising the following steps: (1) Multimodal boundary confidence modeling: For the input 3D point cloud with semantic labels, class probabilities and normal vectors, semantic uncertainty features, geometric mutation features and semantic probability gradient features are extracted respectively, and the boundary confidence score of each point is generated by normalization and weighted fusion. (2) Adaptive boundary point segmentation and type classification: The multi-threshold Otsu algorithm is used to automatically segment strong boundaries, weak boundaries and non-boundary points. The threshold of key areas is dynamically adjusted in combination with trajectory information. Then, based on the degree of geometric change, semantic confusion and height change characteristics, the boundary points are divided into four categories: roadside boundary, wall boundary, complex semantic boundary or weak boundary. (3) Boundary connection optimization based on graph neural network: a weighted graph is constructed with strong boundary points as nodes. The edge weights are fused with feature similarity, spatial distance and trajectory proximity. The connection relationship between points is learned through two layers of graph convolutional network to repair broken boundaries and remove noisy edges. (4) Structured polyline generation and curve optimization: Multi-scale depth-first search is used to extract and merge boundary polylines. For each polyline, a smooth vector boundary is generated by cubic B-spline least squares fitting. (5) Priority ranking of trajectory-guided boundaries: Calculate the comprehensive score of the boundary based on four indicators: average distance, direction consistency, trajectory coverage and boundary strength, and output a structured list of boundaries with type labels and priorities in descending order of scores.

[0027] In the embodiments of the present invention, the semantic uncertainty features in step (1) include the label entropy and the proportion of the largest non-same type labels calculated by the neighborhood label distribution, the geometric mutation features include the normal angle, height step and flatness difference, and the semantic probability gradient features are obtained by calculating the gradient magnitude of the local neighborhood probability matrix using the Sobel operator.

[0028] In an embodiment of the present invention, in step (1), the multimodal features are subjected to min-max normalization and then weighted and fused according to preset weights to generate boundary confidence scores. The preset weights are [0.2, 0.15, 0.25, 0.2, 0.1, 0.1], and the score range is [0, 1].

[0029] In a specific embodiment of the present invention, the multimodal boundary confidence modeling process is as follows: Local neighborhood analysis is performed on the input 3D semantic point cloud to extract three types of features: semantic uncertainty, geometric abrupt change, and semantic probability gradient. The boundary confidence score si∈[0,1] of each point is generated by weighted fusion to quantify the degree of confidence that it belongs to the true boundary.

[0030] The technical solution for the semantic uncertainty analysis module is as follows: given a point cloud and its semantic tags For each point Using a KD-tree to find its k1=30 nearest neighbors Internal statistical label distribution. Calculate label entropy: in The frequency of class c in the neighborhood. Also calculate the maximum proportion of other classes: If point i is an isolated class (with no other classes in its neighborhood), then R i =0. Finally, the semantic feature vector is .

[0031] The technical solution for the geometric mutation detection module is as follows: utilizing point cloud normal vectors. and ground mask (Determined by the "ground" class in the semantic tags), for each point Partition the set of the same type within the nearest neighbors of k2=30. With the collection of outliers .like and Calculate the average normal vector ,like If the data is for the ground class, then the ground point set is fitted using RANSAC. Obtain the global ground normal n g And calculate the height of the steps: otherwise In addition, regarding and Calculate the minimum eigenvalue of the covariance matrix respectively. , Define flatness difference: Geometric eigenvectors are .

[0032] The technical solution for the probability gradient enhancement module is as follows: given semantic probabilities For each point Construct a local 3×3 grid (sorted by spatial coordinates) within its k3=9 nearest neighbors, and extract the probability submatrix for each class c. Calculate the gradient using the Sobel operator: gradient magnitude is Take the maximum value across all categories: Finally, the multimodal features are concatenated into Each dimension is min-max normalized to [0,1] and then weighted according to a preset formula. Fusion: .

[0033] In an embodiment of the present invention, please refer to Figure 1 , Figure 3 In step (2), the adaptive threshold is solved by the multi-threshold Otsu algorithm to find the optimal double threshold, and the threshold of the trajectory-related region is dynamically adjusted by the exponential function.

[0034] In a specific embodiment of the present invention, the adjustment process of the Otsu algorithm is as follows: For confidence score sequence The optimal double threshold is solved using the multi-threshold Otsu algorithm. This maximizes the inter-class variance: in , For the first Class weights and mean The global mean is used. The set of strong boundary points is defined as: If the trajectory is provided Then calculate the shortest distance from point i to the trajectory. And set an adaptive threshold: at this time, .

[0035] The technical solution for the boundary type classification module is as follows: for each Classified according to preset rules: like and 30°), then ; like (45°) and ,but ; like ,but ; otherwise .

[0036] In an embodiment of the present invention, please refer to Figure 1 , Figure 4 In step (3), the graph node features are 6-dimensional multimodal fusion features. The forward propagation of the graph convolutional network uses adjacency matrix with self-loop and degree matrix normalization to output edge retention probability. Edges with retention probability greater than 0.5 constitute the optimized connection graph.

[0037] In a specific embodiment of the present invention, the specific process of step (3) is as follows: by For the node set, construct an 8-nearest neighbor graph. For each edge Calculate the weights: in For point Distance to the trajectory (if the trajectory is not provided, the second item is 1).

[0038] The technical solution for the graph neural network optimization module is as follows: construct a two-layer graph convolutional network (GCN), with the input node feature matrix. Edge index edge weight Forward propagation is as follows: in Given an adjacency matrix with self-loops, Its degree matrix, , For learnable parameters, This is the Sigmoid function. Output. Representing an edge The retention probability, and the final connection decision are: .

[0039] In an embodiment of the present invention, please refer to Figure 1 , Figure 5 In step (4), the multi-scale polyline extraction includes fine scale (retaining all optimized edges), medium scale (retaining edges with weights higher than the median) and coarse scale (retaining edges of a specific type with weights higher than 0.7). During fusion, paths with endpoint distances less than 0.5m are merged through topological consistency verification.

[0040] In an embodiment of the present invention, in step (4), the B spline fitting uses the cumulative chord length normalized parameter node vector, and the control points are solved by least squares optimization. The objective function includes a geometric fidelity term and a third-order difference smoothing term, with a smoothing factor λ=0.1.

[0041] In a specific embodiment of the present invention, step (4) includes the following process: In the figure The above uses depth-first search (DFS) to extract connected paths. Three edge preservation strategies are defined: Fine-scale: All Mesoscale: Only retained The edge; Coarse scale: Only retain those that meet the requirements and The edge; Extract path sets for each scale The final polyline set is obtained by merging lines through topological consistency checks (e.g., merging if the endpoint distance is less than 0.5m). ; The technical solution for the B-spline curve optimization module is as follows: For each broken line ,like Construct a cubic B-spline curve. Let the parameter node vector be... Normalize the accumulated chord lengths: The control points are solved by least squares fitting. , so that: in For cubic B-spline basis functions, It is a third-order difference. This is a smoothing factor. Uniform sampling Points, generate optimized polylines .

[0042] In an embodiment of the present invention, please refer to Figure 1 , Figure 6 The formula for calculating the comprehensive score in step (5) is as follows: Where Dm is the average distance, Sm is the directional consistency, Cm is the trajectory coverage (r=2.0m), and Bm is the boundary strength.

[0043] In a specific embodiment of the present invention, step (5) includes: Based on the trajectory, multi-indicator evaluation and ranking of vector boundaries are performed, and a navigation priority list is output. The technical solution of the multi-indicator comprehensive evaluation module is as follows: for each boundary... Calculate four indicators: average distance: ; Directional consistency: will Divide into segments and calculate the tangent vector for each segment. Segment tangent vector corresponding to the trajectory mean cosine similarity Coverage: , ; Boundary strength: ,in The value is calculated based on the interpolation.

[0044] The overall score is: in Used for normalization.

[0045] The technical solution for the boundary sorting output module is as follows: sort all boundaries according to... Sort in order and output a structured list. It supports downstream systems calling it according to priority.

[0046] Furthermore, embodiments of the present invention also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described in the above embodiments.

[0047] Furthermore, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods described in the above embodiments.

[0048] In summary, the specific implementation of this invention elaborates on the execution logic of each step of the semantic point cloud boundary extraction and vectorization method based on multimodal fusion. The various parameters and thresholds involved, including the multimodal feature fusion weights [0.2, 0.15, 0.25, 0.2, 0.1, 0.1], the endpoint distance setting of multi-scale polyline fusion (0.5m), the edge retention probability threshold of graph connection optimization (0.5), the B-spline fitting smoothing factor λ=0.1, the trajectory coverage radius r=2.0m, and the coefficients in the priority calculation formula, can all be adaptively adjusted according to the characteristics of sensor data, the complexity of the scene, and the system performance requirements in the actual application scenario.

[0049] The specific numerical values ​​listed in this embodiment are merely illustrative demonstrations and clear explanations of the technical solution of the present invention, intended to enable those skilled in the art to accurately reproduce the technical effects of the present invention, and do not constitute the sole limitation of the present invention. It should be understood that any reasonable adjustments, substitutions, or optimizations made to the above parameters within the core conceptual framework of the present invention—"achieving semantic point cloud boundary vectorization through multimodal fusion, graph neural network boundary connection optimization, and trajectory-guided priority sorting"—do not depart from the technical essence and protection scope of the present invention, and should be included within the protection scope defined by the claims of the present invention.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for semantic point cloud boundary extraction and vectorization based on multimodal fusion, characterized in that, Includes the following steps: (1) Multimodal boundary confidence modeling: For the input 3D point cloud with semantic labels, class probabilities and normal vectors, semantic uncertainty features, geometric mutation features and semantic probability gradient features are extracted respectively, and the boundary confidence score of each point is generated by normalization and weighted fusion. (2) Adaptive boundary point segmentation and type classification: The multi-threshold Otsu algorithm is used to automatically segment strong boundaries, weak boundaries and non-boundary points. The threshold of key areas is dynamically adjusted in combination with trajectory information. Then, based on the degree of geometric change, semantic confusion and height change characteristics, the boundary points are divided into four categories: roadside boundary, wall boundary, complex semantic boundary or weak boundary. (3) Boundary connection optimization based on graph neural network: a weighted graph is constructed with strong boundary points as nodes. The edge weights are fused with feature similarity, spatial distance and trajectory proximity. The connection relationship between points is learned through two layers of graph convolutional network to repair broken boundaries and remove noisy edges. (4) Structured polyline generation and curve optimization: Multi-scale depth-first search is used to extract and merge boundary polylines. For each polyline, a smooth vector boundary is generated by cubic B-spline least squares fitting. (5) Priority ranking of trajectory-guided boundaries: Calculate the comprehensive score of the boundary based on four indicators: average distance, direction consistency, trajectory coverage and boundary strength, and output a structured list of boundaries with type labels and priorities in descending order of scores.

2. The method according to claim 1, characterized in that, In step (1), the semantic uncertainty features include the label entropy and the proportion of the largest non-class labels calculated through the neighborhood label distribution, the geometric mutation features include the normal angle, height step and flatness difference, and the semantic probability gradient features are obtained by calculating the gradient magnitude of the local neighborhood probability matrix through the Sobel operator.

3. The method according to claim 1, characterized in that, In step (1), the multimodal features are min-max normalized and then weighted and fused according to preset weights to generate boundary confidence scores.

4. The method according to claim 1, characterized in that, In step (2), the adaptive threshold is solved by the Otsu algorithm to find the optimal double threshold.

5. The method according to claim 1, characterized in that, In step (3), the graph node features are 6-dimensional multimodal fusion features. The forward propagation of the graph convolutional network uses adjacency matrix with self-loop and degree matrix normalization, and outputs edge probability retention to form an optimized connection graph.

6. The method according to claim 1, characterized in that, In step (4), the multi-scale polyline extraction includes fine-scale, medium-scale and coarse-scale. During fusion, paths with endpoint distances less than a set value are merged through topological consistency verification.

7. The method according to claim 1, characterized in that, In step (4), the B-spline fitting uses the cumulative chord length normalized parameter node vector, and the control points are solved by least squares optimization. The objective function includes a geometric fidelity term and a third-order difference smoothing term, with a smoothing factor λ=0.

1.

8. The method according to claim 1, characterized in that, The formula for calculating the overall score in step (5) is as follows: Where Dm is the average distance, Sm is the directional consistency, Cm is the trajectory coverage, and Bm is the boundary strength.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-8.

Images