A method, apparatus, device, medium, and product for semantic segmentation of three-dimensional point clouds.

Abstract

This application discloses a method, apparatus, device, medium, and product for semantic segmentation of 3D point clouds, relating to the fields of computer vision and 3D point cloud processing technology. The method includes: performing sparse voxelization on 3D point cloud data to establish a mapping relationship between points and voxels; extracting geometric features for each non-empty voxel; aggregating voxel features using a clustering algorithm and determining a k-nearest neighbor graph based on the center coordinates of superpoints; performing parallel encoding processing of feature extraction using a dual-stream encoding strategy to obtain voxel stream local features and superpoint stream global features; projecting the superpoint stream global features back to the voxel dimension and concatenating them with the voxel stream local features; enhancing the fused features based on a self-attention mechanism, and concatenating the enhanced attention features with the fused features for semantic classification; and determining the predicted semantic segmentation result based on the classification category. This application can improve segmentation accuracy while ensuring computational efficiency.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D point cloud processing technology, and in particular to a 3D point cloud semantic segmentation method, apparatus, device, medium and product. Background Technology

[0002] With the widespread application of 3D LiDAR and depth cameras in fields such as autonomous driving and robot navigation, how to efficiently and accurately process large-scale 3D point cloud data has become a research hotspot. Related point cloud processing methods are mainly divided into point-based, voxel-based, and graph-based methods.

[0003] Point-based methods (such as PointNet) directly process the raw point cloud, preserving the original geometric information, but their computational complexity is extremely high when processing large-scale data. Voxel-based methods transform the point cloud into a regular 3D mesh, which is convenient for using 3D convolutional networks, but they suffer from the problem that the computational cost increases cubically with resolution, and although sparse convolution alleviates some memory pressure, it is prone to losing long-range contextual information.

[0004] Graph-based methods construct point clouds into graph structures, which can effectively capture geometric relationships in non-Euclidean spaces. However, constructing a global graph in large-scale scenes incurs enormous computational overhead. Therefore, designing a 3D point cloud semantic segmentation method that balances efficiency and accuracy by combining the efficiency of sparse voxels with the context-awareness of graph structures is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a three-dimensional point cloud semantic segmentation method, apparatus, device, medium, and product that can improve segmentation accuracy while ensuring computational efficiency.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a three-dimensional point cloud semantic segmentation method, including: Acquire 3D point cloud data of the target scene; The three-dimensional point cloud data is subjected to sparse voxelization to establish a mapping relationship between points and voxels, and to obtain a set of non-empty voxels. Geometric features are extracted from each non-empty voxel to obtain voxel features; Clustering algorithms are used to aggregate the voxel features to obtain superpoints, and a k-nearest neighbor graph is determined based on the center coordinates of the superpoints; the k-nearest neighbor graph is used to characterize the topological connectivity between the superpoints. Based on the voxel features and the k-nearest neighbor graph, a dual-stream coding strategy is used for parallel coding processing of feature extraction to obtain voxel stream local features and superpoint stream global features. The global features of the superpoint flow are projected back to the voxel dimension and then spliced ​​and fused with the local features of the voxel flow to obtain the fused features. The fused features are enhanced based on a self-attention mechanism, and the enhanced attention features are concatenated with the fused features for semantic classification to obtain the classification category. The semantic segmentation prediction result is determined based on the classification category.

[0007] In one embodiment, the three-dimensional point cloud data is subjected to sparse voxelization to establish a mapping relationship between points and voxels, and to obtain a set of non-empty voxels, specifically including: Set voxel size and scene boundaries ; Mapping the point cloud coordinates of 3D point cloud data to a discrete voxel mesh space, and calculating the points. Corresponding voxel grid coordinates 3D point cloud data ; The total number of points in the 3D point cloud data; For serial numbers; It is the space of real numbers; This is the floor operator; ; Perform a deduplication operation on all voxel mesh coordinates to obtain a set of non-empty voxels. ; ; For the first One non-empty voxel; For serial numbers; is the total number of non-empty voxels in the set of non-empty voxels.

[0008] In one embodiment, geometric feature extraction is performed on each non-empty voxel to obtain voxel features, specifically including: Compute point set The statistical characteristics include the location mean. and location standard deviation ; ; Pair set Perform principal component analysis to calculate the covariance matrix. : ; set up eigenvalues Determine the geometric distribution characteristics; the geometric distribution characteristics include: linearity. Flatness and sphericity : ; Based on the statistical characteristics, the geometric distribution characteristics, and the point density Determine voxel characteristics ; in, For point set The point in the middle; This is a transpose.

[0009] In one embodiment, based on the voxel features and the k-nearest neighbor graph, a dual-stream coding strategy is used for parallel coding processing of feature extraction to obtain voxel stream local features and superpoint stream global features, specifically including: The dual-stream coding strategy includes voxel stream and superpoint stream; Based on voxel flow, linear projection and nonlinear transformation are performed on the voxel features to extract local voxel flow features: ; in, This is a local feature of voxel flow; It is a set of voxel features; and All are learnable parameters; The super-flow uses a graph neural network to perform message passing and extract global context features based on the k-nearest neighbor graph, resulting in global features of the super-flow; specifically including: Aggregate voxel features to supernodes, for supernodes Exceeding the limit initial features The result obtained through average pooling is: ; For the first Local features of voxel flow corresponding to individual voxel features; For serial numbers; For superpoint indexes or identifiers; For mapping functions; A graph neural network based on the k-nearest neighbor graph and initial features is used. Message passing is performed and global context features are extracted to obtain the global features of the super-point flow; among them, in the first... Layer, Super Point The features are updated as follows: ; In the first Layer, Super Point The corresponding global features of the super-point flow; In the first Layer, Super Point The corresponding global features of the super-point flow; It's over the limit. The set of neighbors in the k-nearest neighbor graph; This is the weight matrix; In the first Layer, number The global features of the superpoint flow corresponding to each superpoint.

[0010] In one embodiment, the global features of the superpoint flow are projected back to the voxel dimension and then concatenated and fused with the local features of the voxel flow to obtain fused features, specifically including: Projecting the superflow global features back to the voxel dimension yields the superflow global features in the voxel dimension. : ; Based on the mapping relationship from point to voxel, Distribute to the corresponding voxels and obtain Corresponding super-point context features : ; For the first One non-empty voxel; This is the superpoint feature matrix output by the GNN; This is the weight matrix; This is the bias term for the projection of the superpoint onto the voxel feature; For the first A mapping function from a non-empty voxel to its superpoint index; For based on The characteristics of superpoints to voxels; Integrating local features of voxel flow with superpoint context features Perform splicing and fusion: ; For the first The eigenvectors of non-empty voxels before fusion; For the first Local features of voxel flow corresponding to individual voxel features; For serial numbers; Determine fusion features : ; For weighting; This is for the fusion bias term.

[0011] In one embodiment, determining the semantic segmentation prediction result based on the classification category specifically includes: pass The function is normalized to obtain the probability distribution of each category, and the label corresponding to the maximum value is taken as the semantic segmentation prediction result: ; ; ; in, This represents the prediction result for semantic segmentation; The original score for semantic segmentation classification; An index for semantic categories; Features of fusion; Attention features; This is the weight matrix for the classification layer; For the bias term of the classification layer; This is the weight matrix for the attention mechanism.

[0012] Secondly, this application provides a three-dimensional point cloud semantic segmentation device, comprising: The data acquisition module is used to acquire 3D point cloud data of the target scene; The sparse voxelization processing module is used to perform sparse voxelization processing on the three-dimensional point cloud data to establish a mapping relationship from points to voxels and obtain a set of non-empty voxels. The feature extraction module is used to extract geometric features from each non-empty voxel to obtain voxel features; An aggregation module is used to aggregate the voxel features using a clustering algorithm to obtain superpoints, and to determine a k-nearest neighbor graph based on the center coordinates of the superpoints; the k-nearest neighbor graph is used to characterize the topological connectivity between the superpoints. The parallel encoding processing module is used to perform parallel encoding processing for feature extraction based on the voxel features and the k-nearest neighbor graph using a dual-stream encoding strategy, to obtain voxel stream local features and superpoint stream global features. The stitching and fusion module is used to project the global features of the superpoint flow back to the voxel dimension and stitch and fuse them with the local features of the voxel flow to obtain fused features. The classification module is used to enhance the fused features based on the self-attention mechanism, and then concatenate the enhanced attention features with the fused features to perform semantic classification to obtain the classification category; The semantic segmentation module is used to determine the semantic segmentation prediction result based on the classification category.

[0013] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional point cloud semantic segmentation method described above.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned three-dimensional point cloud semantic segmentation method.

[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned three-dimensional point cloud semantic segmentation method.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, apparatus, device, medium, and product for semantic segmentation of 3D point clouds. Through sparse voxelization and superpoint aggregation, the data size of 3D point clouds can be significantly reduced. A dual-stream coding strategy is employed for parallel encoding processing of feature extraction, yielding voxel stream local features and superpoint stream global features. The superpoint stream global features are projected back to the voxel dimension and concatenated with the voxel stream local features, effectively capturing long-distance dependencies and spatial structure information in the target scene, thus solving the problem of limited receptive field in traditional voxel methods. An attention mechanism further enhances the weights of key features, significantly improving the boundary accuracy and category consistency of semantic segmentation. Therefore, this application can improve segmentation accuracy while maintaining computational efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a 3D point cloud semantic segmentation method; Figure 2 This is a flowchart illustrating the overall process of a 3D point cloud semantic segmentation method in practical applications. Figure 3 This is a schematic diagram of sparse voxelization and superpoint generation. Figure 4 This is a schematic diagram of a hybrid encoder structure; Figure 5 Visualize the comparison of semantic segmentation results; Figure 6 This is a point cloud height distribution map; Figure 7 This is a schematic diagram showing the distribution of the X-coordinates of the superpoint center; Figure 8 This is a schematic diagram showing the distribution of the Y-coordinates of the superpoint center; Figure 9 This is a schematic diagram of the superpoint graph topology; Figure 10 This is a super-clustering graph; Figure 11 This is a semantic segmentation result image; Figure 12 This is a structural diagram of a 3D point cloud semantic segmentation device; Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] In one exemplary embodiment, such as Figure 1 As shown, a 3D point cloud semantic segmentation method is provided, including: Step 100: Obtain 3D point cloud data of the target scene.

[0022] Step 200: Perform sparse voxelization on the 3D point cloud data to establish a mapping relationship from points to voxels and obtain a set of non-empty voxels.

[0023] Specifically, the 3D point cloud data is subjected to sparse voxelization to establish a mapping relationship between points and voxels, resulting in a set of non-empty voxels. Set voxel size and scene boundaries Mapping the point cloud coordinates of 3D point cloud data to a discrete voxel mesh space, and calculating the points... Corresponding voxel grid coordinates 3D point cloud data ; The total number of points in the 3D point cloud data; For serial numbers; It is the space of real numbers. The rounding operator ensures that coordinates are converted into discrete integer indices to achieve the mapping between points and voxels: .

[0024] Perform a deduplication operation on all voxel mesh coordinates to obtain a set of non-empty voxels. : .

[0025] For the first One non-empty voxel; For serial numbers; is the total number of non-empty voxels in the set of non-empty voxels.

[0026] Step 300: Extract geometric features for each non-empty voxel to obtain voxel features.

[0027] Specifically, geometric features are extracted for each non-empty voxel to obtain voxel features, including: Compute point set Statistical characteristics; statistical characteristics include location mean and location standard deviation .

[0028] .

[0029] Pair set Perform principal component analysis to calculate the covariance matrix. : .

[0030] set up eigenvalues Determine the geometric distribution characteristics; the geometric distribution characteristics include: linearity. Flatness and sphericity : .

[0031] Based on statistical characteristics, geometric distribution characteristics, and point density Determine voxel characteristics ;in, For point set The point in the middle; This is a transpose.

[0032] Step 400: Clustering algorithm is used to aggregate voxel features to obtain superpoints, and the k-nearest neighbor graph is determined based on the center coordinates of the superpoints. The k-nearest neighbor graph is used to represent the topological connectivity between superpoints.

[0033] Step 500: Based on voxel features and the k-nearest neighbor graph, a dual-stream coding strategy is used to perform parallel coding processing for feature extraction, resulting in voxel stream local features and superpoint stream global features.

[0034] Specifically, based on voxel features and the k-nearest neighbor graph, a dual-stream coding strategy is used for parallel coding processing of feature extraction to obtain voxel stream local features and superpoint stream global features, including: The dual-stream coding strategy includes voxel stream and superpoint stream; based on the voxel stream, linear projection and nonlinear transformation are performed on voxel features to extract local voxel stream features. .

[0035] in, This is a local feature of voxel flow; It is a set of voxel features; and All of these are learnable parameters.

[0036] Based on super-flow, a graph neural network is used to perform message passing based on the k-nearest neighbor graph and extract global context features to obtain the global features of the super-flow; specifically including: Aggregate voxel features to supernodes, for supernodes Exceeding the limit initial features The result obtained through average pooling is: .

[0037] For the first Local features of voxel flow corresponding to individual voxel features; For serial numbers; For superpoint indexes or identifiers; This is a mapping function.

[0038] A graph neural network based on the k-nearest neighbor graph and initial features is used. Message passing is performed and global context features are extracted to obtain the global features of the super-point flow; among them, in the first... Layer, Super Point The features are updated as follows: .

[0039] In the first Layer, Super Point The corresponding global features of the super-point flow; In the first Layer, Super Point The corresponding global features of the super-point flow; It's over the limit. The set of neighbors in the k-nearest neighbor graph; This is the weight matrix; In the first Layer, number The global features of the superpoint flow corresponding to each superpoint.

[0040] Step 600: Project the global features of the superpoint flow back to the voxel dimension and merge them with the local features of the voxel flow to obtain the fused features.

[0041] The global features of the superpoint flow are projected back to the voxel dimension and then concatenated and fused with the local features of the voxel flow to obtain the fused features, which specifically include: Projecting the global features of the superflow back to the voxel dimension yields the global features of the superflow in the voxel dimension. : .

[0042] Based on the mapping relationship from point to voxel, Distribute to the corresponding voxels and obtain Corresponding super-point context features : .

[0043] For the first One non-empty voxel; This is the superpoint feature matrix output by the GNN; This is the weight matrix; This is the bias term for the projection of the superpoint onto the voxel feature; For the first A mapping function from a non-empty voxel to its superpoint index; For based on The characteristics of superpoint to voxel.

[0044] Integrating local features of voxel flow with superpoint context features Perform splicing and fusion: .

[0045] For the first The eigenvectors of non-empty voxels before fusion; For the first Local features of voxel flow corresponding to individual voxel features; For serial numbers; Determine fusion features : .

[0046] For weighting; This is for the fusion bias term. for Input symbols in the linear transformation layer.

[0047] Step 700: Enhance the fused features based on the self-attention mechanism, and then concatenate the enhanced attention features with the fused features for semantic classification to obtain the classification category.

[0048] Step 800: Determine the semantic segmentation prediction result based on the classification category.

[0049] The prediction result for semantic segmentation based on the classification category specifically includes: pass The function is normalized to obtain the probability distribution of each category, and the label corresponding to the maximum value is taken as the semantic segmentation prediction result: .

[0050] .

[0051] .

[0052] in, This represents the prediction result for semantic segmentation; The original score for semantic segmentation classification; An index for semantic categories; Features of fusion; Attention features; This is the weight matrix for the classification layer; For the bias term of the classification layer; This is the weight matrix for the attention mechanism.

[0053] like Figure 2 As shown, the operational steps of the method mentioned in this application in practical applications can also be as follows: Step S1: Point cloud acquisition and sparse voxelization.

[0054] The main purpose is to acquire 3D point cloud data of the target scene, and to map continuous coordinates to discrete voxel meshes through sparse voxelization technology, generate non-empty voxel sets and establish mapping relationships to achieve data dimensionality reduction and normalization.

[0055] 3D point cloud data Set voxel size and scene boundaries Mapping point cloud coordinates to a discrete voxel mesh space, and calculating the coordinates of each point... Corresponding voxel grid coordinates : (1) Perform a deduplication operation on all voxel mesh coordinates to obtain a set of M non-empty voxels. And establish a mapping relationship from point to voxel.

[0056] Step S2: Voxel geometric feature extraction.

[0057] This step calculates the statistical characteristics (such as mean and variance) and geometric distribution characteristics (such as linearity and flatness) of the internal point set for each non-empty voxel, and constructs a multi-dimensional feature vector to characterize the geometric properties of the voxel.

[0058] For each non-empty voxel Extract its internal point set The multidimensional geometric feature vector, i.e., voxel features . is the dimension of the voxel feature vector.

[0059] First, calculate the statistical characteristics of the point set, including the location mean. and location standard deviation : (2) Then, principal component analysis (PCA) is performed on the point set to obtain geometric distribution characteristics. The covariance matrix is ​​then calculated. : (3) set up eigenvalues Define linearity Flatness and sphericity : (4) Finally, combining height characteristics (maximum, minimum, range, mean, variance), which are statistical characteristics, geometric distribution characteristics, and point density, This constitutes the final voxel characteristics. .

[0060] Step S3: Superpoint generation and graph construction.

[0061] This step uses a clustering algorithm to aggregate voxels into superpoints to reduce computational scale, and constructs a k-nearest neighbor graph based on the superpoint centers to establish topological connections between superpoints. Figure 3 This is a schematic diagram of sparse voxelization and superpoint generation.

[0062] To reduce computational complexity and construct a structured representation, the voxel feature set... Clustering is performed. The K-Means algorithm is used to divide the voxels into groups. Superpoint The objective function is optimized as follows: (5) in, Indicates belonging to the first A set of voxels of superpoints Use these as cluster centers. Obtain the mapping relationship from voxels to superpoints (map). .

[0063] Constructing a map based on the spatial location of superpoints Calculate the center coordinates of each superpoint (Usually the mean of the coordinates of the included voxels), the k-Nearest Neighbors (k-NN) algorithm is used to find the neighbor nodes of each superpoint and construct the edge set. Forming an edge index matrix .in, edge set The number of edges (i.e., the total number of edges).

[0064] Step S4: Mixed encoding processing. Figure 4 This is a schematic diagram of a hybrid encoder structure.

[0065] This step employs a dual-stream coding strategy. The voxel stream extracts local features through linear transformation (voxel stream local features), while the superpoint stream uses a graph neural network for message passing to extract global context features (superpoint stream global features).

[0066] Parallel encoding using a hybrid encoder: Voxel flow: Extracting local voxel flow features by performing linear projection and nonlinear transformation on voxel features. (6) in, . Dimensions are the local features of the voxel flow.

[0067] Super-point flow: First, voxel features are aggregated into super-point nodes. For super-points... Its initial characteristics The result obtained through average pooling is: (7) Then, a graph neural network (GNN) is used for message passing. In the... layer, The features are updated as follows: (8) Step S5: Cross-modal fusion.

[0068] This step projects the global features of the superpoint flow back to the voxel dimension and concatenates them with the local features of the voxel flow. A fusion layer is then used to generate voxel features that combine local details with global semantics, i.e., fused features.

[0069] To fuse local voxel features (voxel flow local features) and global hyperpoint context features (hyperpoint flow global features), the encoded hyperpoint features are projected back to the voxel dimension: (9) Based on the mapping relationship, the global features of the superpoint flow are distributed to the corresponding voxels, resulting in the superpoint context features corresponding to the non-empty voxels: .

[0070] Integrating local features of voxel flow with superpoint context features Perform splicing and fusion: (10) The final voxel fusion feature, or fusion feature, is generated through the fusion layer. : (11) In formula (11) for The input symbol in the linear transformation layer is essentially equivalent to That is, formula (11) can also be expressed as: .

[0071] Step S6: Semantic segmentation prediction.

[0072] This step utilizes an attention mechanism to enhance the key dimensions of the fused features, calculates the probability distribution of each category through a classifier, and finally predicts the semantic category of each voxel. Figure 5 A visual comparison chart of semantic segmentation results.

[0073] To improve classification performance, an attention mechanism is introduced before the classifier to enhance key features. The attention features are calculated as follows: (12) The original fused features and attention features are concatenated and then input into the fully connected classification layer for computation. Logits : (13) Finally passed Function normalization yields the probability distributions for each category, and the label corresponding to the maximum value is taken as the prediction result: (14) This application has the following advantages: 1. Computational efficiency: By using sparse voxelization and super-point aggregation, the data size of the original point cloud is significantly reduced, enabling the network to process point clouds in large-scale urban scenes.

[0074] 2. Strong structural awareness: By constructing a superpoint graph and introducing GNN, it effectively captures long-distance dependencies and spatial structure information in the scene, solving the problem of limited receptive field in traditional voxel methods.

[0075] 3. Rich feature representation: It integrates local voxel features containing PCA geometric features and superpoint features containing global context, and further enhances the weight of key features through an attention mechanism, which significantly improves the boundary accuracy and class consistency of semantic segmentation.

[0076] System initialization and data generation.

[0077] This embodiment focuses on a 3D semantic segmentation task for large-scale urban scenes. First, a 3D point cloud dataset of the scene to be processed is obtained. The point cloud set is defined as follows: Define the 3D spatial region of interest. Its coordinate range is This represents the boundary coordinates of the scene in three-dimensional space (x, y, z directions). In this embodiment, the scene size is set to a cube region of 100m × 100m × 5m. For the scene in Minimum coordinate value of direction For the scene in Maximum coordinate value of direction For the scene in Minimum coordinate value of direction For the scene in Maximum coordinate value of direction For the scene in Minimum coordinate value of direction For the scene in Maximum coordinate value of the direction.

[0078] Sparse voxelization implementation.

[0079] To reduce data redundancy, the original point cloud is sparsed into voxels. The voxel side length is set to... Define the discretization mapping function. Mapping continuous coordinates to voxel indices: (15) in, To be The continuous coordinates are obtained by discretizing the mapping function. Convert to the corresponding voxel index (i.e., the index value of the voxel to which the point belongs). o = ( () is the origin of the scene. By calculating the voxel indices of all points and taking a unique value, a non-empty voxel set is obtained. ,in The number of non-empty voxels. In this embodiment, through this sparsification process, data from approximately 40,000 points is compressed to approximately several thousand voxels.

[0080] Geometric feature extraction implementation.

[0081] For each non-empty voxel Extract its internal point set Geometric features. To point The continuous coordinates are mapped to the corresponding voxel indices to determine the voxel to which the point belongs; first, the covariance matrix is ​​calculated: (16) Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues. Calculate linearity, flatness, and sphericity. Finally, combine the positional statistics, height features, and PCA geometric features to form... The feature vector is the voxel feature. . .

[0082] Use clustering algorithms to group voxel feature sets Divided into Cluster The clustering process aims to minimize the sum of squared errors (SSE) within clusters. (17) For the first The center of each cluster. Each cluster corresponds to a superpoint node. Spatial center coordinates based on the superpoint. , build Nearest neighbor graph Calculate the Euclidean distance between the centers of any two hyperpoints. For each node Choose the one closest to you. Edges are established between neighboring nodes to generate an adjacency matrix. and edge index set . , The first The node and the first The center coordinates (or position vectors) of each node are used to calculate the Euclidean distance between nodes.

[0083] Voxel stream coding: Maps 14-dimensional voxel features to a high-dimensional feature space. Let the projection matrix be... Voxel stream output feature matrix for: (18) Superpoint flow coding: Aggregates voxel features to superpoint nodes. Defines the aggregation function. For average pooling. Initial eigenma matrix of the superpoint. for: (19) Let be the set of voxels contained in the k-th superpoint, that is, all voxels belonging to the k-th superpoint. This represents the number of voxels contained in this superpoint.

[0084] Feature propagation is performed using a graph neural network. In the l-th layer, the node feature update rule is: (20) in, To add a self-loop adjacency matrix, This is the corresponding degree matrix. For example, tanh is an activation function. Let be the adjacency matrix of the superpoint graph; It is an identity matrix. In the first Layer, based on the superpoint initial feature matrix Determined global characteristics of superpoint flow.

[0085] Cross-modal fusion: Projecting superpoint features back into voxel space and fusing them. First, the superpoint features... Projecting onto the voxel dimension yields the corresponding global features. : (twenty one) Based on the mapping relationship, features are distributed to voxels. The local features of the voxels are concatenated with the global features of the superpoints and input into the fusion layer, resulting in the corresponding fused features. It can be transformed into: (twenty two) in, This indicates a vector concatenation operation, where the merged vector retains a 128-dimensional dimension.

[0086] Finally, semantic classification is performed based on the fused features. First, attention-enhanced features are calculated. : (twenty three) The original features and attention features are concatenated and then input into a fully connected classification layer for classification to obtain the classification result. : (twenty four) in, , This represents the number of semantic categories (12 categories in this example). The final voxel label prediction result. for: (25) That is, according to formulas (24) and (25), the corresponding transformations can be obtained. .

[0087] This embodiment provides a multi-dimensional visualization analysis of the processing results: Point cloud height distribution analysis: such as Figure 6 As shown, the point cloud height distribution histogram shows that most points are concentrated in the 0-2 meter range, which is consistent with the characteristic distribution of ground point clouds. At the same time, a small number of points are distributed in higher areas, corresponding to structures such as buildings and poles.

[0088] Superpoint spatial distribution analysis: Figure 7 and Figure 8 The distribution of hyperpoint centers on the XY plane is shown. The frequencies on both axes exhibit an approximately normal distribution, indicating that hyperpoints are uniformly distributed along the X and Y axes. It is evident that hyperpoints are mainly concentrated in the central region of the scene, consistent with the characteristic of concentrated buildings in urban scenes. Superpoint graph structure visualization: Figure 9 The structure of a hypergraph is shown, where nodes represent hypercenters and edges represent connections between hyperpoints. It is evident that the hypergraph forms a connected topological structure, effectively capturing spatial relationships within a scene.

[0089] Super-point clustering results: Figure 10 The image shows a top-down view of the superpoint clustering, with different gray levels representing different superpoints. It can be seen that the superpoints successfully group voxels with similar geometric features together, forming distinct cluster regions.

[0090] Semantic segmentation results: Figure 11 The final semantic segmentation results are shown, with different gray levels representing different semantic categories. The results demonstrate that this invention can accurately distinguish different semantic regions such as ground and buildings, with clear segmentation boundaries and good category consistency.

[0091] Based on the above experimental results, this invention achieves efficient and accurate understanding of complex 3D scenes and verifies the effectiveness of the hybrid representation method based on sparse voxels and superplots.

[0092] In one exemplary embodiment, such as Figure 12 As shown, a three-dimensional point cloud semantic segmentation device is provided, comprising: The data acquisition module is used to acquire 3D point cloud data of the target scene.

[0093] The sparse voxelization processing module is used to perform sparse voxelization processing on the three-dimensional point cloud data to establish a mapping relationship from points to voxels and obtain a set of non-empty voxels.

[0094] The feature extraction module is used to extract geometric features from each non-empty voxel to obtain voxel features.

[0095] The aggregation module is used to aggregate the voxel features using a clustering algorithm to obtain superpoints, and to determine a k-nearest neighbor graph based on the center coordinates of the superpoints; the k-nearest neighbor graph is used to characterize the topological connection relationship between the superpoints.

[0096] The parallel encoding processing module is used to perform parallel encoding processing for feature extraction based on the voxel features and the k-nearest neighbor graph using a dual-stream encoding strategy, to obtain voxel stream local features and superpoint stream global features.

[0097] The splicing and fusion module is used to project the global features of the superpoint flow back to the voxel dimension and splice and fuse them with the local features of the voxel flow to obtain fused features.

[0098] The classification module is used to enhance the fused features based on the self-attention mechanism, and then concatenate the enhanced attention features with the fused features to perform semantic classification to obtain the classification category.

[0099] The semantic segmentation module is used to determine the semantic segmentation prediction result based on the classification category.

[0100] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 13 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores 3D point cloud semantic segmentation data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a 3D point cloud semantic segmentation method.

[0101] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0103] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0104] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0107] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logic devices, etc., and are not limited to these.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A three-dimensional point cloud semantic segmentation method, characterized in that, include: Acquire 3D point cloud data of the target scene; The three-dimensional point cloud data is subjected to sparse voxelization to establish a mapping relationship between points and voxels, and to obtain a set of non-empty voxels. Geometric features are extracted for each non-empty voxel to obtain voxel features; Clustering algorithms are used to aggregate the voxel features to obtain superpoints, and a k-nearest neighbor graph is determined based on the center coordinates of the superpoints; the k-nearest neighbor graph is used to characterize the topological connectivity between the superpoints. Based on the voxel features and the k-nearest neighbor graph, a dual-stream coding strategy is used for parallel coding processing of feature extraction to obtain voxel stream local features and superpoint stream global features. The global features of the superpoint flow are projected back to the voxel dimension and then spliced ​​and fused with the local features of the voxel flow to obtain the fused features. The fused features are enhanced based on a self-attention mechanism, and the enhanced attention features are concatenated with the fused features for semantic classification to obtain the classification category. The semantic segmentation prediction result is determined based on the classification category; Based on the voxel features and the k-nearest neighbor graph, a dual-stream coding strategy is used for parallel coding processing of feature extraction to obtain voxel stream local features and superpoint stream global features, specifically including: The dual-stream coding strategy includes voxel stream and superpoint stream; Based on voxel flow, linear projection and nonlinear transformation are performed on the voxel features to extract local voxel flow features: ； in, This is a local feature of voxel flow; It is a set of voxel features; and All are learnable parameters; The super-flow uses a graph neural network to perform message passing and extract global context features based on the k-nearest neighbor graph, resulting in global features of the super-flow; specifically including: Aggregate voxel features to supernodes, for supernodes Exceeding the limit initial features The result obtained through average pooling is: ； For the first Local features of voxel flow corresponding to individual voxel features; For serial numbers; For superpoint indexes or identifiers; For mapping functions; A graph neural network based on the k-nearest neighbor graph and initial features is used. Message passing is performed and global context features are extracted to obtain the global features of the super-point flow; among them, in the first... Layer, Super Point The features are updated as follows: ； In the first Layer, Super Point The corresponding global features of the super-point flow; In the first Layer, Super Point The corresponding global features of the super-point flow; It's over the limit. The set of neighbors in the k-nearest neighbor graph; This is the weight matrix; In the first Layer, number The global features of the superpoint flow corresponding to each superpoint.

2. The three-dimensional point cloud semantic segmentation method according to claim 1, characterized in that, The 3D point cloud data is subjected to sparse voxelization to establish a mapping relationship between points and voxels, and to obtain a set of non-empty voxels, specifically including: Set voxel size and scene boundaries ; Mapping the point cloud coordinates of 3D point cloud data to a discrete voxel mesh space, and calculating the points. Corresponding voxel grid coordinates 3D point cloud data ; The total number of points in the 3D point cloud data; For serial numbers; It is the space of real numbers; This is the floor operator; ； Perform a deduplication operation on all voxel mesh coordinates to obtain a set of non-empty voxels. ; ； For the first One non-empty voxel; For serial numbers; is the total number of non-empty voxels in the set of non-empty voxels.

3. The three-dimensional point cloud semantic segmentation method according to claim 1, characterized in that, Geometric features are extracted for each non-empty voxel to obtain voxel features, specifically including: Compute point set The statistical characteristics include the location mean. and location standard deviation ; ； Pair set Perform principal component analysis to calculate the covariance matrix. : ； set up eigenvalues Determine the geometric distribution characteristics; the geometric distribution characteristics include: linearity. Flatness and sphericity : ； Based on the statistical characteristics, the geometric distribution characteristics, and the point density Determine voxel characteristics ; in, For point set The point in the middle; This is a transpose.

4. The three-dimensional point cloud semantic segmentation method according to claim 1, characterized in that, The global features of the superpoint flow are projected back to the voxel dimension and then concatenated and fused with the local features of the voxel flow to obtain fused features, specifically including: Projecting the superflow global features back to the voxel dimension yields the superflow global features in the voxel dimension. : ； Based on the mapping relationship from point to voxel, Distribute to the corresponding voxels and obtain Corresponding super-point context features : ； For the first One non-empty voxel; This is the superpoint feature matrix output by the GNN; This is the weight matrix; This is the bias term for the projection of the superpoint onto the voxel feature; For the first A mapping function from a non-empty voxel to its superpoint index; For based on The characteristics of superpoints to voxels; Integrating local features of voxel flow with superpoint context features Perform splicing and fusion: ； For the first The fusion-before-fusion feature vectors of 1 non-empty voxels; For the first Local features of voxel flow corresponding to individual voxel features; For serial numbers; Concatenate the feature vectors; Determine fusion features : ； For weighting; For fusion bias term; for Input symbols in the linear transformation layer.

5. The three-dimensional point cloud semantic segmentation method according to claim 1, characterized in that, The semantic segmentation prediction result is determined based on the classification category, specifically including: pass The function is normalized to obtain the probability distribution of each category, and the label corresponding to the maximum value is taken as the semantic segmentation prediction result: ； ； ； in, This represents the prediction result for semantic segmentation; The original score for semantic segmentation classification; An index for semantic categories; Features of fusion; Attention features; This is the weight matrix for the classification layer; For the bias term of the classification layer; This is the weight matrix for the attention mechanism.

6. A three-dimensional point cloud semantic segmentation device, characterized in that, include: The data acquisition module is used to acquire 3D point cloud data of the target scene; The sparse voxelization processing module is used to perform sparse voxelization processing on the three-dimensional point cloud data to establish a mapping relationship from points to voxels and obtain a set of non-empty voxels. The feature extraction module is used to extract geometric features from each non-empty voxel to obtain voxel features; An aggregation module is used to aggregate the voxel features using a clustering algorithm to obtain superpoints, and to determine a k-nearest neighbor graph based on the center coordinates of the superpoints; the k-nearest neighbor graph is used to characterize the topological connectivity between the superpoints. The parallel encoding processing module is used to perform parallel encoding processing for feature extraction based on the voxel features and the k-nearest neighbor graph using a dual-stream encoding strategy, to obtain voxel stream local features and superpoint stream global features. The stitching and fusion module is used to project the global features of the superpoint flow back to the voxel dimension and stitch and fuse them with the local features of the voxel flow to obtain fused features. The classification module is used to enhance the fused features based on the self-attention mechanism, and then concatenate the enhanced attention features with the fused features to perform semantic classification to obtain the classification category; A semantic segmentation module is used to determine the semantic segmentation prediction result based on the classification category; Based on the voxel features and the k-nearest neighbor graph, a dual-stream coding strategy is used for parallel coding processing of feature extraction to obtain voxel stream local features and superpoint stream global features, specifically including: The dual-stream coding strategy includes voxel stream and superpoint stream; Based on voxel flow, linear projection and nonlinear transformation are performed on the voxel features to extract local voxel flow features: ； in, This is a local feature of voxel flow; It is a set of voxel features; and All are learnable parameters; The super-flow uses a graph neural network to perform message passing and extract global context features based on the k-nearest neighbor graph, resulting in global features of the super-flow; specifically including: Aggregate voxel features to supernodes, for supernodes Exceeding the limit initial features The result obtained through average pooling is: ； For the first Local features of voxel flow corresponding to individual voxel features; For serial numbers; For superpoint indexes or identifiers; For mapping functions; A graph neural network based on the k-nearest neighbor graph and initial features is used. Message passing is performed and global context features are extracted to obtain the global features of the super-point flow; among them, in the first... Layer, Super Point The features are updated as follows: ； In the first Layer, Super Point The corresponding global features of the super-point flow; In the first Layer, Super Point The corresponding global features of the super-point flow; It's over the limit. The set of neighbors in the k-nearest neighbor graph; This is the weight matrix; In the first Layer, number The global features of the superpoint flow corresponding to each superpoint.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the three-dimensional point cloud semantic segmentation method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional point cloud semantic segmentation method according to any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional point cloud semantic segmentation method according to any one of claims 1-5.

Images