Point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood space geometric color feature enhanced coding

Abstract

The invention relates to the technical field of point cloud processing and deep learning, and discloses a point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood space geometric color feature enhancement coding, and the method comprises the following steps: fusing two sampling results according to a preset proportion, and obtaining a sampling point set giving consideration to global and local distribution; the geometric and color features are uniformly input into the multilayer perceptron sharing the weight for feature mapping, an enhanced feature vector is generated, the enhanced feature vector is further fused with the center point features, a final code is output, and end-to-end training is achieved. The method has the beneficial effects that through multi-sparseness fusion sampling, the representativeness of dense and sparse point cloud regions can be dynamically balanced, and the segmentation performance of the network in a complex structure or non-uniform distribution scene is remarkably improved; local neighborhood space geometry and color features are fused, the perception ability of the network to fine structures and texture differences is enhanced, and the category confusion phenomenon is effectively reduced.

Description

Technical Field

[0001] This invention relates to the fields of point cloud processing and deep learning technology, specifically to a point cloud semantic segmentation method based on multi-sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding. Background Technology

[0002] With the widespread application of 3D scanning technologies such as LiDAR and structured light, point cloud data has played a crucial role in fields such as autonomous driving, intelligent inspection, and 3D reconstruction. Typical semantic segmentation methods, such as PointNet++, achieve point cloud classification and segmentation through hierarchical sampling, grouping, and feature extraction.

[0003] However, existing methods often employ only a single sparsity strategy during the sampling stage, such as farthest point sampling (FPS), ignoring the dynamic feature distribution of dense and sparse regions within the point cloud. This leads to a decrease in segmentation accuracy in structurally complex or unevenly dense regions. Furthermore, the feature encoding stage focuses primarily on spatial coordinate information, with insufficient fusion of additional features such as point color or intensity, making it easy to confuse categories with small differences in adjacent structures or textures.

[0004] Therefore, further solutions to the above problems are needed. The applicant proposes a point cloud semantic segmentation method based on multi-sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement coding. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding. This solves the problem that existing methods mentioned in the background ignore the dynamic feature distribution of dense and sparse regions within the point cloud, leading to a decrease in segmentation accuracy in structurally complex or unevenly dense regions. Furthermore, the feature encoding stage focuses primarily on spatial coordinate information, with insufficient fusion of additional features such as point color or intensity, making it easy to confuse categories with small differences in adjacent structures or textures.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding. The method includes the following steps: S1: In each layered sampling stage, the local sparsity index is calculated based on the point cloud density, and the global point cloud is divided into two categories of dense and sparse regions at different levels. A lower sparsity FPS is used for dense regions, and a higher sparsity sampling strategy is used for sparse regions. Finally, the two sampling results are fused according to a preset weight to obtain a sampling point set that takes into account both global and local distribution. S2: For each sampling center point, construct a set of K nearest neighbors, extract its three-dimensional coordinate difference, normal vector, local curvature and other geometric features, and calculate color features such as RGB or point intensity histogram; through concatenation or residual connection, input the geometric and color features into a multilayer perceptron (MLP) with shared weights for feature mapping, generate enhanced feature vectors, and then further fuse them with the center point features to output the final code; S3: Integrate steps S1 and S2 into a network based on segmentation frameworks such as PointNet++. In the encoding stage, first perform multiple sparse fusion sampling, then perform enhanced encoding on the sampled points, and connect them to the downstream segmentation head to achieve end-to-end training.

[0007] As a further aspect of the present invention: step S1 specifically comprises: For raw point clouds Sampling is performed to obtain the sampled point set. The process involves four sampling stages; the hierarchical sampling steps based on multiple sparse fusion sampling include: S11. Input the original point cloud into the input terminal of the improved PointNet++, as shown in the following formula: in, For the spatial coordinates of point cloud data, The color value of the point cloud data; S12. Calculate the local density of each point and obtain the sparsity index by normalizing it using the number of neighbors within the statistical radius. The specific steps are as follows: S121. Neighborhood definition: For each point... At different radii Next, find its neighborhood. The formula is as follows: ; S122, Local Density Measurement: Calculates the number of local points at each scale. The formula is as follows: ; S123. Sparsity definition: Sparsity is measured by the reciprocal of density. The formula is as follows: ; S124, Weighted fusion sparsity: Given the scale weights The fusion yields the overall sparsity. The formula is as follows: .

[0008] As a further aspect of the present invention, the hierarchical sampling step based on multiple sparseness fusion sampling also includes: S13, Set threshold ,Will Divided into dense sets and sparse sets ; S14, to Perform low-sparse FPS sampling, sampling ratio ;right Perform high-sparse FPS sampling, sampling ratio The formula is as follows: ; S15. Merge the two sets of sampling results proportionally to obtain the input point set for the next layer, as shown in the following formula: .

[0009] As a further aspect of the present invention: step S2 specifically includes: Local neighborhood enhancement coding is performed on each layer of sampling point set to obtain the final code, which involves a total of 4 enhancement coding stages; the steps of the local neighborhood enhancement coding module include: S21. For each sampling point in the sampling point set In radius Search for the K nearest neighbors to construct a local neighborhood. The formula is as follows: ; S22, For each neighboring point The specific steps for relative spatial coordinate encoding are as follows: S221, Calculation of relative position vectors of neighboring points: For each pair of neighboring points Calculate its relative position vector And perform a high-dimensional mapping on it, as shown in the following formula: in, For the relative positions of neighboring points, a multilayer perceptron, The dimension for location encoding; S222, Euclidean distance encoding: For each pair of neighboring points Calculate its Euclidean distance And perform a high-dimensional mapping on it, as shown in the following formula: in, It is a multilayer perceptron with Euclidean distance. Distance encoding dimension; S223, Manhattan distance coding: for each pair of neighboring points Calculate its Manhattan distance And it is represented in high dimension, as shown in the following formula: in, For Manhattan distance multilayer sensor.

[0010] As a further aspect of the present invention, the local area enhancement coding module step further includes: S23. For each neighboring point The specific steps for color feature encoding are as follows: S231, Calculation of color difference vector between neighboring points: For each pair of neighboring points Calculate its color difference vector And perform a high-dimensional mapping on it, as shown in the following formula: in, For neighborhood point color difference multilayer perceptron, Color encoding dimension; S232, Neighborhood point color mean encoding: Calculate the neighborhood All neighboring points Color mean And perform a high-dimensional mapping on it, as shown in the following formula: in, A multilayer perceptron for the color mean of neighboring points; S233, Neighborhood point color standard deviation encoding: Calculate the neighborhood All neighboring points Color standard deviation And perform a high-dimensional mapping on it, as shown in the following formula: in, A multilayer perceptron with standard deviation of color for neighboring points.

[0011] As a further aspect of the present invention, the local area enhancement coding module step further includes: S24. Perform feature aggregation on the neighborhood point features obtained from the encoding. The specific steps are as follows: S241. Distance Weight Calculation: Assign weights to each neighboring point based on Euclidean distance. The weights are then normalized to obtain the weights. The formula is as follows: in, This is the weight decay rate parameter, used to control the weight decay rate; S242, Weighted Feature Fusion: This involves concatenating relative spatial coordinate codes, color feature codes, and the original geometric and color features, then performing a weighted summation to obtain the aggregated features. The formula is as follows: in, Original geometric and color features, The original feature dimension; S25, Regarding aggregation features MLP mapping and output are performed to obtain the final enhanced code. The formula is as follows: in, For activation function, For batch normalization operations, These are learnable parameters.

[0012] As a further aspect of the present invention: step S3 specifically involves embedding the final encoding into the encoding module of the PointNet++ network, wherein... Let be a set of points containing N points; d is the coordinate dimension of the points; C is the feature dimension of the points; The number of center points K represents the number of points in the local neighborhood after grouping. The predicted scores are for N points; k is the number of categories. For a feature point set that has undergone local neighborhood spatial geometric and color feature enhancement encoding, similarly... This is a set of two feature points that have been enhanced and encoded with local neighborhood spatial geometry and color features, and so on.

[0013] Compared with the prior art, the beneficial effects of the present invention are: 1. By using multiple sparsity fusion sampling, the representativeness of dense and sparse regions of point cloud can be dynamically balanced, significantly improving the segmentation performance of the network in complex or non-uniformly distributed scenarios. 2. By integrating local neighborhood spatial geometry and color features, the network's ability to perceive subtle structural and textural differences is enhanced, effectively reducing category confusion. Attached Figure Description

[0014] Figure 1 This is a flowchart of the point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding of the present invention. Figure 2 This is a schematic diagram of sparse fusion sampling in the point cloud semantic segmentation method based on multiple sparse fusion sampling and local neighborhood spatial geometric color feature enhancement coding of the present invention. Figure 3 This is a schematic diagram of the local neighborhood spatial geometric and color feature enhancement coding of the point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric and color feature enhancement coding of the present invention. Figure 4 This is a diagram of the improved PointNet++ framework of the point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Example Please see Figure 1-4 A point cloud semantic segmentation method based on multiple sparse fusion sampling and local neighborhood spatial geometric color feature enhancement encoding, the method includes the following steps: Step 1: In each layer sampling stage, the local sparsity index is calculated based on the point cloud density, and the global point cloud is divided into two categories of dense and sparse regions at different levels. A lower sparsity FPS is used for dense regions, and a higher sparsity sampling strategy is used for sparse regions. Finally, the two sampling results are fused according to a preset weight to obtain a sampling point set that takes into account both global and local distribution. Specifically, regarding the original point cloud Sampling is performed to obtain the sampled point set. The process involves four sampling stages; the hierarchical sampling steps based on multiple sparse fusion sampling include: 1. Input the original point cloud into the input terminal of the improved PointNet++, as shown in the following formula: in, For the spatial coordinates of point cloud data, The color value of the point cloud data; 2. Calculate the local density of each point and obtain the sparsity index by normalizing it using the number of neighbors within the statistical radius. The specific steps are as follows: 2-1. Neighborhood definition: For each point At different radii Next, find its neighborhood. The formula is as follows: ; 2-2. Local density measurement: Calculate the number of local points at each scale. The formula is as follows: ; 2-3. Sparsity Definition: Sparsity is measured by the reciprocal of density. The formula is as follows: ; 2-4. Weighted fusion sparsity: Given scale weights The fusion yields the overall sparsity. The formula is as follows: ; 3. Set threshold ,Will Divided into dense sets and sparse sets ; 4. Perform low-sparse FPS sampling, sampling ratio ;right Perform high-sparse FPS sampling, sampling ratio The formula is as follows: ; 5. Merge the two sets of sampling results proportionally to obtain the input point set for the next layer, as shown in the following formula: .

[0017] Step 2: For each sampling center point, construct a set of K nearest neighbors, extract its geometric features such as 3D coordinate difference, normal vector, and local curvature, and calculate color features such as RGB or point intensity histogram; through concatenation or residual connection, input the geometric and color features into a multilayer perceptron (MLP) with shared weights for feature mapping, generate enhanced feature vectors, and then further fuse them with the center point features to output the final encoding; Specifically, local neighborhood enhancement coding is performed on each layer of sampling point set to obtain the final code, which involves a total of 4 enhancement coding stages; the steps of the local neighborhood enhancement coding module include: 1. For each sampling point in the sampling point set In radius Search for the K nearest neighbors to construct a local neighborhood. The formula is as follows: ; 2. For each neighboring point The specific steps for relative spatial coordinate encoding are as follows: 2-1. Calculation of relative position vectors of neighboring points: For each pair of neighboring points... Calculate its relative position vector And perform a high-dimensional mapping on it, as shown in the following formula: in, For the relative positions of neighboring points, a multilayer perceptron, The dimension for location encoding; 2-2. Euclidean distance encoding: For each pair of neighboring points Calculate its Euclidean distance And perform a high-dimensional mapping on it, as shown in the following formula: in, It is a multilayer perceptron with Euclidean distance. Distance encoding dimension; 2-3. Manhattan distance encoding: For each pair of neighboring points Calculate its Manhattan distance And it is represented in high dimension, as shown in the following formula: in, For Manhattan distance multilayer perceptron; 3. For each neighboring point The specific steps for color feature encoding are as follows: 3-1. Calculation of color difference vector between neighboring points: For each pair of neighboring points... Calculate its color difference vector And perform a high-dimensional mapping on it, as shown in the following formula: in, For neighborhood point color difference multilayer perceptron, Color encoding dimension; 3-2. Neighborhood point color mean encoding: Calculate the neighborhood All neighboring points Color mean And perform a high-dimensional mapping on it, as shown in the following formula: in, A multilayer perceptron for the color mean of neighboring points; 3-3. Neighborhood point color standard deviation encoding: Calculate the neighborhood All neighboring points Color standard deviation And perform a high-dimensional mapping on it, as shown in the following formula: in, A multilayer perceptron with the color standard deviation of neighboring points; 4. Perform feature aggregation on the neighborhood point features obtained from the encoding. The specific steps are as follows: 4-1 Distance Weight Calculation: Assign weights to each neighboring point based on Euclidean distance. The weights are then normalized to obtain the weights. The formula is as follows: in, This is the weight decay rate parameter, used to control the weight decay rate; 4-2. Weighted Feature Fusion: The relative spatial coordinate code, color feature code, and original geometric and color features are concatenated and then weighted and summed to obtain the aggregated features. The formula is as follows: in, Original geometric and color features, The original feature dimension; 5. Aggregation features MLP mapping and output are performed to obtain the final enhanced code. The formula is as follows: in, For activation function, For batch normalization operations, These are learnable parameters.

[0018] Step 3: Integrate Step 1 and Step 2 into a network based on segmentation frameworks such as PointNet++. In the encoding stage, first perform multiple sparse fusion sampling, then perform enhanced encoding on the sampled points, and connect them to the downstream segmentation head to achieve end-to-end training. Specifically, the final encoding is embedded into the encoding module of the PointNet++ network, where Let be a set of points containing N points; d is the coordinate dimension of the points; C is the feature dimension of the points; The number of center points K represents the number of points in the local neighborhood after grouping. The predicted scores are for N points; k is the number of categories. For a feature point set that has undergone local neighborhood spatial geometric and color feature enhancement encoding, similarly... This is a set of two feature points that have been enhanced and encoded with local neighborhood spatial geometry and color features, and so on.

[0019] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A point cloud semantic segmentation method based on multi-sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding, characterized in that: The method includes the following steps: S1: In each layered sampling stage, the local sparsity index is calculated based on the point cloud density, and the global point cloud is divided into two categories of dense and sparse regions at different levels. A lower sparsity FPS is used for dense regions, and a higher sparsity sampling strategy is used for sparse regions. Finally, the two sampling results are fused according to a preset weight to obtain a sampling point set that takes into account both global and local distribution. S2: For each sampling center point, construct a set of K nearest neighbors, extract its three-dimensional coordinate difference, normal vector, local curvature and other geometric features, and calculate color features such as RGB or point intensity histogram; through concatenation or residual connection, input the geometric and color features into a multilayer perceptron (MLP) with shared weights for feature mapping, generate enhanced feature vectors, and then further fuse them with the center point features to output the final code; S3: Integrate steps S1 and S2 into a network based on segmentation frameworks such as PointNet++. In the encoding stage, first perform multiple sparse fusion sampling, then perform enhanced encoding on the sampled points, and connect them to the downstream segmentation head to achieve end-to-end training.

2. The point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding as described in claim 1, characterized in that: Step S1 is as follows: For raw point clouds Sampling is performed to obtain the sampled point set. It went through a total of 4 sampling stages; The hierarchical sampling steps based on multiple sparse fusion sampling include: S11. Input the original point cloud into the input terminal of the improved PointNet++, as shown in the following formula: in, For the spatial coordinates of point cloud data, The color value of the point cloud data; S12. Calculate the local density of each point and obtain the sparsity index by normalizing it using the number of neighbors within the statistical radius. The specific steps are as follows: S121. Neighborhood definition: For each point... At different radii Next, find its neighborhood. The formula is as follows: ； S122, Local Density Measurement: Calculates the number of local points at each scale. The formula is as follows: ； S123. Sparsity definition: Sparsity is measured by the reciprocal of density. The formula is as follows: ； S124, Weighted fusion sparsity: Given the scale weights The fusion yields the overall sparsity. The formula is as follows: 。 3. The point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding as described in claim 2, characterized in that: The hierarchical sampling step based on multiple sparse fusion sampling also includes: S13, Set threshold ,Will Divided into dense sets and sparse sets ; S14, to Perform low-sparse FPS sampling, sampling ratio ;right Perform high-sparse FPS sampling, sampling ratio The formula is as follows: ； S15. Merge the two sets of sampling results proportionally to obtain the input point set for the next layer, as shown in the following formula: 。 4. The point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding as described in claim 3, characterized in that: Step S2 is as follows: Local neighborhood enhancement coding is performed on each layer of sampling point set to obtain the final code, which involves a total of 4 enhancement coding stages; the steps of the local neighborhood enhancement coding module include: S21. For each sampling point in the sampling point set In radius Search for the K nearest neighbors to construct a local neighborhood. The formula is as follows: ； S22, For each neighboring point The specific steps for relative spatial coordinate encoding are as follows: S221, Calculation of relative position vectors of neighboring points: For each pair of neighboring points Calculate its relative position vector And perform a high-dimensional mapping on it, as shown in the following formula: in, For the relative positions of neighboring points, a multilayer perceptron, The dimension for location encoding; S222, Euclidean distance encoding: For each pair of neighboring points Calculate its Euclidean distance And perform a high-dimensional mapping on it, as shown in the following formula: in, It is a multilayer perceptron with Euclidean distance. Distance encoding dimension; S223, Manhattan distance coding: for each pair of neighboring points Calculate its Manhattan distance And it is represented in high dimension, as shown in the following formula: in, For Manhattan distance multilayer sensor.

5. The point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding as described in claim 4, characterized in that: The local domain enhancement coding module steps also include: S23. For each neighboring point The specific steps for color feature encoding are as follows: S231, Calculation of color difference vector between neighboring points: For each pair of neighboring points Calculate its color difference vector And perform a high-dimensional mapping on it, as shown in the following formula: in, For neighborhood point color difference multilayer perceptron, Color encoding dimension; S232, Neighborhood point color mean encoding: Calculate the neighborhood All neighboring points Color mean And perform a high-dimensional mapping on it, as shown in the following formula: in, A multilayer perceptron for the color mean of neighboring points; S233, Neighborhood point color standard deviation encoding: Calculate the neighborhood All neighboring points Color standard deviation And perform a high-dimensional mapping on it, as shown in the following formula: in, A multilayer perceptron with standard deviation of color for neighboring points.

6. The point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding as described in claim 5, characterized in that: The local domain enhancement coding module steps also include: S24. Perform feature aggregation on the neighborhood point features obtained from the encoding. The specific steps are as follows: S241. Distance Weight Calculation: Assign weights to each neighboring point based on Euclidean distance. And normalize them to obtain the weights. The formula is as follows: in, This is the weight decay rate parameter, used to control the weight decay rate; S242, Weighted Feature Fusion: This involves concatenating relative spatial coordinate codes, color feature codes, and the original geometric and color features, then performing a weighted summation to obtain the aggregated features. The formula is as follows: in, For primitive geometric and color features, The original feature dimension; S25, Regarding aggregation features MLP mapping and output are performed to obtain the final enhanced code. The formula is as follows: in, For activation function, For batch normalization operations, These are learnable parameters.

7. The point cloud semantic segmentation method based on multiple sparseness fusion sampling and local neighborhood spatial geometric color feature enhancement encoding as described in claim 6, characterized in that: Step S3 specifically involves embedding the final encoding into the encoding module of the PointNet++ network, where Let be a set of points containing N points; d is the coordinate dimension of the points; C is the feature dimension of the points; The number of center points K represents the number of points in the local neighborhood after grouping. The predicted scores are for N points; k is the number of categories. For a feature point set that has undergone local neighborhood spatial geometric and color feature enhancement encoding, similarly... This is a set of two feature points that have been enhanced and encoded with local neighborhood spatial geometry and color features, and so on.

Images