An indoor point cloud semantic segmentation method with curvature coding enhancement introduced

Abstract

The application belongs to the field of point cloud data processing, and discloses an indoor point cloud semantic segmentation method with curvature coding enhancement, wherein firstly, local curvature information of each point in original point cloud data is calculated, principal component analysis is performed on local point cloud data, three principal component directions of the local point cloud and corresponding variance data are generated, and curvature information values representing local curvature of the point cloud are calculated. Then, curvature information expansion is performed on the original point cloud data point by point, and information coding with different depths is performed on the expanded data. Finally, the curvature coding and the point cloud downsampling result are spliced, fused with point cloud attention layer information, and finally the neural network obtains point cloud features at each scale through random upsampling and downsampling of the expanded and fused point cloud information, and the features are fused with semantic information at different levels through skip connection, and finally the neural network outputs a semantic segmentation result. The application significantly improves the recognition and differentiation accuracy of indoor objects and structures.

Description

Technical Field

[0001] This invention belongs to the field of point cloud data processing, and particularly relates to an indoor point cloud semantic segmentation method that incorporates curvature coding enhancement. Background Technology

[0002] With the rapid development of computer vision and machine learning technologies, the processing and understanding of 3D spatial data has become increasingly important. Indoor point cloud semantic segmentation, as one of the key technologies in this field, has become a research hotspot in multiple areas such as intelligent robot navigation, augmented reality, interior design, and building information modeling. By analyzing 3D point cloud data in an indoor environment, point cloud semantic segmentation technology can identify and classify various objects in the environment, such as walls, furniture, doors, and windows, thereby providing data support for advanced applications such as navigation, layout planning, and space optimization.

[0003] The complexity and diversity of indoor environments present additional challenges to point cloud processing. Indoor spaces are typically more enclosed and dynamic, with a greater variety of objects arranged more densely and complexly. This necessitates point cloud semantic segmentation algorithms that can more accurately handle spatial constraints and the relationships between objects. To address these challenges, several deep learning-based point cloud processing methods have emerged in recent years. These methods typically utilize neural networks to learn deep features of point cloud data and then perform object recognition and classification based on these features. However, the performance of many existing methods in indoor environments still needs improvement, especially when dealing with complex scenes. Summary of the Invention

[0004] The purpose of this invention is to provide an indoor point cloud semantic segmentation method that incorporates curvature coding enhancement to solve the aforementioned technical problems.

[0005] To address the aforementioned technical problems, the present invention provides a specific technical solution for an indoor point cloud semantic segmentation method that incorporates curvature coding enhancement, as follows:

[0006] An indoor point cloud semantic segmentation method incorporating curvature coding enhancement includes the following steps:

[0007] Step 1: Calculate point-by-point local curvature information for the input raw point cloud data. When calculating point by point, obtain the local point cloud data according to the absolute distance. Use principal component analysis on the local point cloud data to obtain the directions of the three principal components and the corresponding variances.

[0008] Step 2: Sort the principal component data of the local point cloud from Step 1 according to the variance of the corresponding direction, and calculate the minimum variance percentage. The minimum variance percentage is used to characterize the flatness of the point cloud in the local area of ​​that point, and serves as the curvature information data of that point.

[0009] Step 3: Fuse the original point cloud point cloud point data with the point local curvature information value, and perform corresponding neural network encoding for different depths to obtain curvature information fusion encoding values ​​of different dimensions. The input of the curvature encoding layer is seven-dimensional point cloud data with expanded point local curvature information. After passing through two linear layers and an activation layer, the original data is finally upgraded to obtain the output local curvature information encoding.

[0010] Step 4: Input the point cloud data augmented with point-by-point local curvature information into a U-shaped neural network;

[0011] Step 5: In the U-shaped neural network, the output of each encoder layer is combined with the output of the curvature coding layer at the corresponding depth as the input of the new encoder. Finally, the encoder layer and decoder layer extract the information of the point cloud features and output the final point cloud semantic segmentation result.

[0012] Furthermore, steps 1 and 2 include a point cloud local curvature algorithm, the purpose of which is to input the original point cloud coordinates... In this process, point-by-point local curvature information is obtained based on the sampling radius r of the local neighborhood. and the original point cloud input Integration, to obtain

[0013] Furthermore, the steps of the point cloud local curvature algorithm are as follows:

[0014] enter: r;

[0015] Output:

[0016] First, perform point-by-point neighborhood sampling:

[0017]

[0018] calculate mean μ (i) :

[0019]

[0020] calculate covariance Σ (i) :

[0021]

[0022] Solving for the eigenvalue λ (i) and eigenvector v (i) :

[0023] Σ (i) v (i) =λ(i) v (i) ;

[0024] Calculate curvature information C r :

[0025]

[0026] Obtain the fusion curvature information P c :

[0027]

[0028] Where, N r It represents the number of points contained in the point-by-point envelope sphere of the point cloud; the superscript (i) indicates that this is the i-th point calculated in the point-by-point calculation process of the point cloud; This indicates that after sorting in ascending order, λ (i) The three eigenvalue components.

[0029] Furthermore, step 3, for different depths of the U-shaped neural network, sequentially applies P... c By performing downsampling and encoding, a series of curvature information codes corresponding to different depths are obtained. The details of downsampling and encoding from layer 2 to layer 5 are as follows:

[0030] The algorithm downsamples at a rate of 0.25, based on P. c Given the three-dimensional spatial coordinates of P, recursively calculate the points farthest apart from each other, and then use the farthest point sampling method to calculate the points P. c The included points are downsampled to match the number of points input to the encoder layer of the U-shaped neural network. After downsampling, the point cloud data with fused curvature information changes as follows: For encoder layers of different depths in a U-shaped neural network, the input X (i) P c (i) Always includes X (i) The exact same set of points, for P c (i) The encoding is accomplished through a curvature coding layer, which contains two biased linear layers, a batch normalization layer, and a ReLU activation layer. The output dimensions of the two linear layers are 8 and 16, respectively. The structure of the curvature coding layer remains the same for different depths of the U-shaped neural network, but the weights are not shared between them. After passing through the outputs of different curvature coding layers, the curvature code E corresponding to each depth is obtained. c (i) .

[0031] Furthermore, the curvature code E corresponding to each depth is obtained. c (i) The process is as follows:

[0032]

[0033] The details of the downsampling and encoding in layer 1 are as follows:

[0034] The curvature fusion information of the first layer is not reduced by sampling from the farthest point, but is used unchanged as the input to the curvature coding layer, corresponding to the input point set of the first layer of the U-shaped neural network, that is:

[0035]

[0036] The further encoding is the same as subsequent layers, using a curvature encoding layer with non-shared weights to encode E. c (1) .

[0037] Furthermore, step 4 includes five encoder layers and five decoder layers. The encoder layer is composed of a downsampling layer, a curvature coding layer, and an attention layer, and the decoder layer is composed of an upsampling layer and an attention layer.

[0038] Furthermore, in step 5, and The input to the i-th layer encoder of the U-shaped neural network is obtained by tensor splicing and fusion.

[0039] Furthermore, the input of the i-th layer encoder of the U-shaped neural network is obtained. The process can be expressed as follows:

[0040]

[0041] As a tensor incorporating curvature coding, the input to an encoder employing a self-attention mechanism undergoes further feature extraction and dimensionality upscaling, thereby outputting the input tensor X for the next encoder iteration. (i+1) Repeat X (i+1) With E c (i+1) Tensor concatenation is performed to fuse curvature encoding information at different depths. The final input is then fed into the decoder. Skip connections are used to fuse the outputs of the decoder and encoder at different levels. Due to the multiple downsampling operations, the point cloud is resampled by linear interpolation before the decoder layer, so that the number of points contained in the deep and shallow information of the skip connections corresponds to each other. After encoding and decoding are completed, the network uses two linear layers to map the point-by-point features back to the target dimension and output the classification result.

[0042] The present invention provides an indoor point cloud semantic segmentation method enhanced by curvature coding, which has the following advantages: By calculating local curvature information point-by-point and combining it with principal component analysis, this invention highlights high-curvature areas such as object edges in complex indoor scenes. Furthermore, by fusing curvature coding at different depths of the neural network, the loss of local geometric information is reduced, significantly improving the accuracy of recognizing and distinguishing indoor objects and structures. This technology has broad application potential in fields such as scene understanding and robot navigation. Attached Figure Description

[0043] Figure 1 This is a flowchart of the indoor point cloud semantic segmentation method based on neural networks according to the present invention;

[0044] Figure 2 This is a schematic diagram of the curvature of the point cloud;

[0045] Figure 3 This is a schematic diagram of the neural network constructed in this invention;

[0046] Figure 4 This is a schematic diagram of the semantic segmentation effect. Detailed Implementation

[0047] To better understand the purpose, structure, and function of this invention, the following detailed description of an indoor point cloud semantic segmentation method incorporating curvature coding enhancement is provided in conjunction with the accompanying drawings.

[0048] The present invention provides an indoor point cloud semantic segmentation method incorporating curvature coding enhancement, comprising the following steps:

[0049] Step 1: Calculate point-by-point local curvature information for the input raw point cloud data. During point-by-point calculation, the local point cloud data is obtained according to the absolute distance. Principal component analysis is used on the local point cloud data to obtain the directions of the three principal components and their corresponding variances.

[0050] Step 2: Sort the principal component data of the aforementioned local point cloud according to the variance of the corresponding direction, and calculate the minimum variance percentage. This data can characterize the flatness of the point cloud in the local area of ​​that point, and serve as the curvature information data of that point.

[0051] The purpose of the point cloud local curvature algorithm is to take the original point cloud coordinates as input. In this process, point-by-point local curvature information is obtained based on the sampling radius r of the local neighborhood. and the original point cloud input Integration, to obtain The algorithm is as follows:

[0052] enter: r;

[0053] Output:

[0054] First, perform point-by-point neighborhood sampling:

[0055]

[0056] calculate mean μ (i) :

[0057]

[0058] calculate covariance Σ (i) :

[0059]

[0060] Solving for the eigenvalue λ (i) and eigenvector v (i) :

[0061] Σ (i) v (i) =λ (i) v (i) ;

[0062] Calculate curvature information C r :

[0063]

[0064] Obtain the fusion curvature information P c :

[0065]

[0066] Where, N r It represents the number of points contained in the point-by-point envelope sphere of the point cloud; the superscript (i) indicates that this is the i-th point calculated in the point-by-point calculation process of the point cloud; This indicates that after sorting in ascending order, λ (i) The three eigenvalue components.

[0067] Step 3: Fuse the original point cloud point cloud data point by point with the point-by-point local curvature information values, and perform corresponding neural network encoding for different depths to obtain curvature information fusion encoding values ​​of different dimensions. The input to the curvature encoding layer is seven-dimensional point cloud data augmented with point-by-point local curvature information. After passing through two linear layers and an activation layer, the original data is finally up-dimensioned to obtain the output local curvature information encoding.

[0068] For different depths of the U-shaped neural network, P is sequentially... c By performing downsampling and encoding, a series of curvature information codes corresponding to different depths are obtained. The details of downsampling and encoding from layer 2 to layer 5 are as follows:

[0069] The algorithm downsamples at a rate of 0.25, based on P. c Given the three-dimensional spatial coordinates of P, recursively calculate the points farthest apart from each other, and then use the farthest point sampling method to calculate the points P. c The included points are downsampled to match the number of points input to the encoder layer of the U-shaped neural network. After downsampling, the point cloud data with fused curvature information changes as follows: For encoder layers of different depths in a U-shaped neural network, the input X (i) P c (i) Always includes X (i) Completely identical set of points.

[0070] For P c (i) The encoding is accomplished through a curvature coding layer. This layer contains two linear layers with biases, a batch normalization layer, and a ReLU activation layer. The output dimensions of the two linear layers are 8 and 16, respectively. The structure of the curvature coding layer remains the same for different depths of the U-shaped neural network, but the weights are not shared between them. After passing through the outputs of different curvature coding layers, the curvature code E corresponding to each depth is obtained. c (i) The process is as follows:

[0071]

[0072] Specifically, the details of the downsampling and encoding in layer 1 are as follows:

[0073] The curvature fusion information of the first layer is not reduced by sampling from the farthest point, but is used unchanged as the input to the curvature coding layer, corresponding to the input point set of the first layer of the U-shaped neural network, that is:

[0074]

[0075] In addition, the further encoding is the same as the subsequent layers, using a curvature encoding layer with non-shared weights to encode E. c (1) .

[0076] Step 4: Input the point cloud data augmented with point-by-point local curvature information into a U-shaped neural network, which includes five encoder layers and five decoder layers. The encoder layer is a combination of a downsampling layer, a curvature coding layer and an attention layer, and the decoder layer is a combination of an upsampling layer and an attention layer.

[0077] Step 5: In the U-shaped neural network, the output of each encoder layer is combined with the output of the curvature coding layer at the corresponding depth as the input of the new encoder. Finally, the encoder layer and decoder layer extract the information of the point cloud features and output the final point cloud semantic segmentation result.

[0078] In step 5, and The input to the i-th layer encoder of the U-shaped neural network is obtained by tensor splicing and fusion. The process can be expressed as follows:

[0079]

[0080] further, As a tensor incorporating curvature coding, the input to an encoder employing a self-attention mechanism undergoes further feature extraction and dimensionality upscaling, thereby outputting the input tensor X for the next encoder iteration. (i+1) Similarly, repeat the process for X. (i +1) With E c (i+1) Tensor concatenation is performed to fuse curvature-encoded information at different depths, and the resulting data is input into the decoder. Skip connections are then used to fuse the outputs of the decoder and encoder at different levels. Due to the multiple downsampling operations, the point cloud is resampled using linear interpolation before the decoder layer, ensuring that the number of points contained in the deep and shallow information of the skip connections corresponds to each other. After encoding and decoding, the network uses two linear layers to map the point-by-point features back to the target dimension, outputting the classification result.

[0081] To verify the effectiveness of this method, we used the indoor point cloud semantic segmentation dataset S3DIS. The flowchart of the entire neural network-based indoor point cloud semantic segmentation method is as follows: Figure 1 As shown.

[0082] Step 1: Calculate the curvature information of the 3D point cloud point by point.

[0083] First, neighborhood data was collected point-by-neighborhood samples of the room in the dataset, with a neighborhood radius of r = 0.02m. Then, principal component analysis was performed on the point-by-point coordinates within the neighborhood to obtain the directions of the three principal components and their corresponding variances. The variance of the third principal component direction was selected, and its proportion was calculated as approximate point cloud curvature information, which was then appended to the original point cloud. A schematic diagram of the calculated local curvature of the chair is shown below. Figure 2 As shown.

[0084] Step 2: Curvature information encoding calculation.

[0085] The point cloud curvature information obtained above is fused with the point cloud coordinates and original features through concatenation, and then input into the curvature information encoding layer. Through the curvature information encoding layer, five curvature information codes corresponding to different depths are output, each with a feature vector dimension of 16. Specifically, the size and feature dimension of the codes in the network change as follows: The rate of change in the number of points corresponds to the downsampling rate of the neural network.

[0086] Step 3: Curvature information encoding and fusion with U-shaped neural network.

[0087] Point cloud data augmented with point-by-point local curvature information is input into a U-shaped neural network, which includes five encoder layers and five decoder layers. The encoder layer consists of a downsampling layer, a curvature encoding layer, and an attention layer, while the decoder layer consists of an upsampling layer and an attention layer. Before the input to each encoder layer, the curvature information is encoded and concatenated with the input data; this concatenated data is then used as the input to the encoder layer to supplement local geometric information. Finally, the U-shaped neural network outputs the indoor point cloud semantic segmentation result. A schematic diagram of the calculated point cloud semantic segmentation is shown below. Figure 4 As shown. Specifically, the encoder layer uses a downsampling factor of 0.25, obtains the 16 nearest points in the local neighborhood of the point cloud using KNN, and extracts features using an attention mechanism. The decoder uses linear interpolation for upsampling and extracts features using an attention mechanism.

[0088] Experimental results show that the proposed method demonstrates outstanding semantic segmentation performance on indoor point cloud datasets, achieving better results than traditional models.

[0089] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for semantic segmentation of indoor point clouds by incorporating curvature coding enhancement, characterized in that, Includes the following steps: Step 1: Calculate point-by-point local curvature information for the input raw point cloud data. When calculating point by point, obtain the local point cloud data according to the absolute distance. Use principal component analysis on the local point cloud data to obtain the directions of the three principal components and the corresponding variances. Step 2: Sort the principal component data of the local point cloud from Step 1 according to the variance of the corresponding direction, and calculate the minimum variance percentage. The minimum variance percentage is used to characterize the flatness of the point cloud in the local area of ​​that point, and serves as the curvature information data of that point. Step 3: Fuse the original point cloud point cloud point data with the point local curvature information value, and perform corresponding neural network encoding for different depths to obtain curvature information fusion encoding values ​​of different dimensions. The input of the curvature encoding layer is seven-dimensional point cloud data with expanded point local curvature information. After passing through two linear layers and an activation layer, the original data is finally upgraded to obtain the output local curvature information encoding. Step 4: Input the point cloud data augmented with point-by-point local curvature information into a U-shaped neural network; Step 5: In the U-shaped neural network, the output of each encoder layer is combined with the output of the curvature coding layer at the corresponding depth as the input of the new encoder. Finally, the encoder layer and decoder layer extract the information of the point cloud features and output the final point cloud semantic segmentation result.

2. The indoor point cloud semantic segmentation method with curvature coding enhancement according to claim 1, characterized in that, Steps 1 and 2 include a point cloud local curvature algorithm, the purpose of which is to input the original point cloud coordinates. In this process, point-by-point local curvature information is obtained based on the sampling radius r of the local neighborhood. and the original point cloud input Fusion, to obtain 3. The indoor point cloud semantic segmentation method with curvature coding enhancement according to claim 2, characterized in that, The steps of the point cloud local curvature algorithm are as follows: enter: Output: First, perform point-by-point neighborhood sampling: calculate mean μ (i) : calculate covariance Σ (i) : Solving for the eigenvalue λ (i) and eigenvector v (i) : S (i) v (i) =λ (i) v (i) ; Calculate curvature information C r : Obtain the fusion curvature information P c : Where, N r It represents the number of points contained in the point-by-point envelope sphere of the point cloud; the superscript (i) indicates that this is the i-th point calculated in the point-by-point calculation process of the point cloud; This indicates that after sorting in ascending order, λ (i) The three eigenvalue components.

4. The indoor point cloud semantic segmentation method with curvature coding enhancement according to claim 2, characterized in that, Step 3, for different depths of the U-shaped neural network, sequentially applies P... c By performing downsampling and encoding, a series of curvature information codes corresponding to different depths are obtained. The details of downsampling and encoding from layer 2 to layer 5 are as follows: The algorithm downsamples at a rate of 0.25, based on P. c Given the three-dimensional spatial coordinates of P, recursively calculate the points farthest apart from each other, and then use the farthest point sampling method to calculate the points P. c The included points are downsampled to match the number of points input to the encoder layer of the U-shaped neural network. After downsampling, the point cloud data with fused curvature information changes as follows: For encoder layers of different depths in a U-shaped neural network, the input X (i) P c (i) Always includes X (i) The exact same set of points, for P c (i) The encoding is accomplished through a curvature coding layer, which contains two biased linear layers, a batch normalization layer, and a ReLU activation layer. The output dimensions of the two linear layers are 8 and 16, respectively. The structure of the curvature coding layer remains the same for different depths of the U-shaped neural network, but the weights are not shared between them. After passing through the outputs of different curvature coding layers, the curvature code E corresponding to each depth is obtained. c (i) .

5. The indoor point cloud semantic segmentation method with curvature coding enhancement according to claim 4, characterized in that, Obtain the curvature code E corresponding to each depth c (i) The process is as follows: The details of the downsampling and encoding in layer 1 are as follows: The curvature fusion information of the first layer is not reduced by sampling from the farthest point, but is used unchanged as the input to the curvature coding layer, corresponding to the input point set of the first layer of the U-shaped neural network, that is: The further encoding is the same as subsequent layers, using a curvature encoding layer with non-shared weights to encode E. c (1) .

6. The indoor point cloud semantic segmentation method with curvature coding enhancement according to claim 1, characterized in that, Step 4 includes five encoder layers and five decoder layers. The encoder layer is a combination of a downsampling layer, a curvature coding layer and an attention layer, and the decoder layer is a combination of an upsampling layer and an attention layer.

7. The indoor point cloud semantic segmentation method with curvature coding enhancement according to claim 1, characterized in that, In step 5, and The input to the i-th layer encoder of the U-shaped neural network is obtained by tensor splicing and fusion.

8. The indoor point cloud semantic segmentation method with curvature coding enhancement according to claim 7, characterized in that, Obtain the input of the encoder of the i-th layer of the U-shaped neural network. The process can be expressed as follows: As a tensor incorporating curvature coding, the input to an encoder employing a self-attention mechanism undergoes further feature extraction and dimensionality upscaling, thereby outputting the input tensor X for the next encoder iteration. (i+1) Repeat X (i+1) With E c (i+1) Tensor concatenation is performed to fuse curvature encoding information at different depths. The final input is then fed into the decoder. Skip connections are used to fuse the outputs of the decoder and encoder at different levels. Due to the multiple downsampling operations, the point cloud is resampled by linear interpolation before the decoder layer, so that the number of points contained in the deep and shallow information of the skip connections corresponds to each other. After encoding and decoding are completed, the network uses two linear layers to map the point-by-point features back to the target dimension and output the classification result.

Images