A 3D point cloud semantic segmentation network that integrates normalization and residual connectivity

By constructing a directed graph and introducing a 3D point cloud semantic segmentation network with normalization, channel attention, spatial attention, and residual connections, the shortcomings of existing methods in local and global feature extraction are addressed, thereby improving the accuracy and stability of point cloud segmentation.

CN118351298BActive Publication Date: 2026-11-17CHONGQING UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410236027.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-01
Publication Date
2026-11-17
Estimated Expiration
2044-03-01

AI Technical Summary

Technical Problem

Existing 3D point cloud semantic segmentation methods struggle to effectively capture local and global features when processing point cloud data, resulting in insufficient segmentation accuracy. In particular, when dealing with point clouds that are non-uniform and have varying density, the models lack robustness and stability.

Method used

Design a 3D point cloud semantic segmentation network that integrates normalization and residual connections. Construct a directed graph using KNN and dilated KNN algorithms, extract local features by combining a normalization module and channel attention mechanism, and extract global features by using MLP and spatial attention mechanism. Improve feature representation capability by utilizing residual connections and feature aggregation.

Benefits of technology

It improves the accuracy and stability of point cloud segmentation, enhances the robustness and generalization ability of the model, and can better capture the local and global features of point clouds, thereby improving segmentation accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118351298B_ABST
    Figure CN118351298B_ABST
Patent Text Reader

Abstract

This invention relates to the field of point cloud semantic segmentation accuracy technology, and particularly to a three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections. The steps are as follows: Input an F-dimensional point cloud X with N points; use the KNN algorithm and the extended KNN algorithm to segment point x... i Connect neighboring points x j Two directed graphs representing the local structure of the point cloud were constructed, and edge features were extracted to obtain the edge feature e. ij The invention provides a 3D point cloud semantic segmentation network that integrates normalization and residual connections. By improving the feature extraction process, two branches are introduced to handle local and global feature extraction respectively. The residual concept is used to sum the features as input to the next feature extraction module. The NIBNConv module is used for local feature extraction, while high-dimensional features are extracted directly using an MLP algorithm. After extraction, the final segmentation result is obtained by incorporating a spatial attention mechanism and a fully connected layer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of point cloud semantic segmentation accuracy technology, and in particular to a three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections. Background Technology

[0002] Semantic segmentation, as one of the fundamental tasks of computer vision, has attracted much attention from researchers and is a key step in understanding 3D scenes. When studying an object in 3D space, shape and position information are crucial, and researchers pay more attention to its appearance features, meaning they only need to focus on point cloud information. Point clouds are one form of 3D data representation, and high-level semantic understanding of the spatial information describing point cloud data is receiving increasing attention from researchers.

[0003] The point-wise MLP method utilizes a shared MLP as the basic unit of the network, while introducing some specialized networks to obtain a broader and richer local structure. PointNet++ groups points from different layers and continuously learns from larger local regions. This network proposes a multi-scale and multi-resolution grouping method to address the problems of point cloud inhomogeneity and density variations.

[0004] Engelmann et al. used K-means clustering to define the world space and KNN to define two neighborhoods in the feature space, introducing point-to-point distance loss and centroid loss to achieve typical feature learning.

[0005] randLA-Net proposes a new feature aggregation module that captures and preserves local geometric features by sampling random points. This network achieves very high efficiency in terms of memory and computation.

[0006] PointWeb uses networks to find relationships between every pair of points within a local region and proposes an Adaptive Feature Adjustment (AFA) module for information exchange and feature refinement. Other networks (Edgeconv and NetVLAD) use local-global feature connections to capture both local information and scene-level global features.

[0007] Graph-based methods use graph networks to capture the shape and geometry of point clouds. Superpoint Graph groups point clouds based on their simple shape and the correlation of superpoints, using the superpoint graph to obtain structural and contextual information applicable to large-scale point clouds, decomposing it into geometrically uniform segmentation, context segmentation, and superpoint embedding. Landrieu and Boussaha address the oversegmentation of point clouds into pure superpoints through a novel supervised framework, using contrast loss to identify boundaries between each object. GAC can segment discriminative features and can also assign attention weights based on spatial location and feature differences across different neighborhoods and feature channels to learn relevant features within the local neighborhood set. pointGCR represents point clouds as acyclic graphs along the channel dimension to obtain global contextual information.

[0008] Besides the methods mentioned above, there are many other approaches. Point convolution methods have proposed efficient convolution operators for point clouds. KPConv determines the convolution weights by using the Euclidean distance to the kernel point, and the kernel point's location is formulated as an optimization problem for optimal coverage in spherical space. They use grid subsampling in each layer to achieve high robustness under point clouds of different densities. 3DCNN-DQN-RNN proposes a dilated point convolution (DPC) operation to aggregate dilated neighborhood features, using K-nearest neighbors for aggregation, and demonstrates through numerous experiments that the receptive field has a significant impact on the performance of the aggregation method.

[0009] Using recurrent neural networks (RNNs) to capture intrinsic contextual features is also a common approach in this field. The GB-RCU network first obtains input-level context by transforming point blocks into multi-scale blocks and grid blocks, and then sequentially inputs features extracted using PointNet into composite units (CUs) and recurrent composite units (RCUs) to obtain output-level context. This network demonstrates the importance of spatial context for improving segmentation. 3P-RNN uses pointwise pyramid pooling (3P) modules to obtain coarse-to-fine local structures and employs bidirectional hierarchical RNNs to obtain long-range spatial dependencies, enabling end-to-end learning.

[0010] Attention mechanisms were first proposed in the field of visual image processing and have since been widely used in Natural Language Processing (NLP). To ensure the order of point cloud data, PCT leverages the inherent order invariance of transformers and uses an offset attention mechanism to perform feature learning. Point-TnT combines local and global attention, enabling each point to effectively pay attention to each other.

[0011] To address this issue, a three-dimensional point cloud semantic segmentation network integrating normalization and residual connections is designed to provide an alternative technical solution. Summary of the Invention

[0012] Therefore, it is necessary to provide a three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections to address the aforementioned technical problems.

[0013] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0014] A 3D point cloud semantic segmentation network integrating normalization and residual connectivity is proposed, with the following steps:

[0015] Input an F-dimensional point cloud X containing N points;

[0016] Based on the KNN algorithm and the extended KNN algorithm, for point x i Connect neighboring points x j Two directed graphs representing the local structure of the point cloud were constructed, and edge features were extracted to obtain the edge feature e. ij ;

[0017] Feature extraction is performed on point cloud data using a normalization module;

[0018] After the MLP operation, a channel attention mechanism is added to obtain the features F1' and F2' of the point cloud data;

[0019] The extracted feature information is concatenated and fed into a multilayer perceptron for feature learning, resulting in a more discriminative feature representation F. E .

[0020] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections provided by the present invention, the edge feature e ij The expression is as follows:

[0021]

[0022] Here, θ is a learnable parameter.

[0023] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connectivity provided by the present invention, the feature extraction of point cloud data through the normalization module is as follows:

[0024] Let f i ∈R d Grouping neighborhood is It contains n points, each with a neighborhood point f i,j Given a d-dimensional vector, the expression for transforming local neighborhood points is as follows:

[0025]

[0026] in, These are all learnable parameters, ε = e -5 To ensure the stability of the values.

[0027] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections provided by the present invention, features are aggregated based on the idea of ​​weighted generalized average pooling in images to obtain corresponding new features F1 and F2, as expressed below:

[0028] F1=α×max(e ij )+(1-α)×avg(e ij )

[0029] F2=α×max(e ij k )+(1-α)×avg(e ij k ).

[0030] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections provided by the present invention, the expressions for the features F1' and F2' of the point cloud data are as follows:

[0031] F1′=SE(MLP(F1))

[0032] F2′=SE(MLP(F2)).

[0033] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connectivity provided by the present invention, the feature representation F E The expression is as follows:

[0034] F E ′=MLP(concat(F1′,F2′)).

[0035] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections provided by the present invention, the original point cloud is directly processed by a multilayer perceptron to extract high-dimensional global features F. M The expression is as follows:

[0036] F M =MLP(x i ).

[0037] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connectivity provided by the present invention, the output of the feature extraction module is F. out The expression is as follows:

[0038] F out =concat(F E,F m )

[0039] During feature extraction, the previous output feature F is used... out-1 With the current output feature F out Perform residual connections as input F for the next feature extraction. in The expression is as follows:

[0040] F in =F out-1 +F out .

[0041] As a preferred embodiment of the three-dimensional point cloud semantic segmentation network that integrates normalization and residual connections provided by the present invention, a spatial attention map M of the same size as the input is obtained through the sigmoid function. s (F), the expression is as follows:

[0042] M s (F)=σ(f conv concat(F1,F2))

[0043] Where: M s (F) represents the spatial attention weights of feature F; F1 and F2 are the results of average pooling and max pooling operations; concat is the concatenation operation; conv is the dilated convolution operation; σ is the sigmoid function.

[0044] As a preferred embodiment of the 3D point cloud semantic segmentation network that integrates normalization and residual connections provided by the present invention, the features at all locations are element-wise multiplied with the input features in the channel dimension, and the context information is aggregated to obtain a new feature F', as shown in the following expression:

[0045] F′=x×M s (F)

[0046] Here, F' is the optimized feature map, and its output dimension is the same as the input feature F.

[0047] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0048] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0049] This invention provides a 3D point cloud semantic segmentation network that integrates normalization and residual connections. The feature extraction process is improved by having two branches, one for local feature extraction and the other for global feature extraction. The residual mechanism is used to sum the features as input to the next feature extraction module. Local feature extraction utilizes the NIBNConv module, which employs KNN (Nearest Neighbor) and dilated KNN algorithms to construct a dynamic graph and uses a feature aggregation module and channel attention mechanism to learn more important features. Global feature extraction directly uses the MLP algorithm to extract high-dimensional features. After extraction, the segmentation network further utilizes fully connected layers and channel attention mechanisms to better mine and utilize the internal correlations between features, thereby improving the ability to select and utilize high-quality features. Finally, the final segmentation result is obtained by incorporating spatial attention mechanisms and fully connected layers. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a partial schematic diagram of the present invention;

[0052] Figure 2 This is a detailed structural diagram of the NIBNConv of the present invention;

[0053] Figure 3 This is a schematic diagram of the channel attention (SE) module of the present invention;

[0054] Figure 4 This is a schematic diagram of the spatial attention module of the present invention;

[0055] Figure 5 This is a rendering of the invention;

[0056] Figure 6 This is a flowchart of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0059] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0061] Reference Figures 1-6 A semantic segmentation network for three-dimensional point clouds that integrates normalization and residual connections.

[0062] 1. NIBNCon module

[0063] Input an F-dimensional point cloud X containing N points;

[0064] Using the KNN algorithm and the dilated KNN algorithm for point x i Connect neighboring points x j Two directed graphs representing the local structure of the point cloud were constructed. After constructing the directed graphs, edge features were extracted, and a new edge feature e was proposed. ij :

[0065]

[0066] Where θ is a learnable parameter, which is a nonlinear function obtained using a shared MLP.

[0067] When processing point cloud data, due to the differences in sparsity between local regions and the irregular geometric structure of the point cloud itself, simple deep multilayer perceptron (MLP) structures may not be able to fully capture the complex features of the point cloud data, thus reducing the accuracy and stability of the model and making the model lack sufficient robustness. Furthermore, different local regions possess different geometric features, which may require different feature extraction methods for modeling; however, a shared MLP structure is difficult to meet this need. To address the feature extraction problem of point cloud data, a normalization (NIBNConv) module was designed to solve this challenge.

[0068] Let f i ∈R d Grouping neighborhood is It contains n points, each with a neighborhood point f i,j It is a d-dimensional vector. The local neighborhood points are transformed using the following formula:

[0069]

[0070] in, These are all learnable parameters, ε = e -5This ensures numerical stability. It's important to note that σ here is a scalar, playing the role of measuring and describing the characteristic deviations between all local groups and channels. This property allows for the transformation of points through normalization operations while preserving their original geometric properties.

[0071] Meanwhile, during the implementation of the multilayer perceptron, it was found that traditional batch normalization performs statistics on each batch during training. However, in practical applications, due to memory limitations or other reasons, it is sometimes necessary to use smaller batches than the traditional batch size for training. When the batch size is small, traditional batch normalization may face the problem of inaccurate statistical information in the case of small batches or single samples, thus affecting the model's learning and generalization ability.

[0072] To address this issue and improve performance in mini-batch or single-sample scenarios, a method combining instance-batch normalization and batch normalization is introduced. This approach integrates the advantages of both, controlling their weights through a specified proportional parameter. This allows for the provision of relatively accurate statistics in mini-batch or single-sample scenarios while maintaining the original geometric characteristics, and provides more stable statistics in larger batches. This balance between instance-level and batch-level feature statistics helps improve the model's learning and generalization capabilities.

[0073] The baseline network directly uses max pooling to aggregate the k features extracted from the edge features into a single feature. However, max pooling loses some detailed information. Therefore, this paper borrows the idea of ​​weighted generalized mean pooling from image processing to aggregate the features, resulting in new features F1 and F2, as shown in the following expressions:

[0074] F1=α×max(e ij )+(1-α)×avg(e ij )

[0075] F2=α×max(e ij k )+(1-α)×avg(e ij k )

[0076] Note that GEM pooling is a learnable pooling layer that allows the model to adaptively learn the weights. α is a learnable parameter. F1 represents the features obtained from the k-nn algorithm, and F2 represents the features obtained from the dilated k-nn algorithm.

[0077] To better focus on important feature information, a channel attention mechanism (SE module) is introduced after the MLP operation. This module adaptively learns weights based on the similarity between different points and their features, and weights features with different weights to better represent the features F1' and F2' of the point cloud data, as shown in the following expressions:

[0078] F1′=SE(MLP(F1))

[0079] F2′=SE(MLP(F2))

[0080] The extracted feature information is then concatenated and fed into a multilayer perceptron for feature learning, resulting in a more discriminative feature representation F. E The expression is as follows:

[0081] F E =MLP(concat(F1′,F2′))

[0082] Meanwhile, during the extraction of local features, the construction of a directed graph transforms the entire point cloud into independent groups. However, this operation can easily lose some global information. To compensate for this shortcoming, a multilayer perceptron is used to process the input point cloud, and this operation is used to extract high-dimensional global features F. M The expression is as follows:

[0083] F M =MLP(x i )

[0084] Therefore, the output of the feature extraction module is F. out The expression is as follows:

[0085] F out =concat(F E ,F m )

[0086] It is important to note that during the feature extraction process, the previous output feature F is used... out-1 With the current output feature F out Perform residual connections as input F for the next feature extraction. in The expression is as follows:

[0087] F in =F out-1 +F out

[0088] Segmentation network

[0089] To capture more correlations between points, a spatial attention module is introduced. First, the input features F are subjected to max pooling and average pooling operations. Then, these pooling results are concatenated along the channel dimension and subjected to dilated convolution. Finally, a spatial attention map M of the same size as the input is obtained through the sigmoid function. s (F) performs element-wise multiplication of all positional features with the input features along the channel dimension, thereby selectively aggregating contextual information to obtain a new feature F', as shown in the following expression:

[0090] M s (F)=σ(f conv concat(F1,F2))

[0091] F′=x×M s (F)

[0092] Where: M s (F) represents the spatial attention weights of feature F; F1 and F2 are the results of average pooling and max pooling operations; concat is the concatenation operation; conv is the dilated convolution operation; σ is the sigmoid function. F' is the optimized feature map, whose output dimension is the same as the input feature F. After adjustment by the spatial attention module, more important feature information and location information are enhanced.

[0093] Meanwhile, traditional setups using convolution and dropout operations can lead to the loss of crucial information. To address this issue, an SE module is added to the convolutional layers to selectively highlight important features, thereby improving model performance. Furthermore, residual connections are used to correlate output features and pass them as input features to the next iteration.

[0094] In summary:

[0095] The most important aspect of this embodiment is the design of a novel graph convolutional network architecture, NRA-Net. The designed feature extraction module combines graph-based and point-based methods to extract local and global features from the point cloud, respectively. It then employs residual learning to fuse the extracted features, resulting in significant advantages in feature representation and enriching its capabilities. Furthermore, a feature aggregation module and a channel attention mechanism are used to capture more important information when extracting local features. Finally, to fully explore the relationships between points, a spatial attention module is introduced to extract semantic information between points.

[0096] NIBNConv section: First, the KNN algorithm and the extended KNN algorithm are used for point x. i Connect neighboring points x jTwo directed graphs representing the local structure of the point cloud were constructed. After constructing the directed graphs, edge features were extracted, and a new edge feature e was proposed. ij This paper borrows the concept of weighted generalized average pooling from image processing to aggregate features, obtaining corresponding new features F1 and F2. A normalization (NIBNConv) module is designed to transform local neighborhood points, enabling point transformation while preserving their original geometric properties. Simultaneously, a method combining instance and batch normalization is introduced, integrating the advantages of both and controlling their weights through specified proportional parameters. To better focus on important feature information, a channel attention mechanism (SE module) is introduced after the MLP operation. This module adaptively learns weights based on the similarity between different points and their features, and weights features with different weights to better represent the point cloud data features F1' and F2'. After completing the above operations, the extracted feature information is concatenated and fed into a multilayer perceptron for feature learning, resulting in a more discriminative feature representation F. E Furthermore, it was found that during the extraction of local features, the entire point cloud is divided into independent groups when constructing the directed graph. However, this operation easily loses some global information. To compensate for this shortcoming, a multilayer perceptron is used to directly process the original point cloud and extract high-dimensional global features F. M Therefore, the output of the feature extraction module (NIBNConv module) is F. out .

[0097] Spatial attention component: First, the input features F are subjected to max pooling and average pooling operations. Then, these pooling results are concatenated along the channel dimension and dilated convolution is performed. Finally, a spatial attention map M of the same size as the input is obtained through the sigmoid function. s (F) performs element-wise multiplication of all positional features with the input features along the channel dimension, thereby selectively aggregating contextual information to obtain new features F'.

[0098] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A semantic segmentation method for 3D point clouds that integrates normalization and residual connectivity, characterized in that, The steps are as follows: Input an F-dimensional point cloud X containing N points; Based on the KNN algorithm and the extended KNN algorithm as points and neighboring points Two directed graphs representing the local structure of the point cloud were constructed, and edge features were extracted to obtain edge features. ; Edge features The expression is as follows: ; in, These are learnable parameters; Feature extraction of point cloud data is performed using the normalization module NIBNConv. The steps are as follows: set up Grouping neighborhood is It contains n points, each with a neighborhood point Given a d-dimensional vector, the expression for transforming local neighborhood points is as follows: in, These are all learnable parameters. To ensure numerical stability; σ is a scalar used to measure and describe the characteristic deviations between all local groups and channels; and the normalization module introduces a method that combines instance normalization and batch normalization, controlling the weights of instance normalization and batch normalization through specified scaling parameters; Adding a channel attention mechanism after MLP operations yields features from the point cloud data. , ; The features are aggregated based on the idea of ​​weighted generalized average pooling in images to obtain corresponding new features. , The expression is as follows: ; ; in, It is a learnable parameter; The features obtained by the knn algorithm, Features obtained from the extended knn algorithm; The channel attention mechanism is a SE module, which adaptively learns weights based on the similarity between different points and their corresponding features, and then weights the features with different weights; representing the features of the point cloud data. , The expression is as follows: ; ; The extracted feature information is concatenated and fed into a multilayer perceptron for feature learning, resulting in a more discriminative feature representation. ; Feature representation The expression is as follows: ; High-dimensional global features are extracted by directly processing the original point cloud using a multilayer perceptron. The expression is as follows: ; The output of the feature extraction module is The expression is as follows: ; During feature extraction, the previous output features are... Compared with the output features of this time Residual connections are used as input for the next feature extraction. The expression is as follows: ; It also includes a spatial attention module, which is configured to: perform max pooling and average pooling operations on the input feature F respectively, concatenate the pooling results along the channel dimension and perform dilated convolution operation, and obtain a spatial attention map of the same size as the input through the sigmoid function. The expression is as follows: in: Spatial attention weights for feature F; and It is the result of average pooling and max pooling operations; concat is the concatenation operation; conv is the dilated convolution operation; It is the sigmoid function; The features at all positions are element-wise multiplied with the input features along the channel dimension, and the contextual information is aggregated to obtain new features. The expression is as follows: in, It is the optimized feature map, and its output dimension is the same as the input feature F; The final segmentation result is obtained through spatial attention mechanism and fully connected layer.