A point cloud semantic segmentation system and method for objects and indoor scenes

Abstract

The present invention discloses a point cloud semantic segmentation system for objects and indoor scenes, comprising an acquisition module for acquiring target point cloud data; an encoding module for downsampling the point cloud data, introducing a channel space attention module, and extracting features of the point cloud data by utilizing channel and spatial information; a decoding module for upsampling the point cloud data features processed by the encoding module, introducing an edge information perception module, and fusing and reconstructing the features by means of dynamic weighting. The present invention improves the feature extraction capability of the system when processing point cloud data, optimizes the system's understanding and processing of input data, and thus improves the efficiency of extracting valuable features from point cloud data; it is beneficial to retaining edge information of the task, while suppressing irrelevant edge information, reducing the system's sensitivity to noise, better restoring the original point cloud resolution, and improving the system performance and generalization capability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technologies, and in particular to a point cloud semantic segmentation system and method for objects and indoor scenes. Background Art

[0002] With the rapid development of autonomous driving, augmented reality, and medical image analysis, there is growing interest in 3D scene segmentation technology. Point clouds, as a common form of 3D representation, are widely used in 3D computer vision research. Compared with traditional 2D image segmentation, 3D point cloud segmentation can handle more complex scenes, provide a more comprehensive understanding of scenes, provide richer geometric and spatial information, and better handle occlusions. Traditional point cloud segmentation methods, such as those based on model fitting, region growing, and edge-based methods, mostly rely on manually designed geometric features and are limited by prior knowledge, resulting in unstable segmentation results. To gain a deeper understanding of 3D scenes, researchers have proposed machine learning-based point cloud semantic segmentation methods, such as support vector machines and conditional random fields. Although these methods have improved segmentation accuracy and efficiency to a certain extent, they still have limitations when dealing with complex scenes. As deep neural networks have greatly promoted the advancement of computer vision technology, an increasing number of studies have begun to adopt deep learning methods to achieve point cloud semantic segmentation.

[0003] However, 3D deep learning methods remain challenging because point clouds are unevenly distributed and unstructured in 3D space, making it impossible to directly use convolutional neural networks on point clouds. Early research has converted point clouds into regular structures, such as multi-view projection-based methods and voxel-based methods, but these methods suffer from geometric information loss and high computational and storage overhead. To address these problems, technicians in this field have proposed Pointnet, the first deep learning framework directly on unstructured point clouds. It uses a shared multi-layer perceptron model (MLP) to learn point features and a maximum pooling function to directly process point clouds. Pointnet++ proposes a hierarchical downsampling strategy to capture local geometric detail information and multi-scale grouping, which significantly improves the accuracy of point cloud segmentation. It also proposes the PointSIFT network, which stacks and encodes information from 8 key spatial directions, so that the model can capture information in different directions and achieve multi-scale representation. It also proposes the RandLA-Net network, which first introduces a local spatial encoding unit to compensate for the local structural information lost by random sampling, and then uses attention pooling learning and aggregation of neighborhood point features to strengthen important information, and uses dilated residual blocks to increase the receptive field of each point to extract richer features. It also proposes DGCNN to capture the distance information of each point and its neighboring points through EdgeConv, which can more effectively extract local features. In order to capture the local contextual features of point cloud data, recent research work has explored the direct application of explicit convolution kernels in point space. Among them, KPconv utilizes deformable convolution to more accurately capture local information. PointNeXt also revisits Pointnet++, fully exploring its potential by improving its training strategy, capturing more detailed local information and improving segmentation accuracy. Although most of the above methods have advantages in learning local features, they lack the ability to capture long-range correlations, cannot fully understand the global context, and have difficulty adapting to complex scenarios.

[0004] With the revolutionary progress of Transformers in natural language processing, many recent studies have also applied attention mechanisms to point cloud analysis. Using attention mechanisms enables the model to dynamically focus on important features, effectively overcoming the disorder and irregularity of point cloud data. PointASNL proposed by technicians in this field adopts adaptive sampling block AS and local-non-local L-NL block to capture the long-range correlation of sampling points, and more effectively processes noisy point cloud data; it also proposes an attention-based group relationship aggregator module RPNet to more effectively capture local semantics and position relationships; it also proposes Point Cloud Transformer (PCT) to add the vector mapping structure and offset attention structure of domain points in the encoder to enhance the model's ability to learn local context features; it also proposes PointTransformer to use self-attention mechanism to capture local and global features, and more effectively capture geometric information; it also proposes a hierarchical Transformer (Stratified Transformer) that captures long-range contextual relationships, and the hierarchical strategy more effectively enhances the receptive field; it also proposes a dual-branch Transformer (PointCAT) based on cross-attention mechanism to enhance the model's ability to capture long-range relationships; it also proposes a self-positioning point Transformer (SPoTr), which uses a set of self-positioning points to calculate attention weights to improve the scalability of global self-attention. Although these works are very effective in point cloud segmentation, most of them fail to fully consider the importance of multi-scale and edge information, and cannot make good use of spatial channel information and edge information to optimize the accuracy of point cloud segmentation, resulting in the edges of some objects being difficult to segment or the segmentation being not smooth.

[0005] Therefore, those skilled in the art are committed to developing a point cloud semantic segmentation system and method for objects and indoor scenes. Summary of the Invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is: to fully consider the importance of multi-scale information and edge information, to use spatial channel information and edge information to optimize the accuracy of point cloud segmentation, and to solve the problem that the edges of some objects are difficult to segment or the segmentation is not smooth.

[0007] To achieve the above-mentioned object, the present invention provides a point cloud semantic segmentation system for objects and indoor scenes, an acquisition module for acquiring target point cloud data;

[0008] An encoding module is used to downsample the target point cloud data, introduce a channel-space attention module, and simultaneously utilize the channel and spatial information of the channel-space attention module to extract features of the target point cloud data;

[0009] a decoding module, configured to upsample the target point cloud data features processed by the encoding module, introduce an edge information perception module, and fuse and reconstruct the sampled target point cloud data features through a dynamic weighting method;

[0010] Furthermore, the channel space attention module is a combination of channel attention and spatial attention, and the output formula of the channel space attention module is:

[0011]

[0012] in, is the output of the channel space attention module, is the result of the channel attention output, is the result of the spatial attention output, CA is the channel attention, are the three-dimensional coordinates and feature vectors of the target point cloud data, and SA is the spatial attention;

[0013] Furthermore, the downsampling adopts farthest point sampling;

[0014] Furthermore, the result of the channel attention output is expressed as:

[0015]

[0016] in, yes The corresponding point features are is a set of key points, is the channel attention weight, To obtain the query point based on the normalization process Calculate key points The relative position encoding, is a query point feature With the key point features The linear transformation of = ;

[0017] Furthermore, the channel attention weight is calculated by calculating the attention weight between the query point and the key point of each channel, wherein the query point is selected as a point obtained by randomly sampling the original point cloud data and enhancing it with a random vector generated by a learnable parameter to form a recognizable point; the key point is the point retained after sampling the farthest point;

[0018] Furthermore, the channel attention weight is expressed as:

[0019]

[0020] Among them, c is the channel index, For control The smoothness parameter of the function, is a mapping function, a multi-layer perceptron model consisting of 2 linear layers and ReLu, and is a linear change;

[0021] Furthermore, the spatial attention is performed on the Process again and merge the results of the channel attention and the spatial attention. The result of the spatial attention output is expressed as: ;

[0022] Furthermore, the upsampled features are fused in a dynamic weighted manner, feature selection is performed through a nonlinear activation function Relu and a convolutional layer, a weight coefficient is obtained through a Sigmoid activation function, and the features are dynamically weighted using the weight coefficient;

[0023] Furthermore, the edge information perception module is a new module that acts on edge information in the dynamic weighted manner based on a diffusion unit, wherein the diffusion unit is expressed as:

[0024]

[0025] in, is a point set, are represented as the center point and the neighboring points respectively. Represent the features of the center point and the neighboring points respectively, ReLu, normalization, and convolution operations;

[0026] The result output by the edge information perception module is expressed as follows:

[0027]

[0028] in, is the down-sampled feature, The result output by the edge information perception module EIA;

[0029] The present invention provides a point cloud semantic segmentation method for objects and indoor scenes, which obtains target point cloud data; inputs the point cloud data into the point cloud semantic segmentation system, and outputs the segmented point cloud data.

[0030] The point cloud semantic segmentation system and method for objects and indoor scenes proposed in the present invention improves the system's feature extraction capability when processing point cloud data by introducing a channel spatial attention module, and optimizes the system's understanding and processing of input data, thereby improving the efficiency of extracting valuable features from point cloud data; by introducing an edge information awareness (EIA) module, it is beneficial to retain edge information of the task, while suppressing irrelevant edge information, reducing the system's sensitivity to noise, better restoring the original point cloud resolution, and improving system performance and generalization capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the architecture of the semantic segmentation system of the present invention;

[0032] Figure 2 Schematic diagram of the structure of the channel space attention module of the present invention;

[0033] Figure 3 Schematic diagram of the structure of the edge information perception module of the present invention;

[0034] Figure 4 Schematic diagram of visualization results of partial segmentation of the ShapeNetPart dataset according to an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the visualization results of the semantic segmentation of the S3DIS dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0037] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. For clarity, the thickness of components in some places in the drawings is appropriately exaggerated.

[0038] In the field of point cloud segmentation research, Benfenming proposed a point cloud semantic segmentation system (CSEANet) for objects and indoor scenes, which can effectively handle the complexity of point cloud data; the design of CSEANet adopts the most common U-net encoder-decoder structure in current point cloud semantic segmentation, effectively realizing hierarchical feature learning. Figure 1As shown, the architectural diagram of the semantic segmentation system of the present invention includes an acquisition module for acquiring target point cloud data; an encoding module for downsampling the point cloud data, introducing a channel spatial attention module (CSPA), and extracting features of the point cloud data using channel and spatial information; a decoding module for upsampling the point cloud data features processed by the encoding module, introducing an edge information perception module (EIA), and fusing and reconstructing the features through dynamic weighting.

[0039] In a specific embodiment, the encoder is responsible for extracting and abstracting features from the input point cloud. In the encoder, the point cloud data undergoes a series of downsampling operations. This downsampling reduces the data dimensionality, ensuring efficient encoding of multi-scale features while extracting more discriminative feature representations. Farthest Point Sampling (FPS) is used for downsampling to effectively preserve the geometric and spatial distribution characteristics of the point cloud and minimize information loss. However, due to the structural complexity of point cloud data, downsampling can lose local details, resulting in inadequate feature representation and extraction. To improve the integrity of feature extraction and segmentation accuracy, a channel-wise spatial attention module (CSPA) is added after each downsampling step. This allows the system to more comprehensively understand and utilize feature information, enhancing its ability to represent complex data. Point features processed by the spatial channel module are passed to the next processing stage.

[0040] The decoder gradually restores the point cloud to its original resolution through multiple upsampling steps combined with an edge information perception module. Skip connections mitigate the loss of spatial information, facilitating feature fusion and reconstruction. To better understand contextual information, adapt to data changes, and improve segmentation accuracy, an edge information perception module (EIA) is introduced. EIA acts on upsampled features, preserving edge information while suppressing irrelevant edge information, reducing the system's sensitivity to noise, and ultimately restoring the original point cloud resolution, improving system performance and generalization.

[0041] In some embodiments, in order to better utilize spatial channel information to optimize the accuracy of point cloud segmentation, the channel attention CA is combined with the spatial attention SA to propose an efficient global information attention module, called the channel spatial attention module (CSPA), whose structural diagram is shown in the following figure. Figure 2 As shown. CSPA can simultaneously utilize channel and spatial information to better understand the intrinsic structure and spatial relationship of point cloud data, thereby achieving more comprehensive point cloud feature extraction. The CSPA output formula is shown in formula (1):

[0042]

[0043] in, is the output of the CSPA module, is the result of channel attention output, The result of spatial attention output, CA is the attention of the point based on the channel, are the three-dimensional coordinates and feature vectors of the point, and SA is spatial attention.

[0044] It is worth noting that the way CSPA calculates weights between channels is different from standard channel attention. Channel-based attention CA calculates the attention weights between the query point and the key point of each channel. The query point is selected as the point after randomly sampling the original point cloud and enhancing it with a random vector generated by learnable parameters to form a recognizable point. The key point is the point retained after sampling the farthest point. The output of channel attention CA is shown in formula (2):

[0045]

[0046] in, yes The corresponding point features are is a set of key points, is the channel attention weight. In order to better integrate local and global context information, the relative position encoding function is introduced , It is obtained by normalization based on the query point Calculate key points The relative position encoding, A query point feature With key point features The linear transformation of = .

[0047] is the channel attention weight, defined as shown in formula (3):

[0048]

[0049] Among them, c is the channel index, For control The smoothness parameter of the function, is the mapping function, a multi-layer perceptron model (MLP) consisting of 2 linear layers and ReLu, and It is a linear change.

[0050] In some embodiments, in order to more fully process the local details in the point cloud data, spatial attention is added after the channel attention to help the model capture the detailed geometric structure of the local point cloud. In the spatial attention part, the average pooling and maximum pooling operations are first performed on the Process again and merge the results of channel attention and spatial attention to obtain the final feature. The result of spatial attention SA output is shown in the formula:

[0051]

[0052] CSPA improves the system's feature extraction capabilities when processing point cloud data and optimizes the system's understanding and processing of input data, thereby improving the efficiency of extracting valuable features from point cloud data.

[0053] In some embodiments, in order to better understand contextual information and adapt to changes in point cloud feature data, the present invention proposes a dynamic edge information perception module (EIA), the structural diagram of which is shown in the following figure: Figure 3 As shown in the figure. The EIA module acts on the decoder stage after each upsampling, not only dynamically strengthening the edge features that are beneficial to the task or suppressing irrelevant edge information, but also reducing the system's sensitivity to noise and better restoring the original point cloud resolution, thereby improving the system's performance and generalization ability. EIA is based on the idea of ​​the diffusion unit (DU), a new module that acts on edge information in a dynamically weighted manner. The diffusion unit DU is an edge perception unit extended from the classical diffusion theory. The principle of the diffusion unit DU is shown in Equation (5):

[0054]

[0055] in, is a point set, are represented as the center point and the neighboring points respectively. Represent the features of the center point and the neighboring points respectively, It is ReLu, normalization, and convolution operation.

[0056] In some embodiments, the present invention generates new feature representations by convolution operations on the upsampled local features and the features after global average pooling, and fuses the two feature representations. Feature selection is performed through the nonlinear activation function Relu and the convolution layer, and the weight coefficient is obtained through the Sigmoid activation function, and the features are weighted using the weight coefficient. This dynamic weighting method further optimizes the fusion process, helps to restore local details that may be lost during the downsampling process, and retains global information. The EIA module assigns different weights to different features, strengthens features that are important to the task, and suppresses relatively unimportant information, which helps to segment the edge details of the object point cloud. Finally, edge optimization is further achieved through DU to make the information smoother. The principle of the EIA module is shown in formula (6):

[0057]

[0058] in, is the down-sampled feature, This is the result of the EIA output.

[0059] In some embodiments, a point cloud semantic segmentation method for objects and indoor scenes is also provided, including: obtaining target point cloud data; inputting the point cloud data into a point cloud semantic segmentation system such as the one in the above embodiment, and outputting the segmented point cloud data.

[0060] In some embodiments, the performance of the point cloud semantic segmentation system of the present invention is evaluated on benchmark datasets: including the point cloud part segmentation dataset ShapeNetPart for objects and the point cloud semantic segmentation dataset S3DIS for indoor scenes:

[0061] 1) Object part segmentation:

[0062] Dataset: ShapeNetPart is an object-level dataset for part segmentation. It consists of 16,880 shapes from 16 different shape categories, each with 2-6 parts, and 50 part segmentation labels. In our experiments, we followed the dataset partitioning method used by previous researchers, with 14,006 samples as the training set and 2,847 samples as the validation set. On each shape, 2,048 points were randomly sampled.

[0063] Performance Comparison: The performance of our point cloud semantic segmentation system is evaluated using instance IoU, the average of class IoU, and each class IoU. Table 1 lists some of the segmentation results on ShapeNetPart. Compared with recent studies, including PointNet++, our system achieves competitive class and instance IoU scores of 83.5 and 86.1, respectively.

[0064] Table 1. Results of partial segmentation of ShapeNetPart dataset

[0065]

[0066] Visualization: The visualization results of the ShapeNetPart dataset are shown in Figure 4. The comparison of the visualization results shows that the point cloud semantic segmentation system of the present invention is closer to the true value. In terms of equal categories, the present invention has certain advantages.

[0067] 2) Indoor scene segmentation:

[0068] Dataset: S3DIS (Stanford Large-Scale 3D Indoor Spaces Dataset) is a large-scale indoor 3D point cloud dataset provided by Stanford University. The dataset contains 6 large indoor areas with a total of 271 rooms, including classrooms and offices, with a total coverage area of ​​more than 6000m 2 Each room has a point cloud file and a label file. The annotated objects are divided into 13 semantic classes (such as ceiling, floor, wall, door, etc.) for training and evaluating the semantic segmentation system. In this embodiment, following the settings of previous work, Area5 is used as the test set.

[0069] Performance Comparison: The system performance of the proposed point cloud semantic segmentation system was evaluated using overall accuracy (OA), average class accuracy (mAcc), and average instance IoU (intersection over union). The results are shown in Table 2. The proposed point cloud semantic segmentation system and method were tested in S3DIS's most challenging region, region 5. The proposed CSEANet achieved mIoU / mAcc / OA results of 69.8 / 75.4 / 90.5, respectively, demonstrating a significant advantage.

[0070] Table 2 Semantic segmentation results of the S3DIS dataset

[0071]

[0072] Visualization: The semantic segmentation visualization results of the S3DIS dataset are as follows Figure 5As shown in the figure, the first two columns represent the input point cloud and the ground-truth annotations, respectively, while the last two columns represent the predictions of PointNet++ and our CSEANet. A comparison of the visualization results reveals that our point cloud semantic segmentation system is closer to the ground-truth. In terms of classification, our method has certain advantages.

[0073] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible without inventive effort by those skilled in the art. Therefore, any technical solution that can be derived by one skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A point cloud semantic segmentation system for objects and indoor scenes, characterized by: include: Acquisition module, used to obtain target point cloud data; An encoding module is used to downsample the target point cloud data, introduce a channel-space attention module, and simultaneously utilize the channel and spatial information of the channel-space attention module to extract features of the target point cloud data; The channel-space attention module is a combination of channel attention and spatial attention. The output formula of the channel-space attention module is: in, is the output of the channel space attention module, is the result of the channel attention output, is the result of the spatial attention output, CA is the channel attention, are the three-dimensional coordinates and feature vectors of the target point cloud data, and SA is the spatial attention; A decoding module is used to upsample the target point cloud data features processed by the encoding module, introduce an edge information perception module, and fuse and reconstruct the sampled target point cloud data features through dynamic weighting. The specific process is: feature selection is performed through a nonlinear activation function Relu and a convolutional layer, and a weight coefficient is obtained through a Sigmoid activation function, and the weight coefficient is used to perform the dynamic weighting on the features; The edge information perception module is a new module based on a diffusion unit that acts on edge information in a dynamic weighted manner, wherein the diffusion unit is expressed as: in, is a point set, are represented as the center point and the neighboring points respectively. Represent the features of the center point and the neighboring points respectively, ReLu, normalization, and convolution operations; The result output by the edge information perception module is expressed as follows: in, is the down-sampled feature, It is the result output by the edge information perception module EIA.

2. The point cloud semantic segmentation system for objects and indoor scenes according to claim 1, characterized in that The down sampling adopts farthest point sampling.

3. The point cloud semantic segmentation system for objects and indoor scenes according to claim 2, characterized in that: The result of the channel attention output is expressed as: in, yes The corresponding point features are is a set of key points, is the channel attention weight, To obtain the query point based on the normalization process Calculate key points The relative position encoding, is a query point feature With the key point features The linear transformation of = .

4. The point cloud semantic segmentation system for objects and indoor scenes according to claim 3, characterized in that: The channel attention weight is calculated by calculating the attention weight between the query point and the key point of each channel. The query point is selected as a point after random sampling of the original point cloud data, and is enhanced by a random vector generated by a learnable parameter to form a recognizable point; the key point is the point retained after sampling the farthest point.

5. The point cloud semantic segmentation system for objects and indoor scenes according to claim 4, characterized in that: The channel attention weight is expressed as: Among them, c is the channel index, For control The smoothness parameter of the function, is a mapping function, a multi-layer perceptron model consisting of 2 linear layers and ReLu, and It is a linear change.

6. The point cloud semantic segmentation system for objects and indoor scenes according to claim 5, characterized in that: The spatial attention is performed on the Process again and merge the results of the channel attention and the spatial attention. The result of the spatial attention output is expressed as: .

7. A point cloud semantic segmentation method for objects and indoor scenes, characterized in that: include: Obtain target point cloud data; The point cloud data is input into the point cloud semantic segmentation system according to any one of claims 1 to 6, and the segmented point cloud data is output.

Images