Point cloud semantic segmentation method and device, computing device, and storage medium

Abstract

The application discloses a point cloud semantic segmentation method and device, a computing device and a storage medium. The method comprises the following steps: acquiring point cloud data, performing multi-level encoding on the point cloud data through an encoder network to obtain a multi-level neighborhood feature set; performing step-by-step decoding on the multi-level neighborhood feature set through a decoder network to obtain semantic features of the point cloud data; and performing classification prediction according to the semantic features of the point cloud data to obtain semantic labels corresponding to each point of the point cloud data. The feature aggregation operation is performed at least twice on each first point cloud data to obtain the neighborhood feature set of each first point cloud data, and the center point feature is connected with the neighborhood feature set obtained through each feature aggregation in a residual manner or the neighborhood feature sets obtained through each feature aggregation are densely connected with each other as the input of the lower layer feature aggregation. The embodiment of the application can reduce the number of network parameters and significantly improve the semantic segmentation result of large scene point cloud.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a point cloud semantic segmentation method, apparatus, computing device, and storage medium. Background Technology

[0002] With the continuous development of photogrammetry, remote sensing, computer vision, robotics, and autonomous driving, the processing of 3D point cloud data has become a research hotspot. In photogrammetry, the combination of airborne LiDAR and oblique photogrammetry can achieve 3D scene modeling. In remote sensing, combining radar point cloud data with optical remote sensing data can enhance the geometric information of remote sensing images. In computer vision, point clouds are mainly used for semantic segmentation, instance segmentation, object detection, 3D reconstruction, and point cloud registration. In robotics, point clouds are primarily used for robot grasping recognition, including target localization, pose estimation, grasp detection, and motion planning. Large-scale point clouds can retain their spatial geometry and color information to the greatest extent, and therefore have been widely used in various fields.

[0003] Point cloud data processing, especially semantic segmentation, is a current research hotspot. The goal of semantic segmentation based on 3D point clouds is to divide the point cloud into regions with specific attributes and to segment the labels of each point. It has had beneficial impacts on archaeological analysis, urban planning, virtual tours, and heritage documentation. However, large-scale 3D point clouds are unstructured and highly redundant. How to enable computers to understand large-scale scene point clouds is a major challenge. Creating semantic labels for 3D maps is an effective solution. Semantic labels can be used to visualize objects (such as buildings, terrain, and roads) to generate more easily understood 3D maps. For geographic information science, semantic segmentation helps in the effective and accurate classification of land cover (rivers, trees, mountains, etc.). Furthermore, in terms of cultural heritage documentation, the removal of objects unrelated to monuments is an important step in the 3D modeling process. In particular, the initial data cleaning stage still requires manual processing by professionals. Point cloud semantic segmentation technology can add semantic labels to objects in scanned maps, allowing professionals to remove unwanted objects (such as trees, people, cars, and animals) within a reasonable timeframe. Therefore, semantic segmentation enables intelligent data cleaning, thereby reducing overall processing time. Researchers have proposed a large number of deep learning-based semantic segmentation models to process point clouds, and some of these models have achieved industrial-grade standards in terms of segmentation accuracy and processing time. Summary of the Invention

[0004] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed embodiments section. The summary section of this invention is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0005] In view of the above technical problems, the present invention provides a point cloud semantic segmentation method, apparatus, computing device and storage medium, which can reduce the number of network parameters and significantly improve the semantic segmentation results of large scene point clouds.

[0006] According to one aspect of the present invention, a point cloud semantic segmentation method is provided, comprising:

[0007] Acquire point cloud data, and use an encoder network to perform multi-level encoding on the point cloud data to obtain a multi-level neighborhood feature set;

[0008] The semantic features of the point cloud data are obtained by progressively decoding the multi-level neighborhood feature set through a decoder network; and

[0009] Based on the semantic features of the point cloud data, classification and prediction are performed to obtain the semantic labels corresponding to each point in the point cloud data.

[0010] The multi-level neighborhood feature set is obtained by multi-level encoding of the point cloud data through an encoder network, including:

[0011] The point cloud data is downsampled multiple times to obtain multi-level first point cloud data including different numbers of points;

[0012] Obtain the center point features of the first point cloud data at each level;

[0013] At least two feature aggregation operations are performed on the center point features of the first point cloud data at each level to obtain the neighborhood feature set of the first point cloud data at each level. The current neighborhood feature set output by the current feature aggregation operation, the residual connection or dense connection between the current neighborhood feature set and the features of each center point of the point cloud data, and the dense connection between the current neighborhood feature set and the previous neighborhood feature set output by the previous feature aggregation operation are used as the input for the next feature aggregation.

[0014] In one embodiment of the present invention, the feature aggregation operation includes:

[0015] Local features of the first point cloud data at each level are obtained based on the center point features of the first point cloud data at each level.

[0016] The local feature set is obtained by concatenating the local feature processes of the first point cloud data at each level;

[0017] The attention score of each local feature in the local feature set is obtained, and local features are selected and aggregated according to the attention score to obtain the neighborhood feature set.

[0018] In one embodiment of the present invention, the local features include neighbor features, geometric features, and semantic features.

[0019] In one embodiment of the present invention, obtaining the local features of each point cloud layer based on the center point features of the first point cloud data at each level includes:

[0020] Obtain the neighbor feature set corresponding to each center point, and use it as the neighbor feature of the point cloud data;

[0021] The geometric features of the point cloud data are obtained based on the spatial location information of the neighboring point feature set;

[0022] The semantic features of the point cloud data are obtained based on the features of each center point and the features of its neighboring points.

[0023] In one embodiment of the present invention, the feature aggregation operation is performed three times. The first feature aggregation operation outputs a first neighborhood feature set and a residual connection between the first neighborhood feature set and the features of each center point, which are used as inputs to the second feature aggregation operation. The second feature aggregation operation outputs a second neighborhood feature set, a dense connection between the second neighborhood feature set and the features of each center point, and a dense connection between the second neighborhood feature set and the first neighborhood feature set, which are used as inputs to the third feature aggregation operation.

[0024] In one embodiment of the present invention, obtaining the semantic features of the point cloud data by progressively decoding the multi-level neighborhood feature set through a decoder network includes:

[0025] Perform an upsampling operation on the first point cloud data at each level to obtain multi-level second point cloud data;

[0026] Multi-level semantic features are obtained by operating on the second point cloud data, domain feature set, and semantic features of the corresponding level using a multi-layer perceptron.

[0027] In one embodiment of the present invention, classification prediction is performed based on the semantic features of the point cloud data to obtain semantic labels corresponding to each point in the point cloud data, including:

[0028] A fully connected operation is performed on the semantic features obtained from the last layer decoder to obtain the semantic labels corresponding to each point in the point cloud data.

[0029] According to another aspect of the present invention, a point cloud semantic segmentation apparatus is provided, comprising:

[0030] An encoder is used to perform multi-level encoding on input point cloud data to obtain a multi-level neighborhood feature set of the point cloud data;

[0031] A decoder is used to perform step-by-step decoding of the multi-level neighborhood feature set of the point cloud data to obtain the semantic features of the point cloud data.

[0032] A classifier is used to perform classification and prediction based on the semantic features of the point cloud data, thereby obtaining the semantic label corresponding to each point in the point cloud data.

[0033] The encoder comprises multiple layers of sequentially connected sub-encoders, each sub-encoder including a downsampling module, a center point feature extraction module, and multiple feature aggregation modules connected in sequence.

[0034] The downsampling module is used to perform random downsampling on the input point cloud data to obtain multi-level first point cloud data including different numbers of points;

[0035] The center point feature extraction module obtains the center point features of the first point cloud data;

[0036] The feature aggregation module is used to perform feature aggregation on the center point features of the first point cloud data at each level to obtain a neighborhood feature set;

[0037] The output of each feature aggregation module, the residual connection or dense connection of the output of each feature aggregation module with each center point feature of the point cloud data, and the dense connection of the output of each feature aggregation module with the output of the previous feature aggregation module serve as the input of the next feature aggregation module.

[0038] In one embodiment of the present invention, the feature aggregation module comprises:

[0039] The local feature extraction unit is used to obtain the local features of the first point cloud data at each level based on the center point features of the first point cloud data at each level.

[0040] The feature aggregation unit is used to chain the local feature processes of the first point cloud data at each level to obtain a local feature set, obtain the attention score of each local feature in the local feature set, and select local features for aggregation based on the attention score to obtain the neighborhood feature set.

[0041] In one embodiment of the present invention, the sub-encoder includes a downsampling module, a center point feature extraction module, a first feature aggregation module, a second feature aggregation module, and a third feature aggregation module connected in sequence. The first feature aggregation module outputs a first neighborhood feature set and a residual connection between the first neighborhood feature set and the features of each center point as inputs to the second feature aggregation module. The second feature aggregation module outputs a second neighborhood feature set, a dense connection between the second neighborhood feature set and the features of each center point, and a dense connection between the second neighborhood feature set and the first neighborhood feature set as inputs to the third feature aggregation module.

[0042] In one embodiment of the present invention, the decoder includes multiple layers of sequentially connected sub-decoders, each sub-decoder including an upsampling module and a multiple sensing module.

[0043] The upsampling module is used to perform an upsampling operation on each level of the first point cloud data to obtain multi-level second point cloud data;

[0044] The multi-sensing module is used to operate on the second point cloud data, domain feature set and semantic features of the corresponding level through a multi-layer perceptron to obtain multi-level semantic features.

[0045] According to another aspect of the present invention, a computing device is provided, the device comprising: a processor, and a memory storing computer program instructions: the processor reads and executes the computer program instructions to implement the point cloud semantic segmentation method according to one aspect of the present invention.

[0046] According to another aspect of the present invention, a computer storage medium is provided, wherein computer program instructions are stored thereon, and the computer program instructions, when executed by a processor, implement the point cloud semantic segmentation method according to one aspect of the present invention.

[0047] According to the point cloud semantic segmentation method, apparatus, computing device, and storage medium of the present invention, by introducing residual connections and dense connections into the semantic segmentation of large-scale point clouds, the network depth is increased and distinguishable semantic features are extracted. Secondly, the present invention designs a new local feature extraction module to compensate for the shortcomings of random sampling, and aggregates and optimizes the learned features through an attention mechanism, enabling the network to better adapt to the geometry of the point cloud. Finally, the present invention outperforms the current state-of-the-art large-scene semantic segmentation algorithm RandLA-Net in both OA and mA on the large indoor S3DIS dataset, achieving 88.1% and 82.3% respectively. Similarly, the mA is 9.3% higher than SPG, which also directly processes large-scene point clouds. In the large outdoor Semantic3D dataset, the present invention achieves mIou and OA of 76.5% and 94.4% respectively. Notably, on the large outdoor Toronto-3D dataset, the present invention achieves mIou of 80.27%, surpassing the current state-of-the-art publicly available algorithm MSTGNet by 9.77%. The IoU for the road signs and fence categories exceeded MSTGNet by 42.61% and 19.67%, respectively. This invention delivers optimal semantic segmentation performance across multiple datasets, validating its effectiveness and robustness. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic flowchart of the point cloud semantic segmentation method according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic flowchart of the encoding method in point cloud semantic segmentation according to an embodiment of the present invention;

[0051] Figure 3 This is a schematic structural block diagram of a point cloud semantic segmentation device according to an embodiment of the present invention;

[0052] Figure 4 This is a schematic structural block diagram of a sub-encoder according to an embodiment of the present invention;

[0053] Figure 5 An example of a cloud semantic segmentation apparatus according to an embodiment of the present invention; and

[0054] Figure 6This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Numerous specific details are set forth in the following description to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that embodiments of the present invention can be practiced without one or more of these details. In other instances, to avoid confusion with embodiments of the present invention, some technical features well-known in the art have not been described.

[0056] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0057] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; thus, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0058] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0059] Traditional deep learning segmentation algorithms aim to convert point clouds into voxels or multi-views to extract point features, which can lead to increased data preprocessing time or information loss. PointNet is a landmark algorithm that proposed a neural network that directly takes point clouds as input. To date, many methods based on PointNet and PointNet++ improvements, such as PointCNN, KPCOnV, RSCNN, and DGCNN, have achieved good performance. However, these methods slice the point cloud into small blocks and then sample a certain number of points from each block, such as 4096 or 8192, as input to train the network. Typically, methods based on pruning strategies can increase the time complexity of data preprocessing, and the sampling strategy can destroy the geometry of the point cloud, making it difficult for the network to effectively extract local features of each object.

[0060] Recently, some works have begun to attempt to directly process large-scale point clouds. For example, FCPN combines voxels to process large-scale point clouds. LU-Net proposes an end-to-end architecture for semantic segmentation of LiDAR point clouds. This architecture considers the sensor's topology and efficiently creates a multi-channel range image using learned 3D features. This range image is then used as input to the U-Net architecture, and finally, segmentation can be performed using a simple U-Net segmentation network. VIASeg proposes a multi-scale fully convolutional network based on supercompressed residual modules and semantic connections, which projects fused RGB point clouds onto a 2D sphere. SPG uses superpoints and superpoint graphs to represent large-scene point clouds. The input point cloud is divided into geometrically simple shapes called superpoints. Based on this preprocessing, a superpoint graph (SPG) is constructed by linking hyperedges with rich attributes to nearby superpoints. This is a novel point cloud representation. Each superpoint is embedded by a PointNet network, and information is then transmitted along the hyperedges. The final segmentation result is optimized by a gated recurrent unit (GRU).

[0061] The methods described above have achieved good segmentation results, but they involve preprocessing operations such as voxelization, block segmentation, or conversion to graphics. These operations increase computational cost and memory usage. PASS3D proposes a novel two-stage 3D semantic segmentation framework. In the first stage, an accelerated clustering algorithm is used to perform segmentation of the point cloud without breaks for fine-grained clustering, which improves recall in a short time. In the second stage, the neural network is further processed to estimate the semantic label of each point, and a novel data expansion method is proposed to enhance the network's ability to recognize all categories (especially non-rigid objects). RandLA-Net proposes a feature aggregation module to increase the receptive range of each point, thus better preserving the overall geometric details of the input point cloud. However, it does not extract semantic features. It only performs shortcut connections on the expanded residual blocks without fully utilizing the advantages of the residual network, resulting in the failure to extract many distinguishable features.

[0062] Current methods for semantic segmentation of large-scale 3D point clouds typically involve cutting the massive point cloud into point cloud blocks before processing. However, in actual computation, the geometric features of the cutting boundaries are easily destroyed, resulting in obvious boundary phenomena in the segmentation results. Currently, most networks still choose to train on small point cloud blocks as input, mainly because directly processing large-scale point clouds presents many challenges. First, 3D point clouds collected by sensor scanning are unevenly distributed throughout the space. Direct processing of large scenes may lead to the neglect of sparse points, making semantic segmentation of small objects extremely difficult. Second, the size and number of points in a large scene are uncertain, requiring the network to be insensitive to the number of input points. Third, the geometric structure of large scenes is more complex than that of small-scale point clouds. Increasing the number of categories and instances processed each time significantly increases the difficulty of segmentation. Finally, the surge in computational cost poses a huge challenge to existing GPU memory. Therefore, to address this problem, there is an urgent need for efficient deep learning network models that take the original point cloud as input for semantic segmentation of large-scale point clouds.

[0063] Considering the performance of residual networks and dense convolutional networks in the image domain, we combine residual-dense training for semantic segmentation of 3D point clouds. Point clouds of large scenes contain a large number of low / medium / high-level features. As the number of network layers increases, the features extracted by the network become more discriminative and contain more semantic information. Semantic segmentation of 3D point clouds requires the support of these features to achieve good segmentation results. Residual networks can solve the gradient descent problem caused by the increase in network layers. We find that residual networks and dense networks precisely meet the feature processing needs of different stages of semantic segmentation of point clouds in large scenes. In shallow network layers, the network can only extract a small number of features. We use shortcut connections from ResNet to combine features through feature summation before passing them to the next layer. As the number of network layers increases, the network will extract more features with semantic information. Dense convolutional networks can enhance feature propagation and encourage feature reuse. Therefore, we combine features through dense connections. During the training phase, a complete residual network typically requires a large number of network parameters because their weights in each layer are independently distributed. Dense network structures separate the retained information from the added external information, enabling this invention to reduce the number of network parameters and significantly improve the semantic segmentation results of large-scale point clouds. Therefore, we design a novel framework for large-scale point cloud semantic segmentation based on residual networks and dense convolutional networks.

[0064] Based on the above ideas, this invention provides a point cloud semantic segmentation method, apparatus, computing device, and storage medium, which can detect dirty pages in a database and quickly and accurately detect all dirty pages. The following is a detailed description in conjunction with the accompanying drawings.

[0065] First, the point cloud semantic segmentation method provided in the embodiments of the present invention will be introduced.

[0066] Figure 1 Please refer to the schematic flowchart of the point cloud semantic segmentation method 100 according to an embodiment of the present invention. Figure 1 The point cloud semantic segmentation method 100 disclosed in this embodiment of the invention includes:

[0067] Step S101: Obtain point cloud data, and use an encoder network to perform multi-level encoding on the point cloud data to obtain a multi-level neighborhood feature set.

[0068] For example, the encoder is a multi-level encoder, with each level encoder performing an encoding process once to obtain a level-specific feature set.

[0069] For example, obtaining a multi-level neighborhood feature set by performing multi-level encoding on the point cloud data through an encoder network includes:

[0070] The point cloud data is downsampled multiple times to obtain multi-level first point cloud data including different numbers of points;

[0071] Obtain the center point features of the first point cloud data at each level;

[0072] At least two feature aggregation operations are performed on the center point features of the first point cloud data at each level to obtain the neighborhood feature set of the first point cloud data at each level. The current neighborhood feature set output by the current feature aggregation operation, the residual connection or dense connection between the current neighborhood feature set and the features of each center point of the point cloud data, and the dense connection between the current neighborhood feature set and the previous neighborhood feature set output by the previous feature aggregation operation are used as the input for the next feature aggregation.

[0073] The specific operation of each encoder level will be described below and will not be repeated here.

[0074] Step S102: Obtain the semantic features of the point cloud data by decoding the multi-level neighborhood feature set step by step through the decoder network.

[0075] For example, the decoder is a multi-level decoder, which can obtain the semantic features of the point cloud data by progressively decoding the multi-level neighborhood feature sets obtained by the encoder through a decoder network.

[0076] For example, the encoder obtains a multi-level domain feature set and first point cloud data, which are input into the corresponding sub-decoder in the decoder. The decoder specifically performs the following steps:

[0077] An upsampling operation is performed on the first point cloud data at each level to obtain multi-level second point cloud data. For example, the nearest neighbor interpolation algorithm performs an upsampling operation on the first point cloud data at each level to obtain multi-level second point cloud data.

[0078] Multi-level semantic features are obtained by operating on the second point cloud data, domain feature set, and semantic features of the corresponding level using a multi-layer perceptron.

[0079] Step S103: Classify and predict based on the semantic features of the point cloud data to obtain the semantic labels corresponding to each point in the point cloud data.

[0080] For example, a fully connected operation is performed on the semantic features obtained by the decoder to obtain the semantic labels corresponding to each point in the point cloud data. Specifically, a fully connected operation is performed on the semantic features obtained by the last layer decoder to obtain the semantic labels corresponding to each point in the point cloud data.

[0081] Figure 2 This is a schematic flowchart of an encoding method in point cloud semantic segmentation according to an embodiment of the present invention.

[0082] Please refer to Figure 2The encoding method 200 for point cloud semantic segmentation disclosed in this embodiment of the invention includes:

[0083] Step S201: Obtain point cloud data.

[0084] The point cloud data acquired in S201 is either the original point cloud data or the first point cloud data obtained by downsampling from the upper encoder.

[0085] Step S202: Downsample the point cloud data to obtain the first point cloud data.

[0086] For example, the point cloud data obtained in step 201 is subjected to random downsampling, and the number of points after sampling is a predetermined proportion of the point cloud data obtained in step 201 (e.g., 1 / 4 or other suitable proportions). Then, the points in the first point cloud data obtained by this downsampling are used as the center points.

[0087] Step S203: Obtain the center point features of the first point cloud data.

[0088] For example, the center point features of the first point cloud data are obtained through a fully connected operation.

[0089] Step S204: Obtain the first local feature of the first point cloud data.

[0090] For example, the local features include neighbor features, geometric features, and semantic features.

[0091] Obtaining the first local feature of the first point cloud data based on the center point feature of the first point cloud data can be achieved through the following steps:

[0092] First, the neighbor feature set corresponding to each center point is obtained as the neighbor feature of the point cloud data. For example, the k nearest neighbor algorithm is used to find the k nearest neighbors of the center point (k=16), the coordinates of the neighborhood point set of each center point are obtained, and the neighbor feature set corresponding to each center point is obtained through the neighborhood point index attribute. For example, the neighbor feature set includes point cloud features such as local density, local curvature, linearity, flatness, and scattering.

[0093] Next, the geometric features of the first point cloud data are obtained based on the spatial location information of the neighboring point feature set. For example, the geometric features of the first point cloud data are obtained by encoding the spatial location information of the neighboring point feature set (multilayer perceptron operation). The encoded content includes the three-dimensional coordinates of the center point, the three-dimensional coordinates of the neighboring points, the relative coordinates between the center point and the neighboring points, and the Euclidean distance between the center point and the neighboring points.

[0094] Next, the semantic features of the point cloud data are obtained based on the features of each center point and the features of its neighbors. For example, convolution operations are performed on the features of the center point and the features of its neighbors to obtain the semantic features between the points, enabling the network to extract local neighborhood feature information more deeply and the contextual relationships between the center point and its neighbors.

[0095] Step S205: Perform the first aggregation of the local features of the first point cloud data to obtain the first neighborhood feature set of the first point cloud data.

[0096] For example, the first neighborhood feature set is obtained through the following steps:

[0097] First, the first local feature process of the first point cloud data is chained to obtain the local feature set;

[0098] Then, the attention score of each local feature in the local feature set is obtained. For example, the local feature set is input into the attention module, the attention module performs a fully connected operation, and then the attention score is obtained through, for example, a softmax function.

[0099] Then, important local features are selected based on the attention scores, and these features are weighted, summed, and aggregated to obtain the neighborhood feature set.

[0100] Steps S206-S207 are a repetition of S204-S205, the difference being that the input to S206 includes not only the first neighborhood feature set output from S205, but also the residual connection result between the first neighborhood feature set and the center point feature output from S203. By performing a residual connection between the center point feature and the first neighborhood feature set, the gradient descent problem caused by the increase in network layers can be solved.

[0101] Steps S208-S209 are a repetition of S204-S205, with the difference that the input to S208 includes not only the second neighborhood feature set output from S207, but also the dense connection result between the second neighborhood feature set and the center point feature output from S203, as well as the dense connection result between the second neighborhood feature set and the first neighborhood feature set. Feature combination can be achieved by densely connecting the second neighborhood feature set and the first neighborhood feature set. Furthermore, densely connecting the second neighborhood feature set with the center point feature can increase the network depth while extracting more distinguishable semantic features.

[0102] Finally, the third neighborhood feature set output in step 209 is used as the output of this encoding process.

[0103] It should be understood that although feature aggregation is performed three times in this embodiment (i.e., S204-S205 is repeated three times), feature aggregation may be performed two or more times in other embodiments, and is not limited to three times; this is merely an example.

[0104] According to the point cloud semantic segmentation method of the present invention, by introducing residual connections and dense connections into the semantic segmentation of large-scale point clouds, the network depth is increased and distinguishable semantic features are extracted. Secondly, the present invention designs a novel local feature extraction module to compensate for the shortcomings of random sampling, and aggregates and optimizes the learned features through an attention mechanism, enabling the network to better adapt to the geometry of the point cloud. Finally, the present invention outperforms the current state-of-the-art large-scene semantic segmentation algorithm RandLA-Net in both OA and mA on the large indoor S3DIS dataset, achieving 88.1% and 82.3% respectively. Similarly, the mA is 9.3% higher than SPG, which also directly processes large-scene point clouds. In the large outdoor Semantic3D dataset, the present invention achieves mIou and OA of 76.5% and 94.4% respectively. Notably, on the large outdoor Toronto-3D dataset, the present invention achieves mIou of 80.27%, exceeding the current state-of-the-art publicly available algorithm MSTGNet by 9.77%. Specifically, the IoU for road signs and fences exceeds MSTGNet by 42.61% and 19.67% respectively. This invention delivers optimal semantic segmentation performance across multiple datasets, validating its effectiveness and robustness.

[0105] Figure 3 This is a schematic structural block diagram of a point cloud semantic segmentation device according to an embodiment of the present invention. The following is in conjunction with... Figure 3 A point cloud semantic segmentation apparatus 300 according to an embodiment of the present invention will be described. Figure 4 This is a schematic structural block diagram of a sub-encoder according to an embodiment of the present invention; Figure 5 This is an example of a cloud semantic segmentation device according to an embodiment of the present invention.

[0106] Please refer to Figure 3 The point cloud semantic segmentation apparatus 300 according to an embodiment of the present invention includes an encoder 310, a decoder 320 and a classifier 330.

[0107] Encoder 310 is used to perform multi-level encoding on the input point cloud data to obtain a multi-level neighborhood feature set of the point cloud data. Encoder 310 is used to perform combination. Figure 1 Step S101 in the described point cloud semantic segmentation method, and the combination of Figure 2 The steps S201-S209 in the described encoding method are described in detail in the foregoing. Figures 1-2 The description will not be repeated here.

[0108] Decoder 320 is used to perform step-by-step decoding of the multi-level neighborhood feature set of the point cloud data to obtain the semantic features of the point cloud data. Decoder 320 is used to perform combination. Figure 1 Step S102 in the described point cloud semantic segmentation method is described in detail above. Figure 1 The description will not be repeated here.

[0109] Classifier 330 is used to perform classification prediction based on the semantic features of the point cloud data to obtain the semantic label corresponding to each point in the point cloud data. Classifier 330 is used to perform combination... Figure 1 Step S103 in the described point cloud semantic segmentation method is described in detail above. Figure 1 The description will not be repeated here.

[0110] For example, in this embodiment, the encoder includes multiple layers of sequentially connected sub-encoders (e.g., 5 sub-decoders), such as... Figure 4 As shown, the sub-encoder 400 includes a downsampling module 410, a center point feature extraction module 420, and multiple feature aggregation modules 430 connected in sequence.

[0111] The downsampling module 410 is used to perform random downsampling operations on the input point cloud data to obtain multi-level first point cloud data including different numbers of points.

[0112] The center point feature extraction module 420 acquires the center point features of the first point cloud data.

[0113] The feature aggregation module 430 is used to aggregate the center point features of the first point cloud data at each level to obtain a neighborhood feature set. The output of each feature aggregation module, the residual connection or dense connection between the output of each feature aggregation module and each center point feature of the point cloud data, and the dense connection between the output of each feature aggregation module and the output of the previous feature aggregation module are used as the input of the next feature aggregation module. Exemplarily, in this embodiment, the feature aggregation module 430 includes: a local feature extraction unit, used to obtain local features of the first point cloud data at each level based on the center point features of the first point cloud data at each level; and a feature aggregation unit, used to concatenate the local feature processes of the first point cloud data at each level to obtain a local feature set, obtain the attention score of each local feature in the local feature set, and select local features for aggregation based on the attention score to obtain the neighborhood feature set.

[0114] For example, in this embodiment, the sub-encoder includes a downsampling module, a center point feature extraction module, a first feature aggregation module, a second feature aggregation module, and a third feature aggregation module connected in sequence. The first feature aggregation module outputs a first neighborhood feature set and a residual connection between the first neighborhood feature set and the features of each center point as input to the second feature aggregation module. The second feature aggregation module outputs a second neighborhood feature set, a dense connection between the second neighborhood feature set and the features of each center point, and a dense connection between the second neighborhood feature set and the first neighborhood feature set as input to the third feature aggregation module.

[0115] For example, in this embodiment, the decoder includes multiple layers of sequentially connected sub-decoders. The sub-decoders include an upsampling module and a multiple perception module. The upsampling module is used to perform an upsampling operation on the first point cloud data of each level to obtain multi-level second point cloud data. The multiple perception module is used to operate on the second point cloud data of the corresponding level, the domain feature set, and the semantic features through a multi-layer perceptron to obtain multi-level semantic features.

[0116] It should be understood that in this embodiment, the encoder may include a suitable number of sub-encoders, and the decoder may include a suitable number of sub-decoders. Figure 5 An example of a point cloud semantic segmentation apparatus 300 according to an embodiment of the present invention is provided, which includes five sub-encoders and five word decoders, as well as a classifier composed of fully connected layers. That is, it includes five encoder layers and five decoder layers, wherein the downsampled point cloud data and neighborhood feature set obtained by each encoder layer are input into the decoder of the corresponding layer. Of course, in other embodiments, encoders and decoders with other numbers of layers can also be used.

[0117] Figure 3-5 Each module / unit in the point cloud semantic segmentation device 300 shown has the ability to implement... Figures 1-2 The functions of each step in the process and their corresponding technical effects are described in detail here for the sake of brevity.

[0118] Figure 6 A schematic diagram of the hardware structure of a computing device provided in an embodiment of the present invention is shown.

[0119] The computing device 600 may include a processor 601 and a memory 602 storing computer program instructions.

[0120] Specifically, the processor 601 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0121] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be non-volatile solid-state memory. Memory 602 may be internal or external to the integrated gateway disaster recovery device.

[0122] In one instance, memory 602 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0123] Memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0124] The processor 601 reads and executes computer program instructions stored in the memory 602 to achieve... Figure 1 The methods / steps S101 to S103 in the illustrated embodiments are as follows: Figure 2 The method / steps S201 to S209 in the illustrated embodiment are achieved. Figures 1 to 2 The technical effects achieved by executing the methods / steps shown in the examples are not elaborated here for the sake of brevity.

[0125] The processor 601 reads and executes computer program instructions stored in the memory 602 to achieve... Figure 3 The point cloud semantic segmentation device 300, encoder 310, decoder 320, and classifier 330 in the illustrated embodiment achieve... Figure 3 The corresponding technical effects achieved by the device in the example shown are not elaborated here for the sake of brevity.

[0126] In one example, computing device 600 may further include a communication interface 603 and a bus 610. Wherein, as... Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0127] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0128] Bus 610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0129] The computing device 600 can execute the point cloud semantic segmentation method in this embodiment of the invention using a database anomaly repair algorithm, thereby achieving a combination of... Figures 1 to 2 The point cloud semantic segmentation method is described.

[0130] Furthermore, according to embodiments of the present invention, a storage medium is provided, on which program instructions are stored. When executed by a computer or processor, these program instructions are used to perform corresponding steps of the point cloud semantic segmentation method of the present invention, and to implement corresponding units or modules of the database anomaly repair apparatus according to embodiments of the present invention. The storage medium may, for example, include a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0131] In one embodiment, the computer program instructions, when run by a computer, can implement various functional modules in the database anomaly repair device according to an embodiment of the present invention, and / or can execute the point cloud semantic segmentation method according to an embodiment of the present invention.

[0132] In one embodiment, the computer program instructions, when executed by a computer, perform the following steps: acquiring point cloud data; performing multi-level encoding on the point cloud data using an encoder network to obtain a multi-level neighborhood feature set; performing step-by-step decoding on the multi-level neighborhood feature set using a decoder network to obtain semantic features of the point cloud data; and performing classification prediction based on the semantic features of the point cloud data to obtain semantic labels corresponding to each point in the point cloud data. The step of performing multi-level encoding on the point cloud data using an encoder network to obtain the multi-level neighborhood feature set includes: downsampling the point cloud data multiple times. The method involves obtaining multi-level first point cloud data including different numbers of points; acquiring the center point features of each level of the first point cloud data; performing at least two feature aggregation operations on the center point features of each level of the first point cloud data to obtain the neighborhood feature set of each level of the first point cloud data; and using the current neighborhood feature set output by the current feature aggregation operation, the residual connection or dense connection between the current neighborhood feature set and the features of each center point of the point cloud data, and the dense connection between the current neighborhood feature set and the previous neighborhood feature set output by the previous feature aggregation operation as the input for the next feature aggregation.

[0133] Each module in the point cloud semantic segmentation method according to the embodiments of the present invention can be implemented by the processor of the electronic device, server, or system of the point cloud semantic segmentation method according to the embodiments of the present invention running computer program instructions stored in memory, or by computer instructions stored in a computer-readable storage medium of a computer program product according to the embodiments of the present invention being implemented by a computer running.

[0134] According to the point cloud semantic segmentation method, apparatus, computing device, and storage medium of the present invention, by introducing residual connections and dense connections into the semantic segmentation of large-scale point clouds, the network depth is increased and distinguishable semantic features are extracted. Secondly, the present invention designs a new local feature extraction module to compensate for the shortcomings of random sampling, and aggregates and optimizes the learned features through an attention mechanism, enabling the network to better adapt to the geometry of the point cloud. Finally, the present invention outperforms the current state-of-the-art large-scene semantic segmentation algorithm RandLA-Net in both OA and mA on the large indoor S3DIS dataset, achieving 88.1% and 82.3% respectively. Similarly, the mA is 9.3% higher than SPG, which also directly processes large-scene point clouds. In the large outdoor Semantic3D dataset, the present invention achieves mIou and OA of 76.5% and 94.4% respectively. Notably, on the large outdoor Toronto-3D dataset, the present invention achieves mIou of 80.27%, exceeding the current state-of-the-art publicly available algorithm MSTGNet by 9.77%. The IoU for the road signs and fence categories exceeded that of MSTGNet by 42.61% and 19.67%, respectively. This invention delivers state-of-the-art semantic segmentation performance across multiple datasets, validating its effectiveness and robustness.

[0135] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0136] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0138] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0139] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this method of the invention should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0140] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0141] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0142] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the article analysis device according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0143] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0144] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A point cloud semantic segmentation method, characterized in that, include: Acquire point cloud data, and use an encoder network to perform multi-level encoding on the point cloud data to obtain a multi-level neighborhood feature set; The semantic features of the point cloud data are obtained by decoding the multi-level neighborhood feature set step by step through a decoder network. as well as Based on the semantic features of the point cloud data, classification and prediction are performed to obtain the semantic labels corresponding to each point in the point cloud data. The multi-level neighborhood feature set is obtained by multi-level encoding of the point cloud data through an encoder network, including: The point cloud data is downsampled multiple times to obtain multi-level first point cloud data including different numbers of points; Obtain the center point features of the first point cloud data at each level; At least two feature aggregation operations are performed on the center point features of the first point cloud data at each level to obtain the neighborhood feature set of the first point cloud data at each level. The current neighborhood feature set output by the current feature aggregation operation, the residual connection or dense connection between the current neighborhood feature set and the features of each center point of the point cloud data, and the dense connection between the current neighborhood feature set and the previous neighborhood feature set output by the previous feature aggregation operation are used as the input for the next feature aggregation. The feature aggregation operation includes: Based on the center point features of the first point cloud data at each level, local features of the first point cloud data at each level are obtained, specifically including: Obtain the neighbor feature set corresponding to each center point as the neighbor feature of the point cloud data: find the k nearest neighbor points of the center point through the k nearest neighbor algorithm, obtain the coordinates of the neighbor point set of each center point, and obtain the neighbor feature set corresponding to each center point through the neighbor point index attribute. The neighbor feature set includes local density, local curvature, linearity, flatness and scattering degree. The geometric features of the first point cloud data are obtained by encoding the spatial location information of the neighboring point feature set. The encoding content includes the three-dimensional coordinates of the center point, the three-dimensional coordinates of the neighboring points, the relative coordinates between the center point and the neighboring points, and the Euclidean distance between the center point and the neighboring points. The semantic features of the point cloud data are obtained based on the features of each center point and the features of neighboring points: the semantic features between points are obtained by performing a convolution operation between the center point features and the features of neighboring points. The local feature set is obtained by concatenating the local feature processes of the first point cloud data at each level; The attention score of each local feature in the local feature set is obtained, and local features are selected and aggregated according to the attention score to obtain the neighborhood feature set; The feature aggregation operation is performed three times. The first feature aggregation operation outputs a first neighborhood feature set and a residual connection between the first neighborhood feature set and the features of each center point, which are used as inputs to the second feature aggregation operation. The second feature aggregation operation outputs a second neighborhood feature set, a dense connection between the second neighborhood feature set and the features of each center point, and a dense connection between the second neighborhood feature set and the first neighborhood feature set, which are used as inputs to the third feature aggregation operation.

2. The point cloud semantic segmentation method according to claim 1, characterized in that, The local features include neighbor features, geometric features, and semantic features.

3. The point cloud semantic segmentation method according to any one of claims 1-2, characterized in that, The step of obtaining semantic features from the point cloud data by progressively decoding the multi-level neighborhood feature set through a decoder network includes: Perform an upsampling operation on the first point cloud data at each level to obtain multi-level second point cloud data; Multi-level semantic features are obtained by operating on the second point cloud data, domain feature set, and semantic features of the corresponding level using a multi-layer perceptron.

4. The point cloud semantic segmentation method according to claim 3, characterized in that, Based on the semantic features of the point cloud data, classification and prediction are performed to obtain the semantic labels corresponding to each point in the point cloud data, including: A fully connected operation is performed on the semantic features obtained from the last layer decoder to obtain the semantic labels corresponding to each point in the point cloud data.

5. A point cloud semantic segmentation device, characterized in that, include: An encoder is used to acquire point cloud data. The point cloud data is encoded at multiple levels through an encoder network to obtain a multi-level neighborhood feature set. A decoder is used to obtain semantic features of the point cloud data by decoding the multi-level neighborhood feature set step by step through a decoder network. A classifier is used to perform classification and prediction based on the semantic features of the point cloud data, thereby obtaining the semantic label corresponding to each point in the point cloud data. The multi-level neighborhood feature set is obtained by multi-level encoding of the point cloud data through an encoder network, including: The point cloud data is downsampled multiple times to obtain multi-level first point cloud data including different numbers of points; Obtain the center point features of the first point cloud data at each level; At least two feature aggregation operations are performed on the center point features of the first point cloud data at each level to obtain the neighborhood feature set of the first point cloud data at each level. The current neighborhood feature set output by the current feature aggregation operation, the residual connection or dense connection between the current neighborhood feature set and the features of each center point of the point cloud data, and the dense connection between the current neighborhood feature set and the previous neighborhood feature set output by the previous feature aggregation operation are used as the input for the next feature aggregation. The feature aggregation module includes: The local feature extraction unit is used to obtain local features of the first point cloud data at each level based on the center point features of the first point cloud data at each level, specifically including: Obtain the neighbor feature set corresponding to each center point as the neighbor feature of the point cloud data: find the k nearest neighbor points of the center point through the k nearest neighbor algorithm, obtain the coordinates of the neighbor point set of each center point, and obtain the neighbor feature set corresponding to each center point through the neighbor point index attribute. The neighbor feature set includes local density, local curvature, linearity, flatness and scattering degree. The geometric features of the first point cloud data are obtained by encoding the spatial location information of the neighboring point feature set. The encoding content includes the three-dimensional coordinates of the center point, the three-dimensional coordinates of the neighboring points, the relative coordinates between the center point and the neighboring points, and the Euclidean distance between the center point and the neighboring points. The semantic features of the point cloud data are obtained based on the features of each center point and the features of neighboring points: the semantic features between points are obtained by performing a convolution operation between the center point features and the features of neighboring points. The feature aggregation unit is used to chain the local feature processes of the first point cloud data at each level to obtain a local feature set, obtain the attention score of each local feature in the local feature set, and select local features for aggregation based on the attention score to obtain the neighborhood feature set. The feature aggregation operation is performed three times. The first feature aggregation operation outputs a first neighborhood feature set and a residual connection between the first neighborhood feature set and the features of each center point, which are used as inputs to the second feature aggregation operation. The second feature aggregation operation outputs a second neighborhood feature set, a dense connection between the second neighborhood feature set and the features of each center point, and a dense connection between the second neighborhood feature set and the first neighborhood feature set, which are used as inputs to the third feature aggregation operation.

6. The point cloud semantic segmentation apparatus according to claim 5, characterized in that, The decoder comprises multiple layers of sequentially connected sub-decoders, each sub-decoder including an upsampling module and a multiple sensing module. The upsampling module is used to perform an upsampling operation on each level of the first point cloud data to obtain multi-level second point cloud data; The multi-sensing module is used to operate on the second point cloud data, domain feature set and semantic features of the corresponding level through a multi-layer perceptron to obtain multi-level semantic features.

7. A computing device, characterized in that, The device includes a processor and a memory storing computer program instructions: the processor reads and executes the computer program instructions to implement the point cloud semantic segmentation method as described in any one of claims 1-4.

8. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the point cloud semantic segmentation method as described in any one of claims 1-4.

Images