Three-dimensional point cloud classification and segmentation method and network based on enhanced point cloud rotation invariance

Abstract

The present application relates to the technical field of three-dimensional point cloud classification and segmentation, and discloses a three-dimensional point cloud classification and segmentation method and network based on enhanced point cloud rotation invariance, comprising the following steps: S1, selecting a baseline network, and setting a plurality of rotation invariant enhanced layers in the baseline network; S2, obtaining a neighborhood point set of each point in point cloud data; constructing a local spherical coordinate system; S3, in the local spherical coordinate system, based on the neighborhood point set of the center point, calculating a geometric descriptor SGD with rotation invariance; S4, the SGD expresses the edges between points of the point cloud data; S5, extracting point cloud geometric features through a spherical feature transmission mechanism; integrating into a point cloud network model; S6, training the integrated point cloud network model; S7, completing the classification and segmentation of three-dimensional point cloud through the trained point cloud network model. The present application solves the problem that the prior art cannot universally process rotation disturbance, and has the characteristics of flexibility and universality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional point cloud classification and segmentation, and more particularly to a three-dimensional point cloud classification and segmentation method and network based on enhanced point cloud rotation invariance. BACKGROUND

[0002] Point cloud data is a three-dimensional point set obtained by a laser radar sensor; the point cloud data can be used to describe the geometric shape and spatial position of the environment around the vehicle. The laser radar sensor captures point cloud data in the real world, helping the vehicle to complete the positioning of obstacles, thereby realizing automatic driving. The analysis, classification and segmentation methods of point cloud data include point-based registration methods, point feature-based methods and deep learning methods based on point data, among which the deep learning method is considered to be the most promising method at present. However, the actual obtained point cloud data often has problems such as rotation and missing, which limits the existing methods in the application of automatic driving.

[0003] The rotation invariance of point cloud refers to that after a rotation operation is performed on a point cloud set, the coordinates of all points in the set are changed, but it still represents the same object. How to fully extract the rotation invariance features of point cloud data and better understand and perceive the three-dimensional scene has important significance to the field of three-dimensional vision, and is an important cornerstone for future artificial intelligence technology to better serve human life. The early method of realizing point cloud recognition, classification and segmentation is to use hand-crafted features and apply classical machine learning algorithms to build models to realize the corresponding tasks, such as support vector machine, random forest, conditional random field, Markov random field, etc. However, the method of hand-crafted features mainly depends on the prior knowledge of researchers, which is insufficient for the huge point cloud data today.

[0004] With the support of deep learning technology, the related research on point cloud recognition, classification and segmentation method has rapidly and continuously developed. PointNet pioneered an end-to-end network that directly takes point sets as input and achieved incredible success. After that, inspired by CNN, GCN and Transformers mechanism, researchers developed various end-to-end networks and widely used them in point cloud recognition tasks. However, unlike synthetic data sets, in practical applications, the obtained point cloud data is often not aligned due to different acquisition views, which may show a three-dimensional model with additional offset, such as translation transformation and arbitrary rotation. Three-dimensional coordinates are an effective descriptor in themselves, and many networks directly operate based on the three-dimensional coordinates of point cloud. Due to the irregularity and disordered arrangement of point cloud, for the same object, although the problem of disordered arrangement can be solved by a symmetric function, the obtained point cloud always has unavoidable rotation, and the network performance will be greatly reduced.

[0005] In previous work, it is to improve the defects existing in a certain method, and ideal recognition and classification and segmentation effect is obtained on some specific data set, but in practical application, it lacks certain universality and flexibility. The point cloud recognition and classification and segmentation method also faces the problem that the rotation disturbance leads to the decline of the accuracy of the conventional network and the excessive attention to the rotation transformation causes the limited performance.

[0006] The existing feature processing method, device and terminal equipment based on rotation of point cloud data, wherein the method comprises: obtaining target point cloud data; extracting rotation invariant features from the target point cloud data by using a rotation mapping module; performing multi-dimensional feature processing on the rotation invariant features by using a plurality of clustering modules, the plurality of clustering modules being sequentially connected in order of the number of clusters from large to small; and performing classification on the rotation invariant features after multi-dimensional feature processing by using a classifier module to obtain a classification result of the rotation invariant features.

[0007] However, the existing technology still cannot universally process the rotation disturbance, so how to invent a general method to process the rotation disturbance is a technical problem that needs to be solved in the technical field. SUMMARY

[0008] The present application provides a three-dimensional point cloud classification and segmentation method and network based on enhanced point cloud rotation invariance, which has the characteristics of flexibility and universality.

[0009] To achieve the above-mentioned purposes of the present application, the technical solutions adopted are as follows:

[0010] The three-dimensional point cloud classification and segmentation method based on enhanced point cloud rotation invariance comprises the following steps:

[0011] S1, selecting a baseline network, setting a plurality of rotation invariant enhanced layers corresponding to the baseline network;

[0012] S2, obtaining point cloud data, obtaining the neighborhood point set of each point in the point cloud data by using the k-NN algorithm of the baseline network; dividing the local area with each point in the point cloud data as the center point, and constructing a local spherical coordinate system based on the local area;

[0013] S3, in the local spherical coordinate system, based on the neighborhood point set of the center point, calculating a geometric descriptor SGD with rotation invariance;

[0014] S4, the SGD represents the edges between the points of the point cloud data;

[0015] S5, input edges between points of the SGD expression point cloud data into a neural network based on a graph structure, extract point cloud geometric features of the point cloud data through a graph-based spherical feature transmission mechanism; cascade the point cloud geometric features and the neural network based on the graph structure into a baseline network, and integrate into a point cloud network model;

[0016] S6, train the integrated point cloud network model;

[0017] S7, complete classification and segmentation of the three-dimensional point cloud through the trained point cloud network model.

[0018] Preferably, the selected baseline network can be:

[0019] a network PointNet++ with a downsampling operation;

[0020] a network DGCNN without a downsampling operation;

[0021] a rotation-invariant network RIConv++ with a downsampling operation.

[0022] Further, a corresponding number of rotation-invariant enhancement layers are set for the baseline network:

[0023] For the baseline network PointNet++, which contains a point cloud downsampling operation, a number of single-input-single-output rotation-invariant enhancement layers SISO are set;

[0024] For the baseline network DGCNN, which does not contain a point cloud downsampling operation, a number of single-input-multiple-output rotation-invariant enhancement layers SIMO are set;

[0025] For the baseline network RIConv++, which contains a point cloud downsampling operation, a number of single-input-single-output rotation-invariant enhancement layers SISO are set.

[0026] Further, in step S2, each point in the point cloud data is taken as a center point to divide a local region, and a local spherical coordinate system is constructed based on the local region, specifically:

[0027] S2.1, each point in the point cloud data is taken as , and a plane where is located is taken as an absolute plane;

[0028] S2.2, taking as a center point, k neighbor points of are selected to divide a local region; a plane taking as an origin is recorded as a relative plane, and the plane takes the coordinate value of as a normal vector;

[0029] S2.3, within the aforementioned local area, consider Pointed k Each point includes its own set of neighboring points and establishes a local spherical coordinate system, where z The axis is The coordinate vector value.

[0030] Furthermore, in step S3, in the local spherical coordinate system, a rotationally invariant geometric descriptor is calculated based on the neighborhood point set of the center point, specifically as follows:

[0031] S3.1 Calculate in the local spherical coordinate system. arrive Euclidean distance between :

[0032]

[0033] in Represented as the relative coordinate vector between neighboring points and the center point;

[0034] S3.2, Calculation and The angle between :

[0035]

[0036] in This is to prevent constants with a denominator of zero;

[0037] S3.3, Introduction ,express The projection vector in the corresponding relative plane:

[0038]

[0039] S3.4 After obtaining the corresponding set of projection vectors, calculate the angle between every two adjacent projection vectors. :

[0040]

[0041] Where ±1 indicates whether the calculation is clockwise or counterclockwise.

[0042] Furthermore, regarding the included angle Considering self-loops, when i Sometimes , , .

[0043] Further, in step S4, the point cloud geometry features of the point cloud data are expressed by the SGD, specifically:

[0044] S4.1: The point cloud in the point cloud data is expressed as wherein , is a set of edges with a number of constructed in ;

[0045] S4.2: The edges between the points in the point cloud data are expressed by the SGD and :

[0046] = [ ]

[0047] wherein is the edge between the points expressed by the SGD, [,] is a connection function.

[0048] Further, in step S5, the edges between the points expressed by the SGD are input into the neural network based on the graph structure, and the point cloud geometry features of the point cloud data are extracted by the graph-based spherical feature propagation mechanism; the point cloud geometry features and the neural network based on the graph structure are cascaded into the baseline network to integrate into a point cloud network model; specifically:

[0049] S5.1: The edges between the points expressed by the SGD are input into the neural network based on the graph structure to obtain the point cloud geometry features :

[0050]

[0051] wherein is a nonlinear function realized by the MLP and the learnable parameters ;

[0052] S5.2: The point cloud geometry features are iteratively updated by the graph-based spherical feature propagation mechanism;

[0053] S5.3: A symmetric function is used to reduce the number of dimensions in to adapt to the point cloud features of the feature extractor in the baseline network, and the point cloud geometry features with reduced dimensions are obtained :

[0054]

[0055] wherein MaxPool is set as the maximum pooling function;​​

[0056] For the point cloud of the point cloud data, a set of geometric representations thereof is obtained ;

[0057] S5.4: using an arbitrary nonlinear function representing a feature extractor in the baseline network, with parameters concatenating the point features output by the feature extractor with the learnable parameters, to obtain a point cloud network model, and obtaining final features :

[0058]

[0059] wherein is an arbitrary dimensional input point feature output by the feature extractor, is an arbitrary nonlinear function in the subsequent stage with learnable parameters .

[0060] Further, in step S5.2, the point cloud geometric features are iteratively updated by a graph-based spherical feature propagation mechanism, specifically:

[0061] ;

[0062] wherein m is the number of iterations.

[0063] The three-dimensional point cloud classification and segmentation network based on enhanced point cloud rotation invariance comprises a baseline network module, a local spherical coordinate system module, a geometric descriptor module, a mapping module, a model construction module, a model training module, a classification and segmentation module.

[0064] The baseline network module is configured to select a baseline network, and set a plurality of rotation-invariant enhanced layers corresponding to the baseline network.

[0065] The local spherical coordinate system module is configured to obtain point cloud data, and obtain a neighborhood point set of each point in the point cloud data by using a k-NN algorithm of the baseline network; divide a local region with each point in the point cloud data as a center point, and construct a local spherical coordinate system based on the local region.

[0066] The geometric descriptor module is configured to calculate a geometric descriptor SGD with rotation invariance in the local spherical coordinate system based on the neighborhood point set of the center point.

[0067] The mapping module is configured to express point cloud geometric features of the point cloud data by using the SGD.

[0068] ​The model building module is used to input the edges between points in the SGD-expressed point cloud data into a graph-based neural network, extract the point cloud geometric features of the point cloud data through a graph-based spherical feature transfer mechanism, and cascade the point cloud geometric features with the graph-based neural network into the baseline network to integrate them into a point cloud network model.

[0069] The model training module is used to train the integrated point cloud network model;

[0070] The classification and segmentation module is used to classify and segment 3D point clouds using a trained point cloud network model.

[0071] The beneficial effects of this invention are as follows:

[0072] This invention discloses a method and network for 3D point cloud classification and segmentation based on enhanced rotation invariance. The invention proposes a method to construct a local spherical coordinate system based on a set of neighboring points to calculate rotation-invariant geometric descriptors. A lightweight network is designed to extract rotation-invariant features from the point cloud, and finally, it is cascaded and integrated with a baseline network, thereby improving the network's robustness to rotational perturbations of the input point cloud. Thus, this invention solves the problem that existing technologies cannot universally handle rotational perturbations, and has the characteristics of flexibility and versatility. Attached Figure Description

[0073] Figure 1 This is a flowchart illustrating a 3D point cloud classification and segmentation method based on enhanced point cloud rotation invariance.

[0074] Figure 2 This is a schematic diagram of the specific process of the three-dimensional point cloud classification and segmentation method based on enhanced point cloud rotation invariance in Example 2.

[0075] Figure 3 This is a schematic diagram of the 3D point cloud classification process of the PointNet++ network module in Example 2, which is based on the 3D point cloud classification and segmentation method that enhances the rotation invariance of point clouds.

[0076] Figure 4 This is a schematic diagram of the 3D point cloud classification process of the DGCNN network module in Example 3, which is based on the 3D point cloud classification and segmentation method with enhanced point cloud rotation invariance.

[0077] Figure 5 This is a schematic diagram of the global coordinate system of a 3D point cloud classification and segmentation method based on enhanced point cloud rotation invariance.

[0078] Figure 6 This is a schematic diagram of the local spherical coordinate system of a point cloud based on a 3D point cloud classification and segmentation method that enhances the rotation invariance of point clouds. Detailed Implementation

[0079] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0080] Embodiment 1

[0081] As shown in the following, the three-dimensional point cloud classification and segmentation method based on the enhanced point cloud rotation invariance includes the following steps: Figure 1

[0082] S1, selecting a baseline network, setting a corresponding number of rotation-invariant enhanced layers in the baseline network;

[0083] S2, obtaining point cloud data, obtaining the neighborhood point set of each point in the point cloud data through the k-NN algorithm of the baseline network; dividing a local area with each point in the point cloud data as a center point, and constructing a local spherical coordinate system based on the local area;

[0084] S3, in the local spherical coordinate system, based on the neighborhood point set of the center point, calculating a geometric descriptor SGD with rotation invariance;

[0085] S4, the SGD expresses the edges between the points of the point cloud data;

[0086] S5, inputting the SGD expressing the edges between the points of the point cloud data into a graph structure-based neural network, extracting point cloud geometric features of the point cloud data through a graph-based spherical feature propagation mechanism; cascading the point cloud geometric features and the graph structure-based neural network into the baseline network, and integrating into a point cloud network model;

[0087] S6, training the integrated point cloud network model;

[0088] S7, completing the classification and segmentation of the three-dimensional point cloud through the trained point cloud network model.

[0089] Embodiment 2

[0090] More specifically, in one specific embodiment, the selected baseline network can be:

[0091] a network PointNet++ with downsampling operation;

[0092] a network DGCNN without downsampling operation;

[0093] a rotation-invariant network RIConv++ with downsampling operation.

[0094] In one specific embodiment, as shown in the following, the baseline network is set with a corresponding number of rotation-invariant enhanced layers: Figure 2

[0095] ​​For the baseline network PointNet++, including the downsampling operation of the point cloud, several rotation-invariant enhancement layers (SISO) with single input and single output are set.

[0096] For the baseline network DGCNN, without the downsampling operation of point clouds, several rotation-invariant enhancement layers (SIMO) with single input multiple outputs are set.

[0097] For the baseline network RIConv++, including the downsampling operation of the point cloud, several rotation-invariant enhancement layers (SISO) with single input and single output are set.

[0098] In this embodiment, MIMO includes four dimensions, namely LBRs, corresponding to a certain sampling point ,in Indicates the first Sub-sampling. The output of MIMO is the pooling result after each LBR cascade. SIMO contains four [missing information] Dimensional LBR.

[0099] In one specific embodiment, such as Figure 2 As shown, in step S2, the K-NN algorithm is used to obtain the neighborhood point set of each point, and multiple local spherical coordinate systems are constructed with this as the center point.

[0100] In step S2, such as Figure 5 As shown, local regions are divided with each point in the point cloud data as the center point, and a local spherical coordinate system is constructed based on these local regions, specifically:

[0101] S2.1, Set each point in the point cloud data as... ,Will The plane in which it lies is considered an absolute plane;

[0102] S2.2, with Center point, select k neighboring points ( Define a local area; and use The plane with the origin is called the relative plane, and this plane is defined by... The coordinates are normal vectors;

[0103] S2.3, within the aforementioned local area, consider Pointed k Each point includes its own set of neighboring points and establishes a local spherical coordinate system, where z The axis is The coordinate vector value.

[0104] In this embodiment, as Figure 6As shown, the selected point is used as the origin of the established local spherical coordinate system. ,have The neighboring points include Z represents the Z-axis in this coordinate system. It is Z and The angle between them. It is all The twist (clockwise or counterclockwise) formed after projecting the vector onto this coordinate system.

[0105] In one specific embodiment, in step S3, a rotationally invariant geometric descriptor is calculated based on the neighborhood point set of the center point in the local spherical coordinate system, specifically as follows:

[0106] S3.1 Calculate in the local spherical coordinate system. arrive Euclidean distance between :

[0107]

[0108] in Represented as the relative coordinate vector between neighboring points and the center point;

[0109] S3.2, Calculation and The angle between :

[0110]

[0111] in This is to prevent constants with a denominator of zero;

[0112] S3.3, Introduction ,express The projection vector in the corresponding relative plane:

[0113]

[0114] S3.4 After obtaining the corresponding set of projection vectors, calculate the angle between every two adjacent projection vectors. :

[0115]

[0116] Where ±1 indicates whether the calculation is clockwise or counterclockwise.

[0117] In one specific embodiment, for the included angle Considering self-loops, when i Sometimes , , .

[0118] In one specific embodiment, step S4, expressing the point cloud geometric features of the point cloud data using SGD, specifically involves:

[0119] S4.1: Extract point cloud data from point cloud data Represented as ,in , Is China passed The number of constructs is The set of edges;

[0120] S4.2: Representing points in point cloud data using SGD and The edge between:

[0121] = [ ]

[0122] in, Let [,] be the edge between points in the SGD point cloud data, and [,] be the connection function.

[0123] In this embodiment, points in point cloud data are represented using SGD. and After the edge between, such as Figure 2 As shown, the geometric features of the obtained point cloud data are integrated in a cascaded manner, enabling the limit network to learn the rotation-invariant features of the point cloud without changing its structure.

[0124] In one specific embodiment, in step S5, the edges between points in the SGD-represented point cloud data are input into a graph-based neural network, and the point cloud geometric features are extracted through a graph-based spherical feature transfer mechanism. The point cloud geometric features are then cascaded with the graph-based neural network into a baseline network to integrate them into a point cloud network model. Specifically:

[0125] S5.1: Input the edges between points in the SGD-represented point cloud data into a graph-based network to obtain the geometric features of the point cloud. :

[0126]

[0127] in It is an MLP with learnable parameters Implemented nonlinear functions;

[0128] S5.2: The point cloud geometry features are iteratively updated by a graph-based spherical feature propagation mechanism. are iteratively updated;

[0129] S5.3: A symmetric function is adopted to reduce the number of dimensions of the updated point cloud features in the baseline network to adapt to the point cloud features of the feature extractor, to obtain the reduced-dimension point cloud geometry feature representation :

[0130]

[0131] where MaxPool is set as a max-pooling function;

[0132] For the point cloud data of the point cloud, a set of geometry representations thereof is obtained ;

[0133] S5.4: An arbitrary nonlinear function is adopted to represent the feature extractor in the baseline network, with parameters being learnable parameters, and the is concatenated with the point features output by the feature extractor to obtain a point cloud network model, and to obtain the final feature :

[0134]

[0135] where is an arbitrary f-dimensional input point feature output by the feature extractor, is an arbitrary nonlinear function in the subsequent stage with learnable parameters .

[0136] In one specific embodiment, in the step S5.2, the point cloud geometry features are iteratively updated by a graph-based spherical feature propagation mechanism, specifically as follows:

[0137] ;

[0138] where m is the number of iterations of the iterative update.

[0139] In this embodiment, as shown in Figure 2 , different final features are obtained according to different tasks, and an integrated network output classification probability distribution is obtained.

[0140] In this embodiment, as shown in Figure 2 , a loss function is also set to train the integrated network, to obtain a trained point cloud network model.

[0141] In this embodiment, as shown in Figure 3As shown, for a pre-trained point cloud network model based on the PointNet++ baseline network using SISO, when performing classification:

[0142] N1: Input point cloud (N, d+c), where N represents the number of points and d+c represents the feature dimension of the input points. The downsampling module, grouping module, PointNet feature extraction and local feature extraction are all part of PointNet++.

[0143] N2: Extract the points sampled once (the number of points sampled up to n1 in the figure), extract their set features, and finally cascade them through a pooling layer at the local feature output. Then, it goes through an MLP or fully connected layer to complete the classification or segmentation task of the point cloud.

[0144] This invention, based on existing research methods for point cloud classification and segmentation, aims to address the above problems by studying new deep learning-based point cloud recognition, classification, and segmentation technologies from the following aspects: (1) Rotation-invariant representation: By constructing a local spherical coordinate system, a rotation-invariant geometric descriptor is obtained within the neighborhood point set, and mapped to the feature space through a neural network, thereby further improving the network's robustness to rotational perturbations. (2) General framework: A lightweight network is designed to extract rotation-invariant features and connect them to a conventional point-based network. This makes rotation-invariant representation an additional attribute, rather than a feature that the entire network focuses on. At the same time, considering the differences between different baseline networks, a module is designed to adapt to the changes in point clouds caused by sampling operations in the network.

[0145] This invention can effectively handle rotational disturbances in the real world, thus having wide application value in point cloud recognition and analysis in the field of autonomous driving.

[0146] This invention discloses a method and network for 3D point cloud classification and segmentation based on enhanced rotation invariance. The invention proposes a method to construct a local spherical coordinate system based on a set of neighboring points to calculate rotation-invariant geometric descriptors. A lightweight network is designed to extract rotation-invariant features from the point cloud, and finally, it is cascaded and integrated with a baseline network, thereby improving the network's robustness to rotational perturbations of the input point cloud. Thus, this invention solves the problem that existing technologies cannot universally handle rotational perturbations, and has the characteristics of flexibility and versatility.

[0147] Example 3

[0148] In this embodiment, as Figure 4 As shown, for a pre-trained point cloud network model based on the DGCNN baseline network using SIMO, when performing classification:

[0149] N1: input (N, 3) point cloud, where N represents the number of points, and 3 represents the feature dimension of the input points;

[0150] N2: use the SIMO module to extract its geometric features, and after each MLP layer is subjected to max-pooling, output is cascaded to each EdgeConv in the corresponding DGCNN, and after an MLP and a pooling operation, the classification task of the point cloud can be completed.

[0151] Embodiment 4

[0152] The three-dimensional point cloud classification and segmentation network based on enhanced point cloud rotation invariance comprises a baseline network module, a local spherical coordinate system module, a geometric descriptor module, a mapping module, a model construction module, a model training module, and a classification and segmentation module.

[0153] The baseline network module is used to select a baseline network, and a corresponding number of rotation-invariant enhanced layers are set in the baseline network.

[0154] The local spherical coordinate system module is used to obtain point cloud data, and a k-NN algorithm of the baseline network is used to obtain a neighborhood point set of each point in the point cloud data; a local region is divided with each point in the point cloud data as a center point, and a local spherical coordinate system is constructed based on the local region.

[0155] The geometric descriptor module is used to calculate a geometric descriptor SGD with rotation invariance based on the neighborhood point set of the center point in the local spherical coordinate system.

[0156] The mapping module is used to express point cloud geometric features of the point cloud data through the SGD.

[0157] The model construction module is used to input edges between points expressed by the SGD into a graph structure-based neural network, extract point cloud geometric features of the point cloud data through a graph-based spherical feature transmission mechanism, and integrate the point cloud geometric features and the graph structure-based neural network into a baseline network to form a point cloud network model.

[0158] The model training module is used to train the integrated point cloud network model.

[0159] The classification and segmentation module is used to complete classification and segmentation of three-dimensional point clouds through the trained point cloud network model.

[0160] In this embodiment,

[0161] If the baseline network chooses PointNet++ which contains down-sampling operation of point cloud, the enhanced module MIMO is selected; for the classification network, two 3D point cloud classification and segmentation networks based on enhanced point cloud rotation invariance are used as the first two feature extractors, and the point cloud samples are sampled to 512 and 128 respectively. For the two 3D point cloud classification and segmentation networks based on enhanced point cloud rotation invariance, the number of output channels of the first two layers is set to 32 and 32 respectively, and the number of output channels of the last two layers is set to 64 and 64 respectively, that is, the number of rotation invariant features obtained by sampling is 32+32+64+64=192. =32, that is, the number of rotation invariant features obtained by sampling is 32+32+64+64=192. For the segmentation network, two 3D point cloud classification and segmentation networks based on enhanced point cloud rotation invariance are used, and the number of output channels of the first two layers is set to 32 and 48 respectively, corresponding to the first two feature extractors, that is, the number of rotation invariant features obtained by sampling is 32+32+64+64=192 and 48+48+96+96=288 respectively.

[0162] If the baseline network chooses DGCNN which does not contain down-sampling operation of point cloud, the enhanced module SIMO is selected; for the classification network, considering a static point cloud with =32, and a plurality of MLPs are output by the 3D point cloud classification and segmentation network based on enhanced point cloud rotation invariance, which are finally concatenated into the output of EdgeConv of each layer after maximum pooling; for the segmentation network, since the DGCNN in the segmentation task only uses three EdgeConv layers, the last LBR layer of SIMO needs to be deleted, and =32, and connection is performed after each EdgeConv layer.

[0163] If the baseline network chooses RIConv++ which contains down-sampling operation of point cloud, the enhanced module MIMO is selected; 2 3D point cloud classification and segmentation networks based on enhanced point cloud rotation invariance are used as its feature extractors, and =16, that is, the number of rotation invariant features obtained by sampling is 16+16+32+32=96; for the classification network, the point cloud is sampled to 1024 and 512 on the classification network; for the segmentation network, the point cloud is sampled to 512 and 256 on the segmentation network.

[0164] In this embodiment, classification and segmentation experiments are performed on ModelNet40, ScanObjectNN, ShapeNet and S3DIS datasets. For each task, three cases are set in the present application: (1) training and testing are performed under z-axis (z / z) rotation; (2) training is performed using z-axis rotated data, and testing is performed using randomly rotated data (z / SO3); (3) training and testing are both performed using randomly rotated (SO3 / SO3) data. ​​

[0165] In this embodiment, classification experiments are performed on ModelNet40, which contains 12311 CAD models of 40 classes, with 9843 models for training and 2468 models for testing. In this embodiment, only 1024 points containing 3D coordinate values (x, y, z) in Euclidean space are used as the input of the network, and the batch size is set to 32. Each model is trained for 300 epochs to balance the difference in training time.

[0166] Component segmentation experiments are performed on ShapeNet, which is a dataset containing 16880 CAD models of 16 classes, each containing 2 to 6 components. In this embodiment, 2048 points are selected, and in the case of z / SO3 and SO3 / SO3 rotation, 14006 models are set for training and 2874 models are set for testing.

[0167] In this embodiment, classification experiments are performed on ScanObjectNN, a real-world point cloud dataset captured by an RGB-D camera, which is more challenging than ModelNet40, using the classification network of the three-dimensional point cloud classification and segmentation network based on the enhanced point cloud rotation invariance. This dataset divides 2902 models into 15 classes, with 2319 models for training and 583 models for testing. The dataset includes five difficulty levels, and in the experiment, the most difficult PB_T50_RS with 1024 points and the simplest OBJ_BG under z / SO3 rotation are selected to evaluate the three-dimensional point cloud classification and segmentation network based on the enhanced point cloud rotation invariance.

[0168] Semantic segmentation experiments are performed on S3DIS using the segmentation network of the three-dimensional point cloud classification and segmentation network based on the enhanced point cloud rotation invariance. The dataset includes 6 indoor areas, a total of 272 rooms, and each room can be divided into 13 semantic categories, such as boards, bookshelves, chairs, ceilings, beams, and miscellaneous items. 4096 points are selected as input, each of which is represented as a 9D vector including 3D coordinate values (x, y, z), RGB, and normalized space.

[0169] The experimental results show that the method provided in this embodiment significantly improves the accuracy for classification and segmentation tasks, and the model after introducing the three-dimensional point cloud classification and segmentation network based on the enhanced point cloud rotation invariance successfully achieves the most advanced performance in terms of rotation robustness.

[0170] Embodiment 6

[0171] In this embodiment

[0172] For the segmentation network, the DGCNN in the segmentation task uses three EdgeConv layers, the last LBR layer of SIMO needs to be removed, set = 32, and concatenate after each EdgeConv layer;

[0173] Obviously, the above embodiments of the application are only examples for clearly illustrating the application, and are not intended to limit the implementation modes of the application. Any modification, equivalent replacement, and improvement, etc. within the spirit and principle of the application should be included in the protection scope of the claims of the application.

Claims

1. A method for 3D point cloud classification and segmentation based on enhanced point cloud rotation invariance, characterized in that: The method comprises the following steps: S1, selecting a baseline network, setting a corresponding number of layers of rotation-invariant enhanced layers in the baseline network; S2, obtaining point cloud data, obtaining a neighborhood point set of each point in the point cloud data through a k-NN algorithm of the baseline network; dividing a local area with each point in the point cloud data as a center point, and constructing a local spherical coordinate system based on the local area, which specifically comprises: S2.1, set each point in the point cloud data as , the plane where the point is located as an absolute plane; S2.2、with as the center point, select k neighbors of to divide a local area; the plane with as the origin is recorded as the relative plane, and the plane takes the coordinate value of as the normal vector; S2.3, in the local region, considering the point k including its own neighbor point set and establishing a local spherical coordinate system, wherein z the axis is coordinate vector value; S3, in the local spherical coordinate system, based on the neighborhood point set of the center point, a geometric descriptor SGD with rotation invariance is calculated, which specifically comprises: S3.

1. Calculate the Euclidean distance between the local spherical coordinates of the two points to :​ wherein is expressed as a relative coordinate vector between the neighborhood point and the center point; S3.2, compute between the included angle between : wherein is a constant to prevent the denominator from being zero; S3.3, introducing a vector , representing Projection vector in the corresponding relative plane: S3.4, after getting the corresponding set of projection vectors, calculate the included angle of each two adjacent projection vectors : Wherein, ±1 represents clockwise or counterclockwise calculation; S4, expressing the edges between the points of the point cloud data through SGD; S5, inputting the edges between the points of the point cloud data expressed by SGD into a neural network based on a graph structure, extracting point cloud geometric features of the point cloud data through a graph-based spherical feature transmission mechanism; cascading the point cloud geometric features and the neural network based on the graph structure into the baseline network, and integrating into a point cloud network model; S6, training the integrated point cloud network model; S7, completing classification and segmentation of three-dimensional point cloud through the trained point cloud network model.

2. The method of claim 1, wherein: The selected baseline network can be: A network with downsampling operation PointNet++; A network without downsampling operation DGCNN; A rotation-invariant network with downsampling operation RIConv++.

3. The method of claim 2, wherein: Setting a corresponding number of layers of rotation-invariant enhanced layers in the baseline network: For the baseline network PointNet++, containing a point cloud downsampling operation, a number of single-input single-output rotation-invariant enhanced layers SISO are set; For the baseline network DGCNN, not containing a point cloud downsampling operation, a number of single-input multi-output rotation-invariant enhanced layers SIMO are set; For the baseline network RIConv++, containing a point cloud downsampling operation, a number of single-input single-output rotation-invariant enhanced layers SISO are set.

4. The method of claim 1, wherein: For the included angle , considering the self-loop, when i there are , , .

5. The method of claim 1, wherein: In step S4, the point cloud geometric features of the point cloud data are expressed by SGD, which specifically comprises: S4.1: constructing a graph G = (V, E) from the point cloud data in the point cloud is represented as wherein , is constructed in by the number of edges E is ​ S4.2: Expressing points in the point cloud data by SGD and between edges: = [ ] wherein, is an edge between points of the SGD expressed point cloud data, [, ] is a connection function.

6. The method of claim 1, wherein: In step S5, the edges between the points of the point cloud data expressed by SGD are input into a neural network based on a graph structure, and the point cloud geometric features of the point cloud data are extracted through a graph-based spherical feature transmission mechanism; the point cloud geometric features and the neural network based on the graph structure are cascaded into the baseline network, and integrated into a point cloud network model; specifically comprising: S5.1: input edges between points of the SGD expressed point cloud data into a graph structure based network to obtain point cloud geometry features : wherein is a non-linear function implemented by an MLP with learnable parameters ; S5.2: pass the point cloud geometry features through a graph-based spherical feature propagation mechanism iterative update; S5.3: Adopting symmetric function, reducing the number of dimensions after update The number of dimensions is reduced to adapt to the point cloud features of the feature extractor in the baseline network, to obtain the point cloud geometry feature representation after reducing the dimensions : Wherein, MaxPool is set as a maximum pooling function; For a point cloud of point cloud data, obtaining a set of geometric representations thereof ; S5.4: adopt any nonlinear function representing a feature extractor in the baseline network, with parameters as learnable parameters, concatenate with the point features output by the feature extractor to obtain a point cloud network model, and obtain final features : wherein is an arbitrary f-dimensional input point feature output by the feature extractor, is an arbitrary non-linear function in the subsequent stage with learnable parameters .

7. The three-dimensional point cloud classification and segmentation method based on enhanced point cloud rotation invariance according to claim 6, characterized in that: In the step S5.2, the point cloud geometry features are transferred based on a graph-based spherical feature transfer mechanism Iterative updating is performed, specifically: ； wherein, m is the number of iterations.

8. A three-dimensional point cloud classification and segmentation network based on enhanced point cloud rotation invariance, characterized in that: For realizing the method according to any one of claims 1-7, comprising a baseline network module, a local spherical coordinate system module, a geometric descriptor module, a mapping module, a model construction module, a model training module, a classification and segmentation module; The baseline network module is used for selecting a baseline network, and setting a corresponding number of layers of rotation-invariant enhanced layers in the baseline network; The local spherical coordinate system module is used to obtain point cloud data, and a baseline network is used to obtain a local spherical coordinate system of each point in the point cloud data k The NN algorithm obtains a neighborhood point set of each point in the point cloud data, divides a local region with each point in the point cloud data as a center point, and constructs a local spherical coordinate system based on the local region. The geometric descriptor module is used for calculating a geometric descriptor SGD with rotation invariance in the local spherical coordinate system based on a neighborhood point set of a center point; The mapping module is used to express point cloud geometry features of point cloud data through an SGD; The model construction module is used to input edges between points of the point cloud data expressed by the SGD into a graph structure-based neural network, extract point cloud geometry features of the point cloud data through a graph-based spherical feature transmission mechanism, and cascade the point cloud geometry features and the graph structure-based neural network into a baseline network to integrate into a point cloud network model; The model training module is used to train the integrated point cloud network model; The classification and segmentation module is used to complete classification and segmentation of three-dimensional point cloud through the trained point cloud network model.

Images