A 3D point cloud model classification method and system based on regional correlation modeling

Abstract

The present invention relates to a three-dimensional point cloud model classification method and system based on regional correlation modeling. The method comprises the following steps: S1, using the equal projection area method, projecting the three-dimensional point cloud model onto a two-dimensional plane, and dividing it into n×m regions, where n represents the number of regional divisions in the set azimuth direction, and m represents the number of regional divisions in the corresponding polar angle direction; S2, extracting features from each region respectively to obtain M-dimensional features of each region respectively, and using n×m×M-dimensional features to represent the three-dimensional point cloud model; S3, using the n×m×M-dimensional features as input, and using a convolutional neural network to learn and obtain the correlation features between the regions; S4, the convolutional neural network classifies the three-dimensional point cloud model based on the correlation features. The present invention models the correlation between different regions in the three-dimensional point cloud model, integrates the local features of the sub-regions and the correlation features between different sub-regions, and effectively improves the classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud classification, and in particular relates to a three-dimensional point cloud model classification method and system based on regional correlation modeling. Background Art

[0002] Point cloud is a data form used to describe the topological distribution and surface morphology of an object in three-dimensional space. It is widely used in the fields of autonomous driving, robotics, remote sensing, and medical treatment by representing the set of three-dimensional coordinates of discrete points sampled on the surface of the object. From the perspective of technical implementation, point cloud acquisition methods can be divided into active time-of-flight method, structured light projection method, and passive motion recovery structure, binocular stereo imaging, multi-view imaging, etc. Using images as the information sensing unit of the object, image processing technology can be used to extract the height information of the object expressed by the pixel position, and at the same time, combined with the spatial geometric structure, the three-dimensional coordinates of the corresponding pixel position can be calculated. Therefore, point cloud data has disorder (that is, the information represented by the point cloud does not change when the order of the points is changed) and geometric invariance (the information represented by the point cloud does not change when the point cloud is subjected to scale transformations such as translation, scaling, and rotation). Object classification and recognition is a classic problem in the field of computer vision. The classification and recognition of three-dimensional objects is a basic problem that needs to be solved in the fields of three-dimensional vision-guided robotic arm grasping, unmanned driving, and digital intelligent medical care.

[0003] The classification methods of 3D point cloud models can be divided into those based on artificially designed feature extraction and those based on deep learning. Using the differential geometric features of point clouds, feature descriptors that reflect the local and global nature of point clouds are designed, such as fast point feature histograms, point pair histograms, curvature, local feature descriptors, etc., which are widely used in point cloud classification. However, due to the limitations of artificially designed features, high-level semantic information cannot be reflected, and features that reflect the local and global distribution of point clouds cannot be fully extracted. Deep learning technology can self-learn high-level semantic features corresponding to object labels by constructing a suitable network framework based on sufficient data set training, which can effectively improve feature extraction capabilities and classification effects. There have been extensive studies and applications in the fields of 2D image recognition, classification, and segmentation. For 3D point cloud data, due to the disorder and unstructured nature of point clouds, traditional convolutional neural networks are not applicable. For this reason, researchers have also proposed different methods and models to meet these challenges. For example, the multi-view MVCNN obtains 2D rendering images from different perspectives of the 3D model, uses them as the original training data, uses the classic and mature 2D convolutional neural network for training, and uses the trained model to identify and classify the 3D model; the voxel-based OctNet uses octrees to divide the space, downsamples the point cloud into sparse voxel data, and performs convolution, pooling and other operations to achieve recognition and classification; the pointNet based on point-by-point processing uses a multi-layer perceptron to map each point dimension to a high dimension, and uses maximum pooling for feature extraction, and uses a classifier for classification after dimensionality reduction; the P-based PointNet++, which is improved by the ointNet method, selects several neighborhoods in the point cloud, uses PointNet to extract features in each neighborhood, and performs classification. SPLATNet, based on spatial transformation, transforms the point cloud in Euclidean space into high-dimensional space, transforms the features of the points to the regular lattice vertices in the high-dimensional space, and uses hash tables as indexes for convolution to achieve classification. RS-CNN, based on neighborhood geometric relationships, divides the point cloud into several neighborhoods, calculates the distance between points in the neighborhood, maps it to high dimensions using a multi-layer perceptron, and uses symmetric functions to aggregate features for recognition and classification.

[0004] The key to point cloud classification is to extract local and global features that reflect object classification information in the point cloud. The above-mentioned PointNet uses MLP to extract high-dimensional feature vectors and uses the maximum pooling method to extract global features, which does not fully reflect the features of the local area of ​​the object. In addition to the local and global features of different scales of the point cloud itself, the relationship between different regions in the point cloud model is also an important feature of classification, providing high-level semantic information related to the type. For this reason, a classification method that can consider local features and the correlation between different regions is urgently needed. Summary of the invention

[0005] In view of the above problems existing in the prior art, the present invention provides a three-dimensional point cloud model classification method and system based on regional correlation modeling, which utilizes convolutional neural networks to learn the relationship between regions and perform classification, thereby obtaining the correlation characteristics between regions in the point cloud and improving the classification accuracy.

[0006] The present invention adopts the following technical solutions:

[0007] A three-dimensional point cloud model classification method based on regional correlation modeling includes the following steps:

[0008] S1. Use the equal projection area method to project the three-dimensional point cloud model onto a two-dimensional plane and divide it into n×m regions, where n represents the number of region divisions in the set azimuth direction and m represents the number of region divisions in the corresponding polar angle direction;

[0009] S2, extracting features from each region respectively to obtain M-dimensional features of each region respectively, and using n×m×M-dimensional features to represent the three-dimensional point cloud model;

[0010] S3, taking n×m×M dimensional features as input, using convolutional neural network to learn and obtain the correlation features between regions;

[0011] S4. Convolutional neural network classifies 3D point cloud models based on correlation features.

[0012] As a preferred solution, before step S1, the following steps are also included:

[0013] S1.1. Read and display the 3D point cloud model;

[0014] S1.2, calculate the centroid and main direction of the 3D point cloud model;

[0015] S1.3. Based on the centroid and the main direction, the spatial coordinate system and the three-dimensional point cloud model are corrected in position and posture so that the three-dimensional point cloud model is located on the xOy plane of the spatial coordinate system;

[0016] S1.4. Construct a spherical coordinate system with the origin of the space coordinate system after posture correction as the center, and divide the entire spherical surface into equal areas to obtain n×m regions.

[0017] As a preferred solution, in step S1.2, the principal component analysis method is specifically used to calculate the main direction of the three-dimensional point cloud model.

[0018] As a preferred solution, in step S1.3, posture correction includes:

[0019] Move the origin of the spatial coordinate system to the center of mass;

[0020] The pose of the 3D point cloud model is corrected based on the main direction so that the 3D point cloud model is located on the xOy plane of the spatial coordinate system.

[0021] As a preferred solution, in step S1.4:

[0022] When setting the azimuth The azimuth interval in the direction is The number of uniform divisions is n, then the polar angle θ and the polar angle interval Δθ satisfy:

[0023] c=(θ i+1 -θ i )(sinθ i+1 +sinθ i ),

[0024] Among them, θ i+1 ,θ i Respectively represent the size of the i+1th and ith polar angles, and the polar angle interval Δθ=θ i+1 -θ i , m represents the azimuth angle in the spherical coordinate system The number of regions divided in the range in the direction of the polar angle θ, c represents a constant.

[0025] As a preferred solution, the following steps are further included between step S1 and step S2:

[0026] Count the number of points in each area. For areas with point numbers within the preset range, increase the number of points in the area to the upper limit of the preset range. For areas with point numbers below the preset range, directly discard them.

[0027] As a preferred solution, step S2 includes the following steps:

[0028] S2.1. Use PointNet to train on the ShapeNet dataset to obtain model parameters;

[0029] S2.2, using the trained model parameters to extract features from each region respectively, so as to obtain M-dimensional features of each region respectively;

[0030] S2.3. Use n×m×M dimensional features to represent the three-dimensional point cloud model.

[0031] As a preferred solution, in step S2.1, the training process includes the following steps:

[0032] S2.1.1. PointNet randomly samples 100 points in the ShapeNet dataset without repeatability and represents them as 100×3-dimensional features;

[0033] S2.1.2. Use MLP with shared weights to process each point and map its features to 1024 dimensions to obtain 100×1024-dimensional features.

[0034] S2.1.3, use maximum pooling to extract 1024-dimensional features and obtain 1×1024-dimensional features;

[0035] S2.1.4, through the fully connected layer, the 1×1024-dimensional features are input into the classifier, and the classifier outputs 16 prediction results, and the 16 prediction results are divided into 16 classes;

[0036] S2.1.5. Obtain the trained model parameters.

[0037] As a preferred solution, n=6, m=6, M=1024.

[0038] Also provided is a three-dimensional point cloud model classification system based on regional correlation modeling, based on the above classification method, comprising a region division module, a feature extraction module, and a convolutional neural network module connected in sequence, the convolutional neural network module comprising a connected feature learning unit and a classification unit, the feature learning unit being connected to the feature extraction module;

[0039] The region division module is used to project the three-dimensional point cloud model onto a two-dimensional plane using the equal projection area method, and divide it into n×m regions, where n represents the number of region divisions in the set azimuth direction, and m represents the number of region divisions in the corresponding polar angle direction;

[0040] A feature extraction module is used to extract features from each region to obtain M-dimensional features of each region, and use n×m×M-dimensional features to represent a three-dimensional point cloud model;

[0041] The feature learning unit is used to take the n×m×M dimensional features as input and use the convolutional neural network to learn the correlation features between the regions;

[0042] The classification unit is used to classify the 3D point cloud model based on correlation features.

[0043] The beneficial effects of the present invention are:

[0044] The present invention uses the model parameters trained by PointNet to extract features from the three-dimensional point cloud model, and can obtain deeper features of the point cloud data;

[0045] The present invention divides the three-dimensional point cloud model into n×m areas, uses a convolutional neural network to learn the relationship between the areas, and performs classification, so as to obtain the correlation characteristics between the areas in the point cloud and improve the classification accuracy.

[0046] The present invention converts unstructured point cloud data into a structured image data expression by dividing it into equal surface areas in a spherical coordinate system and combining it with a global feature extraction method of the point cloud. The present invention utilizes the good feature extraction technology of convolutional neural networks in structured image data to model the correlation between different regions in the three-dimensional point cloud model. Since the present method integrates the local features of the sub-regions and the correlation features between different sub-regions, it can effectively improve the classification accuracy of the three-dimensional point cloud model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flow chart of a three-dimensional point cloud model classification method based on regional correlation modeling according to the present invention;

[0049] Figure 2 It is a schematic diagram of the implementation steps of a three-dimensional point cloud model classification method based on regional correlation modeling described in the present invention;

[0050] Figure 3 It is a schematic diagram of projecting a three-dimensional point cloud model onto a two-dimensional plane in a three-dimensional point cloud model classification method based on regional correlation modeling described in the present invention;

[0051] Figure 4 It is a structural schematic diagram of a three-dimensional point cloud model classification system based on regional correlation modeling described in the present invention. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present invention through specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] Reference Figure 1 , 2 This embodiment provides a three-dimensional point cloud model classification method based on regional correlation modeling, specifically for classifying tooth three-dimensional point cloud models, including the steps of:

[0054] S1. Using the equal projection area method, the three-dimensional point cloud model of the tooth is projected onto a two-dimensional plane and divided into n×m regions, where n represents the number of region divisions in the set azimuth direction, and m represents the number of region divisions in the corresponding polar angle direction;

[0055] S2, extracting features from each region respectively to obtain M-dimensional features of each region respectively, and using n×m×M-dimensional features to represent the three-dimensional point cloud model of the tooth;

[0056] S3, taking n×m×M dimensional features as input, using convolutional neural network CNN to learn and obtain the correlation features between regions;

[0057] S4. Convolutional neural network classifies the tooth 3D point cloud model based on correlation features.

[0058] Specifically:

[0059] Before step S1, the step

[0060] S1.1. Read and display the three-dimensional point cloud model of teeth;

[0061] S1.2, calculate the centroid and main direction of the tooth 3D point cloud model;

[0062] S1.3. Based on the centroid and the main direction, the spatial coordinate system and the three-dimensional point cloud model of the tooth are corrected in position so that the occlusal surface of the three-dimensional point cloud model of the tooth is parallel to the xOy plane of the spatial coordinate system and perpendicular to the z axis;

[0063] S1.4. Construct a spherical coordinate system with the origin of the space coordinate system after posture correction as the center, and divide the entire spherical surface into equal areas to obtain n×m regions. The equal projection area method is used for region division to ensure that for point cloud data that approximates a sphere or hemisphere, the area of ​​each divided region is the same, that is, when the density is uniform, the number of points is approximate.

[0064] In step S1.2, the principal component analysis method is specifically used to calculate the main direction of the tooth three-dimensional point cloud model, that is, to calculate the eigenvector direction of its covariance. Furthermore, in step S1.3, the posture of the tooth three-dimensional point cloud model is corrected according to this direction so that the occlusal surface of the tooth three-dimensional point cloud model is parallel to the xOy plane of the spatial coordinate system and perpendicular to the z axis.

[0065] In step S1.3, posture correction also includes: moving the origin of the spatial coordinate system to the center of mass position.

[0066] Reference Figure 3 , in step S1.4:

[0067] When setting the azimuth The azimuth interval in the direction is The number of uniform divisions is n, then the polar angle θ and the polar angle interval Δθ satisfy:

[0068] c=(θ i+1 -θ i )(sinθ i+1 +sinθ i ),

[0069] Among them, θ i+1 ,θ i Respectively represent the size of the i+1th and ith polar angles, and the polar angle interval Δθ=θ i+1 -θ i , m represents the azimuth angle in the spherical coordinate system The number of regions divided in the range in the direction of the polar angle θ, c represents a constant.

[0070] In this embodiment, n=6 and m=6.

[0071] The steps between step S1 and step S2 also include:

[0072] Count the number of points in each area. For areas with point numbers within the preset range, increase the number of points in the area to the upper limit of the preset range. For areas with point numbers below the preset range, directly discard them.

[0073] In this embodiment, the preset range is 50 to 100. Random interpolation is performed on the area with the number of points between 50 and 100 to increase the number of points to 100, and the area with the number of points below 50 is directly discarded.

[0074] Step S2 includes the following steps:

[0075] S2.1. Use PointNet to train on the ShapeNet dataset to obtain model parameters;

[0076] Specifically: ShapeNet dataset is a large-scale public dataset of 3D images with rich annotations, with more than 16,000 samples, each with about 2,500 points, and 16 categories. When using PointNet to train the ShapeNet dataset, the specific process is as follows:

[0077] S2.1.1. PointNet randomly samples 100 points in the ShapeNet dataset without repeatability and represents them as 100×3-dimensional features;

[0078] S2.1.2. Use MLP with shared weights to process each point and map its features to 1024 dimensions to obtain 100×1024-dimensional features.

[0079] S2.1.3, use maximum pooling to extract 1024-dimensional features and obtain 1×1024-dimensional features;

[0080] S2.1.4, through the fully connected layer, the 1×1024-dimensional features are input into the classifier, and the classifier outputs 16 prediction results, and the 16 prediction results are divided into 16 classes;

[0081] S2.1.5. Obtain the trained model parameters, that is, M=1024 in this embodiment.

[0082] The specific parameters during training are set as:

[0083] The number of sampling points is set to 100, training is performed for 100 rounds, the learning rate is 0.001, and the final test accuracy in the ShapeNet dataset is 96.5%.

[0084] S2.2. Use the trained model parameters to extract features from each region to obtain 1024-dimensional features for each region.

[0085] S2.3. Use 6×6×1024-dimensional features to represent the tooth 3D point cloud model.

[0086] In step S3, a convolutional neural network algorithm is used to classify the three-dimensional model, specifically:

[0087] Each type of feature data representing the three-dimensional model is divided into a training set and a test set in a ratio of 7:3. The training set data is input into the convolutional neural network, and the classification accuracy is tested with the test set data. Among them, the feature data of the three-dimensional model is a tooth point cloud data set collected manually and classified by experts, including four categories of upper and lower first and second molars. It is divided into training set and test set according to the needs of the neural network algorithm.

[0088] The input dimension of each tooth 3D point cloud model feature data is 6×6×1024.

[0089] The training process uses Adam as the optimizer, cross entropy loss as the loss function, the learning rate is set to 0.001, and training is performed for 50 rounds.

[0090] The convolutional neural network uses ReLU as the activation function and softmax as the classifier.

[0091] The detailed design of the convolutional neural network is shown in the following table:

[0092]

[0093]

[0094] The final recognition accuracy tested on the test set was 84.3%.

[0095] Since the convolutional neural network uses convolution kernels to scan different positions on the image, after the nonlinear action of the activation function and pooling, and after multiple layers of convolution, the obtained feature vector reflects the information of the image at different scales, that is, it reflects the spatial correlation information of the point cloud data.

[0096] It can be seen that the present invention converts unstructured point cloud data into a structured image data expression form by dividing it into equal surface areas in a spherical coordinate system and combining it with a global feature extraction method of the point cloud. The convolutional neural network uses good feature extraction technology for structured image data to model the correlation between different regions in the three-dimensional point cloud model. Since this method integrates the local features of the sub-regions and the correlation features between different sub-regions, it can effectively improve the classification accuracy of the three-dimensional point cloud model.

[0097] Embodiment 2:

[0098] Reference Figure 4 As shown, this embodiment provides a three-dimensional point cloud model classification system based on regional correlation modeling, based on the classification method described in the above embodiment 1, including a region division module, a feature extraction module, and a convolutional neural network module connected in sequence, the convolutional neural network module includes a connected feature learning unit and a classification unit, and the feature learning unit is connected to the feature extraction module;

[0099] The region division module is used to project the three-dimensional point cloud model onto a two-dimensional plane using the equal projection area method, and divide it into n×m regions, where n represents the number of region divisions in the set azimuth direction, and m represents the number of region divisions in the corresponding polar angle direction;

[0100] A feature extraction module is used to extract features from each region to obtain M-dimensional features of each region, and use n×m×M-dimensional features to represent a three-dimensional point cloud model;

[0101] The feature learning unit is used to take the n×m×M dimensional features as input and use the convolutional neural network to learn the correlation features between the regions;

[0102] The classification unit is used to classify the 3D point cloud model based on correlation features.

[0103] It should be noted that the three-dimensional point cloud model classification system based on regional correlation modeling provided in this embodiment is similar to that in the first embodiment and will not be described in detail here.

[0104] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope of the present invention.

Claims

1. A three-dimensional point cloud model classification method based on regional correlation modeling, characterized in that: Includes steps: S1. Use the equal projection area method to project the three-dimensional point cloud model onto a two-dimensional plane and divide it into n×m regions, where n represents the number of region divisions in the set azimuth direction and m represents the number of region divisions in the corresponding polar angle direction; S2, extracting features from each region respectively to obtain M-dimensional features of each region respectively, and using n×m×M-dimensional features to represent the three-dimensional point cloud model; S3, taking n×m×M dimensional features as input, using convolutional neural network to learn and obtain the correlation features between regions; S4, convolutional neural network classifies 3D point cloud models based on correlation features; Before step S1, the method further includes the following steps: S1.

1. Read and display the 3D point cloud model; S1.2, calculate the centroid and main direction of the 3D point cloud model; S1.

3. Based on the centroid and the main direction, the spatial coordinate system and the three-dimensional point cloud model are corrected in position and posture so that the three-dimensional point cloud model is located on the xOy plane of the spatial coordinate system; S1.4, construct a spherical coordinate system with the origin of the space coordinate system after posture correction as the center, and divide the entire spherical surface into equal areas to obtain n×m areas; In step S1.4: In the direction of the set azimuth angle φ, a uniform division with an azimuth interval of Δφ and a number of n is performed, then the polar angle θ and the polar angle interval Δθ satisfy: c=(θ i+1 -θ i (sinθ) i+1 +sinθ i ), Among them, θ i+1 ,θ i Respectively represent the size of the i+1th and ith polar angles, and the polar angle interval Δθ=θ i+1 -θ i , m represents the number of regions divided in the direction of the polar angle θ within the azimuth angle range of [φ, φ+Δφ] in the spherical coordinate system, and c represents a constant.

2. A three-dimensional point cloud model classification method based on regional correlation modeling according to claim 1, characterized in that: In step S1.2, the principal component analysis method is specifically used to calculate the main direction of the three-dimensional point cloud model.

3. The three-dimensional point cloud model classification method based on regional correlation modeling according to claim 1, characterized in that: In step S1.3, posture correction includes: Move the origin of the spatial coordinate system to the center of mass; The pose of the 3D point cloud model is corrected based on the main direction so that the 3D point cloud model is located on the xOy plane of the spatial coordinate system.

4. The three-dimensional point cloud model classification method based on regional correlation modeling according to claim 1, characterized in that: The steps between step S1 and step S2 also include: Count the number of points in each area. For areas with point numbers within the preset range, increase the number of points in the area to the upper limit of the preset range. For areas with point numbers below the preset range, directly discard them.

5. The three-dimensional point cloud model classification method based on regional correlation modeling according to claim 1, characterized in that: Step S2 includes the following steps: S2.

1. Use PointNet to train on the ShapeNet dataset to obtain model parameters; S2.2, using the trained model parameters to extract features from each region respectively, so as to obtain M-dimensional features of each region respectively; S2.

3. Use n×m×M dimensional features to represent the three-dimensional point cloud model.

6. The three-dimensional point cloud model classification method based on regional correlation modeling according to claim 5, characterized in that: In step S2.1, the training process includes the following steps: S2.1.

1. PointNet randomly samples 100 points in the ShapeNet dataset without repeatability and represents them as 100×3-dimensional features; S2.1.

2. Use MLP with shared weights to process each point and map its features to 1024 dimensions to obtain 100×1024-dimensional features. S2.1.3, use maximum pooling to extract 1024-dimensional features and obtain 1×1024-dimensional features; S2.1.4, through the fully connected layer, the 1×1024-dimensional features are input into the classifier, and the classifier outputs 16 prediction results, and the 16 prediction results are divided into 16 classes; S2.1.

5. Obtain the trained model parameters.

7. The three-dimensional point cloud model classification method based on regional correlation modeling according to claim 6, characterized in that: n=6, m=6, M=1024.

8. A three-dimensional point cloud model classification system based on regional correlation modeling, based on the classification method according to any one of claims 1 to 7, characterized in that: It includes a region division module, a feature extraction module, and a convolutional neural network module which are connected in sequence, wherein the convolutional neural network module includes a connected feature learning unit and a classification unit, and the feature learning unit is connected to the feature extraction module; The region division module is used to project the three-dimensional point cloud model onto a two-dimensional plane using the equal projection area method, and divide it into n×m regions, where n represents the number of region divisions in the set azimuth direction, and m represents the number of region divisions in the corresponding polar angle direction; A feature extraction module is used to extract features from each region to obtain M-dimensional features of each region, and use n×m×M-dimensional features to represent a three-dimensional point cloud model; The feature learning unit is used to take the n×m×M dimensional features as input and use the convolutional neural network to learn the correlation features between the regions; The classification unit is used to classify the 3D point cloud model based on correlation features.

Images