An efficient point cloud classification method and system based on position fusion
By employing a location-based fusion approach that combines global and local features, and utilizing KNN local relationship graphs and multi-scale feature strategies, the high computational complexity in point cloud data processing is addressed, achieving efficient point cloud classification.
Patent Information
- Application Number
- CN202211533503.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-02
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2042-12-02
AI Technical Summary
The unstructured nature of point cloud data and the interactivity between points make it difficult for traditional two-dimensional convolutional neural networks to process. Existing attention mechanisms have high computational complexity, resulting in low efficiency in point cloud classification.
A location-based fusion approach is adopted, which combines global and local features, uses KNN local relationship graphs and multi-scale feature strategies to reduce the number of point cloud points and construct a weight matrix for feature extraction and pooling, thus avoiding high computational requirements.
It achieves efficient point cloud classification, reduces computational complexity, and maintains high classification accuracy. It is suitable for the characteristics of point cloud data and is well-suited for point cloud classification and recognition.
Smart Images
Figure CN115841593B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of artificial intelligence, and relates to an efficient point cloud classification method and system based on position fusion, which can be used in various point cloud generation scenes. BACKGROUND
[0002] Deep learning of 3D point cloud data has promoted the rapid development of the automatic driving field. However, point cloud data is unordered and unstructured. This irregular data has the interaction between points, which makes it difficult for traditional two-dimensional convolutional neural networks to be applied to the point cloud field, and it is a challenging task to process three-dimensional point clouds. In recent years, various processing methods have appeared in the point cloud field, and the most widely used method is the point-based method. In addition, the attention mechanism in deep learning is widely used in various fields of deep learning. As a result, many methods have been introduced into the point cloud field. The attention mechanism of these methods is the point in the point cloud, and various deformations of the attention method are proposed. Although these methods have good results, they have the problem of excessive computational demand. The calculation method of the attention mechanism will result in a calculation complexity proportional to the square of the number of points processed in the point cloud, which greatly increases the calculation amount. Since the number of points in the point cloud is generally one thousand or even larger, the method of processing point cloud using attention often leads to the problem of slow calculation.
[0003] The non-independence and point interaction of point cloud data make it necessary to effectively learn the local features of point cloud. At the same time, the local shape in the point cloud also contributes to the global shape of the point cloud, so a classification method suitable for the characteristics of point cloud data and effectively learning the local relationship of point cloud is of great significance to point cloud data. SUMMARY
[0004] To solve the above problems, the application provides an efficient point cloud classification method based on position fusion, which uses the strategy of fusing the global position and local position information of the point cloud, and fuses the global features and local features, respectively as weights and elements, learns the relationship between the two, without using high computational demand restrictions, and has high classification accuracy. At the same time, the multi-scale feature strategy is introduced, which can further expand the receptive field and learn the shape of the point cloud, solving the problems existing in the prior art.
[0005] Another object of the application is to provide an efficient point cloud classification system based on position fusion.
[0006] The technical scheme adopted by the application is an efficient point cloud classification method based on position fusion, comprising the following steps:
[0007] S1, extracting an initial embedding representation of a point in the point cloud from an input point cloud three-dimensional coordinate, obtaining initial global position information and global feature information of the point in the point cloud;
[0008] S2, reducing the number of points in the point cloud by using farthest point downsampling without changing the basic shape of the point cloud, and taking the downsampled point as a centroid, taking the point before downsampling as a search space, searching for K-Nearest Neighbor (KNN) neighbors centered on the centroid, and establishing a KNN local relationship graph structure;
[0009] S3, constructing F matrix and P matrix connecting feature information and connection position information through KNN local adjacency relationship, obtaining weight matrix and element matrix with global information and local information;
[0010] S4, mapping F matrix and P matrix to the same dimension, and performing Hadamard product operation on the two to obtain a new F' matrix, i.e. a new feature matrix, and inputting it into S5;
[0011] S5, using a linear transformation function and local maximum pooling to obtain a new global feature representation of each centroid from the F' matrix obtained in S4, and inputting it to the next stage;
[0012] S6, receiving the new global feature representation of S5, and repeating steps S2, S3, S4, and S5 to obtain feature representations of points with different sampling numbers;
[0013] S7, inputting the feature representation obtained in S6 into a mapping function and further inputting it into a mapping function, and using a residual structure to connect the results obtained twice in series to obtain the features of all points in the point cloud;
[0014] S8, in order to classify the point cloud, using a maximum pooling layer to pool the features of all points obtained in S7 in the dimension of the number of points to obtain a global feature representation of the point cloud;
[0015] S9, using a classifier to classify and identify the point cloud based on the global feature representation.
[0016] Further, in step S1, a one-time linear, BatchNorm and ReLU layer is used to convert the input three-dimensional coordinates into a new position embedding representation p, and this is copied to the initial feature embedding representation f;
[0017] Further, in step S2, the number of points in the initial point cloud set is defined as N, and the number of points after downsampling at stage i is N i , respectively as shown in formula (1-1) and formula (1-2):
[0018]
[0019]
[0020] S21, a KNN graph of different sizes is established around the same centroid, and different scale features are concatenated, different dimensions are used for different resolution levels of features, and different size receptive fields are obtained;
[0021] Further, in step S3, specifically:
[0022] S31, we define: the absolute position of point i is p i Around this point, a KNN graph is established to obtain the k neighbors of this point. Let one of the neighbors be point j, then the relative position information between point i and point j is as shown in formula (1-3):
[0023] Δp ij =p j -p i (1-3)
[0024] The P matrix is obtained by fusing the relative position information and the absolute position information, and the P matrix block of point i can be represented as shown in formula (1-4):
[0025] P i =R k [concat(p i ,Δp ij )] (1-4)
[0026] Wherein, R k represents repeating the operation k times, j represents the k neighbors of point i, and concat represents concatenation operation.
[0027] S32, the F matrix is obtained by fusing the relative feature information and the absolute feature information, which are defined as f i and Δf ij , wherein the relative feature information between point i and point j is as shown in formula (1-5):
[0028] Δf ij =f j -f i (1-5)
[0029] The F matrix block of point i can be represented as shown in formula (1-6):
[0030] F i =R k [concat(f i ,Δf ij )] (1-6)
[0031] Further, in the step S4, specifically:
[0032] S41, using different mapping functions a and b respectively acting on the matrix P and the matrix F, so that the two mapped matrix dimensions are consistent;
[0033] Further, the different mapping functions a and b are learned and optimized by back propagation.
[0034] S42, the mapped matrix P and the matrix F are respectively regarded as the weight matrix and the feature element matrix of the local structure of the point cloud, and the feature update is realized by two matrix multiplications, as shown in formula (1-7):
[0035] F' = a(P) o b(F) (1-7)
[0036] Where "o" represents the element-wise product of two matrices, a(P) is the weight matrix of the local block in the point cloud, and the new F' feature representation is obtained by fusing the weight, and the feature extraction is performed.
[0037] Further, in the step S5, the linear transformation function is learned and optimized by back propagation.
[0038] Further, in the step S6, in order to obtain multi-level features, the steps S2, S3, S4 and S5 are repeated, and the difference is that the number of points after sampling is half of the number of points in the point set in the last step S2, S3, S4 and S5.
[0039] Further, in the step S7, two learnable and trainable linear mapping functions are used in succession, and a residual structure connection is adopted to avoid overfitting and overlearning.
[0040] Further, in the step S8, the unordered nature of the point cloud and the interaction between points are used, and the feature matrix obtained in S7 is used to obtain the global feature of the point cloud by using maximum pooling, as shown in formula (1-8):
[0041] f global = MaxPooling NUM (F) (1-8)
[0042] Where MaxPooling NUM represents the maximum pooling operation on the F matrix in the point cloud dimension, and further feature fusion is performed to obtain the global feature representation of the point cloud.
[0043] Further, in the step S9, the classifier is used for efficient point cloud learning and feature extraction based on position information fusion to obtain the final classification result.
[0044] The application discloses a high-efficiency point cloud classification system based on position fusion, adopts the high-efficiency point cloud classification method based on position fusion, and comprises the following modules.
[0045] An embedding layer module is used for extracting an initial embedding representation from input point cloud three-dimensional coordinates and assigning the representation to a position embedding representation and a feature embedding representation.
[0046] A feature extraction layer module is used for fusing position information and feature information according to a feature extraction method used in the method, and specifically comprises the following steps: point cloud downsampling, KNN construction of point cloud local area relationship, connection of features by using a multi-scale method, fusion of relative position information and absolute position information according to the KNN adjacency relationship to obtain a P matrix; and relative feature information and absolute feature information are spliced to obtain an F matrix. According to a corresponding relationship, feature updating is performed, and a maximum pooling is used for feature aggregation to obtain a new feature representation of a centroid.
[0047] A mapping layer module is used for inputting the obtained feature representation into a mapping function and performing a series splicing operation by using a residual structure to obtain a more comprehensive feature representation.
[0048] A pooling layer module is used for inputting the feature of all the centroids into a maximum pooling by using a global maximum pooling operation to obtain a global representation of the point cloud.
[0049] A classifier module is used for point cloud classification and recognition based on point cloud three-dimensional coordinates and position fusion. BRIEF DESCRIPTION OF DRAWINGS
[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0051] Figure 1 It is a method flowchart of the high-efficiency point cloud classification based on position information fusion in the embodiment of the present application.
[0052] Figure 2 It is a schematic diagram of the symbol definition of the point cloud local area in the embodiment of the present application.
[0053] Figure 3 It is a matrix P structure diagram for the centroid i based on the KNN local relationship in the embodiment of the present application.
[0054] Figure 4 It is a matrix F structure diagram for the centroid i based on the KNN local relationship in the embodiment of the present application.
[0055] Figure 5 is an efficient point cloud classification and recognition system based on point cloud position information fusion according to an embodiment of the present application. DETAILED DESCRIPTION
[0056] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.
[0057] The embodiments of the present application provide an efficient point cloud classification method based on position fusion. The method can effectively replace the point cloud processing method using attention mechanism, and has low computing requirements, such as Figure 1 As shown in the figure, the method comprises the following steps:
[0058] S1, obtaining an initial embedding representation of a point cloud from three-dimensional coordinates containing the point cloud using an embedding layer, which can be expressed by formula (1):
[0059] embedding=E(x,y,z) (1)
[0060] Wherein, x, y, z represent the three-dimensional coordinates of the input point cloud, that is, the point cloud set E represents a mapping layer, which specifically maps the dimension from 3D to 64D, and assigns it to the position information p and the feature information f of the point cloud.
[0061] S2, for the extraction stage of local features, the number of point sets will gradually decrease. We use the farthest point sampling (FPS) algorithm to downsample the point set to obtain multi-level features. We define si as the number of point sets in stage i. The number of points in stage i after downsampling in different stages is N i The changes are shown in formulas (2) and (3):
[0062]
[0063]
[0064] Since there is no obvious topological relationship between the original point cloud data, after downsampling, the local neighborhood is obtained by KNN on the original point cloud set. For each sampling point, KNN finds a fixed number of neighboring points from the point cloud before sampling. The neighborhood is defined by spatial distance.
[0065] S3, in order to avoid using a single global position information p iWe propose a method to concatenate relative position information (direction vector) with absolute (global) position information, using local position information and global position information, as well as local feature information and global feature information at this stage. As shown in Figure 2 For a point i in the point cloud, the direction vector introduces Δp ij As shown in equation (4):
[0066] Δp ij = p j -p i (4)
[0067] to represent the local neighborhood information between points. Accordingly, the relative feature information Δf ij is used to correspond to the local neighborhood information, and the connection result is as shown in Figure 3 and Figure 4 As described in Figure 3 , Figure 4 Based on the relationship between points in the KNN graph, the information of the center point i is concatenated with the information of the k neighborhoods of the point. We gradually expand the receptive field of the network model to prepare for subsequent feature updates.
[0068] S31, for points with the same centroid, use KNN with different scale sizes to establish neighbor relationships. Specifically, set k to 8, 16, and 32, and concatenate the feature information and position information of the 8 neighbors, 16 neighbors, and 32 neighbors of different scales, respectively.
[0069] In the embodiment, the feature information based on multiple scales has different channel numbers according to different scales. When the number of neighbors is 8, 16, and 32, the number of channels of the feature information is set to 512, 256, and 128, respectively.
[0070] S32, the F matrix block of point i can be represented by the following equation (5):
[0071] F i = R k [concat(f i , Δf ij )] (5)
[0072] The P matrix block of point i can be represented as shown in equation (6):
[0073] P i = R k [concat(p i , Δp ij )] (6)
[0074] R kF and F′ respectively, which are the results of repeating the operation k times according to the KNN neighbor relationship, and F is obtained by this way i F and P i F and P, and send them to the subsequent stage.
[0075] S4, the matrix F from the previous module and the matrix P are input into different mapping trainable functions α and β. We regard α(P) as the corresponding weight of β(F) to enhance the expression ability.
[0076] S41, using different mapping functions α and β to act on the matrix P and the matrix F respectively, so that the dimensions of the two mapped matrices are consistent;
[0077] Further, the different mapping functions α and β are learned and optimized by back propagation.
[0078] S42, the mapped matrix P and the matrix F are regarded as the weight matrix and the feature element matrix of the local structure of the point cloud respectively, and the feature update is realized using two matrix multiplications, as shown in formula (7):
[0079] F′=α(P)⊙β(F) (7)
[0080] Where "⊙" represents the Hadamard product of two matrices, and for each center point, the weighted feature representation of k neighbors is obtained in this way.
[0081] In the embodiment, when this process is implemented after the first downsampling, the α function and the β function both map P and F to 512 dimensions, and when this process is implemented after the second downsampling, the α function and the β function both map P and F to 256 dimensions.
[0082] S5, due to the unordered nature and the transformation invariance of the point cloud, the symmetric aggregation operation is used for the aggregation operation. We choose the local max-pooling operation to aggregate the features in the neighborhood to obtain the new feature representation of the point.
[0083] In order to reduce the computational amount of this process, our method can replace the complex calculation of the attention mechanism in the point cloud. It is more suitable for the characteristics of point cloud data, and compared with the method using attention, it greatly reduces the computational complexity while maintaining the same high accuracy.
[0084] S6, in order to expand the receptive field and gather more feature information, the contents of S2, S3, S4 and S5 are repeated to obtain feature information of different scales.
[0085] S7, in order to gather information from different layers, the feature representation is mapped through γ mapping function and δ mapping function. We connect the results of the two as point features. It can be represented as formula (8):
[0086]
[0087] S8, using global maximum pooling on the feature information obtained in the above step to obtain a global feature representation.
[0088] S9, a classifier for point cloud recognition classification based on point cloud three-dimensional coordinates.
[0089] The classifier uses Linear, BatchNorm and ReLU layers, Dropout layer operations to obtain the final classification result. The class label of the point cloud is determined as the class with the maximum score.
[0090] An embodiment of the application is a high-efficiency point cloud classification and recognition system based on point cloud position information fusion, as shown in Figure 5 The high-efficiency point cloud classification and recognition method based on point cloud position information fusion comprises the following steps:
[0091] An embedding layer module is used to extract an initial embedding representation from the input point cloud three-dimensional coordinates, and the representation is assigned to the position embedding representation and the feature embedding representation.
[0092] A feature extraction layer module is used to fuse the position information and the feature information according to the feature extraction method used in the above method, specifically including: point cloud downsampling, KNN construction of point cloud local relationship, connection of features using a multi-scale method, fusion of relative position information and absolute position information according to the KNN adjacency relationship, corresponding relative feature information and absolute feature information are spliced, feature updating is performed according to the corresponding relationship, and a maximum pooling is used to perform a feature aggregation operation to obtain a new centroid feature representation.
[0093] A mapping layer module is used to input the feature representation obtained in the previous stage into a mapping function, and use a residual structure for concatenation and splicing operation to obtain a more comprehensive feature representation.
[0094] A pooling layer module uses a global maximum pooling operation to input the existing centroid feature into the maximum pooling to obtain a global representation of the point cloud.
[0095] A classifier module is used for point cloud classification and recognition based on point cloud three-dimensional coordinates and position fusion.
[0096] The application is compared with the current most advanced PointTransformer network using the Transformer and attention mechanism applied to point cloud learning on a data set. On the ModelNet40 data set, one of the most widely compared, the best recognition accuracy of the application reaches 93.7%, which is the same as the accuracy of PointTransformer; in terms of parameter quantity, the parameter quantity of the application is only 0.3M, which is one tenth of the parameter quantity of PointTransformer; in terms of FLOPs, the FLOPs of the application is 1.30G, which is much smaller than the FLOPs of PointTransformer. Compared with existing point cloud classification recognition methods based on attention mechanism, the application can directly learn the local structure of the point cloud, avoiding the super large calculation amount of the attention mechanism, and the learned local structure can better explicitly represent the shape information of the point cloud, and can have no complex operation of the Query, Key and Value matrices, which is more conducive to efficient point cloud classification and recognition. Using multi-scale feature connection, the receptive field is expanded while the calculation amount is avoided to increase and the operation of centroid shift is avoided, which is more superior than the point cloud classification using the ordinary attention mechanism, and is more suitable for point cloud data.
[0097] The application is used for point cloud classification and recognition, and the three-dimensional coordinates of the point cloud are used as input, without the supplement of normal vectors, colors and other information of the point cloud, so that the initial embedding representation can be formed under the network of the application, facilitating the later point cloud feature learning. The method of the application is used for recognizing the point cloud, for example, can be applied to automatic driving, and can be applied to augmented reality, robots and the like, and has a wide range of applications.
[0098] The above only describes the preferred embodiments of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.
Claims
1. An efficient point cloud classification method and system based on location fusion, characterized in that, The method comprises the following steps: S1, extracting an initial embedding representation of a point in the point cloud from an input point cloud three-dimensional coordinate, obtaining initial global position information and global feature information of the point in the point cloud; S2, reducing the number of points in the point cloud by using farthest point downsampling without changing the basic shape of the point cloud, taking the downsampled points as centroids, and searching for K-Nearest Neighbor (KNN) neighbors centered on the centroids by taking the points before downsampling as a search space, and establishing a KNN local relationship graph structure; S3, constructing F matrix and P matrix connecting feature information and connection position information through KNN local adjacency relationship, and obtaining a weight matrix and an element matrix with global information and local information; S4, mapping the F matrix and the P matrix to the same dimension, and performing Hadamard product operation on the two, to obtain a new F' matrix, i.e., a new feature matrix, and inputting the new F' matrix into S5; S5, obtaining a new global feature representation of each centroid by using a linear transformation function and local maximum pooling on the F' matrix obtained in S4, and inputting the new global feature representation into the next stage; S6, receiving the new global feature representation in S5, and repeating steps S2, S3, S4 and S5 to obtain feature representations of points with different sampling numbers; S7, inputting the feature representation obtained in S6 into a mapping function, and further inputting the feature representation into the mapping function again, and connecting the results obtained by the two times using a residual structure in series to obtain features of all points in the point cloud; S8, in order to classify the point cloud, using a maximum pooling layer to pool the features of all points obtained in S7 in the dimension of the number of points to obtain a global feature representation of the point cloud; S9, using a classifier to classify and identify the point cloud based on the global feature representation; The step S2 is specifically: Define the number of points in the initial point cloud set as N, and the number of points after sampling as N i may be represented as shown in equation (1), equation (2), respectively: S21, establishing KNN graphs of different scales centered on the same centroid, and connecting features of different scales in series, and using different dimensional representations for features of different resolution levels to obtain receptive fields of different sizes; The step S3 is specifically: S31, we define: the absolute position of point i is p i With this point as the center, establish a KNN graph, get the k neighbors of this point, set one of the neighbors as point j, then the relative position information between point i and point j is as shown in formula (3): Δp ij = p j - p i (3) The P matrix is obtained by fusing relative position information and absolute position information, and a P matrix block of point i can be represented as formula (4): P i = R k [concat(p i , Δp ij )] (4) wherein R k represents repeating the operation k times, j represents the k neighbors of point i, and concat represents a concatenation operation; S32, the F matrix is obtained by fusing relative feature information and absolute feature information, which are defined as f i and Δf ij where the relative feature information between point i and point j is shown in formula (5): Δf ij = f j -f i (5) A F matrix block of point i can be represented as formula (6): F i = R k [concat(f i , Δf ij )] (6) The step S4 is specifically: S41, using different mapping functions α and β to act on the matrix P and the matrix F respectively, so that the dimensions of the two mapped matrices are consistent; The different mapping functions α and β are learned and optimized through back propagation; S42, taking the mapped matrix P and the matrix F as a weight matrix and a feature element matrix of a local structure of the point cloud respectively, and using two matrix multiplications to realize feature updating, as shown in formula (7): F' = α(P) ⊙ β(F) (7) Wherein, "⊙" represents element-wise multiplication of two matrices, α(P) is a weight matrix of a local block in the point cloud, a new F' feature representation is obtained by fusing weights, and feature extraction is performed; In the step S5, the linear transformation function is learned and optimized through back propagation; In the step S6, in order to obtain multi-level features, the strategy of repeating steps S2, S3, S4 and S5 is adopted, except that the number of points after sampling is half of the number of points in the point set in the previous step S2, S3, S4 and S5; In the step S7, specifically: Two learnable and trainable linear mapping functions are used in succession, and a residual structure is used to connect them in order to avoid overfitting and overlearning; In the step S8, the unordered nature of the point cloud and the interaction between points are used to obtain the global features of the point cloud by using maximum pooling on the feature matrix obtained in step S7, as shown in formula (8): f global = MaxPooling NUM (F) (8) MaxPooling NUM denotes the maximum pooling operation on the dimension of the point cloud, and further feature fusion is performed to obtain a global feature representation of the point cloud. In the step S9, the classifier is used to obtain the final classification result based on the efficient point cloud learning and feature extraction of the position information fusion.
2. A high efficient point cloud classification system based on location fusion suitable for point cloud, characterized in that, The efficient point cloud classification method based on position fusion suitable for point clouds according to claim 1 comprises: An embedding layer module is used to extract initial embedding representation from the input point cloud three-dimensional coordinates and assign the representation to the position embedding representation and the feature embedding representation; A feature extraction layer module is used to fuse the position information and the feature information according to the feature extraction method used in the above method, specifically including: point cloud downsampling, KNN construction of point cloud local relationship, multi-scale method connection of features, fusion of relative position information and absolute position information according to the KNN adjacency relationship, corresponding relative feature information and absolute feature information splicing, feature updating according to the corresponding relationship, and feature aggregation operation using maximum pooling to obtain new centroid feature representation; A mapping layer module is used to input the feature representation obtained in the previous stage into the mapping function and use a residual structure for concatenation operation to obtain more comprehensive feature representation; A pooling layer module is used to input all existing centroid features into the maximum pooling using global maximum pooling operation to obtain the global representation of the point cloud; A classifier module is used for point cloud classification and recognition based on point cloud three-dimensional coordinates and position fusion.