A Point Cloud Data Processing Method and Device

Abstract

An embodiment of the present invention discloses a method and device for processing point cloud data. The method includes: fusing the feature information of each point cloud cluster to obtain a feature matrix; wherein, each of the point cloud clusters is obtained by performing clustering processing on the original point cloud data, and the feature information includes shape feature information and position feature information; based on the trained attention mechanism network model, performing weighted processing on the point cloud clusters in the feature matrix, so as to increase the similarity between the point cloud cluster features corresponding to the same object and decrease the similarity between the point cloud cluster features corresponding to different objects; performing clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object. By adopting the above technical solution, the problem of over-segmentation in the point cloud clustering process is solved, and the effectiveness of the clustering result is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of on-vehicle lidar point cloud processing. Specifically, the present invention relates to a method and device for processing point cloud data. Background Art

[0002] Due to the data characteristics of lidar point clouds, such as dense points, diverse distributions, large data volumes, and no distribution rules, coupled with the disordered arrangement of the original point cloud data, in current common processing based on lidar point clouds, there is mostly a step of clustering the point clouds.

[0003] The clustering of traditional methods usually relies on the geometric distance between the points of the point cloud and the prior assumptions about specific objects. For example, pedestrians and tree trunks are cylindrical, and vehicles are rectangular, etc. This clustering method that strongly depends on geometric relationships and prior features usually leads to the problem of over-segmentation, that is, the same object is clustered into multiple point cloud clusters.

[0004] Currently, in order to reduce this over-segmentation situation, the judgment between different point cloud clusters is usually made according to the geometric features of the objects and other artificially set features. Due to the limitations of the artificially set features in this judgment method, the applicability of the judgment result of which object a point cloud cluster specifically belongs to is poor, and the over-segmentation situation cannot be effectively solved. Summary of the Invention

[0005] The embodiments of the present invention provide a method and device for processing point cloud data to overcome the problem of over-segmentation in the process of point cloud clustering and improve the effectiveness of the clustering result.

[0006] In a first aspect, the embodiments of the present invention provide a method for processing point cloud data, and the method includes:

[0007] Fusing the feature information of each point cloud cluster to obtain a feature matrix; wherein, each point cloud cluster is obtained by clustering the original point cloud data, and the feature information includes shape feature information and position feature information;

[0008] Based on the trained attention mechanism network model, performing weighted processing on the point cloud clusters in the feature matrix to increase the similarity between the point cloud cluster features corresponding to the same object and decrease the similarity between the point cloud cluster features corresponding to different objects;

[0009] Performing clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object.

[0010] Optionally, the attention mechanism network model is trained in the following manner:

[0011] Use the feature matrix of the point cloud clusters obtained after feature fusion as training samples, and label the object categories to which each point cloud cluster belongs in the training samples;

[0012] Use the training samples to train the initial attention mechanism network model to obtain a weighted matrix after weighted processing;

[0013] For each point cloud cluster in the weighted matrix, determine the predicted point cloud clusters belonging to the same object as this point cloud cluster according to the distance relationship between this point cloud cluster and other point cloud clusters;

[0014] During the training process, when the loss function value of the similarity between the predicted point cloud clusters and the true point cloud clusters of the corresponding objects reaches convergence, obtain the trained attention mechanism network model.

[0015] Optionally, the attention mechanism network model is a previous encoder-decoder predictor transformer network model.

[0016] Optionally, the fusion of the feature information of each point cloud cluster includes:

[0017] Concatenate the feature information of each point cloud cluster, and merge the feature information of each point cloud cluster after concatenation to form a concatenated matrix;

[0018] Based on a multi-layer perceptron MLP, perform feature fusion processing on the concatenated matrix to obtain a feature matrix.

[0019] Optionally, based on a multi-layer perceptron MLP, perform feature fusion processing on the concatenated matrix to obtain a feature matrix, including:

[0020] For any point cloud cluster, based on a multi-layer perceptron MLP, perform feature fusion processing on the concatenated matrix corresponding to this point cloud cluster to obtain the comprehensive feature information of this point cloud cluster;

[0021] Perform the feature fusion processing on all point cloud clusters according to the number of point cloud clusters to obtain the feature matrix corresponding to all point cloud clusters, and the feature matrix includes the number of point cloud clusters and the feature dimension.

[0022] Optionally, perform clustering processing on the point cloud clusters after weighted processing to determine the point cloud clusters belonging to the same object, including:

[0023] For each cloud cluster after weighted processing, sequentially select a point cloud cluster as the initial clustering center from them, and determine the target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.

[0024] Optionally, determining the target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center includes:

[0025] Calculate the Euclidean distance between other point cloud clusters and the initial clustering center;

[0026] If the value of the Euclidean distance is less than the preset clustering threshold, all point cloud clusters within the preset clustering threshold range are used as candidate point cloud clusters belonging to the same object as the initial clustering center;

[0027] Take the average value of the distances of each candidate point cloud cluster as the new clustering center, and return to perform the operation of calculating the Euclidean distance between other point cloud clusters and the new clustering center. Until the determined new clustering center does not change, all candidate point cloud clusters belonging to the same object as the new clustering center are used as the target point cloud clusters.

[0028] Optionally, the clustering process for the original point cloud data includes:

[0029] Preprocess the original point cloud data, and the preprocessing includes ground point removal and downsampling;

[0030] Cluster the preprocessed point cloud to obtain multiple point cloud clusters;

[0031] Among them, the original point cloud data is obtained after parsing the original lidar data, and the original point cloud data includes the number of all point clouds in the current frame, the three-dimensional coordinates of the point clouds, and the reflection intensity.

[0032] In a second aspect, an embodiment of the present invention further provides a point cloud data processing device, and the device includes:

[0033] A feature fusion module, configured to: fuse the feature information of each point cloud cluster to obtain a feature matrix; wherein, each point cloud cluster is obtained after clustering the original point cloud data, and the feature information includes shape feature information and position feature information;

[0034] A weighted processing module, configured to: based on a trained attention mechanism network model, perform weighted processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object increases, and the similarity between the point cloud cluster features corresponding to different objects decreases;

[0035] A secondary clustering module, configured to: perform clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object.

[0036] Optionally, the attention mechanism network model is trained in the following manner:

[0037] Use the feature matrix of the point cloud clusters obtained after feature fusion as the training sample, and the object category to which each point cloud cluster belongs is marked in the training sample;

[0038] Train the initial attention mechanism network model using the training samples to obtain a weighted matrix after weighted processing;

[0039] For each point cloud cluster in the weighted matrix, determine the predicted point cloud clusters belonging to the same object as this point cloud cluster according to the distance relationship between this point cloud cluster and other point cloud clusters;

[0040] During the training process, when the loss function value of the similarity between the predicted point cloud clusters and the true point cloud clusters of the corresponding objects reaches convergence, obtain the trained attention mechanism network model.

[0041] Optionally, the attention mechanism network model is a previous encoding predictor Transformer network model.

[0042] Optionally, the feature fusion module includes:

[0043] A splicing unit configured to splice the feature information of each point cloud cluster and merge the feature information of each point cloud cluster after splicing to form a splicing matrix;

[0044] A feature fusion unit configured to perform feature fusion processing on the splicing matrix based on a multi-layer perceptron MLP to obtain a feature matrix.

[0045] Optionally, the feature fusion unit is specifically configured to:

[0046] For any point cloud cluster, perform feature fusion processing on the splicing matrix corresponding to this point cloud cluster based on a multi-layer perceptron MLP to obtain the comprehensive feature information of this point cloud cluster;

[0047] Perform the feature fusion processing on all point cloud clusters according to the number of point cloud clusters to obtain the feature matrices corresponding to all point cloud clusters, and the feature matrices include the number of point cloud clusters and the feature dimension.

[0048] Optionally, the secondary clustering module includes:

[0049] A target point cloud cluster determination unit configured to: for each cloud cluster after weighted processing, sequentially select a point cloud cluster as the initial clustering center from them, and determine the target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.

[0050] Optionally, the target point cloud cluster determination unit is specifically configured to:

[0051] Calculate the Euclidean distance between other point cloud clusters and the initial clustering center;

[0052] If the value of the Euclidean distance is less than a preset clustering threshold, all point cloud clusters within the range of the preset clustering threshold are used as candidate point cloud clusters belonging to the same object as the initial clustering center;

[0053] Take the average value of the distances of each candidate point cloud cluster as the new clustering center, and return to perform the operation of calculating the Euclidean distance between other point cloud clusters and the new clustering center until the determined new clustering center does not change, and take all candidate point cloud clusters belonging to the same object as the new clustering center as the target point cloud cluster.

[0054] Optionally, the clustering process of the original point cloud data includes:

[0055] Preprocess the original point cloud data, and the preprocessing includes ground point removal and downsampling;

[0056] Cluster the preprocessed point cloud to obtain multiple point cloud clusters;

[0057] Among them, the original point cloud data is obtained by parsing the original data of the lidar, and the original point cloud data includes the number of all point clouds in the current frame, the three-dimensional coordinates of the point clouds, and the reflection intensity.

[0058] In a third aspect, an embodiment of the present invention further provides a computing device, including:

[0059] A memory storing executable program code;

[0060] A processor coupled to the memory;

[0061] The processor calls the executable program code stored in the memory to execute the point cloud data processing method provided in any embodiment of the present invention.

[0062] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the point cloud data processing method provided in any embodiment of the present invention.

[0063] The technical solution provided by the embodiment of the present invention, based on the first clustering result, can obtain a feature matrix by fusing the feature information of each point cloud cluster. Through the attention mechanism network model, the point cloud clusters in the feature matrix are weighted, so that a single point cloud cluster can contain the feature information of other point cloud clusters, thereby increasing the similarity between the feature of the point cloud cluster corresponding to the same object and reducing the similarity between the feature of the point cloud cluster corresponding to different objects. By clustering the weighted point cloud clusters again, the target point cloud clusters belonging to the same object can be determined, effectively solving the problem of over-segmentation in the point cloud clustering process on the premise of avoiding under-segmentation, and effectively improving the effectiveness of the point cloud clustering result.

[0064] The technical effects of the embodiments of the present invention include:

[0065] 1. Through the attention mechanism network model, each cluster of point clouds can obtain information from other point cloud clusters, which is more conducive to increasing the similarity between the feature vectors of different point cloud clusters belonging to the same object and reducing the similarity between the point cloud clusters belonging to different objects, thereby improving the effect of secondary clustering, avoiding the limitation of the applicable range that may be caused by artificially designed features, and improving the robustness and practicality of the model.

[0066] 2. By fusing the feature information of the point cloud clusters, the feature matrix obtained after feature fusion can simultaneously reflect the comprehensive features of the position and shape of the point cloud clusters, and can also reduce the feature dimension of the point cloud clusters, thereby reducing the computational complexity in the subsequent clustering process. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0068] Figure 1 It is a schematic diagram of the principle of the point cloud secondary clustering method provided by the embodiments of the present invention;

[0069] Figure 2 It is a flowchart of a training method for an attention mechanism network model provided by Embodiment 1 of the present invention;

[0070] Figure 3 It is a flowchart of a point cloud data processing method provided by Embodiment 1 of the present invention;

[0071] Figure 4 It is a flowchart of a point cloud data processing method provided by Embodiment 2 of the present invention;

[0072] Figure 5 It is a structural block diagram of a point cloud data processing device provided by Embodiment 4 of the present invention;

[0073] Figure 6 It is a schematic structural diagram of a computing device provided by Embodiment 5 of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0074] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0075] It should be noted that the terms "including" and "having" in the embodiments of the present invention and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0076] The embodiments of the present invention disclose a method and device for processing point cloud data. To more clearly and clearly explain the content of each embodiment of the present invention, the basic working principle of the embodiments of the present invention will be briefly introduced below.

[0077] Figure 1 It is a schematic diagram of the point cloud secondary clustering method provided by the embodiments of the present invention. As Figure 1 shown. The input object of secondary clustering is the point cloud cluster obtained after the first clustering. Compared with a single point cloud point, the point cloud cluster often has more information, such as shape, size, average reflection intensity, curvature, and the distribution density of points within the cluster. In reality, different point cloud clusters of the same object have similar characteristics, such as reflection intensity and point cloud density. At the same time, if an object is clustered into several clusters, these clusters are often close to each other, or have similar projected areas in a certain direction. The point cloud clusters of different objects, on the other hand, usually have different characteristics. Therefore, by combining the characteristics of the point cloud clusters, secondary clustering can be performed on the basis of the point cloud clusters to merge the over-segmented objects.

[0078] The secondary clustering method provided by the embodiments of the present invention consists of two parts: point cloud cluster feature extraction and point cloud cluster clustering. The difference from the prior art is that in the embodiments of the present invention, the feature extraction uses a feature extraction network based on deep learning. Through this feature extraction network, each point cloud cluster can calculate its shape feature according to its relative position relationship in the point cloud, and can calculate its position feature according to the position information of the point cloud cluster in the entire frame of the point cloud. By combining these two features, that is, splicing, a comprehensive feature including the shape and position of the point cloud cluster can be obtained.

[0079] After obtaining the comprehensive features of the point cloud clusters, the technical solution of the embodiment of the present invention uses a neural network or a deep learning model for secondary clustering. In particular, the attention network model of transformer (pre-sequence encoding predictor) in natural language processing can be used for secondary clustering. This model will take into account the features of all point cloud clusters in each frame, increasing the similarity of the features of point cloud clusters that belong to the same object but are clustered into multiple clusters, and decreasing the similarity of the features of point cloud clusters that do not belong to the same object. The application of this model can avoid the situation of under-segmentation, and at the same time solve the over-segmentation situation in the primary clustering process, improving the effectiveness of subsequent secondary clustering.

[0080] In the prior art, the secondary clustering process usually judges between different point cloud clusters according to the geometric features of the object and other artificially set features, without combining the global information of the entire frame of point cloud. However, the secondary clustering method provided by the embodiment of the present invention can avoid the limitations of artificially set features. Through the attention mechanism network model, each point cloud cluster can effectively combine the feature information of other point cloud clusters, obtain more comprehensive features, and help to make further secondary clustering judgments.

[0081] Next, the training process and application process of the attention mechanism network model in the point cloud secondary clustering process will be described in detail respectively.

[0082] Embodiment 1

[0083] Figure 2 As shown in the flowchart of a training method of an attention mechanism network model provided in Embodiment 1 of the present invention, this method can be applied to the secondary clustering process of point cloud data and can be applied to application scenarios such as target recognition and tracking perception. This method can be executed by a training device of the attention mechanism network model, and this device can be implemented in software and / or hardware. As Figure 2 shown, the method provided in this embodiment includes:

[0084] S100. Fuse the feature information of each point cloud cluster to obtain a feature matrix.

[0085] Among them, the point cloud cluster is obtained after clustering the original point cloud data. The specific clustering process can be:

[0086] Parse the lidar point cloud data to obtain the original point cloud data P0 of the current frame, where P0 can be expressed as [N0, M], where N0 is the number of all point clouds in the current frame, and M is the three-dimensional coordinates and reflection intensity of the point cloud. By preprocessing the original point cloud data P0, such as removing ground points and downsampling, the noise information in the original point cloud data can be removed.

[0087] After preprocessing the original point cloud data, clustering processing can be performed on it to obtain point cloud clusters. Among them, the clustering method can be to determine the point cloud clusters belonging to the same object according to the similarity of the point cloud cluster features. The input of this clustering process is the preprocessed point cloud, and the output is the point cloud cluster. Each cluster of point clouds obtained by clustering contains one or more point clouds, which can be represented by the following formula:

[0088] (c0,c1,c2…,c F )=f(P1)

[0089] Among them, c i represents the i-th cluster of point clouds, which includes (p i0 ,p i1 ,p i2 …), p ij is the j-th point cloud point in the i-th cluster. The number of point clouds contained in each cluster of point clouds is closely related to the clustering algorithm adopted and the specific scenario. The secondary clustering processing flow provided in the embodiments of the present invention does not require the number of clusters and the number of point clouds contained in each cluster of point clouds. The point cloud clusters after clustering have more information than individual point cloud points, such as shape, size, average reflection intensity, curvature, and the distribution density of point clouds within the cluster, etc.

[0090] In this embodiment, the characteristic information of each point cloud cluster includes shape characteristic information and position characteristic information. Among them, the extraction of shape characteristic information can be realized through a feature extraction network, such as pointnet (a deep network framework that takes the original point cloud data without voxelization and rendering as input). Through this feature extraction network, each point cloud cluster can calculate its shape characteristics according to the relative position relationship between its point clouds. The extraction of position characteristic information can be realized through MLP (Multilayer Perceptron). The extraction of the above characteristic information can be represented by the following formula:

[0091] F local_i =net local (c i ),c i =(p i0 ,p i1 …,p ij )

[0092] F global_i =net global (C i )

[0093] Among them, F local_i and F global_i respectively represent the shape characteristic information and position characteristic information extracted from the i-th cluster of point clouds, c i is the i-th cluster of point clouds, p ijis the j-th point of the i-th cluster of point clouds, Ci is the center coordinate of the i-th cluster of point clouds, and net local and net global are the networks for extracting shape features and position features respectively.

[0094] In this embodiment, for the extracted shape feature information and position feature information, the shape feature information and the position feature information can be fused to facilitate subsequent feature processing and clustering operations. Among them, feature fusion can be implemented by MLP. The specific fusion process may include fusing the shape features and position features of each point cloud cluster so that the comprehensive feature information obtained after feature fusion can reflect the comprehensive features of the position and shape of the point cloud cluster. In addition, the feature fusion process also includes fusing the feature information of each point cloud cluster to reduce the feature dimension of the point cloud cluster, thereby reducing the computational amount in the subsequent clustering process. For example, the process of feature fusion processing can be represented by the following formula:

[0095] D i = MLP(cat[F local_i , F global_i )

[0096] In the above formula, D i represents the comprehensive feature information of the i-th cluster of point clouds. The shape feature information and the position feature information are effectively combined through MLP, that is, MLP(cat[F local_i , F global_i ), to obtain the comprehensive feature information that can reflect the position and shape of the point cloud cluster.

[0097] By performing the same processing on all point cloud clusters, the feature matrix M corresponding to all point cloud clusters can be obtained: [F, D], where F is the number of point cloud clusters and D is the feature dimension.

[0098] S110. Use the feature matrix of the point cloud cluster obtained after feature fusion as the training sample, and the object category to which each point cloud cluster belongs is marked in the training sample.

[0099] In this embodiment, the training of the attention mechanism network model adopts the training method of supervised learning, and the object category to which each point cloud cluster belongs is marked in the training sample.

[0100] S120. Use the training sample to train the initial attention mechanism network model to obtain a weighted matrix after weighted processing.

[0101] In this embodiment, the reason for using the attention mechanism network model is as follows: The effectiveness of the attention mechanism network model is based on a basic premise that among the point cloud clusters of the same object, they have more similar features compared to the point cloud clusters of different objects. The attention mechanism network model can achieve information sharing between different point cloud clusters, that is, a certain cluster of point clouds can obtain the information of other point cloud clusters, making it easier for the over-segmented point cloud clusters to have more similar features. For example, during the processing of the attention mechanism network model, the feature value of each point cloud cluster can be made the weighted average of the features of other clusters. By performing such weighted processing on different point cloud clusters, the similarity between the point cloud cluster features corresponding to the same object can be increased, and the similarity between the point cloud cluster features corresponding to different objects can be reduced.

[0102] Among them, when using training samples to train a preset neural network model, the obtained weighted matrix after weighted processing can be represented by the following formula:

[0103] M’ = Transformer(M)

[0104] Where M is the input to the transformer network model, representing a feature matrix where all point cloud clusters are located, and M’ is the weighted matrix obtained after weighted processing. In the existing secondary clustering methods, the transformer network model is not used to perform weighted processing on the point cloud cluster features. Instead, the feature calculation is performed based on the relative position and absolute position of the point clouds in this cluster. Therefore, it is difficult for the existing methods to effectively increase or decrease the similarity of features between different clusters solely using the information of the point cloud clusters themselves. The result is that only the features of each cluster of point clouds calculated at the beginning can be used for secondary clustering, lacking the feature information and global information of other clusters, which will seriously affect the effect of subsequent secondary clustering. In the embodiment of the present invention, the attention mechanism network model can effectively combine the information of the entire frame of point clouds and enable each cluster of point clouds to obtain the information of other point cloud clusters, thus being more conducive to increasing the similarity between the features of different point cloud clusters belonging to the same object and reducing the similarity between the point cloud clusters belonging to different objects.

[0105] Specifically, taking the over-segmented long truck as an example, the long truck is segmented into a cab and a tail. By using the attention mechanism network in this embodiment, it is possible to make a single point cloud cluster contain the feature information of other point cloud clusters. Compared with the method of continuously increasing the similarity between the cab and tail features solely through the constraint of the loss function, the attention mechanism network can increase the similarity between the cab and tail features in the network, thereby increasing the effectiveness of secondary clustering.

[0106] S130. For each point cloud cluster in the weighted matrix, determine the predicted point cloud cluster that belongs to the same object as this point cloud cluster according to the distance relationship between this point cloud cluster and other point cloud clusters.

[0107] Among them, for each point cloud cluster in the weighted matrix, a point cloud cluster can be sequentially selected as the initial clustering center, and according to the distance relationship between other point cloud clusters and the initial clustering center, the point cloud clusters belonging to the same object are determined.

[0108] S140. During the training process, when the loss function value of the similarity between the predicted point cloud cluster and the true point cloud cluster of the corresponding object reaches convergence, the trained attention mechanism network model is obtained.

[0109] Among them, during the training process of the model, after the training samples of each batch are fed into the model, the predicted values are output through forward propagation, that is, the predicted point cloud clusters belonging to the same object. Then, the difference value between the predicted value and the true value is calculated through the loss function, that is, the loss function value. After obtaining the loss function value, the model updates each parameter through backpropagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the model approaches the true value. When the loss function value reaches convergence, the model training is completed.

[0110] The technical solution provided in this embodiment can, by training the attention mechanism network model, during the secondary clustering process of point cloud data, use the trained attention mechanism network model to perform weighted processing on each point cloud cluster, so that the similarity between the point cloud cluster features corresponding to the same object increases, and the similarity between the point cloud cluster features corresponding to different objects decreases. On the premise of reducing under-segmentation, over-segmentation is also reduced at the same time, realizing accurate clustering of point cloud clusters, avoiding the situation of limited application range that may be caused by artificially designed features, thereby improving the robustness and practicality of the model.

[0111] After the attention mechanism network model is trained, it can be applied to the secondary clustering process of point cloud data. For specific content, please refer to the following embodiments.

[0112] Embodiment 2

[0113] Figure 3 It is a flowchart of a point cloud data processing method provided in Embodiment 1 of the present invention. This method can be applied to application scenarios such as target recognition and tracking perception to perform secondary clustering on vehicle-mounted lidar point clouds. This method can be executed by a point cloud data processing device, and the device can be implemented in software and / or hardware. As Figure 3 shown, the method provided in this embodiment includes:

[0114] S210. Fuse the feature information of each point cloud cluster to obtain a feature matrix.

[0115] In this embodiment, the process of fusing the feature information of each point cloud cluster can refer to the description of the above embodiment, and will not be elaborated here.

[0116] Since the secondary clustering method provided in this embodiment is performed based on the result of the primary clustering, the number of point clouds is already at least 2-3 orders of magnitude smaller than the number of the original point clouds. Specifically, taking a mechanical lidar with 32 lines and a horizontal resolution of 0.2° as an example, the number of point cloud clusters after clustering is about 300 clusters, which is nearly 200 times less than the number of the original point clouds. Compared with the method of directly clustering the original point cloud data using a neural network model, the secondary clustering method provided in this embodiment has a smaller computational load.

[0117] It should be noted that the secondary clustering processing flow provided in the embodiments of the present invention does not have requirements on the number of clusters and the number of point clouds included in each cluster of point clouds.

[0118] S220. Based on the trained attention mechanism network model, perform weighted processing on the point cloud clusters in the feature matrix, so as to increase the similarity between the point cloud cluster features corresponding to the same object and decrease the similarity between the point cloud cluster features corresponding to different objects.

[0119] In this embodiment, after the attention mechanism network model is trained, each point cloud cluster can obtain the information of other point cloud clusters, which is more conducive to increasing the similarity between the point cloud cluster features of different point cloud clusters belonging to the same object and decreasing the similarity between the point cloud clusters of different objects. The training process of the attention mechanism network model can refer to the description of the above embodiment and will not be elaborated here.

[0120] S230. Perform clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object.

[0121] Among them, there are various clustering processing methods. In this embodiment, the mean-shift clustering method is preferably adopted. The specific process is as follows:

[0122] For each cloud cluster after weighted processing, select a point cloud cluster as the initial clustering center in turn, and determine the point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center. Among them, the distance relationship can be the Euclidean distance.

[0123] In this embodiment, based on the first clustering result, by fusing the feature information of each point cloud cluster, a feature matrix can be obtained. By using an attention mechanism network model to weight the point cloud clusters in the feature matrix, each individual point cloud cluster can be made to contain the feature information of other point cloud clusters, thereby increasing the similarity between the feature of point cloud clusters corresponding to the same object and decreasing the similarity between the features of point cloud clusters corresponding to different objects. By performing clustering processing on the weighted point cloud clusters again, the target point cloud clusters belonging to the same object can be determined, effectively solving the problem of over-segmentation in the point cloud clustering process on the premise of avoiding under-segmentation, and effectively improving the effectiveness of the point cloud clustering result.

[0124] Embodiment III

[0125] Figure 4 As shown in the flowchart of a point cloud data processing method provided in Embodiment II of the present invention, on the basis of the above embodiment, the steps of "fusing the feature information of each point cloud cluster to obtain a feature matrix" and "performing clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object" in the above embodiment are refined. As Figure 4 shown, the method provided in this embodiment includes:

[0126] S310. Concatenate the feature information of each point cloud cluster, and merge the feature information of each point cloud cluster after concatenation to form a concatenated matrix.

[0127] In this embodiment, before fusing the feature information of each point cloud cluster, the shape feature and the position feature of each point cloud cluster can be concatenated, that is, the two are combined into the same feature space to facilitate subsequent feature fusion operations.

[0128] Specifically, before concatenation, the shape feature and the position feature respectively correspond to a matrix. The concatenation process is to splice these two matrices into the same matrix. For example, if there are 5 clusters of point clouds, the position feature corresponds to a 5×4 matrix, and the shape feature corresponds to a 5×10 matrix, and the concatenated matrix is a 5×14 matrix.

[0129] S320. Based on the multi-layer perceptron MLP, perform feature fusion processing on the concatenated matrix to obtain a feature matrix.

[0130] For example, the process of obtaining the feature matrix can be implemented in the following manner:

[0131] For any point cloud cluster, based on the multi-layer perceptron MLP, perform feature fusion processing on the concatenated matrix corresponding to the point cloud cluster to obtain the comprehensive feature information of the point cloud cluster;

[0132] Perform the above feature fusion processing on all point cloud clusters according to the number of point cloud clusters to obtain the feature matrices corresponding to all point cloud clusters.

[0133] Among them, the process of feature fusion processing through MLP can be expressed by the following formula:

[0134] D i = MLP(cat[F local_i , F global_i )

[0135] In the above formula, D i represents the comprehensive feature information of the i-th cluster of point clouds. The shape feature information and position feature information are effectively merged through MLP, that is, MLP(cat[F local_i , F global_i ), to obtain the comprehensive feature information that can reflect the position and shape of the point cloud cluster.

[0136] By performing the same processing on all point cloud clusters, the feature matrix M corresponding to all point cloud clusters can be obtained: [F, D], where F is the number of point cloud clusters and D is the feature dimension.

[0137] S330. Based on the trained attention mechanism network model, perform weighted processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object increases, and the similarity between the point cloud cluster features corresponding to different objects decreases.

[0138] S340. For each cloud cluster after weighted processing, sequentially select one point cloud cluster as the initial clustering center, and calculate the Euclidean distance between other point cloud clusters and the initial clustering center.

[0139] S350. If the value of the Euclidean distance is less than the preset clustering threshold, then all point cloud clusters within the preset clustering threshold range are used as candidate point cloud clusters belonging to the same object as the initial clustering center.

[0140] Among them, the preset clustering threshold is a parameter value obtained during the training process of the attention mechanism network model. After the attention mechanism network model is trained, this preset clustering threshold is fixed.

[0141] S360. Take the average value of the distances of each candidate point cloud cluster as the new clustering center, and return to perform the operation of calculating the Euclidean distance between other point cloud clusters and the new clustering center until the determined new clustering center does not change, and take all candidate point cloud clusters belonging to the same object as the new clustering center as the target point cloud clusters.

[0142] In this embodiment, by using the average value of the distances of each candidate point cloud cluster as the new clustering center, the position of the clustering center can be updated, improving the accuracy of determining the clustering center, and thus the accuracy of determining the target point cloud cluster can be improved.

[0143] The technical solution provided in this embodiment can make the obtained feature matrix reflect the comprehensive features of the position and shape of the point cloud cluster simultaneously and reduce the feature dimension of the point cloud cluster by splicing the feature information of the point cloud cluster and performing feature fusion on the splicing matrix, thereby reducing the computational amount in the subsequent clustering process. In the secondary clustering process, the accuracy of determining the target point cloud cluster can be improved by updating the position of the clustering center.

[0144] Embodiment Four

[0145] Figure 5 is a structural block diagram of a point cloud data processing device provided in Embodiment Four of the present invention. As Figure 5 shown, the device includes: a feature fusion module 410, a weighted processing module 420, and a secondary clustering module 430; wherein,

[0146] The feature fusion module 410 is configured to: fuse the feature information of each point cloud cluster to obtain a feature matrix; wherein, each point cloud cluster is obtained by performing clustering processing on the original point cloud data, and the feature information includes shape feature information and position feature information;

[0147] The weighted processing module 420 is configured to: based on the trained attention mechanism network model, perform weighted processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object increases, and the similarity between the point cloud cluster features corresponding to different objects decreases;

[0148] The secondary clustering module 430 is configured to: perform clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object.

[0149] Optionally, the attention mechanism network model is trained in the following manner:

[0150] Use the feature matrix of the point cloud clusters obtained after feature fusion as the training sample, and the object category to which each point cloud cluster belongs is marked in the training sample;

[0151] Use the training sample to train the initial attention mechanism network model to obtain a weighted matrix after weighted processing;

[0152] For each point cloud cluster in the weighted matrix, determine the predicted point cloud cluster belonging to the same object as this point cloud cluster according to the distance relationship between this point cloud cluster and other point cloud clusters;

[0153] During the training process, when the loss function value of the similarity between the predicted point cloud clusters and the corresponding real point cloud clusters of the object reaches convergence, a trained attention mechanism network model is obtained.

[0154] Optionally, the attention mechanism network model is a previous encoder-decoder predictor transformer network model.

[0155] Optionally, the feature fusion module 410 includes:

[0156] A splicing unit, configured to splice the feature information of each point cloud cluster, and merge the feature information of each point cloud cluster after splicing to form a splicing matrix;

[0157] A feature fusion unit, configured to perform feature fusion processing on the splicing matrix based on a multi-layer perceptron MLP to obtain a feature matrix.

[0158] Optionally, the feature fusion unit is specifically configured to:

[0159] For any one point cloud cluster, perform feature fusion processing on the splicing matrix corresponding to the point cloud cluster based on a multi-layer perceptron MLP to obtain the comprehensive feature information of the point cloud cluster;

[0160] Perform the feature fusion processing on all point cloud clusters according to the number of point cloud clusters to obtain the feature matrices corresponding to all point cloud clusters, where the feature matrices include the number of point cloud clusters and the feature dimension.

[0161] Optionally, the secondary clustering module 430 includes:

[0162] A target point cloud cluster determination unit, configured to: for each cloud cluster after weighted processing, sequentially select one point cloud cluster as the initial clustering center therefrom, and determine the target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.

[0163] Optionally, the target point cloud cluster determination unit is specifically configured to:

[0164] Calculate the Euclidean distance between other point cloud clusters and the initial clustering center;

[0165] If the value of the Euclidean distance is less than a preset clustering threshold, then all point cloud clusters within the preset clustering threshold range are used as candidate point cloud clusters belonging to the same object as the initial clustering center;

[0166] Use the average value of the distances of each candidate point cloud cluster as the new clustering center, and return to perform the operation of calculating the Euclidean distance between other point cloud clusters and the new clustering center until the newly determined clustering center does not change. Then, use all candidate point cloud clusters belonging to the same object as the new clustering center as the target point cloud cluster.

[0167] Optionally, the clustering process for the original point cloud data includes:

[0168] Preprocess the original point cloud data, where the preprocessing includes ground point removal and downsampling;

[0169] Cluster the preprocessed point cloud to obtain multiple point cloud clusters;

[0170] Among them, the original point cloud data is obtained by parsing the original lidar data, and the original point cloud data includes the number of all point clouds in the current frame, the three-dimensional coordinates of the point clouds, and the reflection intensity.

[0171] The point cloud data processing device provided in the embodiments of the present invention can execute the point cloud data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in the above embodiments, reference can be made to the point cloud data processing method provided in any embodiment of the present invention.

[0172] Embodiment 5

[0173] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a computing device provided in Embodiment 5 of the present invention. As Figure 6 shown, the computing device may include:

[0174] A memory 701 storing executable program code;

[0175] A processor 702 coupled to the memory 701;

[0176] Among them, the processor 702 calls the executable program code stored in the memory 701 to execute the point cloud data processing method provided in any embodiment of the present invention.

[0177] The embodiments of the present invention disclose a computer-readable storage medium storing a computer program, where the computer program causes a computer to execute the point cloud data processing method provided in any embodiment of the present invention.

[0178] In various embodiments of the present invention, it should be understood that the magnitudes of the serial numbers of the above processes do not necessarily mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0179] In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0180] In addition, in each embodiment of the present invention, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0181] When the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc., specifically, the processor in the computer device) to execute some or all of the steps of the above-mentioned methods in each embodiment of the present invention.

[0182] Those of ordinary skill in the art can understand that all or part of the steps in the above-mentioned various methods can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable storage medium. The storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium that can be used to carry or store data.

[0183] Those of ordinary skill in the art will understand that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily essential for implementing the present invention.

[0184] Those of ordinary skill in the art will understand that the modules in the device in the embodiment can be distributed in the device of the embodiment according to the description of the embodiment, or can be correspondingly changed to be located in one or more devices different from the present embodiment. The modules of the above embodiments can be combined into one module, or further split into multiple sub-modules.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing point cloud data, characterized in that, Including: Fusing the feature information of each point cloud cluster to obtain a feature matrix; wherein, each of the point cloud clusters is obtained by clustering the original point cloud data, and the feature information includes shape feature information and position feature information; Based on the trained attention mechanism network model, performing weighted processing on the point cloud clusters in the feature matrix, so as to increase the similarity between the feature of point cloud clusters corresponding to the same object and decrease the similarity between the feature of point cloud clusters corresponding to different objects; Performing clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object; Wherein, the fusing the feature information of each point cloud cluster to obtain a feature matrix includes: Concatenating the feature information of each point cloud cluster, and combining the concatenated feature information of each point cloud cluster to form a concatenated matrix; Based on the multi-layer perceptron MLP, performing feature fusion processing on the concatenated matrix to obtain a feature matrix.

2. The method according to claim 1, characterized in that, The attention mechanism network model is obtained by training in the following manner: Taking the feature matrix of the point cloud clusters obtained after feature fusion as a training sample, and the category of the object to which each point cloud cluster belongs is marked in the training sample; Using the training sample to train the initial attention mechanism network model to obtain a weighted matrix after weighted processing; For each point cloud cluster in the weighted matrix, determining the predicted point cloud cluster belonging to the same object as this point cloud cluster according to the distance relationship between this point cloud cluster and other point cloud clusters; During the training process, when the loss function value of the similarity between the predicted point cloud cluster and the true point cloud cluster of the corresponding object reaches convergence, the trained attention mechanism network model is obtained.

3. The method according to claim 1 or 2, characterized in that, The attention mechanism network model is a pre-order encoding predictor transformer network model.

4. The method according to claim 1, characterized in that, Based on the multi-layer perceptron MLP, performing feature fusion processing on the concatenated matrix to obtain a feature matrix, including: For any point cloud cluster, based on the multi-layer perceptron MLP, performing feature fusion processing on the concatenated matrix corresponding to this point cloud cluster to obtain the comprehensive feature information of this point cloud cluster; Performing the feature fusion processing on all point cloud clusters according to the number of point cloud clusters to obtain the feature matrix corresponding to all point cloud clusters, and the feature matrix includes the number of point cloud clusters and the feature dimension.

5. The method according to claim 1, wherein Performing clustering processing on the weighted point cloud clusters to determine the point cloud clusters belonging to the same object, including: For each of the weighted cloud clusters, sequentially selecting one point cloud cluster as the initial clustering center from them, and determining the target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.

6. The method according to claim 5, wherein Determining the target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center, including: Calculating the Euclidean distance between other point cloud clusters and the initial clustering center; If the value of the Euclidean distance is less than the preset clustering threshold, then taking all the point cloud clusters within the preset clustering threshold range as the candidate point cloud clusters belonging to the same object as the initial clustering center; Use the average value of the distances of each candidate point cloud cluster as the new clustering center, and return to perform the operation of calculating the Euclidean distance between other point cloud clusters and the new clustering center until the determined new clustering center does not change, and regard all candidate point cloud clusters belonging to the same object as the target point cloud cluster.

7. The method according to claim 1, wherein The clustering process for the original point cloud data includes: Preprocess the original point cloud data, and the preprocessing includes ground point removal and downsampling; Cluster the preprocessed point cloud to obtain multiple point cloud clusters; Among them, the original point cloud data is obtained by parsing the original lidar data, and the original point cloud data includes the number of all point clouds in the current frame, the three-dimensional coordinates of the point clouds, and the reflection intensity.

8. A point cloud data processing device, characterized in that, Include: A feature fusion module, configured to: fuse the feature information of each point cloud cluster to obtain a feature matrix; among them, each point cloud cluster is obtained after clustering the original point cloud data, and the feature information includes shape feature information and position feature information; A weighted processing module, configured to: based on the trained attention mechanism network model, perform weighted processing on the point cloud clusters in the feature matrix, so as to increase the similarity between the point cloud cluster features corresponding to the same object and reduce the similarity between the point cloud cluster features corresponding to different objects; A secondary clustering module, configured to: perform clustering processing on the point cloud clusters after weighted processing to determine the point cloud clusters belonging to the same object; Among them, the feature fusion module includes: A splicing unit, configured to: splice the feature information of each point cloud cluster, and merge the feature information of each point cloud cluster after splicing to form a splicing matrix; A feature fusion unit, configured to: based on the multi-layer perceptron MLP, perform feature fusion processing on the splicing matrix to obtain a feature matrix.

9. The device according to claim 8, characterized in that, The attention mechanism network model is trained in the following way: Use the feature matrix of the point cloud clusters obtained after feature fusion as the training sample, and the object category to which each point cloud cluster belongs is marked in the training sample; Use the training sample to train the initial attention mechanism network model to obtain a weighted matrix after weighted processing; For each point cloud cluster in the weighted matrix, determine the predicted point cloud cluster belonging to the same object as the point cloud cluster according to the distance relationship between the point cloud cluster and other point cloud clusters; During the training process, when the loss function value of the similarity between the predicted point cloud cluster and the real point cloud cluster of the corresponding object reaches convergence, obtain the trained attention mechanism network model.

Images