Point cloud segmentation method, device, equipment and storage medium

Abstract

The present application provides a point cloud segmentation method, apparatus, device, and storage medium. Among them, the method includes: slicing a three-dimensional point cloud to obtain multiple point cloud layers of the three-dimensional point cloud; extracting feature points from each point cloud layer to obtain multiple feature points in the point cloud layer; respectively determining the K-nearest neighbor neighborhood and geodesic neighborhood of each feature point, and using the feature expressions of the feature point in the K-nearest neighbor neighborhood and geodesic neighborhood as the local features of the feature point; learning the global features of the feature point through the local features of the feature point, and concatenating the global features and local features to determine the feature expression vector of the feature point; finally, according to the feature expression vectors of the feature points in each point cloud layer, aggregating each point in the three-dimensional point cloud according to the classification result of the point to obtain the point cloud segmentation result of the three-dimensional point cloud. The method of the present application can enhance the local information volume of the point cloud and improve the accuracy of point cloud segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of point cloud segmentation, and more particularly, to a point cloud segmentation method, device, equipment and storage medium. Background Art

[0002] Effective perception and recognition of a three-dimensional environment are the premise and foundation for an intelligent robot to complete autonomous behaviors. Objects in a three-dimensional space are usually expressed in the form of point clouds. Therefore, accurate segmentation of point clouds is crucial.

[0003] In the prior art, sensors are usually used to collect point cloud data of an object, and the collected point cloud data is voxelized, surface meshed, or the three-dimensional point cloud is transformed into an easily expressible form from multiple perspectives to complete object segmentation.

[0004] However, the point clouds collected by sensors in the prior art are sparse, and there is a problem of missing context information. As a result, the independence of the segmented objects is poor, and the accuracy of point cloud segmentation is low. Summary of the Invention

[0005] The purpose of the present application is to provide a point cloud segmentation method, device, equipment and storage medium to solve the problem of low accuracy of point cloud segmentation in the prior art for the deficiencies in the above-mentioned prior art.

[0006] To achieve the above purpose, the technical solutions adopted in the present application are as follows:

[0007] In a first aspect, the present application provides a point cloud segmentation method, the method including:

[0008] Slice a three-dimensional point cloud to obtain a plurality of point cloud layers of the three-dimensional point cloud;

[0009] Extract feature points from each point cloud layer to obtain a plurality of feature points in the point cloud layer;

[0010] Determine the K-nearest neighbor neighborhood and geodesic neighborhood of each feature point respectively, and obtain the local feature of the feature point according to the K-nearest neighbor neighborhood and geodesic neighborhood of the feature point;

[0011] Determine the global feature of the feature point according to the local feature of the feature point, and determine the feature expression vector of the feature point according to the global feature and the local feature, where the feature expression vector is used to characterize the category of the feature point;

[0012] Segment the three-dimensional point cloud according to the feature expression vectors of the feature points in each point cloud layer to obtain the point cloud segmentation result of the three-dimensional point cloud.

[0013] Optionally, respectively determining the K-nearest neighbor neighborhood and the geodesic neighborhood of each feature point, and obtaining the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point, includes:

[0014] Using the K-nearest neighbor algorithm to determine the K-nearest neighbor neighborhood of the feature point;

[0015] Based on the geodesic distance, determining the geodesic neighborhood of the feature point;

[0016] Obtaining the first local feature of the feature point according to the K-nearest neighbor neighborhood, obtaining the second local feature of the feature point according to the geodesic neighborhood, and splicing the first local feature and the second local feature to obtain the local feature of the feature point.

[0017] Optionally, the determining the geodesic neighborhood of the feature point based on the geodesic distance includes:

[0018] Taking the feature point as the starting point, determining K neighboring points whose geodesic distances from the feature point satisfy a preset distance threshold, and taking the region composed of the K neighboring points as the geodesic neighborhood of the feature point.

[0019] Optionally, the obtaining the first local feature of the feature point according to the K-nearest neighbor neighborhood, obtaining the second local feature of the feature point according to the geodesic neighborhood, and splicing the first local feature and the second local feature to obtain the local feature of the feature point includes:

[0020] Inputting the K-nearest neighbor neighborhood, the geodesic neighborhood, and the feature point into a pre-trained neural network model, respectively extracting the first local feature and the second local feature by the neural network model, and splicing the first local feature and the second local feature to obtain the local feature of the feature point.

[0021] Optionally, the determining the global feature of the feature point according to the local feature of the feature point, and determining the feature expression vector of the feature point according to the global feature and the local feature includes:

[0022] The neural network model performs max pooling processing on the local feature of the feature point to obtain the global feature of the feature point, and determines the feature expression vector of the feature point according to the global feature and the local feature.

[0023] Optionally, the determining the feature expression vector of the feature point according to the global feature and the local feature includes:

[0024] Splicing the global feature and the local feature to obtain the feature expression vector of the feature point.

[0025] Optionally, segmenting the 3D point cloud according to the feature expression vectors of the feature points in each point cloud layer to obtain the point cloud segmentation result of the 3D point cloud includes:

[0026] Obtaining the category of the feature points based on the feature expression vectors;

[0027] Aggregating the feature points in the 3D point cloud according to the categories to obtain the point cloud segmentation result of the 3D point cloud.

[0028] In a second aspect, the present application provides a point cloud segmentation device, and the device includes:

[0029] A slicing module, configured to slice the 3D point cloud to obtain multiple point cloud layers of the 3D point cloud;

[0030] An extraction module, configured to extract feature points from each point cloud layer to obtain multiple feature points in the point cloud layer;

[0031] A first determination module, configured to respectively determine the K-nearest neighbor neighborhood and the geodesic neighborhood of each feature point, and obtain the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point;

[0032] A second determination module, configured to determine the global feature of the feature point according to the local feature of the feature point, and determine the feature expression vector of the feature point according to the global feature and the local feature, where the feature expression vector is used to characterize the category of the feature point;

[0033] A segmentation module, configured to segment the 3D point cloud according to the feature expression vectors of the feature points in each point cloud layer to obtain the point cloud segmentation result of the 3D point cloud.

[0034] Optionally, the first determination module is specifically configured to:

[0035] Use the K-nearest neighbor algorithm to determine the K-nearest neighbor neighborhood of the feature point;

[0036] Determine the geodesic neighborhood of the feature point based on the geodesic distance;

[0037] Obtain the first local feature of the feature point according to the K-nearest neighbor neighborhood, obtain the second local feature of the feature point according to the geodesic neighborhood, and splice the first local feature and the second local feature to obtain the local feature of the feature point.

[0038] Optionally, the first determination module is further specifically configured to:

[0039] Taking the feature point as the starting point, determine K neighboring points whose geodesic distance from the feature point satisfies a preset distance threshold, and use the region composed of the K neighboring points as the geodesic neighborhood of the feature point.

[0040] Optionally, the first determination module is further specifically configured to:

[0041] Input the K-nearest neighbor neighborhood, the geodesic neighborhood, and the feature point into a pre-trained neural network model, respectively extract the first local feature and the second local feature from the neural network model, and splice the first local feature and the second local feature to obtain the local feature of the feature point.

[0042] Optionally, the second determination module is further specifically configured to:

[0043] The neural network model performs max pooling processing on the local feature of the feature point to obtain the global feature of the feature point, and determines the feature expression vector of the feature point according to the global feature and the local feature.

[0044] Optionally, the second determination module is further specifically configured to:

[0045] Splice the global feature and the local feature to obtain the feature expression vector of the feature point.

[0046] Optionally, the segmentation module is further specifically configured to:

[0047] Obtain the category of the feature point based on the feature expression vector;

[0048] Aggregate the feature points in the three-dimensional point cloud according to the category to obtain the point cloud segmentation result of the three-dimensional point cloud.

[0049] In a third aspect, the present application provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the above point cloud segmentation method.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it performs the steps of the above point cloud segmentation method.

[0051] The beneficial effects of the present application are as follows: By directly extracting feature points from the point cloud data collected by the sensor, rather than extracting feature points by means of preprocessing operations such as voxelization and meshing, the authenticity and reliability of the point cloud data can be maximally retained. By jointly determining the local features of the feature points through the K-nearest neighbor neighborhood and the geodesic neighborhood, the local features can simultaneously represent the position feature information of the points and the surface geodesic topological association information, thereby enhancing the local information volume of the point cloud and enabling the local features to express more effective information of the object. The feature expression vector determined by the local features and the global features can also more accurately represent the category to which the feature points belong. Therefore, when segmenting the three-dimensional point cloud according to the feature expression vector, the accuracy of the point cloud segmentation can also be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0053] Figure 1 FIG. shows a schematic diagram of an application scenario provided by an embodiment of the present application;

[0054] Figure 2 FIG. shows a flowchart of a point cloud segmentation method provided by an embodiment of the present application;

[0055] Figure 3 FIG. shows a flowchart of determining local features provided by an embodiment of the present application;

[0056] Figure 4 FIG. shows a flowchart of clustering a point cloud provided by an embodiment of the present application;

[0057] Figure 5 FIG. shows a flowchart of another point cloud segmentation method provided by an embodiment of the present application;

[0058] Figure 6 FIG. shows a flowchart of determining second local features provided by an embodiment of the present application;

[0059] Figure 7 FIG. shows a schematic structural diagram of a point cloud segmentation device provided by an embodiment of the present application;

[0060] Figure 8 FIG. shows a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. It should be understood that the accompanying drawings in this application are only for the purposes of illustration and description, and are not used to limit the protection scope of this application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of this application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without a logical context relationship may be reversed or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of this application.

[0062] In addition, the described embodiments are only some embodiments of this application, rather than all embodiments. The components of the embodiments of this application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application claimed, but merely represents the selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative efforts belong to the protection scope of this application.

[0063] It should be noted that the term "including" will be used in the embodiments of this application to indicate the existence of the features stated hereinafter, but does not exclude the addition of other features.

[0064] The point cloud data collected by the sensor is sparse, and during the data collection process, there are problems such as occlusion of object scanning, noise in the sensor, and rigid rotation of the object. Therefore, it is difficult for the point cloud data to effectively express the information of the object.

[0065] Therefore, when performing point cloud segmentation based on the point cloud data collected by the sensor, due to the lack of context information, there is a problem that the independence of the segmented objects is poor, which undoubtedly greatly reduces the accuracy of point cloud segmentation.

[0066] Based on the above problems, this application proposes a point cloud segmentation method, which constructs the topological distance correlation of the point cloud model based on the geodesic distance, combines this information with the features of the point cloud based on the Euclidean distance, realizes the enhancement of the local feature information of the point cloud, and learns the feature expression of the feature points of the point cloud through a deep neural network, thereby realizing point cloud segmentation.

[0067] The point cloud segmentation method of this application can be applied to Figure 1In the application scenario shown, after the sensor collects the point cloud data of object A, the point cloud data is sent to an electronic device. The electronic device divides the point cloud data based on the point cloud segmentation method of the present application and finally outputs the segmentation result of the point cloud data of object A.

[0068] Next, in combination with Figure 2 , the point cloud segmentation method of the present application will be described. The execution subject of this method can be Figure 1 the electronic device shown, such as Figure 2 shown, this method includes:

[0069] S201: Slice the three-dimensional point cloud to obtain multiple point cloud layers of the three-dimensional point cloud.

[0070] Optionally, the three-dimensional point cloud can be the point cloud data of an object, which is used to describe the three-dimensional characteristics of the object. The point cloud data can be collected by a sensor to obtain the point cloud data of the object.

[0071] After the point cloud data is collected, in the present application, the point cloud data can be sliced. Exemplarily, the point cloud model can be sliced into 5 layers along the z-axis, and each point cloud layer includes multiple feature points.

[0072] It should be noted that in the prior art, after the point cloud data is collected, the point cloud data is mostly preprocessed, such as voxelization, surface meshing, or converting the three-dimensional point cloud into an easy-to-express form from multiple perspectives, and then the object segmentation is completed. During the preprocessing of the point cloud data, the point cloud data will inevitably be artificially modified, which will undoubtedly reduce the reliability and authenticity of the point cloud data.

[0073] In the present application, the point cloud data of the object to be collected collected by the sensor is directly layered and the subsequent feature extraction is performed, and no preprocessing work is carried out, thereby improving the authenticity and reliability of the point cloud data.

[0074] Referring to Figure 1 , sensors on the robot carrier, such as depth cameras or lidar, etc., irradiate the laser on the object surface. The laser reflected by the object will carry information such as azimuth and distance. If the laser beam is scanned according to a certain trajectory, the reflected laser point information will be recorded while scanning. Since the scanning is extremely fine, a large number of laser points can be obtained, and thus a laser point cloud, that is, point cloud data, can be formed.

[0075] S202: Extract feature points from each point cloud layer to obtain multiple feature points in the point cloud layer.

[0076] In the present application, the farthest point sampling method (Farthest Point Sample, FPS) can be used to extract feature points from each point cloud layer in the point cloud data collected by the sensor to obtain multiple feature points in the point cloud layer.

[0077] S203: Determine the K-nearest neighbor neighborhood and the geodesic neighborhood of each feature point respectively, and obtain the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point.

[0078] Optionally, the K-nearest neighbor neighborhood can be a region composed of multiple feature points that are the nearest neighbors to the feature point obtained by performing a K-nearest neighbor query on the feature point.

[0079] Among them, the K-nearest neighbor neighborhood can characterize the position feature information of the point cloud based on the Euclidean distance and describe the spatial position of each point in the point cloud.

[0080] Optionally, the geodesic neighborhood can be a region composed of multiple feature points that are the nearest neighbors to the feature point determined based on the geodesic distance. The geodesic refers to the curve that locally minimizes the connection between two feature points in space.

[0081] Among them, the geodesic distance can describe the topological association constraints of local points in the three-dimensional point cloud, effectively separate independent objects in the point cloud, and thus more accurately describe the objects in the point cloud.

[0082] It should be noted that in this application, this step can be executed for each feature point in the three-dimensional point cloud one by one to obtain the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point, and obtain the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point. Among them, the feature point can be any feature point in any point cloud layer of the three-dimensional point cloud.

[0083] In this application, obtaining the local feature based on the K-nearest neighbor neighborhood and the geodesic neighborhood can enhance the local feature information volume of the point cloud, thereby improving the data segmentation accuracy of the three-dimensional point cloud.

[0084] S204: Determine the global feature of the feature point according to the target local feature of the feature point, and determine the feature expression vector of the feature point according to the global feature and the local feature. The feature expression vector is used to characterize the category of the feature point.

[0085] Optionally, the PointNet network can be used to learn the global features of each feature point in the point cloud. The PointNet network is a neural network architecture that can process point cloud data end-to-end. It can perform feature extraction on the input point cloud data, first obtain the local features of the feature points in the point cloud data, and then obtain the final global features according to the local features.

[0086] Optionally, the feature expression vector of the feature point can characterize the point cloud label corresponding to the maximum possibility of the feature point corresponding to the point cloud category in the database, and this point cloud label can describe the category of the feature point.

[0087] It should be noted that in this application, the steps of S203 - S204 can be executed for each feature point in the point cloud data to obtain the local feature and global feature of each feature point, and the feature expression vector of the feature point can be obtained according to the local feature and global feature.

[0088] S205: Segment the three - dimensional point cloud according to the feature expression vectors of the feature points in each point cloud layer to obtain the point cloud segmentation result of the three - dimensional point cloud.

[0089] Optionally, the feature expression vector can describe the category of the feature point. Therefore, based on the feature expression vectors of the feature points, the feature points of the same category can be aggregated together to segment the three - dimensional point cloud to obtain the point cloud segmentation result of the three - dimensional point cloud.

[0090] In the point cloud segmentation result of the three - dimensional point cloud, the points in the same category can represent the same component of the object. Exemplarily, after segmenting the three - dimensional point cloud data of a chair, all the points on the back of the chair will be segmented into feature points of the same category, and the points on the back of the chair and the chair legs will be segmented into feature points of different categories.

[0091] In the embodiments of this application, feature point extraction is performed on the point cloud data collected by the sensor without relying on pre - processing operations such as voxelization and meshing, which can maximize the retention of the authenticity and reliability of the point cloud data. By jointly determining the local feature of the feature point through the K - nearest neighbor neighborhood and geodesic neighborhood, the local feature can simultaneously represent the position feature information of the point and the surface geodesic topological association information, thereby enhancing the local information volume of the point cloud and enabling the local feature to express more effective information of the object. The feature expression vector determined by the local feature and global feature can also more accurately represent the category to which the feature point belongs. Therefore, when segmenting the three - dimensional point cloud according to the feature expression vector, the accuracy of point cloud segmentation can also be greatly improved.

[0092] Next, the steps of respectively determining the K - nearest neighbor neighborhood and geodesic neighborhood of each feature point, and obtaining the local feature of the feature point according to the K - nearest neighbor neighborhood and geodesic neighborhood of the feature point will be described, as Figure 3 shown, the above S203 step includes:

[0093] S301: Use the K - nearest neighbor algorithm to determine the K - nearest neighbor neighborhood of the feature point.

[0094] In the K - nearest neighbor algorithm, first, a feature space is constructed by all the feature points in the point cloud data. The dimension of the feature space is the number of feature points. Then, for a certain feature point, K feature points that are closest to it are searched within the feature space according to the parameter K, thereby determining the K - nearest neighbor neighborhood of the feature point. Among them, when measuring the distance to find the K feature points that are closest to it within the feature space, the Euclidean distance can be used for measurement.

[0095] The K-nearest neighbor neighborhood determined in this application can focus more on spatial position correlation, so as to connect the small part of the missing in the point cloud data, so as to learn more context information and complete the missing part.

[0096] S302: Determine the geodesic neighborhood of the feature point based on the geodesic distance.

[0097] Optionally, the geodesic is the curve that locally connects two points in space with the shortest distance. The geodesic distance can effectively separate independent individuals, and it focuses on learning along the surface topology. Therefore, the geodesic neighborhood in this application can effectively improve the independence of objects in the point cloud data.

[0098] S303: Obtain the first local feature of the feature point according to the K-nearest neighbor neighborhood, obtain the second local feature of the feature point according to the geodesic neighborhood, and splice the first local feature and the second local feature to obtain the local feature of the feature point.

[0099] Optionally, the first local feature may be the local feature extracted by the feature point in the K-nearest neighbor neighborhood, and the second local feature may be the local feature extracted by the feature point in the geodesic neighborhood.

[0100] For each feature point, it will include the first local feature and the second local feature. After splicing the first local feature and the second local feature, the local feature of the feature point can be obtained.

[0101] The following is a further description of the above S302: Determine the geodesic neighborhood of the feature point based on the geodesic distance.

[0102] Taking the feature point as the starting point, determine K neighboring points whose geodesic distance from the feature point satisfies the preset distance threshold, and use the region composed of the K neighboring points as the geodesic neighborhood of the feature point.

[0103] In this application, the Johnson's algorithm can be used to calculate K points that satisfy the preset distance threshold from each feature point as the starting point, and these points are collected as the geodesic neighborhood of the feature point. Among them, the value of K can be any positive integer, and the distance threshold can be determined by those skilled in the art according to the actual situation, and this application does not limit it here.

[0104] Since the geodesic is the curve that locally connects two points in space with the shortest distance, it can effectively separate independent individuals and focus on learning along the surface topology. Therefore, independent individuals cannot be connected by geodesics.

[0105] Exemplarily, assume that the value of K is 64. The point cloud data is sliced along the z-axis. In each layer of the sub-point cloud, any feature point is selected as the starting point, and the geodesic distance from all points that can form a topological connection with the starting point to the starting feature point is calculated.

[0106] If there are more than 64 points in this layer that can be topologically connected to the starting point through geodesics, then the 64 points with the smallest geodesic distances are taken as the geodesic neighborhood of the feature point; if the number of point clouds that can be topologically connected to the starting point in this layer is less than 64, then all the satisfied points are taken and these points are randomly replicated until 64 points are satisfied; if no point in this layer can be topologically connected to the starting point, then the starting point itself is replicated until 64 points are collected. The points collected above are used as the geodesic neighborhood of the feature point, and the geodesic neighborhood is used as the input, and PointNet is used to learn its features to obtain the second local feature of the point cloud based on the geodesic distance.

[0107] Next, the steps of obtaining the first local feature of the feature point according to the K-nearest neighbor neighborhood and the second local feature of the feature point according to the geodesic neighborhood above, and splicing the first local feature and the second local feature to obtain the local feature of the feature point will be described. The above S303 step includes:

[0108] The K-nearest neighbor neighborhood, the geodesic neighborhood, and the feature point are input into a pre-trained neural network model. The first local feature and the second local feature are respectively extracted by the neural network model, and the first local feature and the second local feature are spliced to obtain the local feature of the feature point.

[0109] Optionally, the pre-trained neural network model can be a PointNet network. The PointNet network can respectively extract features from the K-nearest neighbor neighborhood and the geodesic neighborhood of the same feature point to obtain the first local feature and the second local feature of the feature point, and splice the first local feature and the second local feature to obtain the local feature of the feature point.

[0110] The following is the description of the steps of determining the global feature of the feature point according to the local feature of the feature point and determining the feature expression vector of the feature point according to the global feature and the local feature. The above S204 step includes:

[0111] The neural network model performs max-pooling processing on the local feature of the feature point to obtain the global feature of the feature point, and determines the feature expression vector of the target feature point according to the global feature and the local feature.

[0112] Optionally, the neural network model can include a max-pooling layer. Taking the local feature of the feature point as the input of the max-pooling layer, the global feature of the feature point can be obtained.

[0113] After determining the global features of the feature points, in the present application, the feature expression vector of the feature points can also be determined according to the global features and local features. This step includes:

[0114] Concatenate the global feature and the local feature to obtain the feature expression vector of the feature point.

[0115] It should be noted that the neural network model may include two max pooling layers. Among them, the first max pooling layer can obtain the global feature of the feature point according to the local feature of the input feature point. After concatenating the global feature and the local feature and inputting them into the second max pooling layer, the feature expression vector of the feature point can be obtained.

[0116] The following is the description of the steps for segmenting the 3D point cloud according to the feature expression vectors of the feature points in each point cloud layer to obtain the point cloud segmentation result of the 3D point cloud, as Figure 4 shown. The above step S205 includes:

[0117] S401: Obtain the category of the feature point based on the feature expression vector.

[0118] Optionally, the feature expression vector can represent the category to which the feature point belongs, and the category can represent the object to which the feature point belongs, or which component of the object it belongs to.

[0119] S402: Aggregate the feature points in the 3D point cloud according to the category to obtain the point cloud segmentation result of the 3D point cloud.

[0120] It should be noted that the point cloud data collected by the sensor may include multiple objects, and each component of each object can be divided into different categories. Therefore, by aggregating the feature points in the 3D point cloud according to the category, different objects and different components of the same object can be segmented to obtain the segmentation result of the point cloud data.

[0121] As Figure 5 shown, it is a flowchart of point cloud segmentation given in the present application. As Figure 5 shown, after inputting the point cloud data into the electronic device, the electronic device can first slice the point cloud data, and then use the farthest point sampling method to extract feature points for each layer. For each feature point, the local feature of the feature point can be determined respectively based on the K-nearest neighbor algorithm and the geodesic distance, and then the global feature of the feature point is learned through the PointNet network. After connecting the global feature and the local feature and inputting them into the max pooling layer, the feature expression vector of the feature point is obtained. Finally, the point cloud data is segmented based on the feature expression vector to obtain the final segmentation result.

[0122] Among them, Figure 5The MLP in it refers to the Multilayer Perceptron (MLP), which is used for local feature extraction, global feature extraction, and the concatenation of local and global features of feature points.

[0123] As Figure 6 shown, it is a flowchart for determining the second local feature based on geodesic distance given in this application. For each point cloud layer, it can be rasterized first, then the geodesic neighborhood of the feature points is determined based on the topological constraints of the geodesic, and then the local features are learned in the geodesic neighborhood through the PointNet network, and the second local feature is output after maximum pooling of the local features.

[0124] In the embodiments of this application, a three-dimensional scanning device on the robot carrier, such as a depth camera or a lidar, can be used to obtain the surrounding environment information, such as the point cloud data of indoor environment objects. Without relying on conventional preprocessing work such as voxelization and meshing, the x, y, and z coordinates of each point in the point cloud data are directly processed. The transformation matrix learned by the 3D spatial transformation matrix prediction network, that is, T-Net, is used to align the point cloud data to ensure the spatial rotation invariance of the point cloud data. Then, FPS is used to extract feature points from the point cloud data and T-Net is used to align the feature points. Finally, the local and global features of the feature points are connected and maximum pooled to obtain the category of the feature points.

[0125] Based on the same inventive concept, in the embodiments of this application, a point cloud segmentation device corresponding to the point cloud segmentation method is also provided. Since the principle of solving problems by the device in the embodiments of this application is similar to the above-mentioned point cloud segmentation method in the embodiments of this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0126] Referring to Figure 7 shown, it is a schematic diagram of a point cloud segmentation device provided in the embodiments of this application. The device includes: a slicing module 701, an extraction module 702, a first determination module 703, and a second determination module 704; where:

[0127] The slicing module 701 is used to slice the three-dimensional point cloud to obtain multiple point cloud layers of the three-dimensional point cloud;

[0128] The extraction module is used to extract feature points from each point cloud layer to obtain multiple feature points in the point cloud layer;

[0129] The first determination module 702 is used to respectively determine the K-nearest neighbor neighborhood and the geodesic neighborhood of each feature point, and obtain the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point;

[0130] The second determination module 703 is configured to determine the global feature of a feature point according to the local feature of the feature point, and determine the feature expression vector of the feature point according to the global feature and the local feature, where the feature expression vector is used to characterize the category of the feature point;

[0131] The segmentation module 704 is configured to segment the three-dimensional point cloud according to the feature expression vectors of the feature points in each point cloud layer, so as to obtain the point cloud segmentation result of the three-dimensional point cloud.

[0132] Optionally, the first determination module 702 is specifically configured to:

[0133] Use the K-nearest neighbor algorithm to determine the K-nearest neighbor neighborhood of the feature point;

[0134] Determine the geodesic neighborhood of the feature point based on the geodesic distance;

[0135] Obtain the first local feature of the feature point according to the K-nearest neighbor neighborhood, obtain the second local feature of the feature point according to the geodesic neighborhood, and splice the first local feature and the second local feature to obtain the local feature of the feature point.

[0136] Optionally, the first determination module 702 is further specifically configured to:

[0137] Taking the feature point as the starting point, determine K adjacent points whose geodesic distance from the feature point satisfies a preset distance threshold, and use the region composed of the K adjacent points as the geodesic neighborhood of the feature point.

[0138] Optionally, the first determination module 702 is further specifically configured to:

[0139] Input the K-nearest neighbor neighborhood, the geodesic neighborhood, and the feature point into a pre-trained neural network model, and the neural network model extracts the first local feature and the second local feature respectively, and splices the first local feature and the second local feature to obtain the local feature of the feature point.

[0140] Optionally, the second determination module 703 is further specifically configured to:

[0141] The neural network model performs max pooling processing on the local feature of the feature point to obtain the global feature of the feature point, and determines the feature expression vector of the feature point according to the global feature and the local feature.

[0142] Optionally, the second determination module 703 is further specifically configured to:

[0143] Splice the global feature and the local feature to obtain the feature expression vector of the feature point.

[0144] Optionally, the segmentation module 704 is further specifically configured to:

[0145] Obtain the category of the feature point based on the feature expression vector;

[0146] Aggregate the feature points in the three-dimensional point cloud according to categories to obtain the point cloud segmentation result of the three-dimensional point cloud.

[0147] For the processing flow of each module in the device and the interaction flow between modules, reference can be made to the relevant descriptions in the above method embodiments, which will not be elaborated here.

[0148] In the embodiment of the present application, by directly extracting feature points from the point cloud data collected by the sensor, rather than extracting feature points by means of preprocessing operations such as voxelization and meshing, the authenticity and reliability of the point cloud data can be maximally retained. By jointly determining the local features of the feature points through the K-nearest neighbor neighborhood and the geodesic neighborhood, the local features can simultaneously represent the position feature information of the points and the surface geodesic topological association information, thereby enhancing the local information volume of the point cloud and enabling the local features to express more effective information of the object. The feature expression vector determined by the local features and the global features can also more accurately represent the category to which the feature points belong. Therefore, when segmenting the three-dimensional point cloud according to the feature expression vector, the accuracy of the point cloud segmentation can also be greatly improved.

[0149] The embodiment of the present application also provides an electronic device, such as Figure 8 shown in the figure, which is a schematic structural diagram of the electronic device provided by the embodiment of the present application, including: a processor 801, a memory 802, and a bus. The memory 802 stores machine-readable instructions executable by the processor 801 (such as Figure 7 the execution instructions corresponding to the slicing module 701, the extraction module 702, the first determination module 703, and the second determination module 704 in the device in ), when the computer device runs, the processor 801 communicates with the memory 802 through the bus, and when the machine-readable instructions are executed by the processor 801, the processing of the above point cloud segmentation method is executed.

[0150] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the above point cloud segmentation method are executed.

[0151] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the method embodiments, and will not be elaborated herein. In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some communication interfaces. The indirect coupling or communication connection of the devices or modules can be in an electrical, mechanical, or other form.

[0152] In addition, each functional unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical disks, etc., which can store program codes.

[0153] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all should be covered by the protection scope of the present application.

Claims

1. A point cloud segmentation method, characterized in that Including: Slice the three-dimensional point cloud to obtain multiple point cloud slices of the three-dimensional point cloud; Extract feature points from each point cloud slice to obtain multiple feature points in the point cloud slice; Determine the K-nearest neighbor neighborhood and the geodesic neighborhood of each feature point respectively, and obtain the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point. The geodesic neighborhood of the feature point consists of the points topologically connected by geodesics in the point cloud slice; Determine the global feature of the feature point according to the local feature of the feature point, and determine the feature expression vector of the feature point according to the global feature and the local feature. The feature expression vector is used to characterize the category of the feature point; Segment the three-dimensional point cloud according to the feature expression vectors of the feature points in each point cloud slice to obtain the point cloud segmentation result of the three-dimensional point cloud.

2. The method according to claim 1, characterized in that The step of respectively determining the K-nearest neighbor neighborhood and the geodesic neighborhood of each feature point, and obtaining the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point includes: Use the K-nearest neighbor algorithm to determine the K-nearest neighbor neighborhood of the feature point; Determine the geodesic neighborhood of the feature point based on the geodesic distance; Obtain the first local feature of the feature point according to the K-nearest neighbor neighborhood, obtain the second local feature of the feature point according to the geodesic neighborhood, and splice the first local feature and the second local feature to obtain the local feature of the feature point.

3. The method according to claim 2, wherein The step of determining the geodesic neighborhood of the feature point based on the geodesic distance includes: Taking the feature point as the starting point, determine K neighboring points whose geodesic distance from the feature point satisfies a preset distance threshold, and use the region composed of the K neighboring points as the geodesic neighborhood of the feature point.

4. The method according to claim 2, wherein The step of obtaining the first local feature of the feature point according to the K-nearest neighbor neighborhood, obtaining the second local feature of the feature point according to the geodesic neighborhood, and splicing the first local feature and the second local feature to obtain the local feature of the feature point includes: Input the K-nearest neighbor neighborhood, the geodesic neighborhood, and the feature point into a pre-trained neural network model, and the neural network model respectively extracts the first local feature and the second local feature, and splices the first local feature and the second local feature to obtain the local feature of the feature point.

5. The method according to claim 4, characterized in that The step of determining the global feature of the feature point according to the local feature of the feature point, and determining the feature expression vector of the feature point according to the global feature and the local feature includes: The neural network model performs max pooling processing on the local feature of the feature point to obtain the global feature of the feature point, and determines the feature expression vector of the feature point according to the global feature and the local feature.

6. The method according to claim 5, characterized in that, The step of determining the feature expression vector of the feature point according to the global feature and the local feature includes: Splice the global feature and the local feature to obtain the feature expression vector of the feature point.

7. The method according to any one of claims 1 to 6, characterized in that, The step of segmenting the three-dimensional point cloud according to the feature expression vectors of the feature points in each point cloud slice to obtain the point cloud segmentation result of the three-dimensional point cloud includes: Obtain the category of the feature point based on the feature expression vector; Aggregate the feature points in the three-dimensional point cloud according to their categories to obtain the point cloud segmentation result of the three-dimensional point cloud.

8. A point cloud segmentation device, characterized in that, Comprising: A slicing module, configured to slice the three-dimensional point cloud to obtain multiple point cloud slices of the three-dimensional point cloud; An extraction module, configured to extract feature points from each point cloud slice to obtain multiple feature points in the point cloud slice; A first determination module, configured to respectively determine the K-nearest neighbor neighborhood and the geodesic neighborhood of each feature point, and obtain the local feature of the feature point according to the K-nearest neighbor neighborhood and the geodesic neighborhood of the feature point, wherein the geodesic neighborhood of the feature point consists of the points topologically connected by geodesics in the point cloud slice; A second determination module, configured to determine the global feature of the feature point according to the local feature of the feature point, and determine the feature expression vector of the feature point according to the global feature and the local feature, where the feature expression vector is used to characterize the category of the feature point; A segmentation module, configured to segment the three-dimensional point cloud according to the feature expression vectors of the feature points in each point cloud slice to obtain the point cloud segmentation result of the three-dimensional point cloud.

9. An electronic device, characterized in that, Comprising: A processor, a storage medium and a bus, where the storage medium stores program instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to perform the steps of the point cloud segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is run by the processor, it performs the steps of the point cloud segmentation method according to any one of claims 1 to 7.

Images